The wget, bunzip, then dd to disk approach seems to be necessary to avoid memory allocation errors from the kernel. I'm not sure of the root cause.

***> The actual error is included at the end of this comment <***

Re-implemented to keep the image compressed in the ramdisk and to set the max ramdisk size to 50% of memory:

            image_file_base = '/'.join(image_file.split('/')[-1:])

            decompression_cmd = None
            if image_file_base.endswith('.gz'):
                decompression_cmd = '/bin/gzip -dc'
            elif image_file_base.endswith('.bz2'):
                decompression_cmd = '/bin/bzip2 -dc'

            runner.run('mkdir /builddir')
            runner.run('mount -t tmpfs -o size=50% tmpfs /builddir')
            runner.run('wget -O /builddir/%s %s' % (image_file_base, image_url), timeout=1800)

            if decompression_cmd != None:
                cmd = 'dd bs=4M if=%s of=%s' % (image_file_base, device)
            else:
                cmd = '%s %s | dd bs=4M of=%s' % (decompression_cmd, image_file_base, device)

            runner.run(cmd, timeout=1800)
            runner.run('umount /builddir')

--------

> 
> > +            elif image_url.endswith('.bz2'):
> > +                decompression_cmd = '| /bin/bzip2 -dc'
> > +
> > +            runner.run('mkdir /builddir')
> > +            runner.run('mount -t tmpfs -o size=4G tmpfs builddir')
> > +            image = '/builddir/lava.img'
> > +            runner.run('wget -O - %s %s > %s' % (image_url,
> decompression_cmd, image), timeout=1800)
> > +            runner.run('dd bs=4M if=%s of=%s' % (image, device),
> timeout=1800)
> > +            runner.run('umount /builddir')
> 
> Is it always possible to create a 4G tmpfs?
> 
> Instead of saving the image locally, I think you could pipe the wget
> output directl into dd. This way you don't need to to create a tmpfs,
> nor to mount/umount it. This would be something like:
> 
>   runner.run("wget -O - %s %s | dd bs=4M of=%s" % (image_url,
> decompression_cmd, device))
> 
> Also, there are some wget options you can use to get some progress
> indication, which is useful. Look at how master.py does it.
> 
> (if you convince me the tmpfs is needed, make sure to add a missing
> slash in the last token there so it reads `/builddir` instead of
> `builddir`)
> 

----------

root@master [rc=0]# wget -O - http://192.168.1.71/highbank2-tmp/images/tmprBHgDV
/lava.img.bz2 | /bin/bzip2 -dc | dd bs=4M of=/dev/sda
Connecting to 192.168.1.71 (192.168.1.71:80)
-                      1% |                               |  2558k  0:01:16 ETA-                      2% |                               |  5266k  0:01:12 ETA-                      4% |*                              |  8378k  0:01:07 ETA-                      4% |*                              |  8930k  0:01:24 ETA-                      4% |*                              |  9282k  0:01:41 ETA-                      5% |*                              | 10198k  0:01:49 ETA-                      6% |*                              | 

...

143M  0:00:26 ETA-                     75% |***********************        |   145M  0:00:25 ETA-                     76% |***********************        |   147M  0:00:24 ETA-                     77% |************************       |   149M  0:00:23 ETA-                     78% |************************       |   151M  0:00:22 ETA[  179.581460] swapper/0: page allocation failure: order:0, mode:0x120
[  179.587758] [<c00164cc>] (unwind_backtrace+0x0/0x104) from [<c04ec764>] (dump_stack+0x20/0x24)
[  179.596373] [<c04ec764>] (dump_stack+0x20/0x24) from [<c00e725c>] (warn_alloc_failed+0xd8/0x118)
[  179.605159] [<c00e725c>] (warn_alloc_failed+0xd8/0x118) from [<c00e9664>] (__alloc_pages_nodemask+0x524/0x708)
[  179.615159] [<c00e9664>] (__alloc_pages_nodemask+0x524/0x708) from [<c040009c>] (netdev_alloc_frag+0xbc/0xf8)
[  179.625071] [<c040009c>] (netdev_alloc_frag+0xbc/0xf8) from [<c04014dc>] (__netdev_alloc_skb+0x78/0xd0)
[  179.634462] [<c04014dc>] (__netdev_alloc_skb+0x78/0xd0) from [<c0374fec>] (xgmac_rx_refill+0xa4/0x150)
[  179.643763] [<c0374fec>] (xgmac_rx_refill+0xa4/0x150) from [<c0375d2c>] (xgmac_poll+0x370/0x504)
[  179.652547] [<c0375d2c>] (xgmac_poll+0x370/0x504) from [<c040d8dc>] (net_rx_action+0x158/0x240)
[  179.661247] [<c040d8dc>] (net_rx_action+0x158/0x240) from [<c002e448>] (__do_softirq+0xc8/0x1e0)
[  179.670030] [<c002e448>] (__do_softirq+0xc8/0x1e0) from [<c002e9a0>] (irq_exit+0x90/0x98)
[  179.678208] [<c002e9a0>] (irq_exit+0x90/0x98) from [<c000f31c>] (handle_IRQ+0x60/0xc0)
[  179.686123] [<c000f31c>] (handle_IRQ+0x60/0xc0) from [<c0008544>] (gic_handle_irq+0x34/0x68)
[  179.694564] [<c0008544>] (gic_handle_irq+0x34/0x68) from [<c04f6a40>] (__irq_svc+0x40/0x50)
[  179.702907] Exception stack(0xc074bf28 to 0xc074bf70)
[  179.707955] bf20:                   0000001f c0752b68 00000000 00000000 c074a000 c074a000
[  179.716126] bf40: c0788748 c0758050 c074a000 413fc090 c04fe4d8 c074bf7c c074bf80 c074bf70
[  179.724294] bf60: c000f684 c000f688 600f0013 ffffffff
[  179.729346] [<c04f6a40>] (__irq_svc+0x40/0x50) from [<c000f688>] (default_idle+0x38/0x40)
[  179.737521] [<c000f688>] (default_idle+0x38/0x40) from [<c000f8d8>] (cpu_idle+0xa8/0xf4)
[  179.745613] [<c000f8d8>] (cpu_idle+0xa8/0xf4) from [<c04e37b4>] (rest_init+0x6c/0x84)
[  179.753442] [<c04e37b4>] (rest_init+0x6c/0x84) from [<c06fb864>] (start_kernel+0x308/0x364)
[  179.761786] Mem-info:
[  179.764051] Normal per-cpu:
[  179.766838] CPU    0: hi:  186, btch:  31 usd:   0
[  179.771620] CPU    1: hi:  186, btch:  31 usd: 108
[  179.776402] CPU    2: hi:  186, btch:  31 usd: 173
[  179.781184] CPU    3: hi:  186, btch:  31 usd:  18
[  179.785966] HighMem per-cpu:
[  179.788839] CPU    0: hi:  186, btch:  31 usd: 170
[  179.793621] CPU    1: hi:  186, btch:  31 usd:  57
[  179.798404] CPU    2: hi:  186, btch:  31 usd: 153
[  179.803187] CPU    3: hi:  186, btch:  31 usd: 157
[  179.807978] active_anon:1216 inactive_anon:27 isolated_anon:0
[  179.807978]  active_file:462 inactive_file:173221 isolated_file:0
[  179.807978]  unevictable:0 dirty:11408 writeback:2 unstable:0
[  179.807978]  free:852288 slab_reclaimable:3655 slab_unreclaimable:1417
[  179.807978]  mapped:282 shmem:29 pagetables:35 bounce:0
[  179.837258] Normal free:16376kB min:3508kB low:4384kB high:5260kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:685792kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:772160kB mlocked:0kB dirty:45632kB writeback:8kB mapped:0kB shmem:0kB slab_reclaimable:14620kB slab_unreclaimable:5668kB kernel_stack:520kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
[  179.875459] lowmem_reserve[]: 0 26423 26423
[  179.879660] HighMem free:3392776kB min:512kB low:4352kB high:8192kB active_anon:4864kB inactive_anon:108kB active_file:1848kB inactive_file:7092kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3382264kB mlocked:0kB dirty:0kB writeback:0kB mapped:1128kB shmem:116kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:140kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
[  179.918119] lowmem_reserve[]: 0 0 0
[  179.921611] Normal: 14*4kB 6*8kB 5*16kB 2*32kB 0*64kB 0*128kB 1*256kB 1*512kB 1*1024kB 1*2048kB 3*4096kB = 16376kB
[  179.931991] HighMem: 77*4kB 21*8kB 9*16kB 3*32kB 1*64kB 1*128kB 1*256kB 2*512kB 1*1024kB 1*2048kB 827*4096kB = 3392652kB
[  179.942890] 173706 total pagecache pages
[  179.946804] 0 pages in swap cache
[  179.950111] Swap cache stats: add 0, delete 0, find 0/0
[  179.955326] Free swap  = 0kB
[  179.958196] Total swap = 0kB
[  180.012759] 1046784 pages of RAM
[  180.015983] 853238 free pages
[  180.018940] 12531 reserved pages
[  180.022161] 4273 slab pages
[  180.024946] 172310 pages shared
[  180.028079] 0 pages swap cached
[  180.045355] swapper/0: page allocation failure: order:0, mode:0x120