Comment 5 for bug 1619362

Revision history for this message
Steve Langasek (vorlon) wrote : Re: [Bug 1619362] Re: images should not be largely bigger than the actual content

On Thu, Sep 01, 2016 at 06:34:51PM -0000, Oliver Grawert wrote:
> ogra@anubis:~$ ls -l datengrab/images/snappy/u-image-pi2.img
> -rw-r--r-- 1 root root 4000000000 Sep 1 17:30 datengrab/images/snappy/u-image-pi2.img
> ...
> ogra@anubis:~$ du -sh datengrab/images/snappy/u-image-pi2.img
> 3,8G datengrab/images/snappy/u-image-pi2.img

> this image was created using
> http://people.canonical.com/~ogra/snappy/pi2-model.assertion with
> http://people.canonical.com/~ogra/snappy/ubuntu-image_0.5_amd64.snap
> (snapcraft build from a PR from mvo with some extra fixes to master) ...

Ok, I've been able to reproduce the problem now. The issue is that you were
probably using sensible default options, and therefore ubuntu-image was
doing all the building in /tmp; whereas I've been futzing with ubuntu-image
internals and was therefore doing all of my builds in a named working
directory; and the final stage of the build is to move the image from the
working directory to its output location with shutil.move(), which falls
back to shutil.copy2() for a cross-filesystem move (like if /tmp is a
tmpfs), and this function is not sparse-aware.

  https://bugs.python.org/issue10016

So if you're doing all the building in your current directory, you get a
sparse file that is renamed and there's no problem. If you're doing it
under /tmp, you get a sparse file that is then copied by a non-sparse-aware
python function.

I believe this may also explain the report of bug #1619351, and believe that
also means that switching to os.truncate() won't fix the problem.

> the actual content of the image are a 85MB pi2-kernel snap
> and a 56MB ubuntu-core snap plus whatever overhead the image creation adds
> (pre-created writable bits, unpacked gadget, copied kernel and initrd
> files etc), nearly 300MB sounds actually pretty large for that content.

snap prepare-image stores two copies of the snaps, one in
/var/lib/snapd/seed/snaps and one in /var/lib/snapd/snaps. So it doesn't
look like filesystem overhead is the cause of the increase here.