UEC images could be smaller

Just some related information, Matt Zimmerman sent this attached script to
ubuntu-devel to size down the images. This doesn't address the
"out-of-the-box" readiness, but it does give some option.

OK, so the question is more... what should be the default size ? Is the 10Gb default size inherited from some other requirement, like EC2 ?

My main gripe about it is that it seems to require a slightly >10Gb disk allowance to run, so you need to pick the m1.xlarge size to run it.

AMIs for Amazon EC2 should be 10 GB (10240 * 1024 * 1024 bytes)

> AMIs for Amazon EC2 should be 10 GB (10240 * 1024 * 1024 bytes)

The reason for the above is that in ec2 your '/' filesystem is the size of
the filesystem in the image (much as that is in uec). The difference is
that in ec2, the only cost to you is the upload of the larger image
(compressed). So its basically benefit of a larger / with zero cost.

That is obviously different in UEC where that large file image sits on
*your* storage, not amazons.

adding another resize-uec-image script.

This script should work for extending or shrinking an image. At this point I'm leaning towards creating 2G images for uec output, and resizing them up to 10G before bundling for ec2 with this script here.

the 2G images are much more easy to work with, and decompressing a 2G, then resizing up to 10G is quite fast compared to uncompressing the 10G image.

This script does require a 'sudo' call, the other was nicer in that it did not.

Oh yeah, the benefit of this script over the other is that it creates a new filesystem rather than resizing an existing one. This overcomes the one negative of a 2G image that I could think of, in that an 'mke2fs' for a 2G filesystem would possibly create fewer inodes than would reasonably be used on the filesystem after it was resized to 10G.

Hm, I'm slightly concerned about the need for root rights in that script, as it basically forces the "cloud customer" that wants to bundle a UEC image to use a client system on which he has root rights (while he could use any system before). Please post your new script to the ubuntu-devel resize-image thread. Maybe someone creative will come up with a solution that allows to expand the fs in a reliable way without making the whole process dependent on you having root rights.

Thierry: Half joking, I'll point out that anybody can have root on an Ubuntu system for an hour for only $0.10 with EC2.

More seriously: If the benefit of the rsync approach is only to increase the number of inodes, perhaps the original mke2fs could be run with the -N option to increase the number of inodes to something reasonable for both 2GB and 10GB.

The truncate method certainly seems like it would be faster than the rsync.

One of the nice things about the original script was that it didn't require root (to mount filesystems etc.). Regarding the inodes consideration, I *think* that growing the number of inodes works fine, so this shouldn't be an issue if we provide a smallish image and use the script to make it larger if needed.

I just verified that resizing an image now also increases number of inodes.

$ truncate my.img --size 1M
$ mke2fs -F my.img >/dev/null
mke2fs 1.41.9 (22-Aug-2009)
$ dumpe2fs my.img | grep 'Inode count'
dumpe2fs 1.41.9 (22-Aug-2009)
Inode count: 128
$ truncate my.img --size 100M
$ resize2fs my.img 100M
resize2fs 1.41.9 (22-Aug-2009)
Resizing the filesystem on my.img to 102400 (1k) blocks.
The filesystem on my.img is now 102400 blocks long.
$ dumpe2fs my.img | grep 'Inode count'
dumpe2fs 1.41.9 (22-Aug-2009)
Inode count: 1664

So, my proposed script has little to no value. The benefit to it now is that it works on shrink or grow. We'll need to change Matt's script to determine if input size is < or > than current size, and perform the truncate after or before the resize. Then, I think its done.

hopefully final resize script. I know this is a bad place for revision control, so hopefully this will be the last thing put here. Then, we'll put this somewhere in bzr.

Newly attached version:
 - works to increase or decrease size of filesystem
 - catches errors and gives error messages (rather than set -e)
 - only uses zcat when necessary (for copies of our 10G uncompressed images, using straight copy is ~25% faster on clock, uses less sys resources too)
 - changes usage from:
  resize-uec-image <old> <new> <size>
  to
  resize-uec-image <old> <size> [new]
  ie, works in place if new is not given (this will not work on a compressed file, obviously).

The official branch is at lp:uec-tools
https://code.launchpad.net/~ubuntu-core-dev/uec-tools/ubuntu

Can it handle the new default .img.tar.gz format ? At first look it seems it handles only .img or .img.gz...

On Fri, 9 Oct 2009, Thierry Carrez wrote:

> The official branch is at lp:uec-tools
> https://code.launchpad.net/~ubuntu-core-dev/uec-tools/ubuntu
>
> Can it handle the new default .img.tar.gz format ? At first look it
> seems it handles only .img or .img.gz...

I can make it do that, but personally I even felt like gunzip support was
somewhat over-featured. Do you think it needs to support .img.tar.gz ?

right now, the image file is distributed in 2 ways:
- inside a multi-file archive base.tar.gz:
  <base>.img
  <base>-kernel-flavor
  <base>-initrd-flavor
- inside a single-file archive <base>.img.tar.gz

Do you think we need to support resizing from that archive?

I'd rather just drop uncompres/extract support all together.

Sure, there is no point in having .img.gz support while we don't ship anything under that format anymore.
About img.tar.gz support: having it inside the script would just avoid having to discover the magic -S tar option.

i think the use case of downloading a .tar.gz file and then resizing it to an image to upload is probably not that common.

That said, i added code for it. I've also put this into a branch so it at least can be somewhat controlled.

I doubt that its the best place for it, but it needs to go somewhere. so:

https://code.launchpad.net/~ubuntu-on-ec2/ubuntu-on-ec2/uec-tools

I consider this bug fix-released at this point. Builds at http://uec-images.ubuntu.com/karmic/ are now 2G filesystem images. Their sizes are (20091012):
182M karmic-uec-amd64.tar.gz
173M karmic-uec-i386.tar.gz

That is down from a 10G filesystem with download sizes of (20091007):
602M ubuntu-uec-karmic-amd64.img.gz
592M ubuntu-uec-karmic-i386.img.gz

The resize-uec-image tool can resize those with basically no ill effects.

The only applicable change to vm-builder that isn't in the karmic-packaged version is:
1. run apt-get clean
2. change ec2 image sizes from 10G to 2G