Merge lp:~elopio/u1-test-utils/setup-vm into lp:u1-test-utils

[Leo Arias]

Merged the setup_vm project into u1testutils.

[Leo Arias (elopio)]

Currently blocked by: https://bugs.launchpad.net/ubuntuone-servers/+bug/1169218
We can work it around making a new Consumer and User on Ubuntu Pay.

[Leo Arias (elopio)]

Ready for review!

[Vincent Ladeuil (vila)]

Thanks, LGTM

review: Approve

=== added directory 'setup_vm'
=== added file 'setup_vm/ISSUES'
--- setup_vm/ISSUES	1970-01-01 00:00:00 +0000
+++ setup_vm/ISSUES	2013-04-17 01:29:27 +0000
@@ -0,0 +1,9 @@
+* while setting up sso.local .env/db/postgresql.log ends up with:
+
+FATAL:  role "postgres" does not exist
+
+=> This may be "expected" according to mfoord
+
+
+* bzr+ssh://bazaar.launchpad.net/~canonical-isd-hackers/isd-configs/sso-config/ is missing in the sso config and local.cfg should also be generated to refer to that.
+
 
=== added file 'setup_vm/NOTES'
--- setup_vm/NOTES	1970-01-01 00:00:00 +0000
+++ setup_vm/NOTES	2013-04-17 01:29:27 +0000
@@ -0,0 +1,259 @@
+Here we keep notes about the tests we will run on these vms. Once finished all
+the TODOs, this will be probably moved to moztrap, or immediately and 
+magically automated.
+
+== On the host ==
+1. Install the requirements:
+  $ sudo apt-get install libvirt-bin qemu virtinst virt-manager
+
+2. Install the apt cache:
+  $ sudo apt-get install squid-deb-proxy
+
+3. Enable the cache for launchpad's private ppas:
+  See point 9 of the README.
+
+4. Configure the vms:
+
+  # This directory will be used as download cache for the Ubuntu images.
+  $ mkdir ~/installers/ubuntu
+  # This directory will store the disk images for the virtual machines.
+  $ mkdir ~/images
+  $ editor ~/vms.conf
+
+  vm.ram_size=2048
+  vm.cpus=2
+  # Tweak the cpu model according to your needs
+  vm.cpu_model=amd64
+  vm.download_cache=~/installers/ubuntu
+  vm.images_dir=~/images
+  # Tweak according to your squid-deb-proxy setup, 8000 is the default port. 
+  # Use {your-ip} or any address reachable by the vms (keeping in mind that
+  # avahi's .local domain may not be up in the early stages of the install
+  vm.apt_proxy = http://{your-ip}:800
+  vm.launchpad_id=your-launchpad-id
+  # This is the ssh key of your host machine. Make sure that you have
+  uploaded it to https://launchpad.net/~/+editsshkeys
+  vm.ssh_authorized_keys = ~/.ssh/id_rsa.pub
+  # This is the ssh key for your VMs. It might be safer if it's different from
+  # your machine's key. Make sure that you have uploaded it to launchpad too.
+  vm.ssh_keys=~/.ssh/rsa-vms
+  # A default user (ubuntu) is created, here is its password
+  vm.password = you-re-on-you-own-use-a-simple-or-complex-password
+  # Go to https://launchpad.net/~/+archivesubscriptions to get the password for
+  # the Ubuntu One hackers PPA. Click the View link on the PPA row, and on the
+  # sources list entries you will see something like
+  # https://your-launchpad-id:the-password@...
+  ppa.ubuntuone_hackers.password=the-password
+
+  sso.address=sso.local
+  pay.address=pay.local
+  u1.address=u1.local
+
+5. Get the branch:
+
+  $ bzr branch lp:~online-services-qa/u1-test-utils/setup_vm
+  $ cd setup_vm
+
+6. Download the image for the server (do so every time you want to use a
+   fresh image):
+
+  $ ./bin/setup_vm.py precise-server-pristine --download
+
+# TODO use LXCs insteal of virtual machines for the servers.
+7. Install the pristine server vm:
+
+  $ ./bin/setup_vm.py raring-pristine --install
+
+8. Download the image for the desktop (do so every time you want to use a
+   fresh image)::
+
+  $ ./bin/setup_vm.py raring-desktop-pristine --download
+
+8. Install the pristine desktop vm:
+
+  $ ./bin/setup_vm.py raring-pristine --install
+
+# TODO now it might be better to start all the servers on the same VM.
+9. Set the SSO server (Only needed for in-dash payments tests):
+
+  $ ./bin/setup_vm.py sso --install
+  $ virsh start sso
+  $ ssh [e-mail] ~/bin/run-for-u1
+
+10. Set up the Pay server (Only needed for in-dash payments tests):
+
+  $ ./bin/setup_vm.py pay --install
+  $ virsh start pay
+  $ ssh [e-mail] ~/bin/run-for-u1
+
+11. Set up the U1 server (Only needed for in-dash payments tests):
+
+  $ ./bin/setup_vm.py u1 --install
+  $ virsh start u1
+  $ ssh [e-mail] ~/bin/run
+
+12. Set up the Filesync server (Only needed for in-dash payments tests):
+
+  TODO. See below the notes to set it up on the same u1 server.
+  TODO. Explore how to set it up in a different server.
+
+13. Set up the Music Search server:
+
+  TODO. Do we need it?
+
+13. Install the Curucú server (Only needed for Smart Scopes tests):
+
+  TODO.
+
+17. Install the desktop machine that will run the tests:
+
+  $ ./bin/setup_vm.py unity-prevalidation --install
+
+
+Set up the in-dash payments tests against the local servers
+===========================================================
+
+This is for the happy path.
+
+1. Sign in to the unity-prevalidation machine using virt-manager.
+
+2. Kill syncdaemon if it is running. (As this is a pristine machine, this is not necessary)
+
+3. Open seahorse and delete the Ubuntu One credentials, if present. (As this is a pristine machine, this is not necessary)
+
+3. Log in with Staging Ubuntu SSO:
+
+  # TODO: Currently we can just connect to production. This is a regression,
+  # see bug http://pad.lv/1161067
+  $ ~/bin/run-sso-client
+
+  Click the "Log-in with my existing account." link.
+  Fill the form with:
+    # TODO we need to create the user with the API helpers on u1-test-utils.
+    Email address: [e-mail]
+    Password: Hola123*
+  Click the "Sign In" button.
+
+  Autopilot test:
+  http://bazaar.launchpad.net/~elopio/ubuntu-sso-client/autopilot/view/head:/ubuntu_sso/tests/acceptance/test_ubuntu_sso_client.py
+  
+4. Start syncdaemon:
+
+  # TODO we currently don't have a filesync server.
+  $ ~/bin/run-syncdaemon
+
+5. Start the unity musicstore daemon:
+
+  $ ~/bin/run-unity-lens-music
+
+6. Start the control panel.
+
+  # TODO probably not neccessary.
+
+7. Add a credit card to the user.
+
+  # Use the API helpers on u1-test-utils.
+  # We still need to log in to the pay webiste first. See http://pad.lv/1144523
+
+8. Enable the automatic payments for the user.
+
+  # TODO wait for http://ur1.ca/d6z4n to land and then extend the u1-test-utils
+  # API helpers to do this.
+
+# For paypal payments, we would still need to do a lot of stuff on the website.
+# TODO do we need to test paypal payments?
+
+Run the in-dash payments tests against the local servers
+===========================================================
+
+This is the happy path.
+
+1. Super+M.
+2. Search for 'hendrix'.
+3. Wait for the search to complete.
+4. Click the first album
+5. Click the Download button.
+6. Enter the password.
+7. Click the Purchase button.
+
+All the tests are now documented in moztrap.
+
+Set up the filesync server on the same machine as ubuntuone-servers
+====================================================================
+
+  # On the host
+  $ ssh [e-mail]
+  # On the vm.
+  $ bzr branch lp:ubuntuone-filesync
+  $ cd ubuntuone-filesync
+  $ make link-sourcedeps
+  $ editor lib/u1backends/db/config.py
+  Change line 10 from:
+  db_dir = os.path.abspath(os.path.join(get_tmpdir(), 'db1'))
+  To:
+  db_dir = os.path.abspath(
+      os.path.join('/home/ubuntu/ubuntuone-servers/tmp', 'db1'))
+  $ make start
+  # Use ubuntuone-servers S4, statsd and AMQP.
+  $ ln -s ../../ubuntuone-servers/tmp/rabbitmq-ubuntuone.port tmp/rabbitmq-ubuntuone.port
+  $ ln -s ../../ubuntuone-servers/tmp/statsd.port tmp/statsd.port
+  $ ln -s ../../ubuntuone-servers/tmp/s4.port tmp/s4.port
+  $ make start-supervisor start-filesync-dummy-group
+  # TODO we can also start-filesync-oauth-group. Ask #u1-di.
+
+Set up the curucu server
+========================
+
+  # There is a dummy website that accesses the server:
+  # https://productsearch.ubuntu.com/smartscopes/v1/dashmock?geo_store=US
+  # On ~pedronis/curucu/canonistack-deploy, there's a README with the
+  # instructios to deploy curucu on a canonistack machine with Juju.
+  # TODO try to deploy it with juju on our vms.
+  # TODO we need an amazon key.
+  # On the host
+  $ ./bin/setup_vm.py precise-curucu-server --install
+  $ virsh start precise-curucu-server
+  $ ssh ubuntu@precise-curucu-server
+  # On the vm.
+  # TODO install the dependencies, what are the dependencies?
+  $ bzr branch lp:curucu
+  $ cd curucu
+  $ editor try.cfg
+
+  [amazon]
+  key = amazon key
+  secret = amazon secret
+
+  [u1ms]
+  service_url = http://musicsearch.ubuntu.com/v1/
+
+  [feedback_store]
+  interval = 4
+  # when set to empty storing feedback is disabled
+  store_directory = /tmp/feedback
+  
+  $ CURUCU_CFG=try.cfg U1CONFIG=configs/development-lazr.conf PYTHONPATH=.:lib gunicorn -w 2 -k gevent_wsgi -b 0.0.0.0:8000 curucu.wsgi:app
+
+Mounting guest disk images on the host
+======================================
+
+This requires root access (what did you expect ;-p) and the current
+directory should contain the vm disk images.
+
+apt-get install qemu-nbd
+
+root@saw:/# modprobe nbd # once
+root@saw:/# mkdir /mnt/disk1
+root@saw:/# mkdir /mnt/seed
+
+root@saw:/# qemu-nbd -c /dev/nbd0  raring-pristine.qcow2
+root@saw:/# mount /dev/nbd0p1 /mnt/disk1
+root@saw:/# umount /mnt/disk1/
+root@saw:/# qemu-nbd -d /dev/nbd0
+
+root@saw:/# qemu-nbd -c /dev/nbd1  raring-test.seed
+root@saw:/# mount /dev/nbd1 /mnt/seed
+mount: warning: /mnt/seed seems to be mounted read-only.
+root@saw:/# umount /mnt/seed
+root@saw:/# qemu-nbd -d /dev/nbd1
+/dev/nbd1 disconnected
 
=== added file 'setup_vm/README'
--- setup_vm/README	1970-01-01 00:00:00 +0000
+++ setup_vm/README	2013-04-17 01:29:27 +0000
@@ -0,0 +1,241 @@
+Getting started:
+================
+
+1. Install the dependencies:
+
+    sudo apt-get install bzr python-testtools python-yaml
+    sudp apt-get install libvirt-bin qemu qemu-utils virtinst
+    sudo apt-get install qemu-kvm-spice python-spice-client-gtk
+    
+    (Optional):
+    To use a gui manager to see the desktop:
+    sudo apt-get install virt-manager
+    
+    To use Apt proxy to speed up multiple downloads of the same packages:
+    sudo apt-get install squid-deb-proxy 
+    (See point 9 about configuring the apt cache.)
+    
+2. Reboot to allow kvm to activate on the running kernel.
+
+3. Get the code:
+
+    bzr branch lp:~online-services-qa/u1-test-utils/setup_vm
+
+4. Run the tests:
+
+    cd setup_vm
+    ./selftest.py
+
+    (The test_install_from_seed will require you to enter your password
+     because it executes a command with sudo. Your user must be a sudoer.)
+
+5. Configure a virtual machine:
+
+    Write the file ~/vms.conf with something like this:
+
+    vm.ram_size=2048
+    vm.cpus=2
+    vm.cpu_model=amd64
+
+    [raring-pristine]
+    vm.name = raring-pristine
+    vm.update = True
+    vm.packages = bzr, ubuntu-desktop, avahi-daemon
+    vm.release=raring
+    vm.ssh_authorized_keys = {your SSH public key path (~ is allowed, eg ~/.ssh/id_rsa.pub)}
+
+    Create the ~/.config/setup_vm directory where cloud-init configuration
+    files will be stored for each vm. Alternatively, you can create a
+    directory (~/vms) where you want and add the following line in ~/vms.conf:
+
+    vm.vms_dir=~/vms
+
+    Optionally, you can setup scripts to be executed as root or as the
+    ubuntu user just before the vm is powered off:
+
+    vm.root_script = {path to a script on the host}
+    vm.ubuntu_script = {path to a script on the host}
+
+    These scripts *must* specify a shebang line and can be written in any
+    language that can be run from a shebang line.
+
+    You can also ask for some local scripts to be uploaded with:
+
+    vm.uploaded_scripts = sso/run, sso/run-for-pay
+
+    The config options in these scripts will be expanded before the upload.
+
+    PPAS needs a bit of care to setup, as an example, the unity experimental
+    prevalidation PPA is configured by going to
+    https://launchpad.net/~ubuntu-unity/+archive/experimental-prevalidation
+    Click the "Technical details about this PPA" link.
+    Select the distribution from the combo box.
+    Copy the apt line below:
+
+    vm.apt_sources=deb http://ppa.launchpad.net/ubuntu-unity/experimental-prevalidation/ubuntu {vm.release} main|52D62F45
+
+    The page displays Signing Key: 1024R/52D62F45. Please note that only
+    52D62F45 should be specified, and that the url and the key are separated
+    by '|' with no intervening spaces. 
+
+    For a private PPA, make sure to include your launchpad id and your
+    password for that PPA in the URL. It would look something like this:
+
+    vm.apt_sources = deb https://<lp id>:<ppa password>@private-ppa.launchpad.net/a-user/ppa-name/ubuntu {vm.release} main|<ppa key>
+
+6. (Optional) Create a system-wide vms.conf.
+
+   In some cases, some options are better defined in a system-wide config
+   file (/etc/libvirt/vms.conf). This file is queried if no definitions are
+   found in ~/vms.conf and can define a no-name section and vm sections.
+
+7. (Optional) You can configure the location where the image will be
+   downloaded with something like this in the vms.conf file:
+
+    vm.download_cache=~/installers/ubuntu 
+
+8. (Optional) You can configure the location where the virtual machines will
+   be stored with something like this in the vms.conf file:
+
+    vm.images_dir=~/images
+
+9. (Optional) Set up an apt cache, so repeated virtual machine installs will
+   be faster, downloading the packages from the cache instead of an Ubuntu
+   archive mirror:
+    
+    Add this to the vms.conf file:
+    
+    vm.apt_proxy = http://{your-squid-deb-proxy-ip}:8000
+
+    If you need to install packages from non official Ubuntu repositories, you
+    will need to configure the proxy. For example, common tasks would require
+    to access Launchpad public and private PPAs. For that, write the file 
+    /etc/squid-deb-proxy/mirror-dstdomain.acl.d/20-local-vms with:
+
+    # /etc/squid-deb-proxy/mirror-dstdomain.acl.d/20-local-vms
+
+    # network destinations that are allowed by this cache targeted at
+    # locally installed vms
+
+    # launchpad personal package archives
+    ppa.launchpad.net
+    # launchpad private personal package archives
+    private-ppa.launchpad.net
+
+    After that, restart the proxy:
+    
+    sudo restart squid-deb-proxy
+
+    Each time you modify some file under /etc/squid-deb-proxy, don't forget to
+    restart the service.
+
+10. Download the image:
+
+    ./bin/setup_vm.py --download raring-pristine
+
+    (This command will require you to enter your password because the
+     directory where the image will be downloaded might be under control of
+     the root user. Your user must be a sudoer. A pending task is to ask for
+     the password just when needed.)
+
+11. Install the virtual machine:
+
+    ./bin/setup_vm.py --install raring-pristine
+
+    (This command will require you to enter your password because some of the
+     operations it executes require root access. Your user must be a sudoer.)
+
+12. You can ssh into the virtual machine:
+
+    virsh start raring-pristine
+    ssh [e-mail]
+
+    No password is needed because your SSH public key is authorized.
+
+13. You can run the virtual machine from virt-manager to get a graphical user
+    interface:
+
+    Open virt-manager.
+    Right-click on the machine and select Run.
+    You will be presented with the display manager greeter.
+    Log in with the user ubuntu, password ubuntu.
+    
+    (You may need to do the following if virt-manager says it can't connect,
+     sudo usermod -a -G libvirtd $USER (replace $USER with your username)
+     and reboot the system)
+    
+14. (Optional) You can set up a "throw away" virtual machine on top of
+    another. We call it "throw away" because all modifications happening there
+    won't affect the disk image of the backing on virtual machine.
+
+    In the vms.conf described above add:
+
+    [raring-test]
+    vm.name = raring-test
+    vm.update = False
+    vm.release = raring
+    vm.ssh_authorized_keys = {your SSH public key}
+    # The name of the disk image used as a base
+    vm.backing = raring-pristine.qcow2
+
+    Create the new vm with:
+
+    ./bin/setup_vm.py --install raring-test
+
+    The vm creation and boot should be faster.
+    
+15. (Very optional) A few commands for virsh that may be of use:
+    virsh list (shows what is running)
+    virsh start x (start a vm with the name x)
+    virsh destroy x (force shutdowns a vm with the name x)
+    virsh undefine x [--remove-all-storage] (Deletes a vm with the name x)
+
+16. (Optional) Raise sudo timeout.
+
+    If you run into vm installs taking too long and waiting for your
+    password to fisnish, you can change the default value (15 minutes) by
+    adding a file in /etc/sudoers.d containing:
+
+    Defaults:<your login here>	timestamp_timeout=60
+
+    This will setup the timeout to 60 minutes.
+
+17. (Optional) Setup launchpad access for the guests
+
+   If you need to access launchpad private branches from the guests, you'll
+   need to setup ssh launchpad access (if you only need access to public
+   branches, http is good enough and you don't even need to 'bzr
+   launchpad-login'):
+
+   - You need to create an ssh key dedicated to the guests, it has to be
+     passwordless and the public part uploaded to your launchpad profile.
+     You can generate a new key pair (replacing <user> with your launchpad
+     id) with:
+
+     $ (cd ~/.ssh ; ssh-keygen -f <user>@setup_vm -N '' -C '<user>@setup_vm')
+
+     This will create two files: '<user>@setup_vm' and
+     '<user>@setup_vm.pub' in your ~/.ssh directory.
+
+     Upload the later at https://launchpad.net/~/+editsshkeys
+
+     The keys are created in your .ssh directory so you can test that they
+     work against launchpad without involving a vm.
+
+     Note that if you create vms from different hosts, you'll need to either
+     copy the same keys on all the hosts or create a pair on each of them
+     (or any combination as long as the public keys are uploaded to
+     launchpad ;).
+
+   - You need to set vm.launchpad_id to <user>. This will trigger running
+     'bzr launchpad-login <user>' in the guest and copy
+     ~/.ssh/<user>@setup_vm (the private key) from your host to the guest.
+
+   - The bazaar.launchpad.net host ssh key needs to be known or you'll get
+     prompted to add it (which is not nice for scripts). This can be fixed
+     by issuing the following command from the {vm.ubuntu_script}:
+
+     $ ssh-keyscan bazaar.launchpad.net >>~/.ssh/known_hosts
+
+     This will probably be automated at some point in the future.
+
 
=== added file 'setup_vm/TODO'
--- setup_vm/TODO	1970-01-01 00:00:00 +0000
+++ setup_vm/TODO	2013-04-17 01:29:27 +0000
@@ -0,0 +1,129 @@
+* running setup_vm --install I-dont-exist raises an obscure error. Checking
+   that the config section exist for the vm would allow reporting a better
+   error.
+
+* Too many tests become too hard to write because their execution requires a
+   real vm. This can be addressed in much the same way than
+   requires_known_reference_image(), i.e. setup a real vm once (outside of
+   selftest execution for now) and then use throw-away vms. But even that
+   may be too costly and may need to wait for lxc/chroot support.
+
+* running --ssh-keygen twice gives an awful error message.
+
+* Find whether or not we should really support chroots or if lxcs are good
+   enough (roughly: if they can be set up as fast as chroots but provide
+   more features, just optimize the backing-on scenario).
+
+* Investigate btrfs support to use snapshots for nested backing.
+   This may not be appropriate with kvms but will surely shine for
+   lxc/chroot.
+
+* copying a file (with expanded options or not) from the host to the guest
+   is hard (even internally). There should be a way to more simply describe
+   a list of file/directories to install (with user:group and chmod bits).
+
+* Alternatively, we can allow the guest to access the host via ssh (after
+   all, we're installing a private key in the guest so we trust it enough
+   for that already).
+
+* As a first step, we can define vm.scripts as a list of relative paths on
+   the host, that will be option expanded into vm.config_dir and uploaded
+   from there into ~ubuntu/bin.
+
+* Provide the guest with config file containing the values used to build
+   this vm. From there, the guest itself would be able to expanded options
+   in files acquired from the host (including files modified after the vm
+   has been built/started which will help during dev/debug).
+
+* vm.ubuntu_script is kind of an implementation leak from cloud-init, that's
+   the default user there and comes with some nice properties but strictly
+   speaking setup_vm cares about having *a* user, no matter how it is named
+   so the option could be named vm.user_script. In any case, the features we
+   rely on from using ubuntu should be tested if only to document them.
+
+* we need a way to run scripts on the host while expanding the config option
+   for a given vm (see sso/test that needs at least the sso.url).
+
+*  launchpad interaction requires the launchpad host key.
+
+   => ssh-keyscan bazaar.launchpad.net >>~/.ssh/known_hosts does the
+   trick but it would be nice to automate it.
+
+* document the /etc/avahi/ fix required to use avahi with vms
+
+* lag times between significant hosts should be collected (not specific to
+   setup_vm but related to the use of the vms).
+
+* Add a way to display a vm configuration à la 'bzr config'
+
+* from the addresses below, find a way to test if some fixed subspace can be
+   safely used (de:ad:be:ef  or something...) or just steal some unused MAC
+   prefix (vbox's one ? or vmware's one ? or... the sky is the limit ;)
+
+
+$ sudo grep -n 'mac address' /etc/libvirt/qemu/*.xml
+/etc/libvirt/qemu/essex-precise.xml:45:                <mac address='52:54:00:26:3c:20'/>
+/etc/libvirt/qemu/freebsd8.xml:39:                     <mac address='08:00:27:5f:9f:06'/>
+/etc/libvirt/qemu/gentoo.xml:39:                       <mac address='08:00:27:da:65:cd'/>
+/etc/libvirt/qemu/indicator-sync.xml:45:               <mac address='52:54:00:43:15:9c'/>
+/etc/libvirt/qemu/pkgimporter-lucid.xml:45:            <mac address='52:54:00:95:4e:dc'/>
+/etc/libvirt/qemu/precise-cloud.xml:48:                <mac address='52:54:00:68:aa:af'/>
+/etc/libvirt/qemu/precise-pristine.xml:48:             <mac address='52:54:00:d5:52:06'/>
+/etc/libvirt/qemu/precise-pristine.xml:48:             <mac address='52:54:00:70:a3:64'/>
+/etc/libvirt/qemu/precise-server-pristine.xml:51:      <mac address='52:54:00:14:d5:be'/>
+/etc/libvirt/qemu/precise-test.xml:48:                 <mac address='52:54:00:25:8b:56'/>
+/etc/libvirt/qemu/quantal-cloud.xml:48:                <mac address='52:54:00:e6:6a:df'/>
+/etc/libvirt/qemu/quantal-pristine.xml:48:             <mac address='52:54:00:f5:95:0e'/>
+/etc/libvirt/qemu/quantal-pristine.xml:48:             <mac address='52:54:00:b9:68:11'/>
+/etc/libvirt/qemu/quantal-test.xml:48:                 <mac address='52:54:00:21:3b:7b'/>
+/etc/libvirt/qemu/raring-current.xml                   <mac address='52:54:00:d9:ca:70'/>
+/etc/libvirt/qemu/raring-in-dash-pristine.xml:51:      <mac address='52:54:00:c7:f9:ee'/>
+/etc/libvirt/qemu/raring-in-dash-test.xml:51:          <mac address='52:54:00:a5:83:b9'/>
+/etc/libvirt/qemu/raring-pristine.xml:48:              <mac address='52:54:00:07:09:cb'/>
+/etc/libvirt/qemu/raring-pristine.xml:48:              <mac address='52:54:00:34:7f:62'/>
+/etc/libvirt/qemu/raring-scope-base.xml:42:            <mac address='52:54:00:9c:0d:1c'/>
+/etc/libvirt/qemu/raring-scope-test.xml:42:            <mac address='52:54:00:8a:b1:d6'/>
+/etc/libvirt/qemu/raring-test.xml:48:                  <mac address='52:54:00:23:b6:8d'/>
+/etc/libvirt/qemu/raring-test.xml:51:                  <mac address='52:54:00:e3:05:db'/>
+/etc/libvirt/qemu/sso.xml:51:                          <mac address='52:54:00:f1:88:84'/>
+/etc/libvirt/qemu/u1test-quantal.xml:48:               <mac address='52:54:00:04:7b:45'/>
+/etc/libvirt/qemu/u1test-quantal.xml:48:               <mac address='52:54:00:04:7b:45'/>
+/etc/libvirt/qemu/u1test2-quantal.xml:45:              <mac address='52:54:00:df:35:6c'/>
+/etc/libvirt/qemu/u1test2-quantal.xml:45:              <mac address='52:54:00:df:35:6c'/>
+/etc/libvirt/qemu/xp-32bits.xml:58:                    <mac address='52:54:00:86:aa:99'/>
+/etc/libvirt/qemu/xp64bits.xml:46:                     <mac address='52:54:00:0a:5e:7c'/>
+
+* while using fixed IP addresses is one way to address known_hosts
+   stability, another way is to rely on `ssh-keygen -R` when installing the
+   host (to remove the previous mapping between the key and the ip) and then
+   add the new key with ssh-keyscan (or something else that doesn't require
+   the guest to be up and running).
+
+* make actions verbose and obey -q (at least for tests) so we get some
+   feeback about what is executed.
+
+* add a --delete action to make sure we clean up the config_dir
+
+* rework FileMonitor, ConsoleMonitor design, the actual result smells (pass
+   in _wait_for_install_with_seed, really ?).
+
+* investigate using an upstart job like MAAS:
+
+ write_poweroff_job() {
+   cat >/etc/init/maas-poweroff.conf <<EOF
+   description "poweroff when maas task is done"
+   start on stopped cloud-final
+   console output
+   task
+   script
+     [ ! -e /tmp/block-poweroff ] || exit 0
+     poweroff
+   end script
+EOF
+   # reload required due to lack of inotify in overlayfs (LP: #882147)
+   initctl reload-configuration
+}
+
+* look at PXE, interesting read may include:
+
+  http://ubuntuforums.org/archive/index.php/t-1713845.html
 
=== added directory 'setup_vm/bin'
=== added file 'setup_vm/bin/__init__.py'
=== added file 'setup_vm/bin/setup_vm.py'
--- setup_vm/bin/setup_vm.py	1970-01-01 00:00:00 +0000
+++ setup_vm/bin/setup_vm.py	2013-04-17 01:29:27 +0000
@@ -0,0 +1,1203 @@
+#!/usr/bin/env python
+"""
+Setup a virtual machine from a config file.
+
+Note: Most of the operations requires root access and this script uses ``sudo``
+to get them.
+
+"""
+import argparse
+import base64
+from cStringIO import StringIO
+import errno
+import os
+import subprocess
+import sys
+import tempfile
+import time
+
+
+from bzrlib import (
+    config,
+    osutils,
+    transport,
+    urlutils,
+    )
+import yaml
+
+# Work around a bug in bzrlib.config forbidding some constructs in templates.
+# Namely, spaces are invalid as an identifier and therefore should not match
+# below.
+config._option_ref_re = config.lazy_regex.lazy_compile('({[^ {},\n]+})')
+
+
+class VmMatcher(config.NameMatcher):
+
+    def match(self, section):
+        if section.id is None:
+            # The no name section contains default values
+            return True
+        return super(VmMatcher, self).match(section)
+
+    def get_sections(self):
+        matching_sections = super(VmMatcher, self).get_sections()
+        return reversed(list(matching_sections))
+
+
+class VmStore(config.LockableIniFileStore):
+    """A config store for options specific to a directory."""
+
+    def __init__(self, directory, file_name, possible_transports=None):
+        t = transport.get_transport_from_path(
+            directory, possible_transports=possible_transports)
+        super(VmStore, self).__init__(t, file_name)
+        self.id = 'vm'
+
+
+def system_config_dir():
+    return '/etc/libvirt'
+
+
+class VmStack(config.Stack):
+    """Per-directory options."""
+
+    def __init__(self, name):
+        """Make a new stack for a given vm.
+
+        The following sections are queried:
+
+        * the ``name`` section in ./vms.conf,
+        * the no-name section in ./vms.conf
+        * the ``name`` section in ~/vms.conf,
+        * the no-name section in ~/vms.conf
+        * the ``name`` section in /etc/libvirt/vms.conf,
+        * the no-name section in /etc/libvirt/vms.conf
+
+        :param name: The name of a virtual machine.
+        """
+        self.local_store = VmStore('.', 'vms.conf')
+        user_store = VmStore(os.environ['HOME'], 'vms.conf')
+        self.system_store = VmStore(system_config_dir(), 'vms.conf')
+        # FIXME: Only available in bzr-2.6b3 :-/ -- vila 2012-01-31
+        # dstore = self.get_shared_store()
+        super(VmStack, self).__init__(
+            [VmMatcher(self.local_store, name).get_sections,
+             VmMatcher(user_store, name).get_sections,
+             VmMatcher(self.system_store, name).get_sections,
+             ],
+            user_store, mutable_section_id=name)
+
+
+def path_from_unicode(path_string):
+    if not isinstance(path_string, basestring):
+        raise TypeError
+    return os.path.expanduser(path_string)
+
+
+class PathOption(config.Option):
+
+    def __init__(self, *args, **kwargs):
+        """A path option definition.
+
+        This possibly expands the user home directory.
+        """
+        super(PathOption, self).__init__(
+            *args, from_unicode=path_from_unicode, **kwargs)
+
+
+def register(option):
+    config.option_registry.register(option)
+
+
+register(config.Option(
+        'vm', default=None,
+        help='''The name space defining a virtual machine.
+
+This option is a place holder to document the options that defines a virtual
+machine and the options defining the infrastructure used to manage them all.
+
+For qemu based vms, the definition of a vm is stored in an xml file under
+'/etc/libvirt/qemu/{vm.name}.xml'. This is under the libvirt package control
+and is out of scope for setup_vm.py.
+
+There are 3 other significant files used for a given vm:
+
+- a disk image mounted at '/' from '/dev/sda1':
+  '{vm.images_dir}/{vm.name}.qcow2'
+
+- a iso image available from '/dev/sdb' labeled 'cidata':
+  {vm.images_dir}/{vm.name}.seed which contains the cloud-init data used to
+  configure/install/update the vm.
+
+- a console: {vm.images_dir}/{vm.name}.console which can be 'tail -f'ed from
+  the host.
+
+The data used to create the seed above are stored in a vm specific
+configuration directory for easier debug and reference:
+- {vm.config_dir}/user-data
+- {vm.config_dir}/meta-data
+- {vm.config_dir}/ecdsa
+- {vm.config_dir}/ecdsa.pub
+'''))
+
+# The directory where we store vm files related to their configuration with
+# cloud-init (user-data, meta-data, ssh keys).
+register(config.Option(
+        'vm.vms_dir', default='~/.config/setup_vm',
+        help='''Where vm related config files are stored.
+
+This includes user-data and meta-data for cloud-init and ssh server keys.
+
+This directory must exist.
+
+Each vm get a specific directory (automatically created) there based on its
+name.
+'''))
+# The base directories where vms are stored for kvm
+register(PathOption(
+        'vm.images_dir', default='/var/lib/libvirt/images',
+        help="Where vm disk images are stored.",
+        ))
+register(config.Option(
+        'vm.qemu_etc_dir',
+        default='/etc/libvirt/qemu',
+        help="Where libvirt (qemu) stores the vms config files."
+        ))
+
+# Isos and images download handling
+register(config.Option(
+        'vm.iso_url',
+        default='http://cdimage.ubuntu.com/daily-live/current/' ,
+        help="Where an iso can be downloaded from."
+        ))
+register(config.Option(
+        'vm.iso_name',
+        default='{vm.release}-desktop-{vm.cpu_model}.iso',
+        help="The name of the iso."
+        ))
+register(config.Option(
+        'vm.cloud_image_url',
+        default='http://cloud-images.ubuntu.com/{vm.release}/current/',
+        help="Where a cloud image can be downloaded from."
+        ))
+register(config.Option(
+        'vm.cloud_image_name',
+        default='{vm.release}-server-cloudimg-{vm.cpu_model}-disk1.img',
+        help="The name of the cloud image."
+        ))
+register(PathOption(
+        'vm.download_cache',
+        default='{vm.images_dir}',
+        help="Where downloads end up.",
+        ))
+
+# The ubiquitous vm name
+register(config.Option(
+        'vm.name', default=None, invalid='error',
+        help="The vm name, used as a prefix for related files."
+        ))
+# The second most important bit to define a vm: which ubuntu release ?
+register(config.Option(
+        'vm.release', default=None, invalid='error',
+        help="The ubuntu release name."
+        ))
+# The third important piece to define a vm: where to store files like the
+# console, the user-data and meta-data files, the ssh server keys, etc.
+register(config.Option(
+        'vm.config_dir', default='{vm.vms_dir}/{vm.name}',
+        invalid='error',
+        help='''The directory where files specific to a vm are stored.
+
+This includes the user-data and meta-data files used at install time (for
+reference and easier debug) as well as the optional ssh server keys.
+
+By default this is {vm.vms_dir}/{vm.name}. You can put it somewhere else by
+redifining it as long as it ends up being unique for the vm.
+
+{vm.vms_dir}/{vm.release}/{vm.name} may better suit your taste for example.
+'''
+        ))
+# The options defining the vm physical characteristics
+register(config.Option(
+        'vm.ram_size', default='1024',
+        help="The ram size in megabytes."
+        ))
+register(config.Option(
+        'vm.disk_size', default='8G',
+        help='''The disk image size in bytes.
+
+Optional suffixes "k" or "K" (kilobyte, 1024) "M" (megabyte, 1024k) "G"
+(gigabyte, 1024M) and T (terabyte, 1024G) are supported.
+'''))
+register(config.Option(
+        'vm.cpus', default='1',
+        help="The number of cpus."
+        ))
+register(config.Option(
+        'vm.cpu_model', default=None, invalid='error',
+        help="The number of cpus."))
+register(config.Option(
+        'vm.network', default='network=default', invalid='error',
+        help="""The --network parameter for virt-install.
+
+This can be specialized for each machine but the default should work in most
+setups. Watch for your DHCP server exhausting its address space if you create a
+lot of vms with random MAC addresses.
+"""))
+
+register(config.Option(
+        'vm.meta_data', default='''\
+instance-id: {vm.name}
+local-hostname: {vm.name}
+''',
+        invalid='error',
+        help="The meta data for cloud-init to put in the seed."
+        ))
+
+# Some bits that may added to user-data but are optional
+
+register(config.ListOption(
+        'vm.packages', default=None,
+        help='''A list of package names to be installed.
+'''))
+register(config.Option(
+        'vm.apt_proxy', default=None, invalid='error',
+        help='''A local proxy for apt to avoid repeated .deb downloads.
+
+Example:
+
+  vm.apt_proxy = http://192.168.0.42:8000
+
+'''))
+register(config.ListOption(
+        'vm.apt_sources', default=None,
+        help='''A list of apt sources entries to be added to the default ones.
+
+Cloud-init already setup /etc/apt/sources.list with appropriate entries. Only
+additional entries need to be specified here.
+'''))
+register(config.ListOption(
+        'vm.ssh_authorized_keys', default=None,
+        help='A list of paths to public ssh keys to be authorized for'
+        ' the default user.'))
+register(config.ListOption(
+        'vm.ssh_keys', default=None,
+        help='''A list of paths to server ssh keys.
+
+Both public and private keys can be provided. Accepted ssh key types are rsa,
+dsa and ecdsa. The file names should match <type>.*[.pub].
+'''))
+register(config.Option(
+        'vm.update', default=False,
+        from_unicode=config.bool_from_store,
+        help='''Whether or not the vm should be updated.
+Both apt-get update and apt-get upgrade are called if this option is set.
+'''))
+register(config.Option(
+        'vm.password', default='ubuntu', invalid='error',
+        help="The ubuntu user password."
+        ))
+register(config.Option(
+        'vm.launchpad_id',
+        help="The launchpad login used for launchpad ssh access from the guest."
+        ))
+# The scripts that are executed before powering off
+register(PathOption(
+        'vm.root_script', default=None,
+        help='''The path to a script executed as root before powering off.
+
+This script is executed before {vm.ubuntu_script}.
+'''
+        ))
+register(PathOption(
+        'vm.ubuntu_script', default=None,
+        help='''The path to a script executed as ubuntu before powering off.
+
+This script is excuted after {vm.root_script}.
+'''))
+register(config.ListOption(
+        'vm.uploaded_scripts', default=None,
+        help='''A list of scripts to be uploaded to the guest.
+
+Scripts can use config options from their vm, they will be expanded before
+upload. All scripts are uploaded into {vm.uploaded_scripts.guest_dir} under
+their base name.
+'''))
+register(config.Option(
+        'vm.uploaded_scripts.guest_dir', default='~ubuntu/bin',
+        help='''Where {vm.uploaded_scripts} are uploaded on the guest.'''
+        ))
+
+
+class SetupVmError(Exception):
+
+    msg = 'setup_vm Generic Error: %r'
+
+    def __init__(self, msg=None, **kwds):
+        if msg is not None:
+            self.msg = msg
+        for key, value in kwds.items():
+            setattr(self, key, value)
+
+    def __str__(self):
+        return self.msg.format((), **self.__dict__)
+
+    __repr__ = __str__
+
+
+class CommandError(SetupVmError):
+
+    msg = '''
+  command: {joined_cmd}
+  retcode: {retcode}
+  output: {out}
+  error: {err}
+'''
+
+    def __init__(self, cmd, retcode, out, err):
+        super(CommandError, self).__init__(joined_cmd=' '.join(cmd),
+                                           retcode=retcode, err=err, out=out)
+
+class ConfigValueError(SetupVmError):
+
+    msg = 'Bad value "{value}" for option "{name}".'
+
+    def __init__(self, name, value):
+        super(ConfigValueError, self).__init__(name=name, value=value)
+
+
+class ConfigPathNotFound(SetupVmError):
+
+    msg = 'No such file: {path} from {name}'
+
+    def __init__(self, path, name):
+        super(ConfigPathNotFound, self).__init__(path=path, name=name)
+
+
+def run_subprocess(args):
+    proc = subprocess.Popen(args,
+                            stdout=subprocess.PIPE,
+                            stderr=subprocess.PIPE,
+                            stdin=subprocess.PIPE)
+    out, err = proc.communicate()
+    if proc.returncode:
+        raise CommandError(args, proc.returncode, out, err)
+    return proc.returncode, out, err
+
+
+def pipe_subprocess(args):
+    proc = subprocess.Popen(args,
+                            stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
+    return proc
+
+def ssh_infos_from_path(key_path):
+    """Analyze path to find ssh key type and kind.
+
+    The basename should begin with ssh type used to create the key. and end
+    with '.pub' for a public key.
+
+    If the type is neither of rds, dsa or ecdsa, None if returned.
+
+    :param key_path: A path to an ssh key.
+
+    :return: (type, kind) 'type' is the ssh key type or None if neither of rds,
+        dsa or ecdsa. 'kind' is 'public' if the path ends with '.pub',
+        'private' otherwise.
+    """
+    base = os.path.basename(key_path)
+    for p in ('rsa', 'dsa', 'ecdsa'):
+        if base.startswith(p):
+            ssh_type = p
+            break
+    else:
+        ssh_type = None
+    if base.endswith('.pub'):
+        kind = 'public'
+    else:
+        kind = 'private'
+    return ssh_type, kind
+
+
+class ConsoleEOFError(SetupVmError):
+
+    msg = 'Encountered EOF on console, something went wrong'
+
+
+class CloudInitError(SetupVmError):
+
+    msg = 'cloud-init reported: {line} check your config'
+
+    def __init__(self, line):
+        super(CloudInitError, self).__init__(line=line)
+
+
+class ConsoleMonitor(object):
+    """Monitor a console to identify known events."""
+
+    def __init__(self, stream):
+        super(ConsoleMonitor, self).__init__()
+        self.stream = stream
+
+
+    def parse(self):
+        while True:
+            line = self.stream.readline()
+            yield line
+            if not line:
+                raise ConsoleEOFError()
+            elif line.startswith(' * Will now halt'):
+                # That's our final_message, we're done
+                return
+            elif ('Failed loading yaml blob' in line
+                  or 'Unhandled non-multipart userdata starting' in line
+                  or 'failed to render string to stdout:' in line
+                  or 'Failed loading of cloud config' in line):
+                raise CloudInitError(line)
+
+
+class FileMonitor(ConsoleMonitor):
+
+    def __init__(self, path):
+        cmd = ['tail', '-f', path]
+        proc = pipe_subprocess(cmd)
+        super(FileMonitor, self).__init__(proc.stdout)
+        self.path = path
+        self.cmd = cmd
+        self.proc = proc
+        self.lines = []
+
+    def parse(self):
+        try:
+            for line in super(FileMonitor, self).parse():
+                self.lines.append(line)
+                yield line
+        finally:
+            self.proc.terminate()
+
+
+class CIUserData(object):
+    """Maps configuration data into cloud-init user-data.
+
+    This is a container for the data that will ultimately be encoded into a
+    cloud-config-archive user-data file.
+    """
+
+    def __init__(self, conf):
+        super(CIUserData, self).__init__()
+        self.conf = conf
+        # The objects we will populate before creating a yaml encoding as a
+        # cloud-config-archive file
+        self.cloud_config = {}
+        self.root_hook = None
+        self.ubuntu_hook = None
+        self.launchpad_hook = None
+        self.uploaded_scripts_hook = None
+
+    def set(self, ud_name, conf_name=None, value=None):
+        """Set a user-data option from it's corresponding configuration one.
+
+        :param ud_name: user-data key.
+
+        :param conf_name: configuration key, If set to None, `value` should be
+            provided.
+
+        :param value: value to use if `conf_name` is None.
+        """
+        if value is None and conf_name is not None:
+            value = self.conf.get(conf_name)
+        if value is not None:
+            self.cloud_config[ud_name] = value
+
+    def _file_content(self, path, option_name):
+        full_path = os.path.expanduser(path)
+        try:
+            with open(full_path) as f:
+                content = f.read()
+        except IOError, e:
+            if e.args[0] == errno.ENOENT:
+                raise ConfigPathNotFound(path, option_name)
+            else:
+                raise
+        return content
+
+    def set_list_of_paths(self, ud_name, conf_name):
+        """Set a user-data option from its corresponding configuration one.
+
+        The configuration option is a list of paths and the user-data option
+        will be a list of each file content.
+
+        :param ud_name: user-data key.
+
+        :param conf_name: configuration key.
+        """
+        paths = self.conf.get(conf_name)
+        if paths:
+            contents = []
+            for p in paths:
+                contents.append(self._file_content(p, conf_name))
+            self.set(ud_name, None, contents)
+
+    def _key_from_path(self, path, option_name):
+        """Infer user-data key from file name."""
+        ssh_type, kind = ssh_infos_from_path(path)
+        if ssh_type is None:
+            raise ConfigValueError(option_name, path)
+        return '%s_%s' % (ssh_type, kind)
+
+    def set_ssh_keys(self):
+        """Set the server ssh keys from a list of paths.
+
+        Provided paths should respect some coding:
+
+        - the base name should start with the ssh type of their key (rsa, dsa,
+          ecdsa),
+
+        - base names ending with '.pub' are for public keys, the others are for
+          private keys.
+        """
+        key_paths = self.conf.get('vm.ssh_keys')
+        if key_paths:
+            ssh_keys = {}
+            for p in key_paths:
+                key = self._key_from_path(p, 'vm.ssh_keys')
+                ssh_keys[key] = self._file_content(p, 'vm.ssh_keys')
+            self.set('ssh_keys', None, ssh_keys)
+
+    def set_apt_sources(self):
+        sources = self.conf.get('vm.apt_sources')
+        if sources:
+            apt_sources = []
+            for src in sources:
+                # '|' should not appear in urls nor keys so it should be safe
+                # to use it as a separator.
+                parts = src.split('|')
+                if len(parts) == 1:
+                    apt_sources.append({'source': parts[0]})
+                else:
+                    # For PPAs, an additional GPG key should be imported in the
+                    # guest.
+                    apt_sources.append({'source': parts[0], 'keyid': parts[1]})
+            self.cloud_config['apt_sources'] = apt_sources
+
+    def append_cmd(self, cmd):
+        cmds = self.cloud_config.get('runcmd', [])
+        cmds.append(cmd)
+        self.cloud_config['runcmd'] = cmds
+
+    def _hook_script_path(self, user):
+        return '~%s/setup_vm_post_install' % (user,)
+
+    def _hook_content(self, option_name, user, hook_path, mode='0700'):
+        # FIXME: Add more tests towards properly creating a tree on the guest
+        # from a tree on the host. There seems to be three kind of items worth
+        # caring about here: file content (output path, owner, chmod), file
+        # (input and output paths, owner, chmod) and directory (path, owner,
+        # chmod). There are also some subtle traps involved about when files
+        # are created across various vm generations (one vm creates a dir, a mv
+        # on top of that one doesn't, but still creates a file in this dir,
+        # without realizing it can fail in a fresh vm). -- vila 2013-03-10
+        host_path = self.conf.get(option_name)
+        if host_path is None:
+            return None
+        fcontent = self._file_content(host_path, option_name)
+        # Expand options in the provided content so we report better errors
+        expanded_content = self.conf.expand_options(fcontent)
+        # The following will generate an additional newline at the end of the
+        # script. I can't think of a case where it matters and it makes this
+        # code more robust (and/or simpler) if the script/file *doesn't* end up
+        # with a proper newline.
+        # FIXME: This may be worth fixing if we provide a more generic way to
+        # create a remote tree. -- vila 2013-03-10
+        hook_content = '''#!/bin/sh
+cat >{__guest_path} <<'EOSETUPVMCONTENTREALLYUNIQUEDONTBREAKFORFUN'
+{__fcontent}
+EOSETUPVMCONTENTREALLYUNIQUEDONTBREAKFORFUN
+chown {__user}:{__user} {__guest_path}
+chmod {__mode} {__guest_path}
+'''
+        return hook_content.format(__user=user, __fcontent=expanded_content,
+                                   __mode=mode,
+                                   __guest_path=hook_path)
+
+    def set_boot_hook(self):
+        # FIXME: Needs a test ensuring we execute as root -- vila 2013-03-07
+        hook_path = self._hook_script_path('root')
+        content = self._hook_content('vm.root_script', 'root', hook_path)
+        if content is not None:
+            self.root_hook = content
+            self.append_cmd(hook_path)
+
+    def set_ubuntu_hook(self):
+        # FIXME: Needs a test ensuring we execute as ubuntu -- vila 2013-03-07
+        hook_path = self._hook_script_path('ubuntu')
+        content = self._hook_content('vm.ubuntu_script', 'ubuntu', hook_path)
+        if content is not None:
+            self.ubuntu_hook = content
+            self.append_cmd(['su', '-l', '-c', hook_path, 'ubuntu'])
+
+    def set_launchpad_access(self):
+        # FIXME: Needs a test that we can really access launchpad properly via
+        # ssh. Can only be done as a real launchpad user and as such requires
+        # cooperation :) I.e. Some configuration option set by the user will
+        # trigger the test -- vila 2013-03-14
+        lp_id = self.conf.get('vm.launchpad_id')
+        if lp_id is None:
+            return
+        # Use the specified ssh key found in ~/.ssh as the private key. The
+        # user is supposed to have uploaded the public one.
+        local_path = os.path.join('~', '.ssh', '%s@setup_vm' % (lp_id,))
+        # Force id_rsa or we'll need a .ssh/config to point to user@setup_vm
+        # for .lauchpad.net.
+        hook_path = '/home/ubuntu/.ssh/id_rsa'
+        dir_path = os.path.dirname(hook_path)
+        try:
+            fcontent = self._file_content(local_path, 'vm.launchpad_id')
+        except ConfigPathNotFound, e:
+            e.msg = ('You need to create the {p} keypair and upload {p}.pub to'
+                     ' launchpad.\n'
+                     'See vm.launchpad_id in README.'.format(p=local_path))
+            raise e
+        # FIXME: ~Duplicated from _hook_content. -- vila 2013-03-10
+
+        # FIXME: If this hook is executed before the ubuntu user is created we
+        # need to chown/chmod ~ubuntu which is bad. This happens when a
+        # -pristine vm is created and lead to GUI login failing because it
+        # can't create any dir/file there. The fix is to only create a script
+        # that will be executed via runcmd so it will run later and avoid the
+        # issue. -- vila 2013-03-21
+        hook_content = '''#!/bin/sh
+mkdir -p {dir_path}
+chown {user}:{user} ~ubuntu
+chmod {dir_mode} ~ubuntu
+chown {user}:{user} {dir_path}
+chmod {dir_mode} {dir_path}
+cat >{guest_path} <<'EOSETUPVMCONTENTREALLYUNIQUEDONTBREAKFORFUN'
+{fcontent}
+EOSETUPVMCONTENTREALLYUNIQUEDONTBREAKFORFUN
+chown {user}:{user} {guest_path}
+chmod {file_mode} {guest_path}
+'''
+        self.launchpad_hook = self.conf.expand_options(
+            hook_content,
+            env=dict(user='ubuntu', fcontent=fcontent,
+                     file_mode='0400', guest_path=hook_path,
+                     dir_mode='0700', dir_path=dir_path))
+        self.append_cmd(['sudo', '-u', 'ubuntu',
+                         'bzr', 'launchpad-login', lp_id])
+
+    def set_uploaded_scripts(self):
+        script_paths = self.conf.get('vm.uploaded_scripts')
+        if not script_paths:
+            return
+        hook_path = '~ubuntu/setup_vm_uploads'
+        bindir = self.conf.get('vm.uploaded_scripts.guest_dir')
+        out = StringIO()
+        out.write('''#!/bin/sh
+cat >{hook_path} <<'EOSETUPVMCONTENTREALLYUNIQUEDONTBREAKFORFUN'
+mkdir -p {bindir}
+cd {bindir}
+'''.format(**locals()))
+        for path in script_paths:
+            fcontent = self._file_content(path, 'vm.uploaded_scripts')
+            expanded = self.conf.expand_options(fcontent)
+            base = os.path.basename(path)
+            # FIXME: ~Duplicated from _hook_content. -- vila 2012-03-15
+            out.write('''cat >{base} <<'EOF{base}'
+{expanded}
+EOF{base}
+chmod 0755 {base}
+'''.format(**locals()))
+
+        out.write('''EOSETUPVMCONTENTREALLYUNIQUEDONTBREAKFORFUN
+chown {user}:{user} {hook_path}
+chmod 0700 {hook_path}
+'''.format(user='ubuntu',**locals()))
+        self.uploaded_scripts_hook = out.getvalue()
+        self.append_cmd(['su', '-l', '-c', hook_path, 'ubuntu'])
+
+    def set_poweroff(self):
+        # We want to shutdown properly after installing. This is safe to set
+        # here as subsequent boots will ignore this setting, letting us use the
+        # vm ;)
+        if self.conf.get('vm.release') in ('precise', 'quantal'):
+            # Curse cloud-init lack of compatibility
+            self.append_cmd('halt')
+        else:
+            self.set('power_state', None, {'mode': 'poweroff'})
+
+    def populate(self):
+        # Common and non-configurable options
+        if self.conf.get('vm.release') == 'precise':
+            # Curse cloud-init lack of compatibility
+            msg = 'setup_vm finished installing in $UPTIME seconds.'
+        else:
+            msg = 'setup_vm finished installing in ${uptime} seconds.'
+        self.set('final_message', None, msg)
+        self.set('manage_etc_hosts', None, True)
+        self.set('chpasswd', None, dict(expire=False))
+        # Configurable options
+        self.set('password', 'vm.password')
+        self.set_list_of_paths('ssh_authorized_keys', 'vm.ssh_authorized_keys')
+        self.set_ssh_keys()
+        self.set('apt_proxy', 'vm.apt_proxy')
+        # Both user-data keys are set from the same config key, we don't
+        # provide a finer access.
+        self.set('apt_update', 'vm.update')
+        self.set('apt_upgrade', 'vm.update')
+        self.set_apt_sources()
+        self.set('packages', 'vm.packages')
+        self.set_launchpad_access()
+        # uploaded scripts
+        self.set_uploaded_scripts()
+        # The commands executed before powering off
+        self.set_boot_hook()
+        self.set_ubuntu_hook()
+        # This must be called last so previous commands (for precise and
+        # quantal) can be executed before powering off
+        self.set_poweroff()
+
+    def add_boot_hook(self, parts, hook):
+        if hook is not None:
+            parts.append({'content': '#cloud-boothook\n' +  hook})
+
+    def dump(self):
+        parts = [{'content': '#cloud-config\n'
+                  + yaml.safe_dump(self.cloud_config)}]
+        self.add_boot_hook(parts, self.root_hook)
+        self.add_boot_hook(parts, self.ubuntu_hook)
+        self.add_boot_hook(parts, self.launchpad_hook)
+        self.add_boot_hook(parts, self.uploaded_scripts_hook)
+        # Wrap the lot into a cloud config archive
+        return '#cloud-config-archive\n' + yaml.safe_dump(parts)
+
+
+def vm_states(source=None):
+    """A dict of states for vms indexed by name.
+
+    :param source: A list of lines as produced by virsh list --all without
+        decorations (header/footer).
+    """
+    if source is None:
+        retcode, out, err = run_subprocess(['virsh', 'list', '--all'])
+        # Get rid of header/footer
+        source = out.splitlines()[2:-1]
+    states = {}
+    for line in source:
+        caret_or_id, name, state = line.split(None, 2)
+        states[name] = state
+    return states
+
+
+class VM(object):
+
+    def __init__(self, conf):
+        self.conf = conf
+        self._config_dir = None
+
+    def ensure_dir(self, path):
+        try:
+            os.mkdir(path)
+        except OSError, e:
+            # FIXME: Try to create the parent dir ?
+            if e.errno == errno.EEXIST:
+                pass
+            else:
+                raise
+
+    def ensure_config_dir(self):
+        if self._config_dir is None:
+            # FIXME: expanduser is not tested
+            self._config_dir = os.path.expanduser(
+                self.conf.get('vm.config_dir'))
+            self.ensure_dir(self._config_dir)
+
+    def _ssh_keygen(self, key_path):
+        ssh_type, kind = ssh_infos_from_path(key_path)
+        path = os.path.expanduser(key_path) # Just in case
+        if kind == 'private': # public will be generated at the same time
+            run_subprocess(
+                ['ssh-keygen', '-f', path, '-N', '', '-t', ssh_type,
+                 '-C', self.conf.get('vm.name')])
+
+    def ssh_keygen(self):
+        self.ensure_config_dir()
+        keys = self.conf.get('vm.ssh_keys')
+        for key in keys:
+            self._ssh_keygen(key)
+
+
+class Kvm(VM):
+
+    def __init__(self, conf):
+        super(Kvm, self).__init__(conf)
+        # Seed files
+        self._meta_data_path = None
+        self._user_data_path = None
+        # Disk paths
+        self._seed_path = None
+        self._disk_image_path = None
+
+        self._console_path = None
+
+    def _download_in_cache(self, source_url, name, force=False):
+        """Download ``name`` from ``source_url`` in ``vm.download_cache``.
+
+        :param source_url: The url where the file to download is located
+
+        :param name: The name of the file to download (also used as the name
+            for the downloaded file).
+
+        :param force: Remove the file from the cache if present.
+
+        :return: False if the file is in the download cache, True if a download
+            occurred.
+        """
+        source = urlutils.join(source_url, name)
+        download_dir = self.conf.get('vm.download_cache')
+        if not os.path.exists(download_dir):
+            raise ConfigValueError('vm.download_cache', download_dir)
+        target = os.path.join(download_dir, name)
+        # FIXME: By default the download dir may be under root control, but if
+        # a user chose to use a different one under his own control, it would
+        # be nice to not require sudo usage. -- vila 2013-02-06
+        if force:
+            run_subprocess(['sudo', 'rm', '-f', target])
+        if not os.path.exists(target):
+            # FIXME: We do ask for a progress bar but it's not displayed
+            # (run_subprocess capture both stdout and stderr) ! At least while
+            # used interactively, it should. -- vila 2013-02-06
+            run_subprocess(['sudo', 'wget', '--progress=dot:mega','-O',
+                            target, source])
+            return True
+        else:
+            return False
+
+    def download_iso(self, force=False):
+        """Download the iso to install the vm.
+
+        :return: False if the iso is in the download cache, True if a download
+            occurred.
+        """
+        return self._download_in_cache(self.conf.get('vm.iso_url'),
+                                       self.conf.get('vm.iso_name'),
+                                       force=force)
+
+    def download_cloud_image(self, force=False):
+        """Download the cloud image to install the vm.
+
+        :return: False if the image is in the download cache, True if a
+            download occurred.
+        """
+        return self._download_in_cache(self.conf.get('vm.cloud_image_url'),
+                                       self.conf.get('vm.cloud_image_name'),
+                                       force=force)
+
+    def create_meta_data(self):
+        self.ensure_config_dir()
+        self._meta_data_path = os.path.join(self._config_dir, 'meta-data')
+        with open(self._meta_data_path, 'w') as f:
+            f.write(self.conf.get('vm.meta_data'))
+
+    def create_user_data(self):
+        ci_user_data = CIUserData(self.conf)
+        ci_user_data.populate()
+        self.ensure_config_dir()
+        self._user_data_path = os.path.join(self._config_dir, 'user-data')
+        with open(self._user_data_path, 'w') as f:
+            f.write(ci_user_data.dump())
+
+    def create_seed(self):
+        if self._meta_data_path is None:
+            self.create_meta_data()
+        if self._user_data_path is None:
+            self.create_user_data()
+        images_dir = self.conf.get('vm.images_dir')
+        seed_path = os.path.join(
+            images_dir, self.conf.expand_options('{vm.name}.seed'))
+        run_subprocess(
+            # We create the seed in the ``vm.images_dir`` directory, so
+            # ``sudo`` is required
+            ['sudo',
+             'genisoimage', '-output', seed_path, '-volid', 'cidata',
+             '-joliet', '-rock', '-input-charset', 'default',
+             '-graft-points',
+             'user-data=%s' % (self._user_data_path,),
+             'meta-data=%s' % (self._meta_data_path,),
+             ])
+        self._seed_path = seed_path
+
+    def create_disk_image(self):
+        raise NotImplementedError(self.create_disk_image)
+
+    def _wait_for_install_with_seed(self):
+        # The console is created by virt-install which requires sudo but creates
+        # the file 0600 for libvirt-qemu. We give read access to all otherwise
+        # 'tail -f' requires sudo and can't be killed anymore.
+        run_subprocess(['sudo', 'chmod', '0644', self._console_path])
+        # While `virt-install` is running, let's connect to the console
+        console = FileMonitor(self._console_path)
+        try:
+            for line in console.parse():
+# FIXME: We need some way to activate this dynamically (conf var defaulting to
+# env var OR cmdline parameter ? -- vila 2013-02-11
+#                print "read: [%s]" % (line,) # so useful for debug...
+                pass
+        except (ConsoleEOFError, CloudInitError), e:
+            # FIXME: No test covers this path -- vila 2013-02-15
+            err_lines = ['Suspicious line from cloud-init.\n',
+                         '\t' + console.lines[-1],
+                         'Check the configuration:\n']
+            with open(self._meta_data_path) as f:
+                err_lines.append('meta-data content:\n')
+                err_lines.extend(f.readlines())
+            with open(self._user_data_path) as f:
+                err_lines.append('user-data content:\n')
+                err_lines.extend(f.readlines())
+            raise CommandError(console.cmd, console.proc.returncode,
+                               '\n'.join(console.lines),
+                               ''.join(err_lines))
+
+    def install(self):
+        # Create a kvm, relying on cloud-init to customize the base image.
+        #  
+        # There are two processes involvded here:
+        # - virt-install creates the vm and boots it.
+        # - progress is monitored via the console to detect cloud-final.
+        #  
+        # Once cloud-init has finished, the vm can be powered off.
+
+        # FIXME: If the install doesn't finish after $time, emit a warning and
+        # terminate self.install_proc.
+        # FIXME: If we can't connect to the console, emit a warning and
+        # terminate console and self.install_proc.
+        # FIXME: If we don't receive anything on the console after $time2, emit
+        # a warning and terminate console and self.install_proc.
+        # -- vila 2013-02-07
+        if self._seed_path is None:
+            self.create_seed()
+        if self._disk_image_path is None:
+            self.create_disk_image()
+        # FIXME: Install time is probably a good time to delete the
+        # console. While it makes sense to accumulate for all runs for a given
+        # install, keeping them without any limit nor roration is likely to
+        # cause issues at some point... -- vila 2013-02-20
+        self._console_path = os.path.join(
+            self.conf.get('vm.images_dir'),
+            '%s.console' % (self.conf.get('vm.name'),))
+        run_subprocess(
+            ['sudo', 'virt-install',
+             # To ensure we're not bitten again by http://pad.lv/1157272 where
+             # virt-install wrongly detect virtualbox. -- vila 2013-03-20
+             '--connect', 'qemu:///system',
+             # Without --noautoconsole, virt-install will relay the console,
+             # that's not appropriate for our needs so we'll connect later
+             # ourselves
+             '--noautoconsole',
+             # We define the console as a file so we can monitor the install
+             # via 'tail -f'
+             '--serial', 'file,path=%s' % (self._console_path,),
+             '--network', self.conf.get('vm.network'),
+             # Anticipate that we'll need a graphic card defined
+             '--graphics', 'spice',
+             '--name', self.conf.get('vm.name'),
+             '--ram', self.conf.get('vm.ram_size'),
+             '--vcpus', self.conf.get('vm.cpus'),
+             '--disk', 'path=%s,format=qcow2' % (self._disk_image_path,),
+             '--disk', 'path=%s' % (self._seed_path,),
+             # We just boot, cloud-init will handle the installs we need
+             '--boot', 'hd', '--hvm',
+             ])
+        self._wait_for_install_with_seed()
+        # We've seen the console signaling halt, but the vm will need a bit
+        # more time to get there so we help it a bit.
+        if self.conf.get('vm.release') in ('precise', 'quantal'):
+            # cloud-init doesn't implement power_state until raring and need a
+            # gentle nudge.
+            self.poweroff()
+        vm_name = self.conf.get('vm.name')
+        while True:
+            state = vm_states()[vm_name]
+            # We expect the vm's state to be 'in shutdown' but in some rare
+            # occasions we may catch 'running' before getting 'in shutdown'.
+            if state in ('in shutdown', 'running'):
+                # Ok, querying the state takes time, this regulates the polling
+                # enough that we don't need to sleep.
+                continue
+            elif state == 'shut off':
+                # Good, we're done
+                break
+            # FIXME: No idea on how to test the following. Manually tested by
+            # altering the expected state above and running 'selftest.py -v'
+            # which errors out for test_install_with_seed and
+            # test_install_backing. Also reproduced when 'running' wasn't
+            # expected before 'in shutdown' -- vila 2013-02-19
+            # Unexpected state reached, bad.
+            raise SetupVmError('Something went wrong during {name} install\n'
+                               'The vm ended in state: {state}\n'
+                               'Check the console at {path}',
+                               name=vm_name, state=state,
+                               path=self._console_path)
+
+    def poweroff(self):
+        return run_subprocess(
+            ['sudo', 'virsh', 'destroy', self.conf.get('vm.name')])
+
+    def undefine(self):
+        return run_subprocess(
+            ['sudo', 'virsh', 'undefine', self.conf.get('vm.name'),
+             '--remove-all-storage'])
+
+
+class KvmFromCloudImage(Kvm):
+
+    def create_disk_image(self, src_name=None, dst_name=None):
+        """Create a disk image from a cloud one."""
+        if src_name is None:
+            src_name = self.conf.get('vm.cloud_image_name')
+        if dst_name is None:
+            dst_name = self.conf.expand_options('{vm.name}.qcow2')
+        cloud_image_path = os.path.join(
+            self.conf.get('vm.download_cache'), src_name)
+        disk_image_path = os.path.join(
+            self.conf.get('vm.images_dir'), dst_name)
+        run_subprocess(
+            ['sudo', 'qemu-img', 'convert',
+             '-O', 'qcow2', cloud_image_path, disk_image_path])
+        run_subprocess(
+            ['sudo', 'qemu-img', 'resize',
+             disk_image_path, self.conf.get('vm.disk_size')])
+        self._disk_image_path = disk_image_path
+
+
+class KvmFromBacking(Kvm):
+
+    def create_disk_image(self, src_name=None, dst_name=None):
+        """Create a disk image backed by an existing one."""
+        backing_image_path = os.path.join(
+            self.conf.get('vm.images_dir'),
+            self.conf.expand_options('{vm.backing}'))
+        disk_image_path = os.path.join(
+            self.conf.get('vm.images_dir'),
+            self.conf.expand_options('{vm.name}.qcow2'))
+        run_subprocess(
+            ['sudo', 'qemu-img', 'create', '-f', 'qcow2',
+             '-b', backing_image_path, disk_image_path])
+        run_subprocess(
+            ['sudo', 'qemu-img', 'resize',
+             disk_image_path, self.conf.get('vm.disk_size')])
+        self._disk_image_path = disk_image_path
+
+
+class ArgParser(argparse.ArgumentParser):
+    """A parser for the setup_vm script."""
+
+    def __init__(self):
+        description = 'Set up virtual machines from a configuration file.'
+        super(ArgParser, self).__init__(
+            prog='setup_vm.py', description=description)
+        self.add_argument(
+            'name', help='Virtual machine section in the configuration file.')
+        self.add_argument('--download', '-d', action="store_true",
+                          help='Force download of the required image.')
+        self.add_argument('--ssh-keygen', '-k', action="store_true",
+                          help='Generate the ssh server keys (if any).')
+        self.add_argument('--install', '-i', action="store_true",
+                          help='Install the virtual machine.')
+
+    def parse_args(self, args=None, out=None, err=None):
+        """Parse arguments, overridding stdout/stderr if provided.
+
+        Overridding stdout/stderr is provided for tests.
+
+        :params args: Defaults to sys.argv[1:].
+
+        :param out: Defaults to sys.stdout.
+
+        :param err: Defaults to sys.stderr.
+        """
+        out_orig = sys.stdout
+        err_orig = sys.stderr
+        try:
+            if out is not None:
+                sys.stdout = out
+            if err is not None:
+                sys.stderr = err
+            return super(ArgParser, self).parse_args(args)
+        finally:
+            sys.stdout = out_orig
+            sys.stderr = err_orig
+
+
+
+arg_parser = ArgParser()
+
+class Command(object):
+
+    def __init__(self, vm):
+        self.vm = vm
+
+
+class Download(Command):
+
+    def run(self):
+        # FIXME: what needs to be downloaded should depend on the type of the
+        # vm (possibly errors if there is nothing to download). -- vila
+        # 2013-02-06
+        self.vm.download_cloud_image(force=True)
+
+
+class SshKeyGen(Command):
+
+    def run(self):
+        self.vm.ssh_keygen()
+
+
+class Install(Command):
+
+    def run(self):
+        vm_name = self.vm.conf.get('vm.name')
+        state = vm_states().get(vm_name, None)
+        if state == 'shut off':
+            self.vm.undefine()
+        elif state == 'running':
+            raise SetupVmError('{name} is running', name=vm_name)
+        # FIXME: The installation method may vary depending on the vm type.
+        # -- vila 2013-02-06
+        self.vm.install()
+
+
+def build_commands(args=None, out=None, err=None):
+    cmds = []
+    if args is None:
+        args = sys.argv[1:]
+
+    ns = arg_parser.parse_args(args, out=out, err=err)
+
+    conf = VmStack(ns.name)
+    with_backing = conf.get('vm.backing')
+    if with_backing is None:
+        vm = KvmFromCloudImage(conf)
+    else:
+        vm = KvmFromBacking(conf)
+    if ns.download:
+        cmds.append(Download(vm))
+    if ns.ssh_keygen:
+        cmds.append(SshKeyGen(vm))
+    if ns.install:
+        cmds.append(Install(vm))
+    return cmds
+
+
+def run(args=None):
+    cmds = build_commands(args)
+    for cmd in cmds:
+        try:
+            cmd.run()
+        except SetupVmError, e:
+            # Stop on first error
+            print 'ERROR: %s' % e
+            exit(-1)
+
+
+if __name__ == "__main__":
+    run()
 
=== added file 'setup_vm/bin/ubuntu_admin.sh'
--- setup_vm/bin/ubuntu_admin.sh	1970-01-01 00:00:00 +0000
+++ setup_vm/bin/ubuntu_admin.sh	2013-04-17 01:29:27 +0000
@@ -0,0 +1,2 @@
+#!/bin/sh
+adduser ubuntu admin
 
=== added directory 'setup_vm/pay'
=== added file 'setup_vm/pay/install'
--- setup_vm/pay/install	1970-01-01 00:00:00 +0000
+++ setup_vm/pay/install	2013-04-17 01:29:27 +0000
@@ -0,0 +1,40 @@
+#!/bin/sh -ex
+
+# Allow ssh access to launchpad.
+# This should probably be provided by setup_vm. -- vila 2013-03-10
+ssh-keyscan bazaar.launchpad.net >>~/.ssh/known_hosts
+# Get the branch.
+bzr branch lp:canonical-payment-service {pay.src_dir}
+# Get the download cache.
+bzr branch lp:~canonical-isd-hackers/+junk/download-cache
+# Setup the environment.
+cd {pay.src_dir}
+# Get the version controlled configs.
+bzr branch lp:~canonical-isd-hackers/isd-configs/payments-config branches/project
+# Bootstrap the dependencies
+fab bootstrap:download_cache_path=~/download-cache
+# Set up the correct Django configuration.
+cat <<EOF >django_project/local.cfg
+[__noschema__]
+db_host = /home/ubuntu/{pay.src_dir}/.env/db
+hostname = {pay.address}:{pay.port}
+
+[__main__]
+includes =
+    config/devel.cfg
+    ../branches/project/config/acceptance.cfg
+
+[django]
+debug = false
+internal_ips =
+
+[testing]
+imap_server = {sso.address}
+imap_port = {sso.imap_port}
+imap_use_ssl = False
+
+[openid]
+openid_sso_server_url = {sso.url}
+openid_trust_root = {pay.url}
+
+EOF
 
=== added file 'setup_vm/pay/run'
--- setup_vm/pay/run	1970-01-01 00:00:00 +0000
+++ setup_vm/pay/run	2013-04-17 01:29:27 +0000
@@ -0,0 +1,9 @@
+#!/bin/sh
+
+cd {pay.src_dir}
+
+# Setup the database.
+fab setup_postgresql_server
+fab manage:loaddata,test
+# Start the PAY server, accessible from the local network.
+fab run:0.0.0.0:{pay.port}
 
=== added file 'setup_vm/pay/run-for-u1'
--- setup_vm/pay/run-for-u1	1970-01-01 00:00:00 +0000
+++ setup_vm/pay/run-for-u1	2013-04-17 01:29:27 +0000
@@ -0,0 +1,57 @@
+#!/bin/sh
+
+cd {pay.src_dir}
+
+# Setup the database.
+fab setup_postgresql_server
+# Add the U1 consumer.
+cat <<EOF >src/paymentservice/fixtures/consumer.json
+[
+    {
+        "pk": "U1",
+        "model": "paymentservice.consumer",
+        "fields": {
+            "notification_url": "{u1.url}/notifications/",
+            "ip_address": "{u1.address}",
+            "name": "Ubuntu One",
+            "default_business_unit": "Online Services",
+            "email_footer": "Test footer.",
+            "theme": "ubuntuone"
+        }
+    }
+]
+EOF
+fab manage:loaddata,consumer
+# Add the API user for U1.
+# We generated this json file with:
+# Go to {pay.url}/admin
+# Sign in with the admin/admin.
+# Click the more link next to the Model Admin heading.
+# On the Paymentservice section, click the +Add link next to API Users.
+# Fill the form with:
+# username: u1qauser
+# password: u1qapassword
+# Click the Save button.
+# Select the Ubuntu One (U1) Consumer.
+# Click the Save button.
+# $ fab manage:dumpdata,paymentservice.APIUser
+cat <<EOF >src/paymentservice/fixtures/apiuser.json
+[
+    {
+        "pk": 2,
+        "model": "paymentservice.apiuser",
+        "fields": {
+            "username": "u1qauser",
+            "created_at": "2013-04-15 00:09:48",
+            "password": "sha1\$b2a8e\$0e06d9cb46583aa53d3bf144ae07018a7546f737",
+            "consumer": "U1",
+            "updated_at": "2013-04-15 00:09:54"
+        }
+    }
+]
+EOF
+fab manage:loaddata,apiuser
+# Start the PAY server, accessible from the local network.
+# We don't call the run task because it loads a fixture that overwrites our
+# consumer.
+fab manage:runserver,0.0.0.0:{pay.port}
 
=== added file 'setup_vm/pay/test'
--- setup_vm/pay/test	1970-01-01 00:00:00 +0000
+++ setup_vm/pay/test	2013-04-17 01:29:27 +0000
@@ -0,0 +1,5 @@
+#!/bin/sh
+
+cd {pay.src_dir}
+
+SST_BASE_URL={pay.url} fab acceptance:screenshot=true,report=xml,extended=true
 
=== added file 'setup_vm/selftest.py'
--- setup_vm/selftest.py	1970-01-01 00:00:00 +0000
+++ setup_vm/selftest.py	2013-04-17 01:29:27 +0000
@@ -0,0 +1,24 @@
+#!/usr/bin/env python
+
+import sys
+
+import testtools.run
+import unittest
+
+
+class TestProgram(testtools.run.TestProgram):
+
+    def __init__(self, module, argv, stdout=None, testRunner=None, exit=True):
+        if testRunner is None:
+            testRunner = unittest.TextTestRunner
+        super(TestProgram, self).__init__(module, argv=argv, stdout=stdout,
+                                          testRunner=testRunner, exit=exit)
+
+
+# We discover tests under './tests', the python 'load_test' protocol can be
+# used in test modules for more fancy stuff.
+discover_args = ['discover',
+                 '--start-directory', './tests',
+                 '--top-level-directory', '.',
+                 ]
+TestProgram(__name__, argv=[sys.argv[0]] + discover_args + sys.argv[1:])
 
=== added directory 'setup_vm/sso'
=== added file 'setup_vm/sso/install'
--- setup_vm/sso/install	1970-01-01 00:00:00 +0000
+++ setup_vm/sso/install	2013-04-17 01:29:27 +0000
@@ -0,0 +1,45 @@
+#!/bin/sh -ex
+
+# Allow ssh access to launchpad.
+# This should probably be provided by setup_vm. -- vila 2013-03-10
+ssh-keyscan bazaar.launchpad.net >>~/.ssh/known_hosts
+# Get the branch.
+bzr branch lp:canonical-identity-provider {sso.src_dir}
+# Get the download cache.
+bzr branch lp:~canonical-isd-hackers/+junk/download-cache
+# Setup the environment.
+cd {sso.src_dir}
+# Get the version controlled configs.
+bzr branch lp:~canonical-isd-hackers/isd-configs/sso-config branches/project
+# Bootstrap the dependencies
+fab bootstrap:download_cache_path=~/download-cache
+# Set up the correct Django configuration.
+# In order to set the db_host to a directory in .env, we need to use the full
+# path. Otherwise, fab setup_postgresql_server will fail.
+# TODO we can either configure the postgresql authentication and pass db_host
+# as empty, or use cat just to append to the end of the default local.cfg
+# that will contain the full path we need, or pass the user name in a config
+# variable.
+cat <<EOF >django_project/local.cfg
+[__noschema__]
+basedir = .
+db_host = /home/ubuntu/{sso.src_dir}/.env/db
+hostname = {sso.address}:{sso.port}
+
+[__main__]
+includes =
+    config/devel.cfg
+    ../branches/project/config/acceptance-dev.cfg
+
+[django]
+debug = false
+email_port = {sso.smtp_port}
+
+[testing]
+imap_server = {sso.address}
+imap_port = {sso.imap_port}
+# needs to be a full email
+imap_username = [e-mail]
+imap_use_ssl = False
+
+EOF
 
=== added file 'setup_vm/sso/run'
--- setup_vm/sso/run	1970-01-01 00:00:00 +0000
+++ setup_vm/sso/run	2013-04-17 01:29:27 +0000
@@ -0,0 +1,16 @@
+#!/bin/sh
+
+cd ~/{sso.src_dir}
+# We need an SMTP server to send emails.
+.env/bin/twistd localmail --imap {sso.imap_port} --smtp {sso.smtp_port}
+
+# Setup the database.
+fab setup_postgresql_server
+fab manage:loaddata,test
+fab manage:create_test_team
+# get gargoyle flags from their use in the code
+SST_FLAGS=`grep -rho --exclude 'test_*.py' "is_active([\"']\(.*\)[\"']" identityprovider/ webui/ | sed -E "s/is_active\(['\"](.*)['\"]/\1/" | awk '{print tolower($0)}' | sort | uniq | tr '\n' ','`
+# We need to remove the trailing ','
+fab gargoyle_flags:${SST_FLAGS%,}
+# Start the SSO server, accessible from the local network.
+fab run:0.0.0.0:{sso.port}
 
=== added file 'setup_vm/sso/run-for-pay'
--- setup_vm/sso/run-for-pay	1970-01-01 00:00:00 +0000
+++ setup_vm/sso/run-for-pay	2013-04-17 01:29:27 +0000
@@ -0,0 +1,14 @@
+#!/bin/sh
+
+cd ~/{sso.src_dir}
+# We need an SMTP server to send emails.
+.env/bin/twistd localmail --imap {sso.imap_port} --smtp {sso.smtp_port}
+
+# Setup the database.
+fab setup_postgresql_server
+fab manage:loaddata,isdtest
+fab manage:loaddata,allow_unverified
+# Set the allow-unverified config for Pay.
+fab manage:add_openid_rp_config,{pay.url},--allow-unverified,--allowed-sreg="fullname\,nickname\,email\,language"
+# Start the SSO server, accessible from the local network.
+fab run:0.0.0.0:{sso.port}
 
=== added file 'setup_vm/sso/run-for-u1'
--- setup_vm/sso/run-for-u1	1970-01-01 00:00:00 +0000
+++ setup_vm/sso/run-for-u1	2013-04-17 01:29:27 +0000
@@ -0,0 +1,43 @@
+#!/bin/sh
+
+cd ~/{sso.src_dir}
+# We need an SMTP server to send emails.
+.env/bin/twistd localmail --imap {sso.imap_port} --smtp {sso.smtp_port}
+
+# Setup the database.
+fab setup_postgresql_server
+fab manage:loaddata,allow_unverified
+# Set the allow-unverified config for Pay.
+fab manage:add_openid_rp_config,{pay.url},--allow-unverified,--allowed-sreg="fullname\,nickname\,email\,language"
+# Set the allow-unverified config for U1.
+fab manage:add_openid_rp_config,{u1.url},--allow-unverified,--allowed-sreg="fullname\,nickname\,email\,language"
+# Add the API user for U1.
+# We generated this json file with:
+# $ fab manage:createsuperuser
+# Go to {sso.url}/admin
+# Sign in with the super user you have just created.
+# Click the more link next to the Model Admin heading.
+# On the Identityprovider section, click the +Add link next to API Users.
+# Fill the form with:
+# username: u1qauser
+# password: u1qapassword
+# Click the Save button.
+# $ fab manage:dumpdata,identityprovider.APIUser
+cat <<EOF >identityprovider/fixtures/apiuser.json
+[
+    {
+        "pk": 1,
+        "model": "identityprovider.apiuser",
+        "fields": {
+            "username": "u1qauser",
+            "created_at": "2013-04-14 21:09:43",
+            "password": "k1B7nUTaEsrqAPHF/bWsLlNIPUsH7mViraFQBZPgNRRuvsZlRq8CZg==",
+            "updated_at": "2013-04-14 21:09:43"
+        }
+    }
+]
+
+EOF
+fab manage:loaddata,apiuser
+# Start the SSO server, accessible from the local network.
+fab run:0.0.0.0:{sso.port}
 
=== added file 'setup_vm/sso/test'
--- setup_vm/sso/test	1970-01-01 00:00:00 +0000
+++ setup_vm/sso/test	2013-04-17 01:29:27 +0000
@@ -0,0 +1,11 @@
+#!/bin/sh
+
+# FIXME: This should run on the host and get config options expanded
+# -- vila 2013-03-12
+
+cd {sso.src_dir}
+
+# get gargoyle flags from their use in the code
+SST_FLAGS=`grep -rho --exclude 'test_*.py' "is_active([\"']\(.*\)[\"']" identityprovider/ webui/ | sed -E "s/is_active\(['\"](.*)['\"]/\1/" | awk '{print tolower($0)}' | sort | uniq | tr '\n' ';'`
+# run tests                                                                    
+SST_BASE_URL={sso.url} fab acceptance:screenshot=true,report=xml,extended=true,flags=$SST_FLAGS
 
=== added directory 'setup_vm/tests'
=== added file 'setup_vm/tests/__init__.py'
--- setup_vm/tests/__init__.py	1970-01-01 00:00:00 +0000
+++ setup_vm/tests/__init__.py	2013-04-17 01:29:27 +0000
@@ -0,0 +1,75 @@
+import os
+import shutil
+import tempfile
+
+from bzrlib import osutils
+
+def override_env_var(name, value):
+    """Modify the environment, setting or removing the env_variable.
+
+    :param name: The environment variable to set.
+
+    :param value: The value to set the environment to. If None, then
+        the variable will be removed.
+
+    :return: The original value of the environment variable.
+    """
+    orig = os.environ.get(name)
+    if value is None:
+        if orig is not None:
+            del os.environ[name]
+    else:
+        # FIXME: supporting unicode values requires a way to acquire the
+        # user encoding, punting for now -- vila 2013-01-30
+        os.environ[name] = value
+    return orig
+
+
+def override_env(test, name, new):
+    """Set an environment variable, and reset it after the test.
+
+    :param name: The environment variable name.
+
+    :param new: The value to set the variable to. If None, the 
+        variable is deleted from the environment.
+
+    :returns: The actual variable value.
+    """
+    value = override_env_var(name, new)
+    test.addCleanup(override_env_var, name, value)
+    return value
+
+
+isolated_environ = {
+    'HOME': None,
+}
+
+
+def isolate_env(test, env=None):
+    """Isolate test from the environment variables.
+
+    This is usually called in setUp for tests that needs to modify the
+    environment variables and restore them after the test is run.
+
+    :param test: A test instance
+
+    :param env: A dict containing variable definitions to be installed. Only
+        the variables present there are protected. They are initialized with
+        the provided values.
+    """
+    if env is None:
+        env = isolated_environ
+    for name, value in env.items():
+        override_env(test, name, value)
+
+
+def set_cwd_to_tmp(test):
+    """Create a temp dir an cd into it for the test duration.
+
+    This is generally called during a test setup.
+    """
+    test.test_base_dir = tempfile.mkdtemp(prefix='mytests-', suffix='.tmp')
+    test.addCleanup(shutil.rmtree, test.test_base_dir, True)
+    current_dir = os.getcwdu()
+    test.addCleanup(os.chdir, current_dir)
+    os.chdir(test.test_base_dir)
 
=== added file 'setup_vm/tests/test_setup_vm.py'
--- setup_vm/tests/test_setup_vm.py	1970-01-01 00:00:00 +0000
+++ setup_vm/tests/test_setup_vm.py	2013-04-17 01:29:27 +0000
@@ -0,0 +1,1074 @@
+from cStringIO import StringIO
+import os
+
+from bzrlib import errors
+import testtools
+
+import tests
+from bin import setup_vm
+
+
+def requires_known_reference_image(test):
+    # We need a pre-seeded download cache from the user running the tests
+    # as downloading the cloud image is too long.
+    user_conf = setup_vm.VmStack(None)
+    download_cache = user_conf.get('vm.download_cache')
+    if download_cache is None:
+        test.skip('vm.download_cache is not set')
+    # We use some known reference
+    reference_cloud_image_name = 'raring-server-cloudimg-amd64-disk1.img'
+    cloud_image_path = os.path.join(
+        download_cache, reference_cloud_image_name)
+    if not os.path.exists(cloud_image_path):
+        test.skip('%s is not available' % (cloud_image_path,))
+    return download_cache, reference_cloud_image_name
+
+
+class TestCaseWithHome(testtools.TestCase):
+    """Provide an isolated disk-based environment.
+
+    A $HOME directory is setup as well as an /etc/ one so tests can setup
+    config files.
+    """
+
+    def setUp(self):
+        super(TestCaseWithHome, self).setUp()
+        tests.set_cwd_to_tmp(self)
+        tests.isolate_env(self)
+        # Isolate tests from the user environment
+        self.home_dir = os.path.join(self.test_base_dir, 'home')
+        os.mkdir(self.home_dir)
+        os.environ['HOME'] = self.home_dir
+        # Also isolate from the system environment
+        self.etc_dir = os.path.join(self.test_base_dir, 'etc')
+        os.mkdir(self.etc_dir)
+        self.patch(setup_vm, 'system_config_dir', lambda: self.etc_dir)
+
+
+class TestVmMatcher(TestCaseWithHome):
+
+    def setUp(self):
+        super(TestVmMatcher, self).setUp()
+        self.store = setup_vm.VmStore('.', 'foo.conf')
+        self.matcher = setup_vm.VmMatcher(self.store, 'test')
+
+    def test_empty_section_always_matches(self):
+        self.store._load_from_string('foo=bar')
+        matching = list(self.matcher.get_sections())
+        self.assertEqual(1, len(matching))
+
+    def test_specific_before_generic(self):
+        self.store._load_from_string('foo=bar\n[test]\nfoo=baz')
+        matching = list(self.matcher.get_sections())
+        self.assertEqual(2, len(matching))
+        # First matching section is for test
+        self.assertEqual(self.store, matching[0][0])
+        base_section = matching[0][1]
+        self.assertEqual('test', base_section.id)
+        # Second matching section is the no-name one
+        self.assertEqual(self.store, matching[0][0])
+        no_name_section = matching[1][1]
+        self.assertIs(None, no_name_section.id)
+
+
+class TestVmStores(TestCaseWithHome):
+
+    def setUp(self):
+        super(TestVmStores, self).setUp()
+        self.conf = setup_vm.VmStack('foo')
+
+
+    def test_default_in_empty_stack(self):
+        self.assertEqual('1024', self.conf.get('vm.ram_size'))
+
+
+    def test_system_overrides_internal(self):
+        self.conf.system_store._load_from_string('vm.ram_size = 42')
+        self.assertEqual('42', self.conf.get('vm.ram_size'))
+
+    def test_user_overrides_system(self):
+        self.conf.system_store._load_from_string('vm.ram_size = 42')
+        self.conf.store._load_from_string('vm.ram_size = 4201')
+        self.assertEqual('4201', self.conf.get('vm.ram_size'))
+
+    def test_local_overrides_user(self):
+        self.conf.system_store._load_from_string('vm.ram_size = 42')
+        self.conf.store._load_from_string('vm.ram_size = 4201')
+        self.conf.local_store._load_from_string('vm.ram_size = 8402')
+        self.assertEqual('8402', self.conf.get('vm.ram_size'))
+
+
+class TestVmStack(TestCaseWithHome):
+
+    def setUp(self):
+        super(TestVmStack, self).setUp()
+        self.conf = setup_vm.VmStack('foo')
+        self.conf.store._load_from_string('''
+vm.release=raring
+vm.cpu_model=amd64
+''')
+
+    def assertValue(self, expected_value, option):
+        self.assertEqual(expected_value, self.conf.get(option))
+
+    def test_raring_iso_url(self):
+        self.assertValue('http://cdimage.ubuntu.com/daily-live/current/',
+                         'vm.iso_url' )
+
+    def test_raring_iso_name(self):
+        self.assertValue( 'raring-desktop-amd64.iso', 'vm.iso_name')
+
+    def test_raring_cloud_image_url(self):
+        self.assertValue('http://cloud-images.ubuntu.com/raring/current/',
+                         'vm.cloud_image_url')
+
+    def test_raring_cloud_image_name(self):
+        self.assertValue('raring-server-cloudimg-amd64-disk1.img',
+                         'vm.cloud_image_name')
+
+    def test_apt_proxy_set(self):
+        proxy = 'apt_proxy: http://example.org:4321'
+        self.conf.set('vm.apt_proxy', proxy)
+        self.assertEqual(proxy, self.conf.expand_options('{vm.apt_proxy}'))
+
+    def test_download_cache_with_user_expansion(self):
+        download_cache = '~/installers'
+        self.conf.set('vm.download_cache', download_cache)
+        self.assertValue(os.path.join(self.home_dir, 'installers'),
+                         'vm.download_cache')
+
+    def test_images_dir_with_user_expansion(self):
+        images_dir = '~/images'
+        self.conf.set('vm.images_dir', images_dir)
+        self.assertValue(os.path.join(self.home_dir, 'images'),
+                         'vm.images_dir')
+
+
+class TestPathOption(TestCaseWithHome):
+
+    def assertConverted(self, expected, value):
+        option = setup_vm.PathOption('foo', help='A path.')
+        self.assertEquals(expected, option.convert_from_unicode(None, value))
+
+    def test_absolute_path(self):
+        self.assertConverted('/test/path', '/test/path')
+
+    def test_home_path_with_expansion(self):
+        self.assertConverted(self.home_dir, '~')
+
+    def test_path_in_home_with_expansion(self):
+        self.assertConverted(os.path.join(self.home_dir, 'test/path'),
+                             '~/test/path')
+
+
+class TestDownload(TestCaseWithHome):
+
+# FIXME: Needs parametrization against vm.{cloud_image_name|iso_name} and
+# {download_iso|download_cloud_image} -- vila 2013-02-07
+
+    def setUp(self):
+        # Downloading real isos or images is too long for tests, instead, we
+        # fake it by downloading a small but known to exist file: MD5SUMS
+        super(TestDownload, self).setUp()
+        download_cache = os.path.join(self.test_base_dir, 'downloads')
+        os.mkdir(download_cache)
+        self.conf = setup_vm.VmStack('foo')
+        self.conf.store._load_from_string('''
+vm.iso_name=MD5SUMS
+vm.cloud_image_name=MD5SUMS
+vm.release=raring
+vm.cpu_model=amd64
+vm.download_cache=%s
+''' % (download_cache,))
+
+    def test_download_iso(self):
+        vm = setup_vm.Kvm(self.conf)
+        self.assertTrue(vm.download_iso())
+        # Trying to download again will find the file in the cache
+        self.assertFalse(vm.download_iso())
+        # Forcing the download even when the file is present
+        self.assertTrue(vm.download_iso(force=True))
+
+    def test_download_cloud_image(self):
+        vm = setup_vm.Kvm(self.conf)
+        self.assertTrue(vm.download_cloud_image())
+        # Trying to download again will find the file in the cache
+        self.assertFalse(vm.download_cloud_image())
+        # Forcing the download even when the file is present
+        self.assertTrue(vm.download_cloud_image(force=True))
+
+    def test_download_unknown_iso_fail(self):
+        self.conf.set('vm.iso_name', 'I-dont-exist')
+        vm = setup_vm.Kvm(self.conf)
+        self.assertRaises(setup_vm.CommandError, vm.download_iso)
+
+    def test_download_unknown_cloud_image_fail(self):
+        self.conf.set('vm.cloud_image_name', 'I-dont-exist')
+        vm = setup_vm.Kvm(self.conf)
+        self.assertRaises(setup_vm.CommandError, vm.download_cloud_image)
+
+    def test_download_iso_with_unknown_cache_fail(self):
+        dl_cache = os.path.join(self.test_base_dir, 'I-dont-exist')
+        self.conf.set('vm.download_cache', dl_cache)
+        vm = setup_vm.Kvm(self.conf)
+        self.assertRaises(setup_vm.ConfigValueError, vm.download_iso)
+
+    def test_download_cloud_image_with_unknown_cache_fail(self):
+        dl_cache = os.path.join(self.test_base_dir, 'I-dont-exist')
+        self.conf.set('vm.download_cache', dl_cache)
+        vm = setup_vm.Kvm(self.conf)
+        self.assertRaises(setup_vm.ConfigValueError, vm.download_cloud_image)
+
+
+class TestMetaData(TestCaseWithHome):
+
+    def setUp(self):
+        super(TestMetaData, self).setUp()
+        self.conf = setup_vm.VmStack('foo')
+        self.vm = setup_vm.Kvm(self.conf)
+        images_dir = os.path.join(self.test_base_dir, 'images')
+        os.mkdir(images_dir)
+        config_dir = os.path.join(self.test_base_dir, 'config')
+        self.conf.store._load_from_string('''
+vm.name=foo
+vm.images_dir=%s
+vm.config_dir=%s
+''' % (images_dir, config_dir,))
+
+    def test_create_meta_data(self):
+        self.vm.create_meta_data()
+        self.assertTrue(os.path.exists(self.vm._config_dir))
+        self.assertTrue(os.path.exists(self.vm._meta_data_path))
+        with open(self.vm._meta_data_path) as f:
+            meta_data = f.readlines()
+        self.assertEqual(2, len(meta_data))
+        self.assertEqual('instance-id: foo\n', meta_data[0])
+        self.assertEqual('local-hostname: foo\n', meta_data[1])
+
+
+class TestYaml(testtools.TestCase):
+
+    def yaml_load(self, *args, **kwargs):
+        return setup_vm.yaml.safe_load(*args, **kwargs)
+
+    def yaml_dump(self, *args, **kwargs):
+        return setup_vm.yaml.safe_dump(*args, **kwargs)
+
+    def test_load_scalar(self):
+        self.assertEqual({'foo': 'bar'}, self.yaml_load(StringIO('{foo: bar}')))
+        # Surprisingly the enclosing braces are not needed, probably a special
+        # case for the highest level
+        self.assertEqual({'foo': 'bar'}, self.yaml_load(StringIO('foo: bar')))
+
+    def test_dump_scalar(self):
+        self.assertEqual('{foo: bar}\n', self.yaml_dump(dict(foo='bar')))
+
+    def test_load_list(self):
+        self.assertEqual({'foo': ['a', 'b', 'c']},
+                         # Single space indentation is enough
+                         self.yaml_load(StringIO('''\
+foo:
+ - a
+ - b
+ - c
+''')))
+
+    def test_dump_list(self):
+        # No more enclosing braces... yeah for consistency :-/
+        self.assertEqual('foo: [a, b, c]\n',
+                         self.yaml_dump(dict(foo=['a', 'b', 'c'])))
+
+    def test_load_dict(self):
+        self.assertEqual({'foo': {'bar': 'baz'}},
+                          self.yaml_load(StringIO('{foo: {bar: baz}}')))
+        multiple_lines = '''\
+foo: {bar: multiple
+  lines}
+'''
+        self.assertEqual({'foo': {'bar': 'multiple lines'}},
+                          self.yaml_load(StringIO(multiple_lines)))
+
+
+
+class TestLaunchpadAccess(TestCaseWithHome):
+
+    def setUp(self):
+        super(TestLaunchpadAccess, self).setUp()
+        self.conf = setup_vm.VmStack('foo')
+        self.vm = setup_vm.Kvm(self.conf)
+        self.ci_data = setup_vm.CIUserData(self.conf)
+
+    def test_cant_find_private_key(self):
+        self.conf.store._load_from_string('vm.launchpad_id = I-dont-exist')
+        e = self.assertRaises(setup_vm.ConfigPathNotFound,
+                              self.ci_data.set_launchpad_access)
+        key_path = '~/.ssh/I-dont-exist@setup_vm'
+        self.assertEqual(key_path, e.path)
+        self.assertTrue(unicode(e).startswith(
+                'You need to create the {p} keypair'.format(p=key_path)))
+
+    def test_id_with_key(self):
+        ssh_dir = os.path.join(self.home_dir, '.ssh')
+        os.mkdir(ssh_dir)
+        key_path = os.path.join(ssh_dir, 'user@setup_vm')
+        with open(key_path, 'w') as f:
+            f.write('key content')
+        self.conf.store._load_from_string('vm.launchpad_id = user')
+        self.assertEqual(None, self.ci_data.launchpad_hook)
+        self.ci_data.set_launchpad_access()
+        self.assertEqual('''\
+#!/bin/sh
+mkdir -p /home/ubuntu/.ssh
+chown ubuntu:ubuntu ~ubuntu
+chmod 0700 ~ubuntu
+chown ubuntu:ubuntu /home/ubuntu/.ssh
+chmod 0700 /home/ubuntu/.ssh
+cat >/home/ubuntu/.ssh/id_rsa <<'EOSETUPVMCONTENTREALLYUNIQUEDONTBREAKFORFUN'
+key content
+EOSETUPVMCONTENTREALLYUNIQUEDONTBREAKFORFUN
+chown ubuntu:ubuntu /home/ubuntu/.ssh/id_rsa
+chmod 0400 /home/ubuntu/.ssh/id_rsa
+''',
+                         self.ci_data.launchpad_hook)
+        cc = self.ci_data.cloud_config
+        self.assertEquals([['sudo', '-u', 'ubuntu',
+                            'bzr', 'launchpad-login', 'user']],
+                          cc['runcmd'])
+
+
+class TestCIUserData(TestCaseWithHome):
+
+    def setUp(self):
+        super(TestCIUserData, self).setUp()
+        self.conf = setup_vm.VmStack('foo')
+        self.ci_data = setup_vm.CIUserData(self.conf)
+
+    def test_empty_config(self):
+        self.ci_data.populate()
+        # Check default values
+        self.assertIs(None, self.ci_data.root_hook)
+        self.assertIs(None, self.ci_data.ubuntu_hook)
+        cc = self.ci_data.cloud_config
+        self.assertFalse(cc['apt_update'])
+        self.assertFalse(cc['apt_upgrade'])
+        self.assertEqual({'expire': False}, cc['chpasswd'])
+        self.assertEqual('setup_vm finished installing in ${uptime} seconds.',
+                         cc['final_message'])
+        self.assertTrue(cc['manage_etc_hosts'])
+        self.assertEqual('ubuntu', cc['password'])
+        self.assertEqual({'mode': 'poweroff'}, cc['power_state'])
+
+    def test_password(self):
+        self.conf.store._load_from_string('vm.password = tagada')
+        self.ci_data.populate()
+        self.assertEquals('tagada', self.ci_data.cloud_config['password'])
+
+    def test_apt_proxy(self):
+        self.conf.store._load_from_string('vm.apt_proxy = tagada')
+        self.ci_data.populate()
+        self.assertEquals('tagada', self.ci_data.cloud_config['apt_proxy'])
+
+    def test_final_message_precise(self):
+        self.conf.store._load_from_string('vm.release = precise')
+        self.ci_data.populate()
+        self.assertEqual('setup_vm finished installing in $UPTIME seconds.',
+                         self.ci_data.cloud_config['final_message'])
+
+    def test_poweroff_precise(self):
+        self.conf.store._load_from_string('vm.release = precise')
+        self.ci_data.populate()
+        self.assertEqual(['halt'], self.ci_data.cloud_config['runcmd'])
+
+    def test_poweroff_quantal(self):
+        self.conf.store._load_from_string('vm.release = quantal')
+        self.ci_data.populate()
+        self.assertEqual(['halt'], self.ci_data.cloud_config['runcmd'])
+
+    def test_poweroff_other(self):
+        self.conf.store._load_from_string('vm.release = raring')
+        self.ci_data.populate()
+        self.assertEqual({'mode': 'poweroff'},
+                          self.ci_data.cloud_config['power_state'])
+        self.assertIs(None, self.ci_data.cloud_config.get('runcmd'))
+
+    def test_update_true(self):
+        self.conf.store._load_from_string('vm.update = True')
+        self.ci_data.populate()
+        cc = self.ci_data.cloud_config
+        self.assertTrue(cc['apt_update'])
+        self.assertTrue(cc['apt_upgrade'])
+
+    def test_packages(self):
+        self.conf.store._load_from_string('vm.packages = bzr,ubuntu-desktop')
+        self.ci_data.populate()
+        self.assertEqual(['bzr', 'ubuntu-desktop'],
+                         self.ci_data.cloud_config['packages'])
+
+    def test_apt_sources(self):
+        self.conf.store._load_from_string('''\
+vm.release = raring
+# Ensure options are properly expanded (and comments supported ;)
+_archive_url = http://archive.ubuntu.com/ubuntu
+_ppa_url = https://u:[e-mail]/user/ppa/ubuntu
+vm.apt_sources = deb {_archive_url} {vm.release} partner,\
+ deb {_archive_url} {vm.release} main,\
+ deb {_ppa_url} {vm.release} main|ABCDEF
+''')
+        self.ci_data.populate()
+        self.assertEqual(
+            [{'source': 'deb http://archive.ubuntu.com/ubuntu raring partner'},
+             {'source': 'deb http://archive.ubuntu.com/ubuntu raring main'},
+             {'source':
+                  'deb https://u:[e-mail]/user/ppa/ubuntu raring main',
+              'keyid': 'ABCDEF'}],
+            self.ci_data.cloud_config['apt_sources'])
+
+    def create_file(self, path, content):
+        with open(path, 'wb') as f:
+            f.write(content)
+
+    def test_good_ssh_keys(self):
+        paths = ('rsa', 'rsa.pub', 'dsa', 'dsa.pub', 'ecdsa', 'ecdsa.pub')
+        for path in paths:
+            self.create_file(path, '%s\ncontent\n' % (path,))
+        paths_as_list = ','.join(paths)
+        self.conf.store._load_from_string(
+            'vm.ssh_keys = %s' % (paths_as_list,))
+        self.ci_data.populate()
+        self.assertEqual({'dsa_private': 'dsa\ncontent\n',
+                          'dsa_public': 'dsa.pub\ncontent\n',
+                          'ecdsa_private': 'ecdsa\ncontent\n',
+                          'ecdsa_public': 'ecdsa.pub\ncontent\n',
+                          'rsa_private': 'rsa\ncontent\n',
+                          'rsa_public': 'rsa.pub\ncontent\n'},
+                         self.ci_data.cloud_config['ssh_keys'])
+
+    def test_bad_type_ssh_keys(self):
+        self.conf.store._load_from_string('vm.ssh_keys = I-dont-exist')
+        self.assertRaises(setup_vm.ConfigValueError, self.ci_data.populate)
+
+    def test_unknown_ssh_keys(self):
+        self.conf.store._load_from_string('vm.ssh_keys = rsa.pub')
+        self.assertRaises(setup_vm.ConfigPathNotFound, self.ci_data.populate)
+
+    def test_good_ssh_authorized_keys(self):
+        paths = ('home.pub', 'work.pub')
+        for path in paths:
+            self.create_file(path, '%s\ncontent\n' % (path,))
+        paths_as_list = ','.join(paths)
+        self.conf.store._load_from_string(
+            'vm.ssh_authorized_keys = %s' % (paths_as_list,))
+        self.ci_data.populate()
+        self.assertEqual(['home.pub\ncontent\n', 'work.pub\ncontent\n'],
+                         self.ci_data.cloud_config['ssh_authorized_keys'])
+
+    def test_unknown_ssh_authorized_keys(self):
+        self.conf.store._load_from_string('vm.ssh_authorized_keys = rsa.pub')
+        self.assertRaises(setup_vm.ConfigPathNotFound, self.ci_data.populate)
+
+    def test_unknown_root_script(self):
+        self.conf.store._load_from_string('vm.root_script = I-dont-exist')
+        self.assertRaises(setup_vm.ConfigPathNotFound, self.ci_data.populate)
+
+    def test_unknown_ubuntu_script(self):
+        self.conf.store._load_from_string('vm.ubuntu_script = I-dont-exist')
+        self.assertRaises(setup_vm.ConfigPathNotFound, self.ci_data.populate)
+
+    def test_expansion_error_in_script(self):
+        root_script_content = '''#!/bin/sh
+echo Hello {I_dont_exist}
+'''
+        with open('root_script.sh', 'w') as f:
+            f.write(root_script_content)
+        self.conf.store._load_from_string('''\
+vm.root_script = root_script.sh
+''')
+        e = self.assertRaises(errors.ExpandingUnknownOption,
+                              self.ci_data.populate)
+        self.assertEqual(root_script_content, e.string)
+
+    def test_unknown_uploaded_scripts(self):
+        self.conf.store._load_from_string(
+            '''vm.uploaded_scripts = I-dont-exist ''')
+        e = self.assertRaises(setup_vm.ConfigPathNotFound,
+                              self.ci_data.populate)
+
+    def test_root_script(self):
+        with open('root_script.sh', 'w') as f:
+            f.write('''#!/bin/sh
+echo Hello {user}
+''')
+        self.conf.store._load_from_string('''\
+vm.root_script = root_script.sh
+user=root
+''')
+        self.ci_data.populate()
+        # The additional newline after the script is expected
+        self.assertEqual('''\
+#!/bin/sh
+cat >~root/setup_vm_post_install <<'EOSETUPVMCONTENTREALLYUNIQUEDONTBREAKFORFUN'
+#!/bin/sh
+echo Hello root
+
+EOSETUPVMCONTENTREALLYUNIQUEDONTBREAKFORFUN
+chown root:root ~root/setup_vm_post_install
+chmod 0700 ~root/setup_vm_post_install
+''', self.ci_data.root_hook)
+        self.assertEqual(['~root/setup_vm_post_install'],
+                         self.ci_data.cloud_config['runcmd'])
+
+    def test_ubuntu_script(self):
+        with open('ubuntu_script.sh', 'w') as f:
+            f.write('''#!/bin/sh
+echo Hello {user}
+''')
+        self.conf.store._load_from_string('''\
+vm.ubuntu_script = ubuntu_script.sh
+user = ubuntu
+''')
+        self.ci_data.populate()
+        # The additional newline after the script is expected
+        self.assertEqual('''\
+#!/bin/sh
+cat >~ubuntu/setup_vm_post_install <<'EOSETUPVMCONTENTREALLYUNIQUEDONTBREAKFORFUN'
+#!/bin/sh
+echo Hello ubuntu
+
+EOSETUPVMCONTENTREALLYUNIQUEDONTBREAKFORFUN
+chown ubuntu:ubuntu ~ubuntu/setup_vm_post_install
+chmod 0700 ~ubuntu/setup_vm_post_install
+''', self.ci_data.ubuntu_hook)
+        # The command is run as root, so we need to 'su ubuntu' first
+        self.assertEqual([['su', '-l',
+                          '-c', '~ubuntu/setup_vm_post_install',
+                          'ubuntu']],
+                         self.ci_data.cloud_config['runcmd'])
+
+    def test_uploaded_scripts(self):
+        paths = ('foo', 'bar')
+        for path in paths:
+            self.create_file(path, '%s\ncontent\n' % (path,))
+        paths_as_list = ','.join(paths)
+        self.conf.store._load_from_string(
+            'vm.uploaded_scripts = %s' % (paths_as_list,))
+        self.ci_data.populate()
+        self.assertEqual('''\
+#!/bin/sh
+cat >~ubuntu/setup_vm_uploads <<'EOSETUPVMCONTENTREALLYUNIQUEDONTBREAKFORFUN'
+mkdir -p ~ubuntu/bin
+cd ~ubuntu/bin
+cat >foo <<'EOFfoo'
+foo
+content
+
+EOFfoo
+chmod 0755 foo
+cat >bar <<'EOFbar'
+bar
+content
+
+EOFbar
+chmod 0755 bar
+EOSETUPVMCONTENTREALLYUNIQUEDONTBREAKFORFUN
+chown ubuntu:ubuntu ~ubuntu/setup_vm_uploads
+chmod 0700 ~ubuntu/setup_vm_uploads
+''',
+                         self.ci_data.uploaded_scripts_hook)
+        self.assertEqual([['su', '-l',
+                          '-c', '~ubuntu/setup_vm_uploads',
+                          'ubuntu']],
+                         self.ci_data.cloud_config['runcmd'])
+
+
+class TestCreateUserData(TestCaseWithHome):
+
+    def setUp(self):
+        super(TestCreateUserData, self).setUp()
+        self.conf = setup_vm.VmStack('foo')
+        self.vm = setup_vm.Kvm(self.conf)
+
+    def test_empty_config(self):
+        config_dir = os.path.join(self.test_base_dir, 'config')
+        os.mkdir(config_dir)
+        # The config is *almost* empty, we need to set config_dir though as the
+        # user-data needs to be stored there.
+        self.conf.store._load_from_string('vm.config_dir=%s' % (config_dir,))
+        self.vm.create_user_data()
+        self.assertTrue(os.path.exists(self.vm._config_dir))
+        self.assertTrue(os.path.exists(self.vm._user_data_path))
+        with open(self.vm._user_data_path) as f:
+            user_data = f.readlines()
+        # We care about the two first lines only here, checking the format (or
+        # cloud-init is confused)
+        self.assertEqual('#cloud-config-archive\n', user_data[0])
+        self.assertEqual("- {content: '#cloud-config\n", user_data[1])
+
+
+class TestSeed(TestCaseWithHome):
+
+    def setUp(self):
+        super(TestSeed, self).setUp()
+        self.conf = setup_vm.VmStack('foo')
+        self.vm = setup_vm.Kvm(self.conf)
+        images_dir = os.path.join(self.test_base_dir, 'images')
+        os.mkdir(images_dir)
+        config_dir = os.path.join(self.test_base_dir, 'config')
+        self.conf.store._load_from_string('''
+vm.name=foo
+vm.release=raring
+vm.config_dir=%s
+vm.images_dir=%s
+''' % (config_dir, images_dir,))
+
+    def test_create_meta_data(self):
+        self.vm.create_meta_data()
+        self.assertTrue(os.path.exists(self.vm._meta_data_path))
+
+    def test_create_user_data(self):
+        self.vm.create_user_data()
+        self.assertTrue(os.path.exists(self.vm._user_data_path))
+
+    def test_create_seed(self):
+        self.assertTrue(self.vm._seed_path is None)
+        self.vm.create_seed()
+        self.assertFalse(self.vm._seed_path is None)
+        self.assertTrue(os.path.exists(self.vm._seed_path))
+
+
+class TestImageFromCloud(TestCaseWithHome):
+
+    def setUp(self):
+        super(TestImageFromCloud, self).setUp()
+        self.conf = setup_vm.VmStack('foo')
+        self.vm = setup_vm.KvmFromCloudImage(self.conf)
+        images_dir = os.path.join(self.test_base_dir, 'images')
+        os.mkdir(images_dir)
+        download_cache_dir = os.path.join(self.test_base_dir, 'download')
+        os.mkdir(download_cache_dir)
+        self.conf.store._load_from_string('''
+vm.name=foo
+vm.release=raring
+vm.images_dir=%s
+vm.download_cache=%s
+vm.cloud_image_name=fake.img
+vm.disk_size=1M
+''' % (images_dir, download_cache_dir))
+
+    def test_create_disk_image(self):
+        cloud_image_path = os.path.join(self.conf.get('vm.download_cache'),
+                                        self.conf.get('vm.cloud_image_name'))
+        # We need a fake cloud image that can be converted
+        setup_vm.run_subprocess(
+            ['sudo', 'qemu-img', 'create',
+             cloud_image_path, self.conf.get('vm.disk_size')])
+        self.assertTrue(self.vm._disk_image_path is None)
+        self.vm.create_disk_image()
+        self.assertFalse(self.vm._disk_image_path is None)
+        self.assertTrue(os.path.exists(self.vm._disk_image_path))
+
+
+class TestImageWithBacking(TestCaseWithHome):
+
+    def setUp(self):
+        (download_cache_dir,
+         reference_cloud_image_name) = requires_known_reference_image(self)
+        super(TestImageWithBacking, self).setUp()
+        # We'll share the images_dir between vms
+        images_dir = os.path.join(self.test_base_dir, 'images')
+        os.mkdir(images_dir)
+        # Create a shared config
+        conf = setup_vm.VmStack(None)
+        conf.store._load_from_string('''
+vm.release=raring
+vm.images_dir=%s
+vm.download_cache=%s
+vm.disk_size=2G
+[selftest-from-cloud]
+vm.name=selftest-from-cloud
+vm.cloud_image_name=%s
+[selftest-backing]
+vm.name=selftest-backing
+vm.backing=selftest-from-cloud.qcow2
+''' % (images_dir, download_cache_dir, reference_cloud_image_name))
+        conf.store.save()
+        # To bypass creating a real vm, we start from the cloud image that is a
+        # real and bootable one, so we just convert it. That also makes it
+        # available in vm.images_dir
+        temp_vm = setup_vm.KvmFromCloudImage(
+            setup_vm.VmStack('selftest-from-cloud'))
+        temp_vm.create_disk_image()
+
+    def test_create_image_with_backing(self):
+        vm = setup_vm.KvmFromBacking(setup_vm.VmStack('selftest-backing'))
+        self.assertTrue(vm._disk_image_path is None)
+        vm.create_disk_image()
+        self.assertFalse(vm._disk_image_path is None)
+        self.assertTrue(os.path.exists(vm._disk_image_path))
+
+
+class TestVmStates(testtools.TestCase):
+
+    def assertStates(self, expected, lines):
+        self.assertEqual(expected, setup_vm.vm_states(lines))
+
+    def test_empty(self):
+        self.assertStates({},[])
+
+    def test_garbage(self):
+        self.assertRaises(ValueError, self.assertStates, None, [''])
+
+    def test_known_states(self):
+        # From a real life sample
+        self.assertStates({'foo': 'shut off', 'bar': 'running'},
+                          ['-     foo                            shut off',
+                           '19    bar                            running'])
+
+
+class TestConsoleParsing(testtools.TestCase):
+
+    def _parse_console_monitor(self, string):
+        mon = setup_vm.ConsoleMonitor(StringIO(string))
+        lines = []
+        for line in mon.parse():
+            lines.append(line)
+        return lines
+
+    def test_fails_on_empty(self):
+        self.assertRaises(setup_vm.ConsoleEOFError,
+                          self._parse_console_monitor, '')
+
+    def test_fail_on_knwon_cloud_init_errors(self):
+        self.assertRaises(
+            setup_vm.CloudInitError,
+            self._parse_console_monitor,
+            'Failed loading yaml blob\n')
+        self.assertRaises(
+            setup_vm.CloudInitError,
+            self._parse_console_monitor,
+            'Unhandled non-multipart userdata starting\n')
+        self.assertRaises(
+            setup_vm.CloudInitError,
+            self._parse_console_monitor,
+            "failed to render string to stdout: cannot find 'uptime'\n")
+        self.assertRaises(
+            setup_vm.CloudInitError,
+            self._parse_console_monitor,
+            "Failed loading of cloud config "
+            "'/var/lib/cloud/instance/cloud-config.txt'. "
+            "Continuing with empty config\n")
+
+    def test_succeds_on_final_message(self):
+        lines = self._parse_console_monitor('''
+Lalala
+I'm doing my work
+It goes nicely
+setup_vm finished installing in 1 seconds.
+That was fast isn't it ?
+ * Will now halt
+[   33.204755] Power down.
+''')
+        # We stop as soon as we get the final message and ignore the rest
+        self.assertEquals(' * Will now halt\n',
+                          lines[-1])
+
+
+class TestConsoleParsingWithFile(TestCaseWithHome):
+
+    def _parse_file_monitor(self, string):
+        with open('console', 'w') as f:
+            f.write(string)
+        mon = setup_vm.FileMonitor('console')
+        for line in mon.parse():
+            pass
+        return mon.lines
+
+    def test_succeeds_with_file(self):
+        content = '''\
+Yet another install
+Going well
+setup_vm finished installing in 0.5 seconds.
+Wow, even faster !
+ * Will now halt
+Whatever, won't read that
+'''
+        lines = self._parse_file_monitor(content)
+
+    def xtest_fails_on_empty_file(self):
+        # FIXME: We need some sort of timeout there...
+        self.assertRaises(setup_vm.CommandError, self._parse_file_monitor, '')
+
+    def test_fail_on_knwon_cloud_init_errors_with_file(self):
+        self.assertRaises(
+            setup_vm.CloudInitError,
+            self._parse_file_monitor,
+            'Failed loading yaml blob\n')
+        self.assertRaises(
+            setup_vm.CloudInitError,
+            self._parse_file_monitor,
+            'Unhandled non-multipart userdata starting\n')
+        self.assertRaises(
+            setup_vm.CloudInitError,
+            self._parse_file_monitor,
+            "failed to render string to stdout: cannot find 'uptime'\n")
+
+
+class TestInstallWithSeed(TestCaseWithHome):
+
+    def setUp(self):
+        (download_cache,
+         reference_cloud_image_name) = requires_known_reference_image(self)
+        super(TestInstallWithSeed, self).setUp()
+        # We need to allow other users to read this dir
+        os.chmod(self.test_base_dir, 0755)
+        # We also need to sudo rm it as root created some files there
+        self.addCleanup(
+            setup_vm.run_subprocess,
+            ['sudo', 'rm', '-fr',
+             os.path.join(self.test_base_dir, 'home', '.virtinst')])
+        self.conf = setup_vm.VmStack('selftest-seed')
+        self.vm = setup_vm.KvmFromCloudImage(self.conf)
+        images_dir = os.path.join(self.test_base_dir, 'images')
+        os.mkdir(images_dir, 0755)
+        config_dir = os.path.join(self.test_base_dir, 'config')
+        self.conf.store._load_from_string('''
+vm.name=selftest-seed
+vm.update=False # Shorten install time
+vm.cpus=2,
+vm.release=raring
+vm.config_dir=%s
+vm.images_dir=%s
+vm.download_cache=%s
+vm.cloud_image_name=%s
+vm.disk_size=8G
+''' % (config_dir, images_dir, download_cache, reference_cloud_image_name))
+
+    def assertVmState(self, expected):
+        states = setup_vm.vm_states()
+        self.assertEqual(expected, states[self.vm.conf.get('vm.name')])
+
+    def test_install_with_seed(self):
+        self.addCleanup(self.vm.undefine)
+        self.vm.install()
+        self.assertVmState('shut off')
+
+
+class TestInstallWithBacking(TestCaseWithHome):
+
+    def setUp(self):
+        (download_cache_dir,
+         reference_cloud_image_name) = requires_known_reference_image(self)
+        super(TestInstallWithBacking, self).setUp()
+        # We need to allow other users to read this dir
+        os.chmod(self.test_base_dir, 0755)
+        # We also need to sudo rm it as root created some files there
+        self.addCleanup(
+            setup_vm.run_subprocess,
+            ['sudo', 'rm', '-fr',
+             os.path.join(self.test_base_dir, 'home', '.virtinst')])
+        self.conf = setup_vm.VmStack('selftest-backing')
+        self.vm = setup_vm.KvmFromBacking(self.conf)
+        # We'll share the images_dir between vms
+        images_dir = os.path.join(self.test_base_dir, 'images')
+        os.mkdir(images_dir, 0755)
+        config_dir = os.path.join(self.test_base_dir, 'config')
+        # Create a shared config
+        conf = setup_vm.VmStack(None)
+        conf.store._load_from_string('''
+vm.release=raring
+vm.config_dir=%s
+vm.images_dir=%s
+vm.download_cache=%s
+vm.disk_size=2G
+vm.update=False # Shorten install time
+[selftest-from-cloud]
+vm.name=selftest-from-cloud
+vm.cloud_image_name=%s
+[selftest-backing]
+vm.name=selftest-backing
+vm.backing=selftest-from-cloud.qcow2
+''' % (config_dir, images_dir, download_cache_dir, reference_cloud_image_name))
+        conf.store.save()
+        # Fake a previous install by just re-using the reference cloud image
+        temp_vm = setup_vm.KvmFromCloudImage(
+            setup_vm.VmStack('selftest-from-cloud'))
+        temp_vm.create_disk_image()
+
+    def assertVmState(self, vm, expected):
+        states = setup_vm.vm_states()
+        self.assertEqual(expected, states[vm.conf.get('vm.name')])
+
+    def test_install_with_backing(self):
+        vm = setup_vm.KvmFromBacking(setup_vm.VmStack('selftest-backing'))
+        self.addCleanup(vm.undefine)
+        vm.install()
+        self.assertVmState(vm, 'shut off')
+
+
+class TestSshKeyGen(TestCaseWithHome):
+
+    def setUp(self):
+        super(TestSshKeyGen, self).setUp()
+        self.conf = setup_vm.VmStack(None)
+        self.vm = setup_vm.VM(self.conf)
+        self.config_dir = os.path.join(self.test_base_dir, 'config')
+
+    def load_config(self, more):
+        content = '''\
+vm.config_dir=%s
+vm.name=foo
+''' % (self.config_dir,)
+        self.conf.store._load_from_string(content + more)
+
+    def generate_key(self, ssh_type, upper_type=None):
+        if upper_type is None:
+            upper_type = ssh_type.upper()
+        self.load_config('vm.ssh_keys={vm.config_dir}/%s' % (ssh_type,))
+        self.vm.ssh_keygen()
+        private_path = 'config/%s' % (ssh_type,)
+        self.assertTrue(os.path.exists(private_path))
+        public_path = 'config/%s.pub' % (ssh_type,)
+        self.assertTrue(os.path.exists(public_path))
+        public = file(public_path).read()
+        private = file(private_path).read()
+        self.assertTrue(private.startswith(
+                '-----BEGIN %s PRIVATE KEY-----\n' % (upper_type,)))
+        self.assertTrue(private.endswith(
+                '-----END %s PRIVATE KEY-----\n' % (upper_type,)))
+        return private, public
+
+    def test_dsa(self):
+        private, public = self.generate_key('dsa')
+        self.assertTrue(public.startswith('ssh-dss '))
+        self.assertTrue(public.endswith(' foo\n'))
+
+    def test_rsa(self):
+        private, public = self.generate_key('rsa')
+        self.assertTrue(public.startswith('ssh-rsa '))
+        self.assertTrue(public.endswith(' foo\n'))
+
+    def test_ecdsa(self):
+        private, public = self.generate_key('ecdsa', 'EC')
+        self.assertTrue(public.startswith('ecdsa-sha2-nistp256 '))
+        self.assertTrue(public.endswith(' foo\n'))
+
+
+class TestOptionParsing(testtools.TestCase):
+
+    def setUp(self):
+        super(TestOptionParsing, self).setUp()
+        self.out = StringIO()
+        self.err = StringIO()
+
+    def parse_args(self, args):
+        return setup_vm.arg_parser.parse_args(args, self.out, self.err)
+
+    def test_nothing(self):
+        self.assertRaises(SystemExit, self.parse_args, [])
+
+    def test_install(self):
+        ns = self.parse_args(['foo', '--install'])
+        self.assertEquals('foo', ns.name)
+        self.assertTrue(ns.install)
+        self.assertFalse(ns.download)
+
+    def test_download(self):
+        ns = self.parse_args(['foo', '--download'])
+        self.assertEquals('foo', ns.name)
+        self.assertFalse(ns.install)
+        self.assertTrue(ns.download)
+
+class TestBuildCommands(testtools.TestCase):
+
+    def setUp(self):
+        super(TestBuildCommands, self).setUp()
+        self.out = StringIO()
+        self.err = StringIO()
+
+    def build_commands(self, args):
+        return setup_vm.build_commands(args, self.out, self.err)
+
+    def test_install(self):
+        cmds = self.build_commands(['--install', 'foo'])
+        self.assertEqual(1, len(cmds))
+        self.assertTrue(isinstance(cmds[0], setup_vm.Install))
+
+    def test_download(self):
+        cmds = self.build_commands(['--download', 'foo'])
+        self.assertEqual(1, len(cmds))
+        self.assertTrue(isinstance(cmds[0], setup_vm.Download))
+
+    def test_ssh_keygen(self):
+        cmds = self.build_commands(['--ssh-keygen', 'foo'])
+        self.assertEqual(1, len(cmds))
+        self.assertTrue(isinstance(cmds[0], setup_vm.SshKeyGen))
+
+    def test_download_and_install(self):
+        cmds = self.build_commands(['--install', '--download', 'foo'])
+        self.assertEqual(2, len(cmds))
+        # Download comes first
+        self.assertTrue(isinstance(cmds[0], setup_vm.Download))
+        self.assertTrue(isinstance(cmds[1], setup_vm.Install))
+
+
+# FIXME: This needs to be parametrized for KvmFromCloudImage and
+# KvmFromBacking. Since we don't define vm.backing below, we're only testing
+# KvmFromCloudImage for now. -- vila 2013-02-13
+class TestInstall(TestCaseWithHome):
+
+    def setUp(self):
+        super(TestInstall, self).setUp()
+        self.conf = setup_vm.VmStack('I-dont-exist')
+        self.conf.store._load_from_string('''
+vm.name=I-dont-exist
+vm.release=raring
+vm.cpu_model=amd64
+''')
+        self.states = []
+
+        def vm_states(source=None):
+            return self.states
+        self.patch(setup_vm, 'vm_states', vm_states)
+        self.vm = None
+
+    def install(self):
+        class FakeKvm(setup_vm.Kvm):
+
+            def __init__(self, conf):
+                super(FakeKvm, self).__init__(conf)
+                self.undefine_called = False
+                self.install_called = False
+
+            # Make sure we avoid dangerous or costly calls
+            def poweroff(self):
+                pass
+
+            def undefine(self):
+                self.undefine_called = True
+
+            def install(self):
+                self.install_called = True
+
+
+        self.vm = FakeKvm(self.conf)
+        cmd = setup_vm.Install(self.vm)
+        cmd.run()
+
+    def test_install_while_running(self):
+        self.conf.set('vm.name', 'foo')
+        self.states = {'foo': 'running'}
+        self.assertRaises(setup_vm.SetupVmError, self.install)
+        self.assertFalse(self.vm.install_called)
+        self.assertFalse(self.vm.undefine_called)
+
+    def test_install_unknown(self):
+        self.states = {}
+        self.install()
+        self.assertTrue(self.vm.install_called)
+        self.assertFalse(self.vm.undefine_called)
+
+    def test_install_shutoff(self):
+        self.conf.set('vm.name', 'foo')
+        self.states = {'foo': 'shut off'}
+        self.install()
+        self.assertTrue(self.vm.install_called)
+        self.assertTrue(self.vm.undefine_called)
 
=== added file 'setup_vm/tests/test_test.py'
--- setup_vm/tests/test_test.py	1970-01-01 00:00:00 +0000
+++ setup_vm/tests/test_test.py	2013-04-17 01:29:27 +0000
@@ -0,0 +1,58 @@
+import os
+
+import testtools
+
+import tests
+
+
+def assertTestSuccess(test, inner):
+    """The received test runs successfully."""
+    result = testtools.TestResult()
+    inner.run(result)
+    test.assertEqual(0, len(result.errors) + len(result.failures))
+    test.assertEqual(1, result.testsRun)
+    return result
+
+
+class TestEnv(testtools.TestCase):
+
+
+    def test_env_preserved(self):
+        os.environ['NOBODY_USES_THIS'] = 'foo'
+
+        class Inner(testtools.TestCase):
+
+            def  test_overridden(self):
+                tests.isolate_env(self, {'NOBODY_USES_THIS': 'bar'})
+                self.assertEqual('bar', os.environ['NOBODY_USES_THIS'])
+
+        assertTestSuccess(self, Inner('test_overridden'))
+        self.assertEqual('foo', os.environ['NOBODY_USES_THIS'])
+
+    def test_env_var_deleted(self):
+        os.environ['NOBODY_USES_THIS'] = 'foo'
+
+        class Inner(testtools.TestCase):
+
+            def  test_deleted(self):
+                tests.isolate_env(self, {'NOBODY_USES_THIS': None})
+                self.assertIs('deleted',
+                              os.environ.get('NOBODY_USES_THIS', 'deleted'))
+        assertTestSuccess(self, Inner('test_deleted'))
+        self.assertEqual('foo', os.environ['NOBODY_USES_THIS'])
+
+
+class TestTmp(testtools.TestCase):
+
+    def test_cwd_in_tmp(self):
+
+        class Inner(testtools.TestCase):
+
+            def setUp(self):
+                super(Inner, self).setUp()
+                tests.set_cwd_to_tmp(self)
+
+            def test_cwd_in_tmp(self):
+                self.assertEqual(os.getcwdu(), self.test_base_dir)
+
+        assertTestSuccess(self, Inner('test_cwd_in_tmp'))
 
=== added directory 'setup_vm/u1'
=== added file 'setup_vm/u1/install'
--- setup_vm/u1/install	1970-01-01 00:00:00 +0000
+++ setup_vm/u1/install	2013-04-17 01:29:27 +0000
@@ -0,0 +1,71 @@
+#!/bin/sh -ex
+
+# Allow ssh access to launchpad.
+# This should probably be provided by setup_vm. -- vila 2013-03-10
+ssh-keyscan bazaar.launchpad.net >>~/.ssh/known_hosts
+# Use the openjdk.
+sudo update-alternatives --set java /usr/lib/jvm/java-7-openjdk-amd64/jre/bin/java
+# Get the branch.
+bzr branch lp:ubuntuone-servers {u1.src_dir}
+# Setup the environment.
+cd {u1.src_dir}
+# Set up the correct configuration.
+cat <<EOF >configs/local.conf
+[meta]
+extends: development-appserver-lazr.conf
+
+[general]
+port: {u1.port}
+django_module: u1servers.web.localsettings
+
+[upay]
+consumer_id: U1
+port: {pay.port}
+hostname: {pay.address}
+url_format: http://%(host)s:%(port)d/api/2.0
+
+[upay_u1ms]
+consumer_id: U1
+port: {pay.port}
+hostname: {pay.address}
+url_format: http://%(host)s:%(port)d/api/2.0
+
+[url]
+openid_sso_server: {sso.url}
+
+EOF
+# XXX The secrets file is overlayed, so we can't use the config file.
+# This is an ugly way to overwrite the default values.
+cat <<EOF >>configs/dev_secrets-lazr.conf
+ubuntu_pay_username: u1qauser
+ubuntu_pay_password: u1qapassword
+ubuntu_pay_username_u1ms: u1qauser
+ubuntu_pay_password_u1ms: u1qapassword
+
+EOF
+cat <<EOF >servers/u1servers/web/localsettings.py
+from u1servers.web.devsettings import *
+
+OPENID_SSO_SERVER_URL = config.url.openid_sso_server
+OPENID_SSO_LOGOUT_URL = '%s/+logout?return_to=%s' % (
+    OPENID_SSO_SERVER_URL, BASE_URL)
+
+if __name__ == os.environ.get("DJANGO_SETTINGS_MODULE"):
+    # This only gets executed if the configured DJANGO_SETTINGS_MODULE matches  
+    # the current module name.                                                  
+    from ubuntuone import dispatch
+    dispatch.connect_all(async=True)
+
+    from u1servers.web import email
+    email.connect_receivers()
+
+    # Triggered when the env variable U1_PAY_HOST is defined with               
+    # "<hostname>:<port>"                                                       
+    if config.upay.hostname or config.upay_u1ms.hostname:
+        from u1backends.account.upayclient import init_payclient
+        init_payclient()
+EOF
+make update-sourcedeps
+# TODO ask on #u1-ops if there's a better way.
+sed -i 's/development-lazr.conf/local.conf/g' utilities/supervisor-dev.conf.tpl
+sed -i 's/development-appserver-lazr.conf/local.conf/g' utilities/supervisor-dev.conf.tpl
 
=== added file 'setup_vm/u1/run'
--- setup_vm/u1/run	1970-01-01 00:00:00 +0000
+++ setup_vm/u1/run	2013-04-17 01:29:27 +0000
@@ -0,0 +1,4 @@
+#!/bin/sh
+
+cd ~/{u1.src_dir}
+HOSTNAME={u1.address} DJANGO_SETTINGS_MODULE=u1servers.web.localsettings U1CONFIG=`pwd`/configs/local.conf make start
 
=== added file 'setup_vm/u1/test'
--- setup_vm/u1/test	1970-01-01 00:00:00 +0000
+++ setup_vm/u1/test	2013-04-17 01:29:27 +0000
@@ -0,0 +1,15 @@
+#!/bin/sh
+
+
+cd {u1.src_dir}
+# When run from the host against the u1 guest:
+# sudo apt-get install python-mocker
+# scp ubuntu@{u1.address}:~/ubuntuone-servers/configs/local.conf configs/local.conf 
+# scp ubuntu@{u1.address}:~/ubuntuone-servers/servers/u1servers/web/localsettings.py servers/u1servers/web/localsettings.py
+# TODO ask on #u1-ops if there's a better way.
+# sed -i 's/development-lazr.conf/local.conf/g' utilities/supervisor-dev.conf.tpl
+# sed -i 's/development-appserver-lazr.conf/local.conf/g' utilities/supervisor-dev.conf.tpl
+# echo 9999 >tmp/statsd.port
+# make update-sourcedeps
+U1CONFIG=`pwd`/configs/local.conf make smoke-test
+U1CONFIG=`pwd`/configs/local.conf make acceptance-test
 
=== added directory 'setup_vm/unity'
=== added file 'setup_vm/unity/install-sources'
--- setup_vm/unity/install-sources	1970-01-01 00:00:00 +0000
+++ setup_vm/unity/install-sources	2013-04-17 01:29:27 +0000
@@ -0,0 +1,19 @@
+#!/bin/sh
+# Allow ssh access to launchpad.
+# This should probably be provided by setup_vm. -- vila 2013-03-10
+ssh-keyscan bazaar.launchpad.net >>~/.ssh/known_hosts
+# Install sst and dependencies from source
+mkdir src
+cd src
+bzr branch lp:~ubuntuone-hackers/ubuntuone-servers/selenium
+bzr branch lp:~canonical-isd-qa/selenium-simple-test/trunk selenium-simple-test
+cd ~/src/selenium
+python setup.py install --user
+cd ~/src/selenium-simple-test
+python setup.py install --user
+
+# If the need arise to use sst-run, it may become necessary to create a
+# symlink to /home/ubuntu/.local/bin/sst-run
+
+# Also note that unittest2 is installed as a side-effect of installing sst
+# even if we're already using pyton-2.7 (which includes unittest2 features).
 
=== added file 'setup_vm/unity/run-sso-client'
--- setup_vm/unity/run-sso-client	1970-01-01 00:00:00 +0000
+++ setup_vm/unity/run-sso-client	2013-04-17 01:29:27 +0000
@@ -0,0 +1,11 @@
+#!/bin/sh -ex
+
+# We can use U1_DEBUG=True to get debug messages on the console.
+USSOC_SERVICE_URL={sso.url}/api/1.0/ /usr/lib/ubuntu-sso-client/ubuntu-sso-login &
+# XXX ugly sleep.
+sleep 5s
+# TODO x86_64 sounds like trouble.
+# This has just stopped working on raring. See http://pad.lv/1161067
+# TODO in order for the application to be accessible with testability, we need
+# TESTABILITY=1
+/usr/lib/ubuntu-sso-client/ubuntu-sso-login-qt --app_name 'Ubuntu One' --help_text '...' --ping_url '{u1.url}/oauth/sso-finished-so-get-tokens/%7Bemail%7D?platform_version=3.8.0-2-generic&platform=Linux&client_version=4.1.90&platform_arch=x86_64' --policy_url {u1.url}/privacy/ --tc_url {u1.url}/terms/
 
=== added file 'setup_vm/unity/run-syncdaemon'
--- setup_vm/unity/run-syncdaemon	1970-01-01 00:00:00 +0000
+++ setup_vm/unity/run-syncdaemon	2013-04-17 01:29:27 +0000
@@ -0,0 +1,4 @@
+#!/bin/sh -ex
+
+/usr/lib/ubuntuone-client/ubuntuone-syncdaemon --disable_ssl_verify  --dns_srv=None --host={filesync.address} &
+u1sdtool --connect
 
=== added file 'setup_vm/unity/run-unity-lens-music'
--- setup_vm/unity/run-unity-lens-music	1970-01-01 00:00:00 +0000
+++ setup_vm/unity/run-unity-lens-music	2013-04-17 01:29:27 +0000
@@ -0,0 +1,6 @@
+#!/bin/sh -ex
+
+# TODO x86_64 sounds like trouble.
+# We can use G_MESSAGES_DEBUG=all to get debug messages on the console.
+# TODO change the name of the environment variables, it's not just staging.
+pkill unity-music; U1_STAGING_WEBAPI={u1.url} U1_STAGING_AUTHENTICATION={sso.url} /usr/lib/x86_64-linux-gnu/unity-lens-music/unity-musicstore-daemon
 
=== added file 'setup_vm/unity/transient-dist-upgrade'
--- setup_vm/unity/transient-dist-upgrade	1970-01-01 00:00:00 +0000
+++ setup_vm/unity/transient-dist-upgrade	2013-04-17 01:29:27 +0000
@@ -0,0 +1,4 @@
+#!/bin/sh
+# For an unclear reason (probably a transient raring issue) we need to
+# dist-upgrade instead of just upgrade
+apt-get dist-upgrade -y
 
=== added file 'setup_vm/vms.conf'
--- setup_vm/vms.conf	1970-01-01 00:00:00 +0000
+++ setup_vm/vms.conf	2013-04-17 01:29:27 +0000
@@ -0,0 +1,96 @@
+# This must be defined in some other vms.conf file (user or system)
+# sso.address=sso.local
+# pay.address=pay.local
+# u1.address=u1.local
+# ppa.ubuntuone-hackers.password
+
+sso.src_dir=canonical-identity-provider
+sso.port=8001
+sso.url=http://{sso.address}:{sso.port}
+sso.imap_port=2143
+sso.smtp_port=2025
+
+pay.src_dir=canonical-payment-service
+pay.port=8002
+pay.url=http://{pay.address}:{pay.port}
+
+ppa.ubuntuone_hackers=deb https://{vm.launchpad_id}:{ppa.ubuntuone_hackers.password}@private-ppa.launchpad.net/ubuntuone/hackers/ubuntu {vm.release} main|4BD0ECAE
+
+u1.src_dir=ubuntuone-servers
+u1.port=8003
+u1.url=http://{u1.address}:{u1.port}
+
+[precise-server-pristine]
+vm.name=precise-server-pristine
+vm.release=precise
+vm.packages=bzr, avahi-daemon, emacs23
+vm.update=True
+
+[sso]
+vm.name=sso
+vm.release=precise
+vm.backing=precise-server-pristine.qcow2
+vm.packages=config-manager, fabric, libpq-dev, make, memcached, postgresql-plpython, python-m2crypto, python-dev, python-setuptools, python-virtualenv, swig, wget, libxml2-dev, libxslt1-dev
+vm.ubuntu_script=sso/install
+vm.update=True
+vm.uploaded_scripts=sso/run, sso/run-for-pay, sso/run-for-u1
+
+[pay]
+vm.name=pay
+vm.release=precise
+vm.backing=precise-server-pristine.qcow2
+vm.packages=config-manager, fabric, libpq-dev, make, postgresql-plpython, python-dev, python-setuptools, python-virtualenv, wget, libxml2-dev, libxslt1-dev
+vm.ubuntu_script=pay/install
+vm.update=True
+vm.uploaded_scripts=pay/run, pay/run-for-u1
+
+[u1]
+vm.name=u1
+vm.release=precise
+vm.backing=precise-server-pristine.qcow2
+vm.apt_sources={ppa.ubuntuone_hackers}
+vm.packages=openjdk-7-jre,ubuntuone-developer-dependencies
+vm.ubuntu_script=u1/install
+vm.update=True
+vm.uploaded_scripts=u1/run
+
+[raring-desktop-pristine]
+vm.name=raring-desktop-pristine
+vm.release=raring
+# python-unittest2 is not strictly required here but works around sst
+# insisting on installing it locally.
+vm.packages=bzr, emacs23, python-setuptools, python-unittest2, python-autopilot, unity-autopilot, ubuntu-desktop, avahi-daemon
+vm.update=True
+# Roughly all vms installing ubuntu-desktop need to complete the
+# installation by making the ubuntu user part of the admin group.
+vm.root_script = bin/ubuntu_admin.sh
+
+[purchase-testing]
+vm.name=purchase-testing
+vm.release=raring
+vm.backing=raring-desktop-pristine.qcow2
+vm.apt_sources=deb http://ppa.launchpad.net/ubuntuone/dashpurchase-testing/ubuntu {vm.release} main|4BD0ECAE
+vm.update=True
+vm.ubuntu_script=purchase-testing/install
+
+[unity-prevalidation]
+vm.name=unity-prevalidation
+vm.release=raring
+vm.backing=raring-desktop-pristine.qcow2
+vm.apt_sources=deb http://ppa.launchpad.net/ubuntu-unity/experimental-prevalidation/ubuntu {vm.release} main|52D62F45
+vm.uploaded_scripts=unity/run-sso-client, unity/run-unity-lens-music
+# TODO unity/run-syncdaemon. We don't yet have the hermetic filesync server.
+vm.update=True
+vm.root_script=unity/transient-dist-upgrade
+vm.ubuntu_script=unity/install-sources
+
+[indash-didrocks]
+vm.name=indash-didrocks
+vm.release=raring
+vm.backing=raring-desktop-pristine.qcow2
+vm.apt_sources=ppa:didrocks/ppa
+vm.uploaded_scripts=unity/run-sso-client, unity/run-unity-lens-music
+# TODO unity/run-syncdaemon. We don't yet have the hermetic filesync server.
+vm.update=True
+vm.root_script=unity/transient-dist-upgrade
+vm.ubuntu_script=unity/install-sources