Merge lp:~vila/uci-engine/britney-images into lp:uci-engine

[Vincent Ladeuil]

Document how glance images have been created for uci-britney

This describes how the glance images for uci-britney have been created.

In the long term, this should become part of the image builder.

In the mean time, it documents how a nova instance can be snapshotted to produce a glance image which opens the doors to any setup we may want to do inside a running instance which is easier than using a chroot in a qemu mounted image.

[lp:~vila/uci-engine/britney-images]

880. By Vincent Ladeuil on 2014-11-03

Chmod 755 the script

881. By Vincent Ladeuil on 2014-11-03

Cleanup

[PS Jenkins bot (ps-jenkins)]

PASSED: Continuous integration, rev:881
http://s-jenkins.ubuntu-ci:8080/job/uci-engine-ci/1663/
Executed test runs:

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/uci-engine-ci/1663/rebuild

review: Approve (continuous-integration)

[Joe Talbott (joetalbott)]

This looks okay to me.

review: Approve

=== modified file 'britney_proxy/README.images'
--- britney_proxy/README.images	2014-10-08 07:35:31 +0000
+++ britney_proxy/README.images	2014-11-03 13:13:03 +0000
@@ -19,49 +19,47 @@
 
 Create a directory to hold cloud images
 
-ubuntu@juju-bootstack-machine-1:~$ mkdir images
+ubuntu@juju-bootstack-machine-1:~$ mkdir -p images
 ubuntu@juju-bootstack-machine-1:~$ cd images
 
 Get some images
 
-ubuntu@juju-bootstack-machine-1:~$ wget http://cloud-images.ubuntu.com/utopic/current/utopic-server-cloudimg-amd64-disk1.img --output-document utopic-amd64.img
-
-ubuntu@juju-bootstack-machine-1:~$ wget http://cloud-images.ubuntu.com/trusty/current/trusty-server-cloudimg-amd64-disk1.img --output-document trusty-amd64.img
-
-ubuntu@juju-bootstack-machine-1:~$ wget http://cloud-images.ubuntu.com/precise/current/precise-server-cloudimg-amd64-disk1.img --output-document precise-amd64.img
-
-ubuntu@juju-bootstack-machine-1:~$ wget http://cloud-images.ubuntu.com/utopic/current/utopic-server-cloudimg-i386-disk1.img --output-document utopic-i386.img
-
-ubuntu@juju-bootstack-machine-1:~$ wget http://cloud-images.ubuntu.com/trusty/current/trusty-server-cloudimg-i386-disk1.img --output-document trusty-i386.img
-
-ubuntu@juju-bootstack-machine-1:~$ wget http://cloud-images.ubuntu.com/precise/current/precise-server-cloudimg-i386-disk1.img --output-document precise-i386.img
+for series in utopic trusty precise
+do
+    for arch in amd64 i386
+    do
+        wget http://cloud-images.ubuntu.com/${series}/current/${series}-server-cloudimg-${arch}-disk1.img --output-document ${series}-${arch}.img
+    done
+done
 
 
 Create images used by tests
 
-ubuntu@juju-bootstack-machine-1:~$ glance image-create --name uci/cloudimg/utopic-amd64.img --file utopic-amd64.img --disk-format qcow2 --container-format bare --is-public True
-
-ubuntu@juju-bootstack-machine-1:~$ glance image-create --name uci/cloudimg/trusty-amd64.img --file trusty-amd64.img --disk-format qcow2 --container-format bare --is-public True
-
-ubuntu@juju-bootstack-machine-1:~$ glance image-create --name uci/cloudimg/precise-amd64.img --file precise-amd64.img --disk-format qcow2 --container-format bare --is-public True
-
-ubuntu@juju-bootstack-machine-1:~$ glance image-create --name uci/cloudimg/utopic-i386.img --file utopic-i386.img --disk-format qcow2 --container-format bare --is-public True
-
-ubuntu@juju-bootstack-machine-1:~$ glance image-create --name uci/cloudimg/trusty-i386.img --file trusty-i386.img --disk-format qcow2 --container-format bare --is-public True
-
-ubuntu@juju-bootstack-machine-1:~$ glance image-create --name uci/cloudimg/precise-i386.img --file precise-i386.img --disk-format qcow2 --container-format bare --is-public True
-
-Create images used for britney (just the cloud image to start with, further
-optimizations will happen later)
-
-ubuntu@juju-bootstack-machine-1:~$ glance image-create --name uci/britney/utopic-amd64.img --file utopic-amd64.img --disk-format qcow2 --container-format bare --is-public True
-
-ubuntu@juju-bootstack-machine-1:~$ glance image-create --name uci/britney/trusty-amd64.img --file trusty-amd64.img --disk-format qcow2 --container-format bare --is-public True
-
-ubuntu@juju-bootstack-machine-1:~$ glance image-create --name uci/britney/precise-amd64.img --file precise-amd64.img --disk-format qcow2 --container-format bare --is-public True
-
-ubuntu@juju-bootstack-machine-1:~$ glance image-create --name uci/britney/utopic-i386.img --file utopic-i386.img --disk-format qcow2 --container-format bare --is-public True
-
-ubuntu@juju-bootstack-machine-1:~$ glance image-create --name uci/britney/trusty-i386.img --file trusty-i386.img --disk-format qcow2 --container-format bare --is-public True
-
-ubuntu@juju-bootstack-machine-1:~$ glance image-create --name uci/britney/precise-i386.img --file precise-i386.img --disk-format qcow2 --container-format bare --is-public True
+PREFIX="uci/cloudimg/"
+for series in utopic trusty precise
+do
+    for arch in amd64 i386
+    do
+        BASE_NAME=${series}-${arch}.img
+        IMG_NAME=${PREFIX}${BASE_NAME}
+        glance image-delete ${IMG_NAME}
+        glance image-create --name ${IMG_NAME} --file ${BASE_NAME} --disk-format qcow2 --container-format bare --is-public True
+    done
+done
+
+Create images used for britney, this should be run from your local host (not the image builder one) and requires a known keypair to be registered with nova.
+
+nova keypair-add --pub-key <path> <your key name>
+
+We care only about utopic and trusty for now (precise is not targeted, vivid doesn't yet provide a cloud image).
+
+for series in utopic trusty 
+do
+    for arch in amd64 i386
+    do
+        BASE_NAME=${series}-${arch}.img
+        glance image-delete "uci/britney/${BASE_NAME}"
+        ./image-builder/create-nova-adt-image.sh "uci/cloudimg/${BASE_NAME}" vila-uci-vms
+        glance image-update --is-public True "uci/britney/${BASE_NAME}"
+    done
+done

=== added file 'image-builder/create-nova-adt-image.sh'
--- image-builder/create-nova-adt-image.sh	1970-01-01 00:00:00 +0000
+++ image-builder/create-nova-adt-image.sh	2014-11-03 13:13:03 +0000
@@ -0,0 +1,55 @@
+#!/bin/sh
+# create an uci/britney/${release}-${arch}.img nova image based on a generic
+# image and adt-setup-vm
+# Author: Martin Pitt <[e-mail]>
+set -eu
+BASEIMG="${1:-}"
+
+if [ -z "$BASEIMG" ]; then
+    echo "Usage: $0 <base image name> [key name]" >&2
+    exit 1
+fi
+
+opts=""
+[ -z "${2:-}" ] || opts="$opts --key_name $2"
+[ -z "${EPHEMERAL_CLOUD_NET_ID:-}" ] || opts="$opts --nic net-id=$EPHEMERAL_CLOUD_NET_ID"
+
+# unbreak my server option :-(
+userdata=`mktemp`
+trap "rm $userdata" EXIT TERM INT QUIT PIPE
+/bin/echo -e "#cloud-config\nmanage_etc_hosts: True" > $userdata
+
+# create new instance
+INSTNAME="$BASEIMG-adt-prepare"
+nova boot --flavor m1.smaller --image $BASEIMG --user-data $userdata $opts --poll $INSTNAME
+
+# determine IP address
+ipaddr=$(nova show --minimal $INSTNAME | awk 'BEGIN {FS="|"} /network/ {n=split($3,i,/,\s*/); gsub(" ", "", i[n]); print i[n]}')
+SSH_CMD="ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no ubuntu@$ipaddr"
+echo "Waiting for ssh (may cause some error messages)..."
+timeout 5m sh -c "while ! $SSH_CMD true; do sleep 5; done"
+
+echo "Waiting until cloud-init is done..."
+timeout 10m $SSH_CMD 'while [ ! -e /var/lib/cloud/instance/boot-finished ]; do sleep 1; done'
+
+# Hack to create a vivid image from an utopic one
+
+# $SSH_CMD 'sudo sed -i "s/utopic/vivid/g" /etc/apt/sources.list /etc/apt/sources.list.d/*.list || true'
+# $SSH_CMD sudo apt-get update
+# $SSH_CMD sudo apt-get -y dist-upgrade
+
+echo "Running adt setup script..."
+$SSH_CMD sudo sh -x < /usr/share/autopkgtest/adt-setup-vm
+release=$($SSH_CMD lsb_release -sc)
+arch=$($SSH_CMD dpkg --print-architecture)
+
+echo "Powering off to get a clean file system..."
+$SSH_CMD sudo poweroff
+while ! nova show --minimal $INSTNAME | grep -q 'vm_state.*stopped'; do sleep 1; done
+
+# determine new image name and build it
+IMAGE_NAME="uci/britney/${release}-${arch}.img"
+nova image-create --show --poll $INSTNAME "$IMAGE_NAME"
+nova delete $INSTNAME
+echo
+echo "Successfully built image $IMAGE_NAME"

=== modified file 'tests/test_britney.py'
--- tests/test_britney.py	2014-10-16 10:31:16 +0000
+++ tests/test_britney.py	2014-11-03 13:13:03 +0000
@@ -60,8 +60,7 @@
 
     scenarios = scenarii.multiply_scenarios(
         # series
-        ([('precise', dict(series='precise', result='FAILED')),
-          ('trusty', dict(series='trusty', result='PASSED')),
+        ([('trusty', dict(series='trusty', result='PASSED')),
           ('utopic', dict(series='utopic', result='PASSED'))]),
         # architectures
         ([('amd64', dict(arch='amd64')), ('i386', dict(arch='i386'))]))