Merge lp:~thomir-deactivatedaccount/adt-cloud-worker/uci-nova-testbed-fixes into lp:adt-cloud-worker

[Thomi Richards]

cloud-data improvements.

Include fixes from pitti's vm-setup script. Should improve pass rate.

[lp:~thomir-deactivatedaccount/adt-cloud-worker/uci-nova-testbed-fixes]

8. By Thomi Richards on 2015-04-15

Fix typo.

9. By Thomi Richards on 2015-04-15

Always reboot.

Unmerged revisions

9. By Thomi Richards on 2015-04-15

Always reboot.

8. By Thomi Richards on 2015-04-15

Fix typo.

7. By Thomi Richards on 2015-04-15

Integrate fixes for cloud-init userdata from pitti's autopkgtest vm-setup script.

6. By Celso Providelo on 2015-04-06

Return console-logging cmd-line option back to adt on open(), so it can be properly used when calling cleanup() later on. [r=Francis Ginther]

5. By Francis Ginther on 2015-04-02

Add missing ':' for console in long opts list. [r=Thomi Richards]

4. By Francis Ginther on 2015-04-02

Add support for saving the nova console-log to the specified location. Also increase the number of ipaddr retries as spawning can be a bit slow. [r=Thomi Richards, Paul Larson]

3. By Paul Larson on 2015-03-26

Support snappy images by enabling ssh in the cloud-init data for snappy

2. By Celso Providelo on 2015-03-17

Coping with missing neutron security-groups capabilities (cannonistack and stackystack) and falling back to nova security-groups without egress traffic control.

1. By Celso Providelo on 2015-03-12

Initial import of the neutron-security-group-based setup.

=== added file 'README.rst'
--- README.rst	1970-01-01 00:00:00 +0000
+++ README.rst	2015-04-15 16:34:08 +0000
@@ -0,0 +1,30 @@
+Ubuntu CI nova ssh setup for autopkgtest
+########################################
+
+
+This branch contains a custom autopkgtest ssh-setup script for creating
+nova testbeds with the same network restrictions applied on the Ubuntu
+adt-cloud production system.
+
+It basically restricts ingress network traffic to SSH (tcp:22) using an
+ephemeral keypair created exclusively for the current test. Egress network
+traffic is restricted to DNS (on 8.8.8.8) and APT and NTP (from
+nova.clouds.archive.ubuntu.com and ntp.ubuntu.com respectively, as required
+for the stock ubuntu cloud-images).
+
+Getting the code::
+
+  $ bzr branch lp:~canonical-ci-engineering/adt-cloud-worker/uci-nova-ssh-setup
+
+Enabling it involves copying 'uci-nova' to the default autopkgtest ssh-setup
+location::
+
+  $ sudo cp uci-nova-ssh-setup/uci-nova /usr/share/autopkgtest/ssh-setup/
+
+A example usage command-line is::
+
+  $ adt-run libpng -d \
+    --- ssh -s uci-nova -d \
+      -- -d \
+      --flavor m1.small \
+      --image ubuntu-trusty-14.04-amd64-server-20150305-disk1.img

=== renamed file 'README.rst' => 'README.rst.moved'
=== added file 'uci-nova'
--- uci-nova	1970-01-01 00:00:00 +0000
+++ uci-nova	2015-04-15 16:34:08 +0000
@@ -0,0 +1,393 @@
+#!/bin/sh
+#
+# This script is part of autopkgtest
+# autopkgtest is a tool for testing Debian binary packages
+#
+# This script sets up a nova instance to use as an autopkgtest testbed. It
+# assumes that the host system is already prepared to run nova commands.
+# WARNING: This is mostly a proof of concept and not very robust.
+
+# Options:
+#
+# -f flavor | --flavor=flavor
+#        Name or ID of flavor (see 'nova flavor-list'), mandatory
+# -i image | --image=image
+#        Name or ID of image (see 'nova image-list'), mandatory
+# -N net-id | --net-id=net-id
+#        UUID of the network that should be used for the instance
+# -n name | --name=name
+#        Name for the new server. A name will be generated if not specified.
+# -c console-log | --console=file-name
+#        Save the nova console-log of the server to the specified file.
+#
+#
+# Authors:
+# Celso Providelo <[e-mail]>
+#
+#
+# autopkgtest is Copyright (C) 2006-2015 Canonical Ltd.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+#
+# See the file CREDITS for a full list of credits information (often
+# installed as /usr/share/doc/autopkgtest/CREDITS).
+set -eu
+
+CAPABILITIES='isolation-machine,reboot,revert,revert-full-system'
+
+SUDO_PASSWORD=''
+SSH_USER=ubuntu
+
+FLAVOR=""
+IMAGE=""
+
+SRVNAME=""
+NET_ID=""
+CONSOLE=""
+DEBUG=""
+
+
+debug() {
+    [ -z "$DEBUG" ] && return
+    echo "$@">&2
+}
+
+warning() {
+    echo "$@">&2
+}
+
+error() {
+    echo "$@">&2
+}
+
+parse_args() {
+    # Parse command line argument and populate environment
+
+    SHORTOPTS="f:,i:,N:,n:,c:,d"
+    LONGOPTS="flavor:,image:,net-id:,name:,console:,debug"
+
+    TEMP=$(getopt -o $SHORTOPTS --long $LONGOPTS -- "$@")
+    eval set -- "$TEMP"
+
+    while true; do
+        case "$1" in
+            -f|--flavor)
+                FLAVOR=$2
+                shift 2;;
+            -i|--image)
+                IMAGE=$2
+                shift 2;;
+            -N|--net-id)
+                NET_ID="$2"
+                shift 2;;
+            -n|--name)
+                SRVNAME=$2
+                shift 2;;
+            -c|--console)
+                CONSOLE=$2
+                shift 2;;
+            -d|--debug)
+                DEBUG=1; shift;;
+            --)
+                shift;
+                break;;
+            *)
+                error "E: $(basename $0): Unsupported option $1"
+                exit 1;;
+        esac
+    done
+
+    if [ -z "$FLAVOR" ]; then
+        error "Argument 'flavor' is mandatory. Run 'nova flavor-list' to "\
+            "print a list of available flavors."
+        exit 1
+    fi
+    if [ -z "$IMAGE" ]; then
+        error "Argument 'image' is mandatory. Run 'nova image-list' to "\
+            "print a list of available images to boot from."
+        exit 1
+    fi
+}
+
+security_setup_nova() {
+    debug "Creating specific nova security-group: $SRVNAME"
+    nova secgroup-create $SRVNAME "$SRVNAME testbed"
+
+    debug "Allowing ingress SSH ..."
+    nova secgroup-add-rule $SRVNAME tcp 22 22 0.0.0.0/0
+
+    debug "Egress traffic will respect external Firewall rules ..."
+}
+
+security_setup_neutron() {
+    debug "Creating specific neutron security-group: $SRVNAME"
+    neutron security-group-create $SRVNAME \
+	--description "$SRVNAME testbed"
+
+    debug "Cleaning up default egress rules ..."
+    DEFAULT_RULES=$(neutron security-group-rule-list \
+	--format csv -c id -c security_group --quote none \
+	| grep $SRVNAME | cut -d',' -f1)
+    for rule_id in $DEFAULT_RULES; do
+	neutron security-group-rule-delete $rule_id
+    done
+
+    debug "Allowing ingress SSH ..."
+    neutron security-group-rule-create \
+	--direction ingress \
+	--ethertype ipv4 \
+	--protocol tcp \
+	--port-range-min 22 \
+	--port-range-max 22 \
+	--remote-ip-prefix 0.0.0.0/0 \
+	$SRVNAME
+
+    debug "Allowing DNS egress traffic ..."
+    neutron security-group-rule-create \
+    	--direction egress \
+    	--remote-ip-prefix 8.8.8.8 \
+    	$SRVNAME
+
+    debug "Allowing APT and NTP egress traffic ..."
+    neutron security-group-rule-create \
+    	--direction egress \
+    	--remote-ip-prefix 91.189.88.0/21 \
+    	$SRVNAME
+}
+
+# create a testbed (if necessary), configure ssh, copy ssh key into it,
+# configure sudo, etc.; print a list of "key=value" parameters to stdout on
+# success
+open() {
+    # Boot a nova instance and returns its connection parameters
+    [ -n "$SRVNAME" ] || SRVNAME=`mktemp -u adt-nova-XXXXXX`
+
+    mkdir /tmp/$SRVNAME
+
+    debug "Creating new SSH key on /tmp/$SRVNAME"
+    SSH_IDENTITY=/tmp/$SRVNAME/id_rsa
+    ssh-keygen -f $SSH_IDENTITY -q -N ""
+
+    debug "Creating specific nova keypair: $SRVNAME"
+    nova keypair-add --pub-key $SSH_IDENTITY.pub $SRVNAME
+
+    # Setup testbed security with nova or neutron according to their
+    # availability in the target cloud.
+    if ! neutron security-group-list; then
+	security_setup_nova
+    else
+        security_setup_neutron
+    fi
+        
+    # generate cloud-init user data; mostly for manage_etc_hosts, but also get
+    # rid of some unnecessary stuff in the VM
+    #
+    # NOTE: The "#cloud-config" at the top of this file tells cloud-init that this is
+    # a config file, as opposed to a shell script (which must start with "#!""). Don't 
+    # Do what I did and think "commented out code! DELETE!" and then wonder why nothing
+    # works. - Thomi
+    local userdata=`mktemp`
+    cat <<EOF > $userdata
+#cloud-config
+manage_etc_hosts: true
+apt_update: true
+apt_upgrade: false
+snappy:
+    ssh_enabled: True
+
+runcmd:
+ # Make apt faster:
+ - echo 'Acquire::Languages "none";' > /etc/apt/apt.conf.d/90nolanguages
+ - echo 'force-unsafe-io' > /etc/dpkg/dpkg.cfg.d/autopkgtest
+ # provides kmods like scsi_debug or mac80211_hwsim
+ - apt-get install -y linux-generic
+ # some tests use a lot of /dev/random, avoid hangs
+ - apt-get install -y haveged
+ # We need python to run the auxverb helper - install it iff it's missing:
+ - if ! sh -c 'type python3 >/dev/null 2>&1 || type python >/dev/null 2>&1'; then apt-get install -y --no-install-recommends python3-minimal; fi
+ # Remove some packages we don't need
+ - for p in accountsservice apt-xapian-index cryptsetup landscape-client landscape-common open-vm-tools w3m vim-runtime aptitude-common command-not-found-data manpages ntfs-3g sosreport ubuntu-release-upgrader-core; do apt-get --auto-remove -y purge \$p || true; done
+ - apt-get clean
+ # Do a dist-upgrade:
+ - DEBIAN_FRONTEND=noninteractive apt-get -y dist-upgrade
+EOF
+
+    EXTRA_OPTS=''
+    if [ -n "$NET_ID" ]; then
+        EXTRA_OPTS="$EXTRA_OPTS --nic net-id=$NET_ID"
+    fi
+
+    # Boot the instance
+    debug "Creating nova instance $SRVNAME ..."
+    OUT=$(nova boot --config-drive=1 \
+	  --flavor $FLAVOR --image $IMAGE --user-data $userdata \
+          --key_name $SRVNAME --security-groups $SRVNAME \
+	  $EXTRA_OPTS $SRVNAME 2>&1) || {
+        error "nova boot failed:"
+        error "$OUT"
+        exit 1
+    }
+    debug "Nova boot succeeded"
+    rm $userdata
+
+    # Find IP address
+    ipaddr=""
+    retry=60
+    while [ -z "$ipaddr" ]; do
+        OUT=$(nova show --minimal $SRVNAME)
+        ipaddr=$(echo "$OUT" | awk 'BEGIN {FS="|"} /network/ {n=split($3,i,/,\s*/); gsub(" ", "", i[n]); print i[n]}')
+        retry=$(( retry - 1 ))
+        if [ $retry -le 0 ]; then
+            error "Failed to acquire an IP address. Aborting!"
+            error "$OUT"
+            cleanup
+            exit 1
+        fi
+        sleep 3
+    done
+    debug "Finding IP address succeeded: $ipaddr"
+
+    # purge the device host key so that SSH doesn't print a scary warning
+    ssh-keygen -f ~/.ssh/known_hosts -R $ipaddr >/dev/null 2>&1 || true
+
+    ADT_EXTRA_OPTS="-n $SRVNAME"
+    if [ -n "$CONSOLE" ]; then
+        ADT_EXTRA_OPTS="$ADT_EXTRA_OPTS -c $CONSOLE"
+    fi
+
+    # Return access information to adt-virt-ssh.
+    cat<<EOF
+identity=$SSH_IDENTITY
+login=$SSH_USER
+hostname=$ipaddr
+capabilities=$CAPABILITIES
+extraopts=$ADT_EXTRA_OPTS
+EOF
+    if [ -n "$SUDO_PASSWORD" ]; then
+        echo "password=$SUDO_PASSWORD"
+    fi
+
+    # wait until ssh is available and cloud-config is done
+    debug "Waiting until ssh becomes available"
+    SSH="ssh -q -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -i /tmp/$SRVNAME/id_rsa -l $SSH_USER $ipaddr"
+    retry=60
+    while ! $SSH true; do
+        retry=$(( retry - 1 ))
+        if [ $retry -le 0 ]; then
+            error "Timed out waiting for ssh. Aborting!"
+            cleanup
+            exit 1
+        fi
+        sleep 5
+    done
+
+    debug "Waiting for cloud-init to finish"
+    if ! timeout 30m $SSH 'while [ ! -e /var/lib/cloud/instance/boot-finished ]; do sleep 1; done'; then
+        error "Timed out waiting for cloud-init to finish. Aborting!"
+        cleanup
+        exit 1
+    fi
+
+}
+
+cleanup() {
+    if [ -z "$SRVNAME" ]; then
+        error "Cannot determine server name. Instance won't be deleted!"
+        exit 0
+    fi
+
+    if [ -n "$CONSOLE" ]; then
+        debug "Saving console-log for $SRVNAME"
+        nova console-log $SRVNAME > $CONSOLE
+    fi
+
+    debug "Deleting $SRVNAME instance"
+    nova delete $SRVNAME || true
+
+    debug "Deleting $SRVNAME keypair"
+    nova keypair-delete $SRVNAME || true
+
+    debug "Deleting /tmp/$SRVNAME SSH keys"
+    rm -rf /tmp/$SRVNAME || true
+
+    DELETE_CMD="neutron security-group-delete $SRVNAME"
+    if ! neutron security-group-list; then
+	DELETE_CMD="nova secgroup-delete $SRVNAME"
+    fi
+
+    debug "Deleting $SRVNAME security-group"
+    retry=3
+    while ! eval "$DELETE_CMD"; do
+        retry=$(( retry - 1 ))
+        if [ $retry -le 0 ]; then
+            error "Timed out deleting secgroup. Aborting!"
+            cleanup
+            exit 1
+        fi
+        sleep 5
+    done
+
+    SRVNAME=""
+}
+
+revert() {
+    if [ -z "$SRVNAME" ]; then
+        echo "Needs to be called with -n <server name>" >&2
+        exit 1
+    fi
+    cleanup
+    open
+}
+
+reboot() {
+    if [ -z "$SRVNAME" ]; then
+        error "Cannot determine server name. Instance won't be rebooted!"
+        exit 1
+    fi
+
+    nova reboot --poll $SRVNAME >/dev/null 2>&1||true
+}
+
+# ########################################
+# Main procedure
+#
+if [ $# -eq 0 ]; then
+    error "Invalid number of arguments, command is missing"
+    exit 1
+fi
+cmd=$(echo $1|tr [[:upper:]] [[:lower:]])
+shift
+parse_args "$@"
+
+# Don't leave stuff behind ...
+trap "cleanup" 1 2 6 15
+
+case $cmd in
+    open)
+        open;;
+    cleanup)
+        cleanup;;
+    revert)
+        revert;;
+    reboot)
+        reboot;;
+    '')
+        echo "Needs to be called with command as first argument" >&2
+        exit 1
+        ;;
+    *)
+        echo "invalid command $cmd" >&2
+        exit 1
+esac