Merge ~smoser/ubuntu/+source/open-iscsi:bug/1785108-xenial-net-interface-handler-runs-always into ubuntu/+source/open-iscsi:ubuntu/xenial-devel

[Scott Moser]

net-interface-handler: Apply changes only for the iscsi-root interface.

net-interface-handler was effectively bringing "up" the iscsi-root
interface for all network device adds and it "down" for all the removes.
The quotes are used above because the design point of the script is to
basically fake ifupdown into believing the event already occurred.

The change here is to only operate when the INTERFACE provided by
udev is the iscsi root device.

I've also pulled back the functioning dep8 tests from cosmic.
This allows us to actually have a test that shows the failure and
passes with the changes.

LP: #1785108

see commit message

03dc4a4... by Scott Moser on 2018-08-02

releasing package open-iscsi version 2.0.873+git0.3b4b4500-14ubuntu3.5

f5cfc6b... by Scott Moser on 2018-08-02

copy debian/tests cloud image test back from cosmic.

The open-iscsi boot test has been quite problematic in the past.
There are a number of fixes since it was last touched in 16.04 and
it is now running consistently in cosmic.

So bring back the tests and tools from cosmic.

beeb3a5... by Scott Moser on 2018-08-02

debian/tests/test-open-iscsi.py: attempt to trigger 1785108

Without the fix applied, this would cause test failure.

710ebad... by Scott Moser on 2018-08-02

update changelog pull test back

7a7fad1... by Scott Moser on 2018-08-07

ammend tests to have depends open-iscsi

diff --git a/debian/changelog b/debian/changelog
index b23a821..e6dbe18 100644
--- a/debian/changelog
+++ b/debian/changelog
@@ -1,3 +1,11 @@
+open-iscsi (2.0.873+git0.3b4b4500-14ubuntu3.5) xenial; urgency=medium
+
+  * d/tests: pull back cloud image test updates from cosmic.
+  * d/net-interface-handler: Apply changes only for the iscsi-root
+    (LP: #1785108)
+
+ -- Scott Moser <[e-mail]>  Thu, 02 Aug 2018 15:09:53 -0400
+
 open-iscsi (2.0.873+git0.3b4b4500-14ubuntu3.4) xenial; urgency=medium
 
   * d/p/08_Parse-origin-value-from-iBFT.patch
diff --git a/debian/net-interface-handler b/debian/net-interface-handler
index 7717f61..15d35c6 100755
--- a/debian/net-interface-handler
+++ b/debian/net-interface-handler
@@ -10,12 +10,23 @@
 # ifupdown appears to have no way to do this without also running
 # /etc/network/*.d/ scripts.
 
+assert_interface() {
+    # udev sets INTERFACE to the name of the currently-processed nic.
+    [ -n "$INTERFACE" ] && return 0
+    echo "environment variable INTERFACE not set." 1>&2;
+    return 1
+}
+
 start() {
     CR="
 "
+    assert_interface || return
     ifile=/run/initramfs/open-iscsi.interface
-    if [ -f "$ifile" ] && read iface < "$ifile" &&
-       ! grep -qs "^$iface=" /run/network/ifstate; then
+
+    [ -f "$ifile" ] && read iface < "$ifile" || return 0
+    [ "$INTERFACE" = "$iface" ] || return
+
+    if ! grep -qs "^$iface=" /run/network/ifstate; then
         mkdir -p /run/network
         echo "$iface=$iface" >>/run/network/ifstate
 
@@ -51,9 +62,12 @@ EOF
 }
 
 stop() {
+    assert_interface || return
     ifile=/run/initramfs/open-iscsi.interface
-    if [ -f "$ifile" ] && read iface < "$ifile" &&
-       grep -qs "^$iface=" /run/network/ifstate; then
+    [ -f "$ifile" ] && read iface < "$ifile" || return 0
+    [ "$INTERFACE" = "$iface" ] || return
+
+    if grep -qs "^$iface=" /run/network/ifstate; then
         grep -v "^$iface=" /run/network/ifstate >/run/network/.ifstate.tmp || true
         mv /run/network/.ifstate.tmp /run/network/ifstate
 
diff --git a/debian/tests/README-boot-test.md b/debian/tests/README-boot-test.md
new file mode 100644
index 0000000..e45bbf0
--- /dev/null
+++ b/debian/tests/README-boot-test.md
@@ -0,0 +1,139 @@
+## open-iscsi boot test
+The purpose of this test (`CloudImageTest`) is to test the boot of a system
+using an iscsi root target.  In order to accomplish that, the tests does
+
+  1. Download Ubuntu cloud image
+  2. installs the open-iscsi deb inside
+  3. collect kernel and initramfs from inside
+  4. register the image as a read-only iscsi target served by tgt
+  5. boot kernel and initramfs with a command line to use the image as root.
+     And additionally attach a local disk for collecting output.
+  6. provide user-data that executes commands, colects files and writes them
+     to the output disk and then shuts the system down.
+  7. extract the collected files from the output disk and inspect them.
+
+The `CloudImageTest` uses qemu user networking.
+
+
+## Caveats
+
+  1. It depends on a cloud-image being present.
+
+     Cloud-images are often not available for the first few weeks of a cycle.
+     If no cloud-image of 'REL' is available, then boot-test will skip.
+     If 'REL' is not known in distro-info-data, then test will fail.
+
+     This means that uploads of open-iscsi (or its dependencies) will not
+     be properly tested until a cloud-image is available, and will fail
+     until distro-info-data is uploaded.
+
+  2. Installation of large packages via patch-image may fail.
+     An ubuntu image downloaded has only a small amount of extra space.
+     Installation of a new kernel into the target would probably fail.
+
+     If this becomes a problem, we could grow the disk like done at
+       https://gist.github.com/smoser/6a048a0e2795b48221fc44962202fa14
+
+
+### testing manually ###
+The test case in `debian/tests/test-open-iscsi.py` uses some helper tools.
+
+  * **patch-image**: this is used to install packages into the pristine image
+     and collect the kernel and initramfs out of the image.  This allows us to
+     test the portions of open-iscsi that update the initramfs.  Without
+     using the updated initramfs we wouldn't really be testing the new
+     open-iscsi.
+
+     It will upgrade any packages inside that are mentioned in
+     ADT_TEST_TRIGGERS environment.  It will also install open-iscsi if
+     it is not in that list.
+
+     It installs packages into the target by copying the host system's
+     /etc/apt content in, and also includes copying local (file://) apt
+     repos into the target.  This is necessary for the autopackage test
+     environment that adds local package repositories to sources.list.d.
+
+     It may also make other changes to the image to workaround bugs.
+     See --help output for list of bugs.
+
+  * **get-image**: This downloads an image from cloud-images.ubuntu.com.  See
+     its Usage for more information.  it downloads via stream data and verifies
+     download.  One thing to note is that it does not overwrite existing files.
+
+  * **tgt-boot-test**: this registers an image in tgt locally, and then boots
+     kernel and initramfs to mount that.  It knows how to build iscsi kernel
+     command lines.
+
+     By default, the xkvm process that is started will be allowed `60m` to
+     complete.  This can be adjusted by setting `BOOT_TIMEOUT` in the
+     environment.
+
+  * **xkvm**: this is a helper/wrapper around qemu.  It is taken from the curtin
+     projects tools/ directory.  It allows some simplified command lines, and
+     most usefully, the '--netdev=<bridge>' argument will create a tun/tap
+     device and attach it to the bridge.
+
+     If the host system is not bare metal, kvm will not be
+     enabled in the guest that is lauched.  To force use of kvm, set
+     _USE_KVM=1 in the environment.  See 'should_try_kvm' in xkvm for details.
+
+Testing manually looks like this:
+
+    ## set up path to include debian/tests directory.
+    $ PATH=$PWD/debian/tests:$PATH
+
+    ## Get the image you want.  This creates out.d/disk.img and disk.img.dist
+    $ get-image xenial.d xenial
+
+    ## patch the image with an open-iscsi, which creates xenial.d/kernel
+    ## and xenial.d/initrd from the kernel and initramfs inside the image.
+    $ apt-get download open-iscsi
+    $ deb=$(ls open-iscsi_*.deb | tail -n 1)
+    $ sudo ./debian/tests/patch-image \
+         --kernel=xenial.d/kernel --initrd=xenial.d/initrd
+         xenial.d/disk.img "$deb"
+
+    ## Boot the system, log in, look around.
+    $ tgt-boot-test -v xenial.d/disk.img xenial.d/kernel xenial.d/initrd
+
+
+### Features of tgt-boot-test ###
+
+tgt-boot-test does a number of useful things.
+
+ * determines the host address that the guest will use.
+   This should support ipv6 and ipv4 addresses on bridges, and
+   knows values that qemu's user networking uses.  Flags passed to `--netdev`
+   are read intelligently.  This can be overriden with `--host-addr`, but
+   it does a good job of determining what the right values are.
+
+ * provides a nocloud metadata service with a python web server that
+   supports ipv4 and ipv6.
+
+ * provides the ability to provide additional kernel command line options
+   or to provide a 'template' that references variables it knows such as
+   {iserver} (iscsi server) or {seed_url}.
+
+ * Sets ubuntu (passw0rd) and root password (root) and imports users
+   ssh keys to the ubuntu user.
+
+One thing to note is that yakkety's version of qemu does not run an ipv6
+dhcp server on its user-network, so a stateful dhclient request will not
+work.
+
+In order to create a bridge easily with a ipv6 dhcp server, you can use
+lxd at sufficent version (https://github.com/lxc/lxd/issues/2481).
+Assuming that bug is fixed, to create an ipv6 only bridge:
+
+    $ netname="ipv6-only"
+    $ lxc network create $netname
+    $ lxc network unset $netname ipv4.address
+    $ lxc network unset $netname ipv4.nat
+    $ lxc network set $netname ipv6.dhcp.stateful true
+
+Then, you can use tgt-boot-test with that:
+
+    $ PATH=$PWD/debian/tests:$PATH
+    $ ./debian/tests/tgt-boot-test -vv --netdev=ipv6-only \
+       --cmdline-ip="ip=off ip6=dhcp" \
+       disk.img kernel initramfs
diff --git a/debian/tests/control b/debian/tests/control
index 320757e..3662819 100644
--- a/debian/tests/control
+++ b/debian/tests/control
@@ -1,3 +1,7 @@
-Tests: install sysvinit-install daemon testsuite
+Tests: install sysvinit-install daemon
 Restrictions: needs-root isolation-machine breaks-testbed
-Depends: open-iscsi, python, tgt, qemu-system, ubuntu-cloudimage-keyring, simplestreams, python-netifaces, distro-info, cloud-image-utils
+Depends: open-iscsi
+
+Tests: testsuite
+Restrictions: needs-root isolation-machine breaks-testbed
+Depends: open-iscsi, python, tgt, qemu-system, ubuntu-cloudimage-keyring, simplestreams, python-netifaces, distro-info, cloud-image-utils, dctrl-tools, rsync
diff --git a/debian/tests/get-maas-eph b/debian/tests/get-image
index 660a72d..41b903c 100755
--- a/debian/tests/get-maas-eph
+++ b/debian/tests/get-image
@@ -2,23 +2,38 @@
 Usage() {
    cat <<EOF
 ${0##*/} out-directory release
+
+By default, the cloud image for 'release' is downloaded from the
+daily cloud image stream.
+
+Additional arguments can change behavior of what is downloaded.
+Some supported arguments:
+   * cloud-daily: use daily stream [default]
+   * cloud-release: use release stream rather than daily stream.
+   * maas-release, maas-daily: download maas images instead of cloud-image
+   * hwe-N: download the hwe kernel from release 'N' instead of default
+     this is only respected if maas stream is used.
+   * YYYYMMDD[.X] : download the specific build version
+
 example:
- ${0##*/} trusty_dir trusty daily
+ ${0##*/} xenial_dir xenial cloud-daily
 EOF
 }
 TEMP_D=""
 
+NO_IMAGE_AVAILABLE_RC=3
+
 cleanup() {
    [ ! -d "${TEMP_D}" ] || rm -Rf "${TEMP_D}"
 }
-VERBOSITY=0
+VERBOSITY=1
 inargs() {
   local n=$1 x="";
   shift;
   for x in "$@"; do [ "$n" = "$x" ] && return 0; done;
   return 1
 }
-debug() { [ $1 -lt $VERBOSITY ] || return 0; shift; echo "$@" 1>&2; }
+debug() { [ $1 -ge $VERBOSITY ] || return 0; shift; echo "$@" 1>&2; }
 error() { echo "$@" 1>&2; }
 fail() { [ $# -eq 0 ] || error "$@"; exit 1; }
 
@@ -29,8 +44,11 @@ trap cleanup EXIT
 arches=( ppc64el i386 amd64 )
 releases=( $(ubuntu-distro-info --all) )
 mbase="http://maas.ubuntu.com/images/ephemeral-v2"
-def_stream="${mbase}/daily"
-def_rel="trusty"
+cbase="http://cloud-images.ubuntu.com"
+def_stream_type="cloud"
+stream_src=""
+curbase="$mbase"
+def_rel="xenial"
 vername=""
 
 [ "$1" = "-h" -o "$1" = "--help" ] && { Usage; exit 0; }
@@ -50,28 +68,52 @@ pt=( )
 for x in "$@"; do
    inargs "$x" "${arches[@]}" && pt[${#pt[@]}]=arch=$x && arch="$x" && continue
    inargs "$x" "${releases[@]}" && pt[${#pt[@]}]="release=$x" && release=$x && continue
+   [ "$x" = "released" -o "$x" = "release" ] && x="releases"
    case "$x" in
-      daily) stream="${mbase}/daily";;
-      released|release|releases) stream="${mbase}/releases";;
+      cloud-released|cloud-release) x="cloud-releases";;
+      maas-released|maas-release) x="maas-releases";;
+   esac
+
+   case "$x" in
+      maas|cloud) stream_type="$x";;
+      cloud-daily|maas-daily|cloud-releases|maas-releases)
+          stream_type="${x%-*}"
+          stream_sub=${x#*-};;
       http://*) stream=$x;;
-      hwe-*) subarch="$x"; pt[${#pt[@]}]="subarch=$x";;
-      subarch=*) pt[${#pt[@]}]="$x"; subarch=${x#*=};;
+      hwe-*) subarch="$x";;
+      subarch=*) subarch=${x#*=};;
       *=*) pt[${#pt[@]}]="$x";;
       *~*) pt[${#pt[@]}]="$x";;
       [0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]|[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9].[0-9])
          pt[${#pt[@]}]="version_name=$x"
          vername="$x";;
+      *) fail "Confused by input token '$x'";;
    esac
 done
 
-[ -n "$stream" ] || stream="$def_stream"
+[ -n "$stream_type" ] || stream_type="${def_stream_type}"
+if [ -z "$stream" ]; then
+    stream_sub=${stream_sub:-daily}
+    case "$stream_type" in
+        maas) stream="$mbase/$stream_sub";;
+        cloud) stream="$cbase/$stream_sub";;
+        *) fail "unknown stream_type '$stream_type'";;
+    esac
+fi
+
 [ -n "$arch" ] || pt[${#pt[@]}]="arch=$def_arch"
-if [ -z "$subarch" ]; then
-  t=${release#?}
-  first_letter=${release%${t}}
-  subarch="hwe-${first_letter}"
-  pt[${#pt[@]}]="subarch=$subarch"
-  error "selected subarch=${subarch} for $release"
+
+if [ "$stream_type" = "maas" ]; then
+    if [ -z "$subarch" ]; then
+        t=${release#?}
+        first_letter=${release%${t}}
+        subarch="hwe-${first_letter}"
+        error "selected subarch=${subarch} for $release"
+    fi
+    pt[${#pt[@]}]="subarch=$subarch"
+    ofmt="%(sha256)s %(ftype)s %(subarch)s %(item_url)s"
+else
+    ofmt="%(sha256)s %(ftype)s %(item_url)s"
 fi
 
 OIFS="$IFS"
@@ -97,39 +139,89 @@ if [ -n "${needs# }" ]; then
 fi
 
 qcmd=( sstream-query ${keyring:+"--keyring=$keyring"} ${vername:---max=1}
-       --output-format="%(sha256)s %(subarch)s %(ftype)s %(item_url)s" "${stream}" "${pt[@]}" )
+       --output-format="$ofmt" "${stream}" datatype="image-downloads" 
+       "${pt[@]}" )
 echo "${qcmd[@]}"
 "${qcmd[@]}" > "${TEMP_D}/qout"
-roots=$(awk '$3 == "root-image.gz" { print $2 }' "${TEMP_D}/qout" | sort -u | wc -l)
-[ "$roots" = "1" ] || {
+roots=$(awk '$2 == "root-image.gz" || $2 == "disk1.img" { print $2 }' \
+        "${TEMP_D}/qout" | sort -u | wc -l)
+if [ "$roots" = "0" ]; then
+   error "No image available for $release in $stream${pt:+ (${pt[*]})}"
+   exit ${NO_IMAGE_AVAILABLE_RC}
+elif [ "$roots" != "1" ]; then
    error "query resulted in '$roots' root images. expect 1 and only 1"
+   error "cmd was ${qcmd[*]}"
    cat "${TEMP_D}/qout"
    fail
-}
-image="$out_d/root-image"
+fi
+
 set -f
 while read line; do
     set -- $line
-    sha256="$1"; subarch="$2"; ftype="$3"; url="$4"
-    [ "$ftype" = "di-kernel" -o "$ftype" = "di-initrd" ] && continue
+    if [ "$stream_type" = "maas" ]; then
+        sha256="$1"; ftype="$2"; subarch="$3"; url="$4"
+    else
+        sha256="$1"; ftype="$2"; url="$3"
+    fi
+    case "$ftype" in
+        boot-kernel|boot-initrd|root-image.gz|disk1.img) :;;
+        *) continue
+    esac
     if [ "$ftype" = "root-image.gz" ]; then
        fname="$ftype"
-       [ -f "$image" ] && continue
+       image_src="$out_d/$fname"
+       image="$out_d/root-image"
+    elif [ "$ftype" = "disk1.img" ]; then
+       fname="disk.img.dist"
+       image_src="$out_d/$fname"
+       image="$out_d/disk.img"
     else
        fname="$subarch/$ftype"
     fi
     out_f="${out_d}/$fname"
-    [ -f "$out_f" ] && continue
+    if [ -f "$out_f" ]; then
+        debug 1 "reusing existing '$out_f' rather than downloading $url."
+        continue
+    fi
     [ -d "${out_f%/*}" ] || mkdir -p "${out_f%/*}"
-    wget -c "$url" -O "$out_f.tmp" && mv "$out_f.tmp" "$out_f" || { rm -f "$out_f.tmp"; exit 3; }
-    echo "$out_d/$fname" "$url" >> "$out_d/contents"
+    tmp="${out_f}.tmp"
+    wget --progress=dot:mega -O "$tmp" "$url" || {
+        error "Failed to download $url to $tmp."
+        rm -f "$tmp"
+        exit 3
+    }
+    debug 1 "checksumming $tmp from $url"
+    out=$(sha256sum "$tmp") || {
+        error "Failed to checksum $tmp"
+        rm -f "$tmp"
+        exit 3;
+    }
+    found=${out%  *}
+    if [ "$found" != "$sha256" ]; then
+        error "$tmp from $url found sha256 of $found. expected $sha256."
+        ls -l "$tmp" 1>&2
+        rm -f "$tmp"
+        exit 3
+    fi
+    mv "$tmp" "$out_f" || fail "failed renaming $tmp to $out_f"
+    echo "$fname" "$url" >> "$out_d/contents"
 done < "${TEMP_D}/qout"
 
 if [ ! -f "$image" ]; then
-   debug 1 "decompressing $image.gz"
-   zcat "$image.gz" > "$image.tmp" &&
-     mv "$image.tmp" "$image" ||
-     { rm -f "$image.tmp"; exit 1; }
+    case "$image_src" in
+        */root-image.gz)
+           debug 1 "decompressing $image_src"
+           zcat "$image_src" > "$image.tmp" &&
+             mv "$image.tmp" "$image" ||
+             { rm -f "$image.tmp"; exit 1; }
+           ;;
+        */disk.img.dist)
+           debug 1 "converting qcow2 in $image_src -> raw in $image"
+           qemu-img convert -O raw "$image_src" "$image.tmp" &&
+               mv "$image.tmp" "$image" ||
+               { rm -f "$image.tmp"; exit 1; }
+           ;;
+    esac
 fi
 
 # vi: ts=4 expandtab
diff --git a/debian/tests/patch-image b/debian/tests/patch-image
new file mode 100755
index 0000000..b9b34e3
--- /dev/null
+++ b/debian/tests/patch-image
@@ -0,0 +1,288 @@
+#!/bin/bash
+
+VERBOSITY=1
+TEMP_D=""
+
+error() { echo "$@" 1>&2; }
+fail() { local r=$?;  [ $r -eq 0 ] && r=1; failrc "$r" "$@"; }
+failrc() { local r=$1; shift; [ $# -eq 0 ] || error "$@"; exit $r; }
+
+Usage() {
+    cat <<EOF
+Usage: ${0##*/} [options] image [packages]
+  Patch image, upgrading [packages].
+    --kernel FILE             copy kernel out to FILE
+    --initrd FILE             copy initrd out to FILE
+
+    --krd-only                only copy out kernel/initrd do not change image.
+                              this is incompatible with 'packages'
+                              it is a short-cut to specifying all the '--no-*'
+                              options below.
+
+    --no-copy-apt             do not copy host's apt repos in.
+
+  image modifications:
+    --no-update-fstab         do not modify fstab (LP: #1732028)
+
+  if no packages are provided, and ADT_TEST_TRIGGERS is set
+  in environment, then it will be read for the list of packages.
+  to override that behavior pass package 'none'.
+EOF
+
+}
+
+bad_Usage() { Usage 1>&2; [ $# -eq 0 ] || error "$@"; return 1; }
+cleanup() {
+    [ -z "${TEMP_D}" -o ! -d "${TEMP_D}" ] || rm -Rf "${TEMP_D}"
+}
+
+debug() {
+    local level=${1}; shift;
+    [ "${level}" -gt "${VERBOSITY}" ] && return
+    error "${@}"
+}
+
+adt_test_triggers_to_bin_pkgs() {
+    # ADT_TEST_TRIGGERS is space delimited <src>/version
+    # returns a scalar '_RET', caller has to expand
+    local tok src="" ver="" binpkgs=""
+    for tok in "$@"; do
+        src=${tok%/*}
+        ver=${tok#*/}
+        bin_packages_from_source_pkg "$src" "$ver" || return
+        if [ -z "${_RET}" ]; then
+            error "$tok: (source=$src ver=$ver) No packages available!"
+            return 1
+        fi
+        binpkgs="${binpkgs} ${_RET}"
+    done
+    _RET=${binpkgs# }
+}
+
+bin_packages_from_source_pkg() {
+    local pkg=$1 ver=${2} cmd="" out="" p="" ret=""
+    if ! command -v grep-aptavail >/dev/null; then
+        error "No command grep-aptavail: apt-get install dctrl-tools"
+        return 1
+    fi
+    cmd=( grep-aptavail --show-field=Package
+        --exact-match --field=Source:Package "$pkg" )
+    if [ -n "$ver" ]; then
+       cmd=( "${cmd[@]}" --and --field=Version "$ver" )
+    fi
+    out=$("${cmd[@]}" | sed 's,Package: ,,')
+    ret=""
+    for p in $out; do
+        ret="$ret ${p#Package: }${ver:+=${ver}}"
+    done
+    # if no version was provided, we could have multiple results
+    _RET=$(set -f; for i in ${ret}; do echo "$i"; done | sort -u)
+}
+
+update_fstab() {
+    # update_fstab(path) modify the fstab at path
+    # to remove problematic entries (LP: #1732028)
+    local fstab="$1"
+    if [ ! -e "$fstab.patch-image-dist" ]; then
+        cp "$fstab" "$fstab.patch-image-dist" ||
+            { error "failed backing up $fstab to $fstab.dist"; return 1; }
+    fi
+    sed -i '/^LABEL=UEFI/s/^/#/' "$fstab"
+}
+
+main() {
+    local short_opts="hv"
+    local long_opts="help,no-copy-apt,initrd:,kernel:,verbose"
+    local getopt_out="" orig_args=""
+    orig_args=( "$@" )
+    getopt_out=$(getopt --name "${0##*/}" \
+        --options "${short_opts}" --long "${long_opts}" -- "$@") &&
+        eval set -- "${getopt_out}" ||
+        { bad_Usage; return; }
+
+    local cur="" next=""
+    local kernel="" initrd="" copy_apt=true krd_only=false
+    local update_fstab=true
+
+    while [ $# -ne 0 ]; do
+        cur="$1"; next="$2";
+        case "$cur" in
+            -h|--help) Usage ; exit 0;;
+               --krd-only) krd_only=true; shift;;
+               --no-copy-apt) copy_apt=false; shift;;
+               --no-update-fstab) update_fstab=false; shift;;
+               --kernel) kernel=$next; shift;;
+               --initrd) initrd=$next; shift;;
+            -v|--verbose) VERBOSITY=$((${VERBOSITY}+1));;
+            --) shift; break;;
+        esac
+        shift;
+    done
+
+    [ $# -ne 0 ] || { bad_Usage "must provide at least image"; return; }
+
+    [ "$(id -u)" = "0" ] || fail "not root"
+    target="$1"
+    shift
+    if [ -f "$target" ]; then
+        local nargs=""
+        nargs=( "${orig_args[@]}" )
+        # replace the first occurance of target in orig args with _MOUNTPOINT_
+        for((i=0;i<${#nargs[@]};i++)); do
+            [ "${nargs[$i]}" = "$target" ] && nargs[$i]=_MOUNTPOINT_ && break
+        done
+        debug 1 "mic $target -- $0 ${nargs[*]}"
+        IMAGE="$target" exec mount-image-callback \
+            --system-resolvconf "$target" -- "$0" "${nargs[@]}"
+    fi
+
+    if [ ! -d "$target" ]; then
+        fail "$target: not a directory or file"
+    fi
+
+    local tsrc=${IMAGE:-${target}}
+    local packages=( )
+    packages=( "$@" )
+
+    if $krd_only; then
+        if [ "${#packages[@]}" != 0 -a "${packages[*]}" != "none" ]; then
+            error "--krd-only is incompatible with packages."
+            return 1
+        fi
+        debug 1 "--krd-only provided no changes will be done."
+        copy_apt=false
+        update_fstab=false
+        packages=( "none" )
+    fi
+
+    if [ "${#packages[@]}" = "1" -a "${packages[0]}" = "none" ]; then
+        packages=( )
+    elif [ "${#packages[@]}" -eq 0 -a -n "${ADT_TEST_TRIGGERS}" ]; then
+        adt_test_triggers_to_bin_pkgs ${ADT_TEST_TRIGGERS} ||
+            fail "failed to find binary packages " \
+                "from ADT_TEST_TRIGGERS=$ADT_TEST_TRIGGERS"
+        packages=( $_RET )
+        debug 1 "read ADT_TEST_TRIGGERS=$ADT_TEST_TRIGGERS to set" \
+            "packages=${packages[*]}"
+    else
+        local opackages="" debs="" pkg=""
+        opackages=( "${packages[@]}" )
+        packages=( )
+        debs=( )
+        for pkg in "${opackages[@]}"; do
+            [ -f "${pkg}" ] && debs[${#debs[@]}]="$pkg" ||
+                packages[${packages[@]}]="$pkg"
+        done
+    fi
+
+    # if open-iscsi is not in the packages list above, we handle it specifically
+    # so that even if the image did not have open-iscsi, it will get it.
+    local pkg="" mypkg="open-iscsi"
+    for pkg in "${packages[@]}"; do
+        case "$pkg" in
+            open-iscsi|open-iscsi/*) mypkg="none";;
+        esac
+    done
+
+    TEMP_D=$(mktemp -d "${TMPDIR:-/tmp}/${0##*/}.XXXXXX") ||
+        fail "failed to make tempdir"
+    trap cleanup EXIT
+
+    if $copy_apt; then
+        local distdir="${target}/etc/apt.dist.${0##*/}"
+        if [ ! -d "$distdir" ]; then
+            mv "$target/etc/apt" "$distdir" ||
+                fail "failed to backup /etc/apt in $tsrc"
+        fi
+        cp -a /etc/apt "$target/etc/" ||
+            fail "failed to replace /etc/apt in $tsrc"
+
+        # find file:// repos in apt sources and copy them in
+        local file_repos=""
+        file_repos=$(
+            fm='s,.*[[:space:]]file:///*\([^[[:space:]]*\).*,/\1,p'
+            shopt -s nullglob
+            for f in /etc/apt/sources.list /etc/apt/sources.list.d/*.list; do
+                sed -n -e 's/#.*//' -e "$fm" "$f"
+            done
+        )
+        local dir="" bdir="" tdir=""
+        for dir in ${file_repos}; do
+            tdir="${target}$(dirname $dir)/$(basename ${dir})"
+            if [ -d "$dir" ]; then
+                debug 1 "copying $dir -> $tdir"
+                mkdir -p "$tdir" ||
+                    fail "failed to create $tdir"
+                rsync --delete -a "$dir/" "$tdir" ||
+                    fail "failed copy from $dir/ to $tdir"
+            fi
+        done
+    fi
+
+    if $update_fstab; then
+        update_fstab "$target/etc/fstab" ||
+            { error "failed updating /etc/fstab in target"; return 1; }
+    fi
+
+    mount -o bind /sys "$target"/sys || :
+    mount -o bind /proc "$target"/proc || :
+    mount -o bind /dev "$target"/dev || :
+    mount -o bind /dev/pts "$target"/dev/pts || :
+
+    if [ "${#packages[@]}" != "0" ]; then
+        DEBIAN_FRONTEND=noninteractive chroot "$target" sh -exc '
+            mypkg=$1
+            shift
+            apt-get update -q
+            if [ "$mypkg" != "none" ]; then
+                apt-get install -qy "$mypkg"
+            fi
+            apt-get install -qy --only-upgrade "$@"' \
+                chroot-inst "$mypkg" "${packages[@]}" ||
+            fail "failed to install ${packages[*]} in $target"
+    fi
+
+    if [ "${#debs[@]}" != "0" ]; then
+        local tmpd=""
+        tmpd=$(mktemp -d "${target}/tmp/${0##*/}.XXXXXX")
+        cp "${debs[@]}" "$tmpd"
+        DEBIAN_FRONTEND=noninteractive chroot "$target" sh -exc \
+            'cd "$1"; shift; dpkg -i *.deb' debinstalls "${tmpd#${target}}"
+        rm -Rf "${tmpd}"
+    fi
+
+    umount "$target"/dev/pts || :
+    umount "$target"/dev || :
+    umount "$target"/proc || :
+    umount "$target"/sys || :
+
+    local kpath="" ipath=""
+    for f in "$target/boot/"*; do
+        case "${f##*/}" in
+            *.signed) continue;;
+            vmlinu?-*) kpath="$f";;
+            initrd.img-*) ipath="$f";;
+        esac
+    done
+
+    if [ -n "$kernel" ]; then
+        [ -n "$kpath" ] || fail "unable to find kernel in $tsrc"
+        debug 1 "using kernel ${kpath#${target}}"
+        cp "$kpath" "$kernel" ||
+            fail "failed copy $kpath to $kernel"
+        [ -z "$IMAGE" ] || chown "--reference=$IMAGE" "$kernel"
+    fi
+
+    if [ -n "$initrd" ]; then
+        [ -n "$ipath" ] || fail "unable to find initrd in $tsrc"
+        debug 1 "using initrd ${ipath#${target}}"
+        cp "$ipath" "$initrd" ||
+            fail "failed copy $ipath to $initrd"
+        [ -z "$IMAGE" ] || chown "--reference=$IMAGE" "$initrd"
+    fi
+
+}
+
+main "$@"
+
+# vi: ts=4 expandtab
diff --git a/debian/tests/test-open-iscsi.py b/debian/tests/test-open-iscsi.py
index 5358a9d..4da193e 100644
--- a/debian/tests/test-open-iscsi.py
+++ b/debian/tests/test-open-iscsi.py
@@ -44,8 +44,10 @@
 
 
 from netifaces import gateways, AF_INET
-import unittest, subprocess, sys, os
+import unittest, subprocess, sys, os, glob
+import shutil
 import testlib
+import textwrap
 from tempfile import mkdtemp
 import time
 
@@ -57,6 +59,46 @@ username_in = 'ubuntu'
 password_in = 'ubuntupasswd'
 initiatorname = 'iqn.2009-10.com.example.hardy-multi:iscsi-01'
 
+COLLECT_USER_DATA = """\
+#cloud-config
+bukket:
+ - &get_resolved_status |
+   if [ "${1:--}" != "-" ]; then
+      exec >"$1" 2>&1
+   fi
+   if ! command -v systemd-resolve >/dev/null 2>&1; then
+       echo "systemd-resolve: not available."
+       exit
+   fi
+   if ! ( systemd-resolve --help | grep -q -- --status ); then
+       echo "systemd-resolve: no --status."
+       exit
+   fi
+   systemd-resolve --status --no-pager
+ - &add_and_remove_tuntap |
+   #!/bin/sh
+   # LP: #1785108 would break dns when any device was removed.
+   tapdev="mytap0"
+   echo ==== Adding $tapdev ====
+   ip tuntap add mode tap user root $tapdev
+   udevadm settle
+   echo ==== Removing $tapdev ====
+   ip tuntap del mode tap $tapdev
+   udevadm settle
+
+runcmd:
+ - [ sh, -c, *add_and_remove_tuntap ]
+ - [ mkdir, -p, /output ]
+ - [ cp, /etc/resolv.conf, /output]
+ - [ sh, -c, *get_resolved_status, --, /output/systemd-resolve-status.txt]
+ - [ tar, -C, /output, -cf, /dev/disk/by-id/virtio-output-disk, . ]
+
+power_state:
+ mode: poweroff
+ message: cloud-init finished. Shutting down.
+ timeout: 60
+"""
+
 try:
     from private.qrt.OpenIscsi import PrivateOpenIscsiTest
 except ImportError:
@@ -158,89 +200,163 @@ discovery.sendtargets.auth.password_in = %s
         nih_path = '/lib/open-iscsi/net-interface-handler'
         self.assertTrue(os.access(nih_path, os.X_OK))
 
-reason = (
-    "Skipped MAAS ephemeral test in ppc64el because it lacks nested "
-    "virtualization support.")
[e-mail](testlib.manager.dpkg_arch == "ppc64el", reason)
-class MAASEphemeralTest(testlib.TestlibCase, PrivateOpenIscsiTest):
-    '''Test MAAS ephemeral image can boot.
-
-    Downloads MAAS ephemeral image, patches the image to use dep8's built
-    open-iscsi package, sets the image to be served by tgt and then boot that
-    image under (nested) kvm. It runs cloud-init in the booted VM to collect
-    some data and verify the image worked as expected.
+class CloudImageTest(testlib.TestlibCase, PrivateOpenIscsiTest):
+    '''Test the cloud image can boot on iscsi root.
+
+    See README-boot-test.md for more information.
     '''
 
-    def setUp(self):
-        self.here = os.path.dirname(os.path.abspath(__file__))
-        self.release = testlib.ubuntu_release()
-        self.subarch = 'hwe-{}'.format(self.release[0])
+    @classmethod
+    def setUpClass(cls):
+        reason = (
+            "Skipped Cloud Image test on {arch}. whitelisted are: {whitelist}")
+        whitelisted = os.environ.get('WHITELIST_ARCHES', 'amd64').split(",")
+        curarch = testlib.manager.dpkg_arch
+        if testlib.manager.dpkg_arch not in whitelisted:
+            raise unittest.SkipTest(
+                reason.format(arch=curarch, whitelist=whitelisted))
+
+        here = os.path.dirname(os.path.abspath(__file__))
+        os.environ['PATH'] = "%s:%s" % (here, os.environ['PATH'])
+        release = os.environ.get(
+            "ISCSI_TEST_RELEASE", testlib.ubuntu_release())
+        image_d = os.path.join(here, '{}.d'.format(release))
         # Download MAAS ephemeral image.
-        self.get_maas_ephemeral()
-        # Patch the image to use the dep8 built open-iscsi.
-        self.patch_image()
-        self.output_disk_path = self.create_output_disk(self.here)
-
-    def create_output_disk(self, base_path):
-        output_disk_path = os.path.join(base_path, 'output_disk.img')
-        subprocess.call([
-            'qemu-img', 'create', '-f',  'raw', output_disk_path, '10M'])
-        return output_disk_path
-
-    def get_maas_ephemeral(self):
-        get_eph_cmd = os.path.join(self.here, 'get-maas-eph')
-        subprocess.call([
-            get_eph_cmd, os.path.join(self.here, '{}.d'.format(self.release)),
-            self.release, self.subarch])
-
-    def patch_image(self):
-        '''Patch root-image with dep8 built open-iscsi package.'''
-        root_image_path = os.path.join(
-            self.here, '{}.d'.format(self.release), 'root-image')
-        open_iscsi_deb_path = os.path.join(
-        # XXX: matsubara fix this path hackery.
-        #   os.environ['ADTTMP'], 'binaries', 'open-iscsi.deb')
-            self.here, '..', '..', '..', '..', 'binaries', 'open-iscsi.deb')
-        open_iscsi_deb = open(open_iscsi_deb_path)
-        install_cmd = (
-            'cat >/tmp/my.deb && dpkg -i /tmp/my.deb; ret=$?;'
-            'rm -f /tmp/my.deb; exit $ret')
-        subprocess.check_output([
-            'mount-image-callback', '--system-mounts',
-            root_image_path, '--', 'chroot', '_MOUNTPOINT_', '/bin/sh', '-c',
-            install_cmd], stdin=open_iscsi_deb)
+        info = {'release': release,
+                'image_d': image_d,
+                'root_image': os.path.join(image_d, 'disk.img'),
+                'kernel': os.path.join(image_d, 'kernel'),
+                'initrd': os.path.join(image_d, 'initrd')}
+        try:
+            get_image(info['image_d'], release)
+        except CalledProcessError as e:
+            if e.return_code != 3:
+                raise e
+            raise unittest.SkipTest(
+                "Cloud Image not available for release '%s'." % release)
+        if os.environ.get("NO_PATCH_IMAGE", "0") == "0":
+            patch_image(info['root_image'],
+                        kernel=info['kernel'], initrd=info['initrd'])
+        cls.info = info
+        cls.here = here
+
+    def tearDown(self):
+        super(CloudImageTest, self).tearDown()
+        if os.path.exists(self.tmpdir):
+            shutil.rmtree(self.tmpdir)
+
+    def setUp(self):
+        super(CloudImageTest, self).setUp()
+        self.tmpdir = mkdtemp()
+        udp = os.path.join(self.tmpdir, 'user-data')
+        with open(udp, "w") as fp:
+            fp.write(COLLECT_USER_DATA)
+        self.info['user-data'] = udp
+
+    def create_output_disk(self):
+        path = os.path.join(self.tmpdir, 'output-disk.img')
+        subprocess.check_call([
+            'qemu-img', 'create', '-f',  'raw', path, '10M'])
+	return path
 
     def extract_files(self, path):
+        # get contents in a dictionary of tarball at 'path'
         tmpdir = mkdtemp()
-        subprocess.call(['tar', '-C', tmpdir, '-xf', path])
-        return [os.path.join(tmpdir, f) for f in os.listdir(tmpdir)]
-
-    def get_default_route_ip(self):
-        gws = gateways()
-        return gws['default'][AF_INET][0]
+        try:
+            subprocess.check_call(['tar', '-C', tmpdir, '-xf', path])
+            flist = {}
+            prefix = len(tmpdir) + 1
+            for root, dirs, files in os.walk(tmpdir):
+                for fname in files:
+                    fpath = os.path.join(root, fname)
+                    key = fpath[prefix:]
+                    with open(fpath, "r") as fp:
+                        flist[key] = fp.read()
+            return flist
+        finally:
+            shutil.rmtree(tmpdir)
 
     def test_tgt_boot(self):
         tgt_boot_cmd = os.path.join(self.here, 'tgt-boot-test')
-        env = os.environ
-        env['HOST_IP'] = self.get_default_route_ip()
         # Add self.here to PATH so xkvm will be available to tgt-boot-test
-        env['PATH'] = env['PATH'] + ":%s" % self.here
-        env['OUTPUT_DISK'] = self.output_disk_path
-        rel_dir = '{}.d'.format(self.release)
-        subprocess.call([
-            tgt_boot_cmd,
-            os.path.join(self.here, rel_dir, 'root-image'),
-            os.path.join(self.here, rel_dir, self.subarch, 'boot-kernel'),
-            os.path.join(self.here, rel_dir, self.subarch, 'boot-initrd')],
-            env=env)
-        files = self.extract_files(self.output_disk_path)
-        for f in files:
-            if f.endswith('resolv.conf'):
-                resolvconf_contents = open(f).read()
-        self.assertIn(
-            '10.0.2.3', resolvconf_contents,
-            msg="10.0.2.3 not in resolvconf contents: \n{}".format(
-                resolvconf_contents))
+        dns_addr = '10.1.1.4'
+        rel_dir = '{}.d'.format(self.info['release'])
+        dns_search = 'example.com'
+        info = {'host': '10.1.1.2', 'dns': dns_addr,
+                'dnssearch': dns_search, 'network': '10.1.1.0/24'}
+        netdev = ("--netdev=user,net={network},host={host},dns={dns},"
+                  "dnssearch={dnssearch}").format(**info)
+
+        output_disk = self.create_output_disk()
+        cmd = [
+            tgt_boot_cmd, '-v', netdev,
+            '--disk=%s,serial=output-disk' % output_disk,
+            '--user-data-add=%s' % self.info['user-data'],
+            self.info['root_image'], self.info['kernel'],
+            self.info['initrd']]
+        sys.stderr.write(' '.join(cmd) + "\n")
+
+        env = os.environ.copy()
+        env['BOOT_TIMEOUT'] = env.get('BOOT_TIMEOUT', '60m')
+        subprocess.check_call(cmd, env=env)
+
+        files = self.extract_files(output_disk)
+        print("files: %s" % files)
+        resolvconf = files.get('resolv.conf', "NO_RESOLVCONF_FOUND")
+        resolve_status = files.get('systemd-resolve-status.txt')
+
+        resolvconf_id = 'generated by resolvconf'
+        resolved_addr = "127.0.0.53"
+        if resolvconf_id in resolvconf:
+            print("resolvconf manages resolvconf.\n")
+            self.assertIn(
+                dns_addr, resolvconf,
+                msg = ("%s not in resolvconf contents: \n%s" %
+                       (dns_addr, resolvconf)))
+            if dns_search in resolvconf:
+                print("%s was found in resolv.conf." % dns_search)
+            elif resolved_addr in resolvconf and dns_search in resolve_status:
+                # zesty has resolvconf and systemd-resolved.
+                print("%s was in resolve_status and %s in resolv.conf" %
+                      (resolved_addr, dns_search))
+            else:
+                raise AssertionError(
+                    "%s domain is not being searched." % dns_search)
+
+        else:
+            print("systemd-resolved managing resolve.conf\n")
+            self.assertIn(
+                resolved_addr, resolvconf,
+                msg="%s not in resolved resolv.conf: %s" % (resolved_addr,
+                                                            resolvconf))
+            self.assertIn(dns_addr, resolve_status,
+                msg=("%s not in systemd-resolve status: %s" %
+                     (dns_addr, resolve_status)))
+            self.assertIn(dns_search, resolve_status,
+                msg=("search addr '%s' not in systemd-resolve status: %s" %
+                     (dns_search, resolve_status)))
+
+
+def get_image(top_d, release):
+    cmd = ['get-image', top_d, 'cloud-daily', release]
+    subprocess.check_call(cmd)
+
+
+def patch_image(image, packages=None, kernel=None, initrd=None):
+    '''Patch root-image with dep8 built open-iscsi package.'''
+
+    if packages is None:
+        # an empty 'packages' to patch-image still installs open-iscsi
+        packages = []
+
+    cmd = ['patch-image', image]
+    if kernel:
+        cmd.append('--kernel=%s' % kernel)
+    if initrd:
+        cmd.append('--initrd=%s' % initrd)
+    cmd.extend(packages)
+
+    subprocess.check_call(cmd)
 
 
 if __name__ == '__main__':
@@ -283,7 +399,7 @@ if __name__ == '__main__':
     suite = unittest.TestSuite()
     suite.addTest(unittest.TestLoader().loadTestsFromTestCase(OpenIscsiTest))
     suite.addTest(unittest.TestLoader().loadTestsFromTestCase(
-        MAASEphemeralTest))
+        CloudImageTest))
     rc = unittest.TextTestRunner(verbosity=2).run(suite)
     if not rc.wasSuccessful():
         sys.exit(1)
diff --git a/debian/tests/tgt-boot-test b/debian/tests/tgt-boot-test
index 538477c..6ed7a9b 100755
--- a/debian/tests/tgt-boot-test
+++ b/debian/tests/tgt-boot-test
@@ -1,27 +1,27 @@
 #!/bin/bash
 
-## we run a python simple web server on this port.
-WPORT=${WPORT:-32600}
-## HOST_IP has to be an IP address that the vm will be able to access
-## ie, it can't be "127.0.0.1".  If empty (default) then the ip from
-## the device HOST_IP_NIC (default=eth0) will be used.
-HOST_IP=${HOST_IP:-""}
-HOST_IP_NIC="eth0"
-## set OVERLAY_DISK in environment to get this.
-## set to 'temp' to just get a temp file to excercise that path
-## set to some other path and it will be created if non-existant
-## the overlay will then be put on this drive and read from there
-## instead of a tmpfs
-OVERLAY_DISK=${OVERLAY_DISK:-""}
-##
-## XKVM_BRIDGE: set to 'user' to use qemu user networking
-## set to name of a bridge such as virbr0 to use tap on that bridge
-XKVM_BRIDGE=${XKVM_BRIDGE:-"user"}
+VERBOSITY=0
+TEMP_D=""
+DEF_WPORT=32600
+IPV4_NET="10.0.12"
+DEF_NETDEV="user,net=${IPV4_NET}.0/24,host=${IPV4_NET}.2"
 
+ISCSI_PORT=3260
 TGT_CONF=""
 HTTP_PID=""
 TGT_NAME=""
 
+_DEF_CMDLINE_TMPL=(
+    nomodeset {icmdline_parms} {cmdline_ip} ro
+    net.ifnames=0 BOOTIF_DEFAULT=eth0
+    root={root} {overlay_drive}
+    console=ttyS0 {cmdline_ds}
+)
+DEF_CMDLINE_TMPL="${_DEF_CMDLINE_TMPL[*]}"
+
+error() { echo "$@" 1>&2; }
+fail() { [ $# -eq 0 ] || error "$@"; exit 1; }
+
 cleanup() {
     [ ! -d "$TEMP_D" ] || rm -Rf "$TEMP_D"
     if [ -f "$TGT_CONF" ]; then
@@ -33,165 +33,500 @@ cleanup() {
        kill $HTTP_PID
     fi
 }
+
 Usage() {
-   cat <<EOF
-${0##*/} image kernel initrd
+    cat <<EOF
+Usage: ${0##*/} [ options ] image kernel initrd
+
+   put 'image' into tgt, boot 'kernel' and 'initrd' in a kvm
+   guest pointing to that iscsi target.
 
-puts 'image' into tgt, then boots kernel and initrd in kvm to use it as root.
+   options:
+   -p | --webserv-port PORT  serve cloud-init seed on port PORT
+                             default: ${DEF_WPORT}
+   -O | --overlay-disk DISK  attach DISK to guest and use it for the overlayroot
+                             target.  Use 'temp' to use a temp file.
+                             use an existing file to use that.  Use a
+                             non-existing file to create.
+   -n | --netdev DEV         passed through to xkvm.
+                             if none passed, default is:
+                               $DEF_NETDEV
+   -a | --host-addr ADDR     will be used for iscsi host address and
+                             cloud-init seed address.  Default is read
+                             from 'host=' in --netdev arg.
+   -l | --serial-log FILE    write serial log to FILE default is to let output
+                             go to console.
+        --cmdline    TMPL    template for kernel command line.
+                             default: ${DEF_CMDLINE_TMPL}
+        --cmdline-add ARGS   add the args to the kernel command line. default: ""
+        --user-data  FILE    user-data to provide to instance.
+                             default: builtin
+        --user-data-add FILE append FILE to default user-data.
 EOF
 }
-error() { echo "$@" 1>&2; }
-fail() { [ $# -eq 0 ] || error "$@"; exit 1; }
 
+bad_Usage() { Usage 1>&2; [ $# -eq 0 ] || error "$@"; return 1; }
+
+debug() {
+    local level=${1}; shift;
+    [ "${level}" -gt "${VERBOSITY}" ] && return
+    error "${@}"
+}
 
-[ "$1" = "-h" -o "$1" = "--help" ] && { Usage ; exit 0; }
-image_in=$1
-kernel_in=$2
-initrd_in=$3
-[ $# -ge 3 ] || { Usage 1>&2; fail "must give image, kernel, initrd"; }
-
-image=$(readlink -f "$image_in") ||
-   fail "cannot get full path to $image_in"
-kernel=$(readlink -f "$kernel_in") ||
-   fail "cannot get full path to $kernel_in"
-initrd=$(readlink -f "$initrd_in") ||
-   fail "cannot get full path to $initrd_in"
-
-TEMP_D=$(mktemp -d "${TMPDIR:-/tmp}/${0##*/}-XXXXXX") || exit 1
-TEMP_D=$(cd "$TEMP_D" && pwd)
-trap cleanup EXIT
-
-t_image="${TEMP_D}/image"
-t_kernel="${TEMP_D}/kernel"
-t_initrd="${TEMP_D}/initrd"
-
-# this is the IP address of the tgtd
-if [ -n "$HOST_IP" ]; then
-  ipaddr=${HOST_IP}
-else
-  ipaddr=$(ifconfig ${HOST_IP_NIC} |
-           sed -n '/inet addr/s/\([^:]*:\)\([^ ]*\)\( .*\)/\2/p')
-  [ $? -eq 0 -a -n "$ipaddr" ] ||
-    fail "failed to get ipaddr from '$HOST_IP_NIC'. set HOST_IP or HOST_IP_NIC"
-fi
-
-## set up a user-data and meta-data for cloud-init
-## and we'll seed that through the kernel command line
-##
-## The user data we provide will allow us to log in
-## as the 'ubuntu' user with 'passw0rd'.
-keys=$( (ssh-add -L 2>/dev/null ;
-        [ -f ~/.ssh/id_rsa.pub ] && cat ~/.ssh/id_rsa.pub ) | sort -u )
-if [ -n "$keys" ]; then
-   d=$(IFS=$'\n'; for i in ${keys}; do echo "'$i',"; done)
-   keys="[${d%,}]"
-else
-   keys="[]"
-fi
-mkdir "${TEMP_D}/ci-seed"
-cat > "${TEMP_D}/ci-seed/user-data" <<EOF
+write_userdata() {
+    ## set up a user-data and meta-data for cloud-init
+    ## and we'll seed that through the kernel command line
+    ##
+    ## The user data we provide will allow us to log in
+    ## as the 'ubuntu' user with 'passw0rd'.
+    local out="$1" keys="${2:-[]}"
+    cat > "$out" <<"EOF"
 #cloud-config
-password: passw0rd
-chpasswd: { expire: False }
+chpasswd:
+  expire: False
+  list: |
+    ubuntu:passw0rd
+    root:root
 ssh_pwauth: True
-ssh_authorized_keys: $keys
 EOF
+}
 
-### XXX: matsubara fix user data to collect stuff
-## just an example of what you could do here.
-## added when debugging resolvconf LP: #1501033
-if true; then
-   cat >> "${TEMP_D}/ci-seed/user-data" <<EOF
-runcmd:
- - [ mkdir, -p, /mnt/output ]
- - [ cp, /etc/resolv.conf, /mnt/output]
- - [ tar, -C, /mnt/output, -cf, /dev/vda, . ]
-power_state:
- mode: poweroff
- message: cloud-init finished. Shutting down.
- timeout: 60
-EOF
-fi
+get_ssh_pubkeys() {
+    # get users ssh pubkeys in a yaml list format
+    local keys="" d=""
+    keys=$( (ssh-add -L 2>/dev/null ;
+            [ -f ~/.ssh/id_rsa.pub ] && cat ~/.ssh/id_rsa.pub ) | sort -u )
+    if [ -n "$keys" ]; then
+        d=$(IFS=$'\n'; for i in ${keys}; do echo "'$i',"; done)
+        keys="[${d%,}]"
+    else
+        keys="[]"
+    fi
+    _RET="$keys"
+}
 
-iid=$(uuidgen 2>/dev/null || echo i-abcdefg)
-cat > "${TEMP_D}/ci-seed/meta-data" <<EOF
-instance-id: $iid
+write_metadata() {
+    local iid="" out="$1" pubkeys="$2"
+    iid=$(uuidgen 2>/dev/null || echo i-abcdefg)
+    {
+    echo "instance-id: $iid"
+    if [ -n "$pubkeys" ]; then
+        echo "public-keys: $pubkeys"
+    fi
+    } > "$out"
+}
+
+register_image() {
+    local conf="$1" name="$2" image="$3"
+    sudo tee "$conf" >/dev/null <<EOF
+<target ${name}>
+    readonly 1
+    backing-store "$image"
+    allow-in-use yes
+</target>
 EOF
+    [ $? -eq 0 ] || { error "failed to write '$conf'"; return 1; }
 
-## run a python web server to seed this
-( cd "${TEMP_D}/ci-seed" && exec python -m SimpleHTTPServer $WPORT ) &
-HTTP_PID=$!
+    sudo tgt-admin "--update=$name" ||
+        { error "failed tgt-admin update=$name"; return 1; }
+}
 
-if [ "${image%.gz}" != "$image" ]; then
-   error "uncompressing $image_in -> temp-dir"
-   gzcat "$image" > "${TEMP_D}/image" ||
-      fail "failed uncompress of $image_in"
-else
-   ln -s "$image" "$t_image" || fail "symlink to $image failed"
-fi
+address_on_dev() {
+    local dev="$1" atype="${2:-inet}" out="" addr=""
+    out=$(ip address show dev "$dev" 2>/dev/null) ||
+        { error "could not get address on $1"; return 1; }
+    addr=$(echo "$out" |
+        awk '$1 == atype && $4 == "global" {
+                sub(/\/.*/, "", $2); print $2; exit(0); }' "atype=$atype" )
+    [ -n "$addr" ] || return
+    _RET="$addr"
+}
 
-ln -s "$kernel" "$t_kernel" &&
-   ln -s "$initrd" "$t_initrd" ||
-   fail "failed link to kernel or initrd"
+find_host_addr() {
+    # run through --netdev args and fine 'host=' or make one up base
+    # on there being a net= and assume .2 for host.
+    # example: type=user,id=net00,net=10.0.12.0/24,host=10.0.12.2
+    #   ipv6-net=fec0::/64,ipv6-host=fec0::2
+    #   lxcbr0
+    local nd="" atype="" bridge=""
+    local oifs="$IFS"
+    for nd in "$@"; do
+        local host="" netaddr="" cur="" ipv4="x" ipv6="x" ntype=""
+        bridge=${nd%%,*}
+        if [ "$bridge" != "user" -a -e "/sys/class/net/$bridge" ]; then
+            for atype in inet inet6; do
+                address_on_dev "$bridge" "$atype" || continue
+                debug 2 "took $atype address on bridge '$bridge': $_RET";
+                return 0;
+            done
+        fi
+        cur=${nd}
+        while :; do
+            first=${cur%%,*}
+            rest=${cur#*,}
+            case "$first" in
+                user) ntype=user;;
+                ipv4=on|ipv4) ipv4=on;;
+                ipv4=off) ipv4=off;;
+                ipv6=on|ipv6) ipv6=on;;
+                ipv6=off) ipv6=off;;
+                ipv6-net=*) tmp=${first#*=};
+                    tmp=${tmp%/*}
+                    netaddr="${tmp%.*}:2"
+                    ;;
+                net=*) tmp=${first#*=};
+                    tmp=${tmp%/*}
+                    netaddr="${tmp%.*}.2"
+                    ;;
+                host=*|ipv6-host=) host=${first#*=};;
+            esac
+            if [ "$first" = "$rest" ]; then
+                # there was no ','
+                break
+            fi
+            cur=$rest
+        done
+        local found="" msg=""
+        if [ -n "$host" ]; then
+            found="$host"
+            msg="host= in '$nd'"
+        elif [ -n "$netaddr" ]; then
+            found="$netaddr"
+            msg="second address on net= in '$nd'";
+        elif [ "$ntype" = "user" ]; then
+            case "$ipv4/$ipv6" in
+                */on)
+                    msg="known user net default host in ipv6 in '$nd'"
+                    found="fec0::2";;
+                x/x|on/*)
+                    msg="known user net default host in ipv4 in '$nd'"
+                    found="10.0.2.2";;
+            esac
+        fi
+        if [ -n "$found" ]; then
+            debug 2 "found host address '$found' via $msg"
+            _RET="$found"
+            return 0
+        fi
+    done
+    return 1
+}
 
 
-TGT_NAME=${TEMP_D##*/}
-TGT_CONF="/etc/tgt/conf.d/$TGT_NAME.conf"
+get_rfc4173() {
+    # protocol would normally be '6' (tcp but empty is fine)
+    # https://tools.ietf.org/html/rfc4173
+    # https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=804162
+    local server="$1" name="$2" port="$3" lun="$4" user="$5" pass="$6" proto="$7"
+    local rfc="" auth="${user:+${user}${pass:+:${pass}}}"
+    rfc="iscsi:"
+    if [ -n "${user}" ]; then
+        rfc="$rfc:${user}${pass:+:${pass}}"
+    fi
+    case "$server" in
+        \[*\]) :;;
+        *:*) server="[$server]";;
+    esac
+    rfc="${server}:$proto:${port}:${lun}:${name}"
+    _RET="$rfc"
+}
 
-sudo tee "$TGT_CONF" >/dev/null <<EOF
-<target ${TGT_NAME}>
-    readonly 1
-    backing-store "$t_image"
-    allow-in-use yes
-</target>
+get_root_parm() {
+    local server="$1" name="$2" port="${3:-$ISCSI_PORT}"
+    local lun="$4" user="$5" pass="$6" proto="$7" part="$8"
+    local ret=""
+    # for ipv6 addresses, path just contains : for the server
+    _RET="/dev/disk/by-path/ip-$server:$port-iscsi-${name}-lun-${lun}${part:+-part$part}"
+}
+
+get_cmdline_params() {
+    local server="$1" name="$2" port="$3" lun="$4" user="$5" pass="$6"
+    local proto="$7" initiator="$8" ret=""
+    ret="${initiator:+iscsi_initiator=${initiator} }"
+    ret="${ret}iscsi_target_name=$name "
+    ret="${ret}iscsi_target_ip=$server "
+    ret="${ret}${port:+iscsi_target_port=$port }"
+    ret="${ret}${initiator:+iscsi_initiator=$initiator }"
+    _RET=${ret% }
+}
+
+get_iscsi_cmd() {
+    local server="$1" name="$2" port="$3" lun="$4" user="$5" pass="$6"
+    local proto="$7" initiator="$8" ret="" group=${9:-1}
+    _RET=( iscsistart -i "$initiator"
+            -t "$name" -g "$group" -a "$server" -p "$port"
+            ${user:+-u "$user"} ${pass:+-w "${pass}"} )
+}
+
+serv_http() {
+    local dir="$1" port="$2" check="$3"
+    local python="" kid=""
+    command -v python3 >/dev/null 2>&1 && python=python3 || python=python
+    cat > "$TEMP_D/webserv" <<EOF
+import os, socket, sys
+try:
+    from BaseHTTPServer import HTTPServer
+    from SimpleHTTPServer import SimpleHTTPRequestHandler
+except ImportError:
+    from http.server import HTTPServer, SimpleHTTPRequestHandler
+
+class HTTPServerV6(HTTPServer):
+  address_family = socket.AF_INET6
+
+if __name__ == "__main__":
+    dir = sys.argv[1]
+    port = int(sys.argv[2])
+    os.chdir(dir)
+    server = HTTPServerV6(("::", port), SimpleHTTPRequestHandler)
+    server.serve_forever()
 EOF
-[ $? -eq 0 ] || fail "failed to write '$TGT_CONF'"
-
-sudo tgt-admin "--update=${TGT_NAME}" || fail "failed tgt-admin update=$TGT_NAME"
-
-overlay_disk=${OVERLAY_DISK}
-overlay_drive_kernel="overlayroot=tmpfs"
-overlay_drive_xkvm=""
-if [ -n "$overlay_disk" ]; then
-  if [ "${overlay_disk}" = "temp" ]; then
-     overlay_disk="${TEMP_D}/overlay.img"
-  fi
-  if [ ! -f "$overlay_disk" ]; then
-    qemu-img create -f raw "$overlay_disk" 1G
-    mkfs.ext2 -L "delta" -F "$overlay_disk" >/dev/null 2>&1 ||
-       fail "failed mkfs on $overlay_disk"
-    overlay_drive_kernel="overlayroot=device:dev=LABEL=delta"
-    overlay_drive_xkvm="--disk=${overlay_disk},format=raw"
-  fi
-fi
-
-## Now boot a kvm from it
-iport="3260"
-root=LABEL=cloudimg-rootfs
-root=/dev/disk/by-path/ip-$ipaddr:$iport-iscsi-${TGT_NAME}-lun-1
-parms=(
-  nomodeset
-  iscsi_target_name=${TGT_NAME} iscsi_target_ip=$ipaddr
-  iscsi_target_port=3260
-  iscsi_initiator=maas-enlist
-  ip=::::maas-enlist:BOOTIF ro
-  net.ifnames=0
-  BOOTIF_DEFAULT=eth0
-  "root=${root}"
-  ${overlay_drive_kernel}
-  console=ttyS0 console=tty1
-  "ds=nocloud-net;seedfrom=http://$ipaddr:$WPORT/"
-)
+    $python "$TEMP_D/webserv" "$dir" "$port" &
+    HTTP_PID=$!
+    local i=0 site="127.0.0.1"
+    local url="http://$site:$port/$check"
+    while [ -d /proc/$HTTP_PID -a "$i" -lt 10 ] && i=$(($i+1)); do
+        no_proxy=$site wget -q -O /dev/null "$url" && return 0
+        sleep .5
+    done
+    return 1
+}
+
+render_templ() {
+    local tmpl="$1"
+    shift
+    local key val i="" cur="$tmpl"
+    for i in "$@"; do
+        key=${i%%=*}
+        val=${i#*=}
+        cur=${cur//\{${key}\}/${val}}
+    done
+    _RET="$cur"
+}
+
+get_partition() {
+    # return in _RET the 'auto' partition for a image.
+    # return partition number for a partitioned image
+    # return 0 for unpartitioned
+    # return 0 if image is partitioned, 1 if not
+    local img="$1"
+    out=$(LANG=C sfdisk --list -uS "$img" 2>&1) || {
+        error "failed determining if partitioned: $out";
+        return 1;
+    }
+    if echo "$out" | grep -q 'Device.*Start.*End'; then
+        _RET=1
+    else
+        _RET=0
+    fi
+}
+
+main() {
+    local short_opts="a:hn:O:p:l:v"
+    local long_opts="help,cmdline:,cmdline-add:,cmdline-ip:,disk:,host-addr:,overlay-disk:,netdev:,serial-log:,user-data:,user-data-add:,verbose,webserv-port:"
+    local getopt_out=""
+
+    getopt_out=$(getopt --name "${0##*/}" \
+        --options "${short_opts}" --long "${long_opts}" -- "$@") &&
+        eval set -- "${getopt_out}" ||
+        { bad_Usage; return; }
+
+    local cur="" next=""
+    local overlay_disk="" webserv_port="${DEF_WPORT}" netdevs="" host_addr=""
+    local serial_log="" cmdline_tmpl="${DEF_CMDLINE_TMPL}" user_data_file=""
+    local cmdline_add="" user_data_add="" pt=""
+    # The string 'BOOTIF' is replaced at runtime with the interface
+    # name by cloud-initramfs-dyn-netconf.
+    local cmdline_ip="ip=::::{iinitiator}:BOOTIF"
+    netdevs=( )
+    pt=( )
+
+    while [ $# -ne 0 ]; do
+        cur="$1"; next="$2";
+        case "$cur" in
+            -h|--help) Usage ; exit 0;;
+               --cmdline) cmdline_tmpl=$next; shift;;
+               --cmdline-add) cmdline_add=$next; shift;;
+               --cmdline-ip) cmdline_ip=$next; shift;;
+               --disk) pt[${#pt[@]}]="$1=$2"; shift;;
+            -a|--host-addr) host_addr=$next; shift;;
+            -O|--overlay-disk) overlay_disk=$next; shift;;
+            -n|--netdev) netdevs[${#netdevs[@]}]="$next"; shift;;
+            -v|--verbose) VERBOSITY=$((${VERBOSITY}+1));;
+            -l|--serial-log) serial_log="$next"; shift;;
+               --user-data-file) user_data_file="$next"; shift;;
+               --user-data-add) user_data_add="$next"; shift;;
+            -p|--webserv-port) webserv_port=$port; shift;;
+            --) shift; break;;
+        esac
+        shift;
+    done
+
+    [ $# -eq 3 ] || { bad_Usage "expected 3 args, got $# ($*)"; return 1; }
+    local image_in=$1 kernel_in=$2 initrd_in=$3
+    for f in "$image_in" "$kernel_in" "$initrd_in"; do
+        [ -f "$f" ] || fail "$f: not a file"
+    done
+
+    local image="" kernel="" initrd=""
+    image=$(readlink -f "$image_in") ||
+        fail "cannot get full path to $image_in"
+    kernel=$(readlink -f "$kernel_in") ||
+        fail "cannot get full path to $kernel_in"
+    initrd=$(readlink -f "$initrd_in") ||
+        fail "cannot get full path to $initrd_in"
+
+    TEMP_D=$(mktemp -d "${TMPDIR:-/tmp}/${0##*/}.XXXXXX") ||
+        fail "failed to make tempdir"
+    TEMP_D=$(cd "$TEMP_D" && pwd)
+    trap cleanup EXIT
+
+    local t_image t_kernel t_initrd
+    t_image="${TEMP_D}/image"
+    t_kernel="${TEMP_D}/kernel"
+    t_initrd="${TEMP_D}/initrd"
+
+    if [ "${#netdevs[@]}" -eq 0 ]; then
+        netdevs=( "$DEF_NETDEV" )
+    fi
+
+    local keys=""
+    mkdir -p "$TEMP_D/ci-seed"
+    if [ -n "${user_data_file}" ]; then
+        cat "${user_data_file}" > "$TEMP_D/ci-seed/user-data" ||
+            fail "failed to copy '$user_data_file' to ci-seed/user-data"
+    else
+        write_userdata "$TEMP_D/ci-seed/user-data" ||
+            fail "failed to write user-data"
+    fi
+    if [ -n "$user_data_add" ]; then
+        cat "$user_data_add" >> "$TEMP_D/ci-seed/user-data" ||
+            fail "failed to write additional user-data"
+    fi
+
+    get_ssh_pubkeys || fail "failed getting pubkeys"
+    keys="$_RET"
+    write_metadata "$TEMP_D/ci-seed/meta-data" "$keys" ||
+        fail "failed writing metadata"
+
+    if [ -z "$host_addr" ]; then
+        find_host_addr "${netdevs[@]}" ||
+            fail "failed to find host address. try --host-addr."
+        host_addr="$_RET"
+    fi
+    debug 1 "using host address: '${host_addr}'"
+
+    ## run a python web server to seed this
+    serv_http "$TEMP_D/ci-seed" "$webserv_port" "user-data" ||
+        fail "failed to web serv $TEMP_D/ci-seed on $webserv_port"
+    debug 1 "web server in pid $HTTP_PID serving ${TEMP_D}/ci-seed/ on port $webserv_port"
+    local seed_url="http://$host_addr:$webserv_port/"
+    if [ "${host_addr#*:}" != "$host_addr" ]; then
+        seed_url="http://[$host_addr]:$webserv_port/"
+    fi
+    debug 1 " ${seed_url}meta-data"
+
+    if [ "${image%.gz}" != "$image" ]; then
+        error "uncompressing $image_in -> temp-dir"
+        gzcat "$image" > "${TEMP_D}/image" ||
+        fail "failed uncompress of $image_in"
+    else
+        ln -s "$image" "$t_image" || fail "symlink to $image failed"
+    fi
+
+    ln -s "$kernel" "$t_kernel" &&
+        ln -s "$initrd" "$t_initrd" ||
+        fail "failed link to kernel or initrd"
+
+    local partition=""
+    get_partition "$t_image" ||
+        fail "failed to read partition from $image"
+    partition=${_RET}
+    [ "$part" = "0" ] && partition=""
+
+    TGT_NAME=${TEMP_D##*/}
+    TGT_NAME=${TGT_NAME//./-}
+    TGT_CONF="/etc/tgt/conf.d/$TGT_NAME.conf"
+    register_image "$TGT_CONF" "$TGT_NAME" "$t_image" ||
+        fail "failed to register image $t_image in tgt"
+
+    overlay_drive_kernel="overlayroot=tmpfs"
+    overlay_drive_xkvm=""
+    if [ -n "${overlay_disk}" ]; then
+        if [ "${overlay_disk}" = "temp" ]; then
+            overlay_disk="${TEMP_D}/overlay.img"
+        fi
+        if [ ! -f "$overlay_disk" ]; then
+            qemu-img create -f raw "$overlay_disk" 1G
+            mkfs.ext2 -L "delta" -F "$overlay_disk" >/dev/null 2>&1 ||
+                fail "failed mkfs on $overlay_disk"
+            overlay_drive_kernel="overlayroot=device:dev=LABEL=delta"
+            overlay_drive_xkvm="--disk=${overlay_disk},format=raw"
+        fi
+    fi
+
+    local iserver="${host_addr}" iname="$TGT_NAME" iport="$ISCSI_PORT"
+    local ilun="1" iuser="" ipass="" iproto="" iinitiator="maas-enlist"
+    local rfc="" root="" iscsi_cmdline_parms="" iscsi_start_cmd=""
+    local cmdline=""
+    get_rfc4173 "${iserver}" "$iname" "$iport" "$ilun" \
+        "$iuser" "$ipass" "$iproto"
+    rfc="$_RET"
+
+    get_cmdline_params "${iserver}" "$iname" "$iport" "$ilun" \
+        "$iuser" "$ipass" "$iproto" "$iinitiator"
+    iscsi_cmdline_parms="$_RET"
+
+    get_root_parm "$iserver" "$iname" "$iport" "$ilun" \
+        "$iuser" "$ipass" "$iproto" "$partition"
+    root="$_RET"
+
+    get_iscsi_cmd "$iserver" "$iname" "$iport" "$ilun" \
+        "$iuser" "$ipass" "$iproto" "$iinitiator"
+    iscsi_start_cmd=( "${_RET[@]}" )
+
+    render_templ "${cmdline_ip}" iinitiator="$iinitiator" ||
+        { error "failed rendering cmdline ip: ${cmdline_ip}"; return 1; }
+    cmdline_ip="$_RET"
+
+    render_templ "$cmdline_tmpl${cmdline_add:+ ${cmdline_add}}" \
+        iserver="$host_addr" iname="$TGT_NAME" iport="$ISCSI_PORT" \
+        ilun="$ilun" iuser="$iuser" ipass="$ipass" iproto="$iproto" \
+        iinitiator="$iinitiator" "rfc4173=$rfc" "root=$root" \
+        "seed_url=$seed_url" icmdline_parms="$iscsi_cmdline_parms" \
+        "overlay_drive=${overlay_drive_kernel}" \
+        "cmdline_ds=ds=nocloud-net;seedfrom=${seed_url}" \
+        "cmdline_ip=${cmdline_ip}" ||
+        { error "failed rendering cmdline"; return 1; }
+    cmdline="$_RET"
+
+    debug 1 "rfc4173: $rfc"
+    debug 1 "iscsi_cmdline_parms: $iscsi_cmdline_parms"
+    debug 1 "root: $root"
+    debug 1 "iscsi_start_cmd: ${iscsi_start_cmd[*]}"
+    debug 1 "cmdline=$cmdline"
+
+    local netdev_args=""
+    netdev_args=( )
+    for i in "${netdevs[@]}"; do
+        netdev_args[${#netdev_args[@]}]="--netdev=$i"
+    done
+
+    local mem="512" start="$SECONDS" ret=""
+    local cmd=""
+    cmd=(
+        ${BOOT_TIMEOUT:+timeout --kill-after=1m --signal=TERM $BOOT_TIMEOUT}
+        xkvm -v -v "${netdev_args[@]}" ${overlay_drive_xkvm} "${pt[@]}" --
+        -echr 0x5
+        -m $mem ${serial_log:+-serial "file:$serial_log"} -nographic
+        -kernel "${t_kernel}" -initrd "${t_initrd}"
+        -append "${cmdline}"
+    )
+    debug 1 "executing: ${cmd[*]}"
+    "${cmd[@]}"
+    ret=$?
+    debug 1 "xkvm returned $ret in $((SECONDS-start))s"
+    return $ret
+}
 
-## The console is set to ttyS0, and serial output is
-## logged to the file 'serial.log'
-## in the ncurses path, you'll eventually get a login prompt
-## but you can also ssh in.
-[ "${NO_LOG:-0}" = "0" ] && serial_log="serial" || serial_log=""
-mem="1024"
-xkvm  --netdev ${XKVM_BRIDGE} \
-   ${overlay_drive_xkvm} \
-   ${OUTPUT_DISK:+--disk "$OUTPUT_DISK"} --\
-   -m 1024 ${serial_log:+-serial "file:$serial_log"} -nographic \
-   -kernel "${t_kernel}" -initrd "${t_initrd}" \
-   -append "${parms[*]}"
+main "$@"
+# vi: ts=4 expandtab
diff --git a/debian/tests/xkvm b/debian/tests/xkvm
index e260988..9513723 100755
--- a/debian/tests/xkvm
+++ b/debian/tests/xkvm
@@ -11,6 +11,7 @@ TAPDEVS=( )
 # OVS_CLEANUP gets populated with bridge:devname pairs used with ovs
 OVS_CLEANUP=( )
 MAC_PREFIX="52:54:00:12:34"
+KVM="kvm"
 declare -A KVM_DEVOPTS
 
 error() { echo "$@" 1>&2; }
@@ -27,7 +28,10 @@ randmac() {
 
 cleanup() {
     [ -z "${TEMP_D}" -o ! -d "${TEMP_D}" ] || rm -Rf "${TEMP_D}"
-    [ -z "${KVM_PID}" ] || kill "$KVM_PID"
+    if [ -n "${KVM_PID}" ]; then
+        debug 1 "${0##*/} cleanup killing kvm pid '$KVM_PID'"
+        kill "$KVM_PID"
+    fi
     if [ ${#TAPDEVS[@]} -ne 0 ]; then
         local name item
         for item in "${TAPDEVS[@]}"; do
@@ -74,6 +78,10 @@ Usage: ${0##*/} [ options ] -- kvm-args [ ... ]
       -d | --disk  DISK.img   attach DISK.img as a disk (via virtio)
            --dry-run          only report what would be done
 
+           --uefi             boot with efi
+           --uefi-nvram=FILE  boot with efi, using nvram settings in FILE
+                              if FILE not present, copy from defaults.
+
    NETDEV:
     Above, 'NETDEV' is a comma delimited string
     The first field must be
@@ -102,13 +110,13 @@ isdevopt() {
     local model="$1" input="${2%%=*}"
     local out="" opt="" opts=()
     if [ -z "${KVM_DEVOPTS[$model]}" ]; then
-        out=$(kvm -device "$model,?" 2>&1) &&
-            out=$(echo "$out" | sed -e "s,${model}[.],," -e 's,=.*,,') &&
+        out=$($KVM -device "$model,?" 2>&1) &&
+            out=$(echo "$out" | sed -e "s,[^.]*[.],," -e 's,=.*,,') &&
             KVM_DEVOPTS[$model]="$out" ||
             { error "bad device model $model?"; exit 1; }
     fi
     opts=( ${KVM_DEVOPTS[$model]} )
-    for opt in "${KVM_DEVOPTS[@]}"; do
+    for opt in "${opts[@]}"; do
         [ "$input" = "$opt" ] && return 0
     done
     return 1
@@ -223,19 +231,126 @@ ovs_cleanup() {
     return $errors
 }
 
+quote_cmd() {
+    local quote='"' x="" vline=""
+    for x in "$@"; do
+        if [ "${x#* }" != "${x}" ]; then
+            if [ "${x#*$quote}" = "${x}" ]; then
+                x="\"$x\""
+            else
+                x="'$x'"
+            fi
+        fi
+        vline="${vline} $x"
+    done
+    echo "$vline"
+}
+
+get_bios_opts() {
+    # get_bios_opts(bios, uefi, nvram)
+    # bios is a explicit bios to boot.
+    # uefi is boolean indicating uefi
+    # nvram is optional and indicates that ovmf vars should be copied
+    # to that file if it does not exist. if it exists, use it.
+    local bios="$1" uefi="${2:-false}" nvram="$3"
+    local ovmf_dir="/usr/share/OVMF"
+    local bios_opts="" pflash_common="if=pflash,format=raw"
+    unset _RET
+    _RET=( )
+    if [ -n "$bios" ]; then
+        _RET=( -drive "${pflash_common},file=$bios" )
+        return 0
+    elif ! $uefi; then
+        return 0
+    fi
+
+    # ovmf in older releases (14.04) shipped only a single file
+    #   /usr/share/ovmf/OVMF.fd
+    # newer ovmf ships split files
+    #   /usr/share/OVMF/OVMF_CODE.fd
+    #   /usr/share/OVMF/OVMF_VARS.fd
+    # with single file, pass only one file and read-write
+    # with split, pass code as readonly and vars as read-write
+    local joined="/usr/share/ovmf/OVMF.fd"
+    local code="/usr/share/OVMF/OVMF_CODE.fd"
+    local vars="/usr/share/OVMF/OVMF_VARS.fd"
+    local split="" nvram_src=""
+    if [ -e "$code" -o -e "$vars" ]; then
+        split=true
+        nvram_src="$vars"
+    elif [ -e "$joined" ]; then
+        split=false
+        nvram_src="$joined"
+    elif [ -n "$nvram" -a -e "$nvram" ]; then
+        error "WARN: nvram given, but did not find expected ovmf files."
+        error "      assuming this is code and vars (OVMF.fd)"
+        split=false
+    else
+        error "uefi support requires ovmf bios: apt-get install -qy ovmf"
+        return 1
+    fi
+
+    if [ -n "$nvram" ]; then
+        if [ ! -f "$nvram" ]; then
+            cp "$nvram_src" "$nvram" ||
+                { error "failed copy $nvram_src to $nvram"; return 1; }
+            debug 1 "copied $nvram_src to $nvram"
+        fi
+    else
+        debug 1 "uefi without --uefi-nvram storage." \
+            "nvram settings likely will not persist."
+        nvram="${nvram_src}"
+    fi
+
+    if [ ! -w "$nvram" ]; then
+        debug 1 "nvram file ${nvram} is readonly"
+        nvram_ro="readonly"
+    fi
+
+    if $split; then
+        # to ensure bootability firmware must be first, then variables
+        _RET=( -drive "${pflash_common},file=$code,readonly" )
+    fi
+    _RET=( "${_RET[@]}"
+           -drive "${pflash_common},file=$nvram${nvram_ro:+,${nvram_ro}}" )
+}
+
+should_try_kvm() {
+    local name="should_try_kvm" msg="set _USE_KVM=1 to force."
+    case "${_USE_KVM-notset}" in
+        0) debug 1 "$name = no. _USE_KVM=${_USE_KVM}"; return 1;;
+        1) debug 1 "$name = yes. _USE_KVM=${_USE_KVM}"; return 0;;
+        notset) :;;
+        *) debug 1 "$name = no. unknown _USE_KVM=${_USE_KVM}"; return 1;;
+    esac
+    local virt="no-detect-virt"
+    if command -v systemd-detect-virt >/dev/null; then
+        virt="$(systemd-detect-virt --vm)"
+    fi
+    case "$virt" in
+        none) :;;
+        kvm) debug 1 "$name = no. virt=$virt (nested kvm is finicky). $msg";
+            return 1;;
+        *) debug 1 "$name = no. virt=$virt. $msg"; return 1;;
+    esac
+    debug 1 "$name = yes."
+    return 0
+}
+
 main() {
     local short_opts="hd:n:v"
-    local long_opts="help,dowait,disk:,dry-run,kvm:,no-dowait,netdev:,verbose"
+    local long_opts="bios:,help,dowait,disk:,dry-run,kvm:,no-dowait,netdev:,uefi,uefi-nvram:,verbose"
     local getopt_out=""
     getopt_out=$(getopt --name "${0##*/}" \
         --options "${short_opts}" --long "${long_opts}" -- "$@") &&
         eval set -- "${getopt_out}" || { bad_Usage; return 1; }
 
-    local bridge="$DEF_BRIDGE"
+    local bridge="$DEF_BRIDGE" oifs="$IFS"
     local netdevs="" need_tap="" ret="" p="" i="" pt="" cur="" conn=""
     local kvm="" kvmcmd="" archopts=""
-    local def_diskif=${DEF_DISKIF:-"virtio"}
+    local def_disk_driver=${DEF_DISK_DRIVER:-"virtio-blk"}
     local def_netmodel=${DEF_NETMODEL:-"virtio-net-pci"}
+    local bios="" uefi=false uefi_nvram=""
 
     archopts=( )
     kvmcmd=( )
@@ -259,13 +374,16 @@ main() {
             -h|--help) Usage; exit 0;;
             -d|--disk)
                 diskdevs[${#diskdevs[@]}]="$next"; shift;;
-               --dry-run) DRY_RUN=true;;
-               --kvm) kvm="$next"; shift;;
+            --dry-run) DRY_RUN=true;;
+            --kvm) kvm="$next"; shift;;
             -n|--netdev)
                 netdevs[${#netdevs[@]}]=$next; shift;;
             -v|--verbose) VERBOSITY=$((${VERBOSITY}+1));;
-               --dowait) dowait=true;;
-               --no-dowait) dowait=false;;
+            --dowait) dowait=true;;
+            --no-dowait) dowait=false;;
+            --bios) bios="$next"; shift;;
+            --uefi) uefi=true;;
+            --uefi-nvram) uefi=true; uefi_nvram="$next"; shift;;
             --) shift; break;;
         esac
         shift;
@@ -274,7 +392,7 @@ main() {
     [ ${#netdevs[@]} -eq 0 ] && netdevs=( "${DEF_BRIDGE}" )
     pt=( "$@" )
 
-    local kvm_pkg=""
+    local kvm_pkg="" virtio_scsi_bus="virtio-scsi-pci"
     [ -n "$kvm" ] && kvm_pkg="none"
     case $(uname -m) in
         i?86)
@@ -285,28 +403,119 @@ main() {
             [ -n "$kvm" ] ||
                 { kvm="qemu-system-x86_64"; kvm_pkg="qemu-system-x86"; }
             ;;
+        s390x)
+            [ -n "$kvm" ] ||
+                { kvm="qemu-system-s390x"; kvm_pkg="qemu-system-misc"; }
+            def_netmodel=${DEF_NETMODEL:-"virtio-net-ccw"}
+            virtio_scsi_bus="virtio-scsi-ccw"
+            ;;
         ppc64*)
             [ -n "$kvm" ] ||
                 { kvm="qemu-system-ppc64"; kvm_pkg="qemu-system-ppc"; }
-            def_netmodel="spapr-vlan"
+            def_netmodel="virtio-net-pci"
             # virtio seems functional on in 14.10, but might want scsi here
             #def_diskif="scsi"
             archopts=( "${archopts[@]}" -machine pseries,usb=off )
             archopts=( "${archopts[@]}" -device spapr-vscsi )
             ;;
-        *) kvm=qemu-system-$(uname -r);;
+        *) kvm=qemu-system-$(uname -m);;
     esac
-    kvmcmd=( $kvm -enable-kvm )
+    KVM="$kvm"
+    kvmcmd=( $kvm )
+    if should_try_kvm; then
+        if [ ! -e /dev/kvm ]; then
+            # if /dev/kvm didn't exist, then try to make it so.
+            local mod=""
+            for mod in kvm-intel kvm-amd kvm-hv kvm-pv; do
+                modprobe --quiet $mod && [ -w /dev/kvm ] &&
+                    debug 1 "loaded module $mod and got kvm" && break
+            done
+        fi
+        if [ -w /dev/kvm ]; then
+            kvmcmd[${#kvmcmd[@]}]="-enable-kvm"
+        else
+            error "WARNING: no kvm support it seems."
+        fi
+    fi
 
-    local out="" fmt=""
-    for i in "${diskdevs[@]}"; do
-        if [ -f "$i" ]; then
-            out=$(LANG=C qemu-img info "$i") &&
+    local bios_opts=""
+    if [ -n "$bios" ] && $uefi; then
+        error "--uefi (or --uefi-nvram) is incompatible with --bios"
+        return 1
+    fi
+    get_bios_opts "$bios" "$uefi" "$uefi_nvram" ||
+        { error "failed to get bios opts"; return 1; }
+    bios_opts=( "${_RET[@]}" )
+
+    local out="" fmt="" bus="" unit="" index="" serial="" driver="" devopts=""
+    local busorindex="" driveopts="" cur="" val="" file=""
+    for((i=0;i<${#diskdevs[@]};i++)); do
+        cur=${diskdevs[$i]}
+        IFS=","; set -- $cur; IFS="$oifs"
+        driver=""
+        id=$(printf "disk%02d" "$i")
+        file=""
+        fmt=""
+        bus=""
+        unit=""
+        index=""
+        serial=""
+        for tok in "$@"; do
+            [ "${tok#*=}" = "${tok}" -a -f "${tok}" -a -z "$file" ] && file="$tok"
+            val=${tok#*=}
+            case "$tok" in
+                driver=*) driver=$val;;
+                if=virtio) driver=virtio-blk;;
+                if=scsi) driver=scsi-hd;;
+                if=pflash) driver=;;
+                if=sd|if=mtd|floppy) fail "do not know what to do with $tok on $cur";;
+                id=*) id=$val;;
+                file=*) file=$val;;
+                fmt=*|format=*) fmt=$val;;
+                serial=*) serial=$val;;
+                bus=*) bus=$val;;
+                unit=*) unit=$val;;
+                index=*) index=$val;;
+            esac
+        done
+        [ -z "$file" ] && fail "did not read a file from $cur"
+        if [ -f "$file" -a -z "$fmt" ]; then
+            out=$(LANG=C qemu-img info "$file") &&
                 fmt=$(echo "$out" | awk '$0 ~ /^file format:/ { print $3 }') ||
-                { error "failed to determine format of $i"; return 1; }
+                { error "failed to determine format of $file"; return 1; }
+        else
+            fmt=raw
         fi
-        diskargs[${#diskargs[@]}]="-drive";
-        diskargs[${#diskargs[@]}]="if=${def_diskif},file=$i,format=$fmt";
+        if [ -z "$driver" ]; then
+            driver="$def_disk_driver"
+        fi
+        if [ -z "$serial" ]; then
+            serial="${file##*/}"
+        fi
+
+        # make sure we add either bus= or index=
+        if [ -n "$bus" -o "$unit" ] && [ -n "$index" ]; then
+            fail "bus and index cant be specified together: $cur"
+        elif [ -z "$bus" -a -z "$unit" -a -z "$index" ]; then
+            index=$i
+        elif [ -n "$bus" -a -z "$unit" ]; then
+            unit=$i
+        fi
+
+        busorindex="${bus:+bus=$bus,unit=$unit}${index:+index=${index}}"
+        diskopts="file=${file},id=$id,if=none,format=$fmt,$busorindex"
+        devopts="$driver,drive=$id${serial:+,serial=${serial}}"
+        for tok in "$@"; do
+            case "$tok" in
+                id=*|if=*|driver=*|$file|file=*) continue;;
+                fmt=*|format=*) continue;;
+                serial=*|bus=*|unit=*|index=*) continue;;
+            esac
+            isdevopt "$driver" "$tok" && devopts="${devopts},$tok" ||
+                diskopts="${diskopts},${tok}"
+        done
+
+        diskargs=( "${diskargs[@]}" -drive "$diskopts" -device "$devopts" )
     done
 
     local mnics_vflag=""
@@ -316,7 +525,7 @@ main() {
     # now go through and split out options
     # -device virtio-net-pci,netdev=virtnet0,mac=52:54:31:15:63:02
     # -netdev type=tap,id=virtnet0,vhost=on,script=/etc/kvm/kvm-ifup.br0,downscript=no
-    local oifs="$IFS" netopts="" devopts="" id="" need_taps=0 model=""
+    local netopts="" devopts="" id="" need_taps=0 model=""
     local device_args netdev_args
     device_args=( )
     netdev_args=( )
@@ -403,6 +612,15 @@ main() {
 
     if [ $need_taps -ne 0 ]; then
         local missing="" missing_pkgs="" reqs="" req="" pkgs="" pkg=""
+        for i in "${connections[@]}"; do
+            [ "$i" = "user" -o -e "/sys/class/net/$i" ] ||
+                missing="${missing} $i"
+        done
+        [ -z "$missing" ] || {
+            error "cannot create connection on ${missing# }."
+            error "bridges do not exist.";
+            return 1;
+        }
         error "creating tap devices: ${connections[*]}"
         if $DRY_RUN; then
             error "sudo $0 tap-control make-nics" \
@@ -444,15 +662,21 @@ main() {
                   -netdev "${netdev_args[$i]}")
     done
 
-    cmd=( "${kvmcmd[@]}" "${archopts[@]}" "${netargs[@]}"
-           "${diskargs[@]}" "${pt[@]}" )
-    error "${cmd[@]}"
+    local bus_devices
+    bus_devices=( -device "$virtio_scsi_bus,id=virtio-scsi-xkvm" )
+    cmd=( "${kvmcmd[@]}" "${archopts[@]}"
+          "${bios_opts[@]}"
+          "${bus_devices[@]}"
+          "${netargs[@]}"
+          "${diskargs[@]}" "${pt[@]}" )
+    local pcmd=$(quote_cmd "${cmd[@]}")
+    error "$pcmd"
     ${DRY_RUN} && return 0
 
     if $dowait; then
         "${cmd[@]}" &
         KVM_PID=$!
-        debug 1 "kvm pid=$KVM_PID. my pid=$$"
+        debug 1 "kvm pid=$KVM_PID. my pid=$$. ppid=$PPID."
         wait
         ret=$?
         KVM_PID=""