Merge lp:~chad.smith/vmbuilder/jenkins_kvm_azure_netplan_hotplug into lp:vmbuilder

[Chad Smith]

Update Azure's nic hotplug script to use netplan if available instead of ENI

Also avoid appending unnecessary include directives in
/etc/network/interfaces on netplan-enabled systems.

WIP: I think I targeted the wrong branch, will resubmit tomorrow
diff should be http://paste.ubuntu.com/p/Kx8YrXv7cp/

Azure images deliver a script /usr/local/sbin/ephemeral_eth.sh which is called from udev add rules on nic hotplug events for nics named eth[1-9]*. This script was created when netplan wasn't a 'thing' and, as such, only cared about /etc/network/interfaces.

In Bionic and later, cloud-init writes a fallback interface config in /etc/netplan/50-cloud-init.yaml configuration dhcp on eth0 as a primary/mandatory NIC (optional: false). So boot will wait on that device to come up.

This changeset adds a test whether netplan command exists in ephemeral_eth.sh:
  - If netplan is present, a separate /etc/netplan/90-hotplug-<nicname>.yaml will be generated for each network device eth[1-9]n* that is attached after cloud-init's initial network configuration.
  - After the netplan yaml is created, call 'netplan apply' to bring up that device with dchp. The netplan config will specify that these nics are "optional: true" so that subsequent boots will not wait on them to come up in case they are subsequently detached.

Attaching nics in Azure is done through their UI or API. The attaching/detaching operation in Azure requires an instance to be stopped before attach/detach and started after the operation.

  Potental gap:
     There is no attempt to clean up old netplan yaml files, or to designate a new primary/mandatory nic because this original hotplug script didn't deal with udev rules for hotplug removal of nics (via Azure network interface detachment).
     This could present a minor issue if eth1 is attached (and optional by design) and eth0 gets detached. In that case, systemd may still wait for eth0 to come up because of the mandatory eth0 definition in /etc/netplan/50-cloud-init.yaml.

[Daniel Axtens (daxtens)]

A few things:

1) netplan is the default on Artful too. I think your detection code is right, but your commit message is potentially wrong?

2) If I understand cloud-init and netplan correctly, couldn't you achieve the same effect by just adding this as /etc/netplan/99-azure-hotplug.yaml? Then you could drop ephemeral_eth.sh entirely on Artful and Bionic.

network:
....version: 2
....ethernets:
........ephemeral:
............dhcp4: true
............match:
................driver: hv_netvsc
................name: "eth*"
............optional: true

3) Looking at the code itself, you should probably use /run/netplan for ephemeral files, rather than /etc/netplan. That also solves your cleanup problem.

4) And it's worth knowing that netplan apply will look for network devices that are 'down' and them from their drivers and rebind them. With your approach, netplan apply will be run for each extra device, so if there are 4 extra devices, the first one configured won't be replugged, the second will be replugged once, the third will be replugged twice and so on. This *probably* isn't problematic, but it makes me nervous, especially doing it in rapid succession.

[lp:~chad.smith/vmbuilder/jenkins_kvm_azure_netplan_hotplug]

800. By Chad Smith on 2018-05-31

Revert changes to ephemeral_eth.sh and emit a netplan 90-hotplug-azure.yaml

cloud-init only sets up a network configuration at initial boot pinned to
the original macaddress. If we are building a netplan enabled image,
emit a static netplan yaml which will complement the orignal /etc/netplan/50-cloud-init.yaml fallback definition. If the original eth0 is no longer attached to vm, cloud-init's netplan yaml will not match by macaddress and system will fall through to match the following hotpluggedeth0 definition:

        hotpluggedeth0:
            dhcp4: true
            match:
                driver: hv_netvsc
                name: 'eth0'

801. By Chad Smith on 2018-06-01

Move /etc/network/interfaces include directive back out of config_udev.

Appended include directive in /etc/network/interfaces needs to exist for
both upstart and udev solutions. So, it can't live exclusively within
config_udev_or_netplan function. It needs to be present on all non-netplan
environments (upstart and ENI), but test we are not a netplan enabled
image before manipulating /etc/network/interfaces.

Unmerged revisions

801. By Chad Smith on 2018-06-01

Move /etc/network/interfaces include directive back out of config_udev.

Appended include directive in /etc/network/interfaces needs to exist for
both upstart and udev solutions. So, it can't live exclusively within
config_udev_or_netplan function. It needs to be present on all non-netplan
environments (upstart and ENI), but test we are not a netplan enabled
image before manipulating /etc/network/interfaces.

800. By Chad Smith on 2018-05-31

Revert changes to ephemeral_eth.sh and emit a netplan 90-hotplug-azure.yaml

cloud-init only sets up a network configuration at initial boot pinned to
the original macaddress. If we are building a netplan enabled image,
emit a static netplan yaml which will complement the orignal /etc/netplan/50-cloud-init.yaml fallback definition. If the original eth0 is no longer attached to vm, cloud-init's netplan yaml will not match by macaddress and system will fall through to match the following hotpluggedeth0 definition:

        hotpluggedeth0:
            dhcp4: true
            match:
                driver: hv_netvsc
                name: 'eth0'

799. By Chad Smith on 2018-05-31

Update Azure's nic hotplug script to use netplan if available instead of ENI

Also avoid appending unnecessary include directives in
/etc/network/interfaces on netplan-enabled systems.

798. By Dan Watkins on 2018-04-24

Install Azure model assertion in Azure bionic images

797. By Dan Watkins on 2018-03-22

Install linux-azure in bionic Azure images [a=Odd_Bloke][r=fginther,tribaal]

MP: https://code.launchpad.net/~ubuntu-on-ec2/vmbuilder/jenkins_kvm-oddbloke/+merge/341846

796. By Philip Roche on 2018-03-15

Merge lp:~ubuntu-on-ec2/vmbuilder/jenkins_kvm-oddbloke into lp:~ubuntu-on-ec2/vmbuilder/jenkins_kvm [a=daniel-thewatkins] [r=fginther,philroche]

Use HTTPS for Vagrant box redirects (LP: #1754948)

MP: https://code.launchpad.net/~ubuntu-on-ec2/vmbuilder/jenkins_kvm-oddbloke/+merge/341339

795. By Dan Watkins on 2018-02-22

Drop unscd from bionic Azure images [a=Odd_Bloke][r=fginther,philroche]

MP: https://code.launchpad.net/~daniel-thewatkins/vmbuilder/jenkins_kvm-drop-unscd/+merge/337830

794. By Dan Watkins on 2017-12-07

do not explicitly install cloud-init [a=mwhudson][r=fginther,Odd_Bloke,philroche]

MP: https://code.launchpad.net/~mwhudson/vmbuilder/jenkins_kvm.mwhudson/+merge/334878

793. By Francis Ginther on 2017-11-06

Update source image and package set for artful. Dropping packages that no longer exist.
[a=fginther][r=daniel-thewatkins,philroche,tribaal]

MP: https://code.launchpad.net/~fginther/vmbuilder/new-artful-builder/+merge/332487

792. By Francis Ginther on 2017-10-18

Add a bb-series version of the Azure suite specific template files, img-azure-18.04-addin.tmpl.
[a=fginther][r=daniel-thewatkins,rcj]

MP: https://code.launchpad.net/~fginther/vmbuilder/jenkins_kvm-add-azure-18.04/+merge/332368

=== added file 'azure_config.sh'
--- azure_config.sh	1970-01-01 00:00:00 +0000
+++ azure_config.sh	2018-05-31 04:33:07 +0000
@@ -0,0 +1,95 @@
+#!/bin/bash
+
+# Load up some libraries
+my_dir="$( cd "$( dirname "$0" )" && pwd )"
+source "${my_dir}/functions/locker"
+source "${my_dir}/functions/common"
+source "${my_dir}/functions/retry"
+source "${my_dir}/functions/merge_templates"
+
+usage() {
+    cat <<EOM
+${0##/} - Populated values in build temple.
+
+    Required:
+    --template      Template file
+    --extra         Extra, arbitrary addin
+    --serial        The build serial
+    --out           The output file
+    --tar           Name of tar file
+    --tar-d         Name of directory to tar up
+    --version       The version number of the distro
+    --proposed      Build against proposed
+    --docker        Install Docker/Docker compose
+EOM
+}
+
+short_opts="h"
+long_opts="out:,template:,serial:,tar:,tar-d:,version:,proposed,docker,extra:"
+getopt_out=$(getopt --name "${0##*/}" \
+    --options "${short_opts}" --long "${long_opts}" -- "$@") &&
+    eval set -- "${getopt_out}" || { echo "BAD INVOCATION!"; usage; exit 1; }
+
+serial=${SERIAL:-$(date +%Y%m%d)}
+
+# Standard templates
+template_f="$(readlink -f ${0%/*}/templates/img-azure.tmpl)"
+template_netaddin_f="$(readlink -f ${0%/*}/templates/img-extra-nets.tmpl)"
+template_extra_f="$(readlink -f ${0%/*}/templates/img-azure-extra.tmpl)"
+extra_addins=()
+
+while [ $# -ne 0 ]; do
+  cur=${1}; next=${2};
+  case "$cur" in
+    --template)                 template_f=$2; shift;;
+    --extra)                    extra_addins+=($2); shift;;
+    --serial)                   serial=$2; shift;;
+    --tar)                      tar_f=$2; shift;;
+    --tar-d)                    tar_d=$2; shift;;
+    --out)                      out_f=$2; shift;;
+    --version)                  version=$2; shift;;
+    --proposed)                 proposed="true";;
+    --docker)                   docker="1";;
+    --) shift; break;;
+  esac
+  shift;
+done
+
+fail() { echo "${@}" 2>&1; exit 1;}
+fail_usage() { fail "Must define $@"; }
+
+# Create the template file for image conversion
+sed -e "s,%S,${serial},g" \
+    -e "s,%v,${version},g" \
+    -e "s,%P,${proposed:-false},g" \
+    ${template_f} > ${out_f}.base ||
+        fail "Unable to write template file"
+
+# Support per-suite addins
+net_addin=1
+
+# Disable the extra nets for Azure due due to the systemd changes
+dist_ge ${version} vivid && net_addin=0
+
+# Order the addins
+default_addin="${template_f//.tmpl/}-${version}-addin.tmpl"
+docker_addin="${template_f//.tmpl/}-${version}-docker.tmpl"
+
+addins=(${default_addin})
+[ ${net_addin:-0} -eq 1 ] && addins+=("${template_netaddin_f}")
+[ ${docker:-0} -eq 1 -a -f "${docker_addin}" ] && addins+=("${docker_addin}")
+addins+=("${extra_addins[@]}" "${template_extra_f}")
+
+merge_templates ${out_f}.base ${out_f} ${addins[@]}
+
+debug "=================================================="
+debug "Content of template:"
+cat ${out_f}
+debug "=================================================="
+
+if [ -n "${tar_d}" ]; then
+   tar -C "${tar_d}" -cf "${tar_f}" . &&
+        debug "TAR'd up ${tar_d}" ||
+        fail  "Failed to tar up ${tar_d}"
+fi
+exit 0

=== added file 'base_indicies.sh'
--- base_indicies.sh	1970-01-01 00:00:00 +0000
+++ base_indicies.sh	2018-05-31 04:33:07 +0000
@@ -0,0 +1,28 @@
+#!/bin/bash -xe
+#
+# Simple job for creating indicies
+suite="${1:-$SUITE}"
+serial="${2:-$SERIAL}"
+
+umask 022
+cronrun="/srv/builder/vmbuilder/bin/cronrun"
+
+# Override and set some home variables
+export HOME="/srv/builder/vmbuilder"
+export CDIMAGE_BIN="${CDIMAGE_BIN:-$HOME/cdimage/bin}"
+export CDIMAGE_ROOT="${CDIMAGE_ROOT:-$HOME/cdimage}"
+export PUBLISH_SCRIPTS="${PUBLISH_SCRIPTS:-$HOME/ec2-publishing-scripts}"
+export PATH="${PUBLISH_SCRIPTS}:${CDIMAGE_BIN}:${PATH}"
+
+fail() { echo "${@}" 2>&1; exit 1;}
+
+echo "Checksumming result directories"
+work_d="${WORKD:-/srv/ec2-images}/${suite}/${serial}"
+
+${CDIMAGE_BIN}/checksum-directory "${work_d}" &&
+    checksum-directory "${work_d}/unpacked" ||
+    fail "Failed to checksum result directories"
+
+${PUBLISH_SCRIPTS}/update-build-indexes daily ${work_d} ${suite} &&
+    update-build-indexes daily ${work_d} ${suite} ||
+    fail "Failed to make the indexes for ${work_d}"

=== added file 'build-juju-local.sh'
--- build-juju-local.sh	1970-01-01 00:00:00 +0000
+++ build-juju-local.sh	2018-05-31 04:33:07 +0000
@@ -0,0 +1,108 @@
+#!/bin/bash
+
+# Read in the common files
+myname=$(readlink -f ${0})
+mydir=$(dirname ${myname})
+mypdir=$(dirname ${mydir})
+
+# Scope stuff locally here
+# Create a temporary directory for the fun
+tmp_dir=$(mktemp -d builder.XXXXX --tmpdir=${TMPDIR:-/tmp})
+export TMPDIR=${tmp_dir}
+export WORKSPACE=${mydir}
+export HOME=${mydir}
+export LOCAL_BUILD=1
+
+clean() { [ -d ${tmp_dir} ] && rm -rf ${tmp_dir};
+          [ -d "${mydir}/Virtualbox\ VMS" ] && rm -rf "${mydir}/Virtualbox\ VMS";
+          exit "${@}"; }
+error() { echo "$@"; }
+debug() { error "$(date -R):" "$@"; }
+fail()  { debug "${1:-Something bad happend}"; clean 1; }
+
+# Fly with the safety on!
+trap fail EXIT
+trap fail SIGINT
+
+test_cmd_exists() {
+    which $1 >> /dev/null || fail "Command $1 does not exist! Please install $2"
+}
+
+if [ "$(lsb_release -r -s | sed 's/\.//')" -lt 1404 ]; then
+    fail "This must be run on Ubuntu 14.04 or higher"
+fi
+
+test_cmd_exists qemu-nbd qemu-utils
+test_cmd_exists vboxmanage virtualbox
+test_cmd_exists bzr bzr
+test_cmd_exists sstream-query simplestreams
+
+# This defines what gets built
+build_for=(${BUILD_FOR:-trusty:amd64 precise:amd64})
+[ -n "${JUJU_CORE_PKG}" -o -n "${JUJU_LOCAL_PKG}" ] && \
+    [ ${#build_for[@]} -ge 2 ] && \
+    fail "JUJU_CORE_PKG and JUJU_LOCAL_PKG can be specified only for a single build target."
+
+for build in ${build_for[@]};
+do
+    suite=${build%%:*}
+    arch=${build##*:}
+    builder_img="${mydir}/${suite}-builder-${arch}.img"
+    results_d_arch="${mydir}/${suite}-${arch}"
+    built_img="${suite}-server-cloudimg-${arch}-juju-vagrant-disk1.img"
+
+    [ ! -e "${results_d_arch}" ] &&
+        mkdir -p "${results_d_arch}"
+
+    cmd=(
+       "${mydir}/standalone.sh"
+        "--cloud_cfg    ${mydir}/config/cloud-vps.cfg"
+        "--template     ${mydir}/templates/img-juju.tmpl"
+        "--suite        ${suite}"
+        "--arch         ${arch}"
+        "--use_img      ${builder_img}"
+        "--final_img    ${built_img}"
+        "--resize_final 40"
+        )
+
+    [ ! -e "${builder_img}" ] && cmd+=("--fetch_new")
+    if [ -n "${JUJU_CORE_PKG}" -o -n "${JUJU_LOCAL_PKG}" ]; then
+        cmd+=("--cloud-init-file ${mydir}/templates/handle-xdeb.py:text/part-handler")
+        if [ -n "${JUJU_CORE_PKG}" ]; then
+            cmd+=("--cloud-init-file ${JUJU_CORE_PKG}:application/x-deb")
+            echo "JUJU_CORE_PKG=$(basename $JUJU_CORE_PKG)" > ${tmp_dir}/juju-sources.sh
+        fi
+        if [ -n "${JUJU_LOCAL_PKG}" ]; then
+            cmd+=("--cloud-init-file ${JUJU_LOCAL_PKG}:application/x-deb")
+            echo "JUJU_LOCAL_PKG=$(basename $JUJU_LOCAL_PKG)" >> ${tmp_dir}/juju-sources.sh
+        fi
+        cmd+=("--cloud-init-file ${tmp_dir}/juju-sources.sh:application/x-shellscript")
+    fi
+
+    [ -e "${results_d_arch}/${suite}-server-cloudimg-${arch}-juju-vagrant-disk1.img" ] ||
+        ( cd ${results_d_arch} && ${cmd[@]} )
+
+    # The following Vagrant-ifies the build
+    SUITE=${suite} \
+    	ARCH_TYPE=${arch} \
+        SERIAL="current" \
+        SRV_D="${mydir}/${suite}-${arch}" \
+        OUTPUT_D="${mydir}/${suite}-${arch}" \
+        WORKSPACE="${mydir}/${suite}-${arch}" \
+    	${mydir}/jenkins/CloudImages_Juju.sh
+
+    expected_box="${results_d_arch}/${suite}-server-cloudimg-${arch}-juju-vagrant-disk1.box"
+    [ -f "${expected_box}" ] || fail "unable to find ${expected_box}; build failed!"
+    results_out+=("${build} ${expected_box}")
+done
+
+# Clear the traps
+trap - EXIT
+trap - SIGINT
+trap
+
+debug "Results are in following locations"
+echo -e "${results_out[@]}"
+
+debug "Done with the build!"
+clean 0

=== added file 'builder_config.sh'
--- builder_config.sh	1970-01-01 00:00:00 +0000
+++ builder_config.sh	2018-05-31 04:33:07 +0000
@@ -0,0 +1,78 @@
+#!/bin/bash
+short_opts="h"
+long_opts="distro:,arch:,build-type:,bzr-automated-builds:,bzr-pubscripts:,bzr-livebuild:,bzr-vmbuilder:,out:,template:,serial:,proposed"
+getopt_out=$(getopt --name "${0##*/}" \
+    --options "${short_opts}" --long "${long_opts}" -- "$@") &&
+    eval set -- "${getopt_out}" || { echo "BAD INVOCATION!"; usage; exit 1; }
+
+usage() {
+    cat <<EOM
+${0##/} - Populated values in build temple.
+
+    Required:
+    --distro        Distro code name, i.e. precise
+    --arch          Arch, i.e. amd64, i386, armel, armhf
+    --template      Template file
+    --serial        The build serial
+    --out           The output file
+    --proposed      Build against -proposed
+
+    Optional:
+    --bzr-automated-builds  bzr branch for automated ec2 builds
+    --bzr-pubscripts        bzr branch of EC2 Publishing Scripts
+    --bzr-livebuild         bzr branch of live-builder
+    --bzr-vmbuilder         bzr branch of vmbuilder
+EOM
+}
+
+
+fail() { echo "${@}" 2>&1; exit 1;}
+
+serial=$(date +%Y%m%d)
+bzr_automated_builds="http://bazaar.launchpad.net/~ubuntu-on-ec2/vmbuilder/automated-ec2-builds"
+bzr_pubscripts="http://bazaar.launchpad.net/~ubuntu-on-ec2/ubuntu-on-ec2/ec2-publishing-scripts"
+bzr_livebuild="http://bazaar.launchpad.net/~ubuntu-on-ec2/live-build/cloud-images"
+bzr_vmbuilder="http://bazaar.launchpad.net/~ubuntu-on-ec2/vmbuilder/0.11a"
+template_f="${PWD}/img-build.tmpl"
+proposed=0
+
+while [ $# -ne 0 ]; do
+  cur=${1}; next=${2};
+  case "$cur" in
+    --distro)                   distro=$2;  shift;;
+    --arch)                     arch=$2;    shift;;
+    --build-type)               build_type=$2;  shift;;
+    --bzr-automated-builds)     bzr_automated_builds=$2; shift;;
+    --bzr-pubscripts)           bzr_pubscripts=$2; shift;;
+    --bzr-livebuild)            bzr_livebuild=$2; shift;;
+    --bzr-vmbuilder)            bzr_vmbuilder=$2; shift;;
+    --template)                 template_f=$2; shift;;
+    --serial)                   serial=$2; shift;;
+    --out)                      out_f=$2; shift;;
+    --proposed)                 proposed=1;;
+    --) shift; break;;
+  esac
+  shift;
+done
+
+fail_usage() { fail "Must define $@"; }
+
+[ -z "${distro}" ] && fail_usage "--distro"
+[ -z "${arch}" ] && fail_usage "--arch"
+[ -z "${build_type}" ] && fail_usage "--build-type"
+[ -z "${out_f}" ] && fail_usage "--out"
+
+sed -e "s,%d,${distro},g" \
+    -e "s,%a,${arch},g" \
+    -e "s,%b,${build_type},g" \
+    -e "s,%A,${bzr_automated_builds},g" \
+    -e "s,%P,${bzr_pubscripts},g" \
+    -e "s,%L,${bzr_livebuild},g" \
+    -e "s,%V,${bzr_vmbuilder},g" \
+    -e "s,%S,${serial},g" \
+    -e "s,%p,${proposed:-0},g" \
+    -e "s,%C,$(awk 1 ORS='\\n' < "${HOME}/.lp_creds")," \
+    ${template_f} > ${out_f} ||
+        fail "Unable to write template file"
+
+exit 0

=== added file 'checksum.sh'
--- checksum.sh	1970-01-01 00:00:00 +0000
+++ checksum.sh	2018-05-31 04:33:07 +0000
@@ -0,0 +1,27 @@
+# Override and set some home variables
+export HOME="${USE_HOME:-/srv/builder}"
+export CDIMAGE_BIN="${HOME}/cdimage/bin"
+export CDIMAGE_ROOT="${HOME}/cdimage"
+export PATH="${CDIMAGE_BIN}:${PATH}"
+
+SUITE_DIR="${BASE_D}/${SUITE}"
+[ -n "${STREAM}" -a "${STREAM}" != "daily" ] &&
+    SUITE_DIR="${BASE_D}/${STREAM}/${SUITE}"
+SERIAL_DIR="${SUITE_DIR}/${SERIAL}"
+
+echo "Checksumming the new version..."
+checksum-directory "${SERIAL_DIR}"
+
+if [ ! -d ${SERIAL_DIR}/unpacked ]; then
+    echo "Adding build info to the new version..."
+    mkdir -p ${SERIAL_DIR}/unpacked
+    cat << EOF > ${SERIAL_DIR}/unpacked/build-info.txt
+SERIAL=$SERIAL
+EOF
+fi
+
+checksum-directory ${SERIAL_DIR}/unpacked
+
+if [ "${UPDATE_CURRENT:-false}" = "true" ]; then
+   ./update_release_directory.sh "${SUITE_DIR}"
+fi

=== added directory 'config'
=== added file 'config/cloud-azure.cfg'
--- config/cloud-azure.cfg	1970-01-01 00:00:00 +0000
+++ config/cloud-azure.cfg	2018-05-31 04:33:07 +0000
@@ -0,0 +1,9 @@
+#cloud-config
+package_upgrade: true
+password: ubuntu
+chpasswd: { expire: False }
+ssh_pwauth: True
+packages:
+- pastebinit
+- zerofree
+- ubuntu-dev-tools

=== added file 'config/cloud-maas.cfg'
--- config/cloud-maas.cfg	1970-01-01 00:00:00 +0000
+++ config/cloud-maas.cfg	2018-05-31 04:33:07 +0000
@@ -0,0 +1,11 @@
+#cloud-config
+package_upgrade: true
+password: ubuntu
+packages:
+- bzr
+- kpartx
+- qemu-kvm
+- qemu-kvm-extras
+- qemu-kvm-extras-static
+- zerofree
+

=== added file 'config/cloud-maasv2.cfg'
--- config/cloud-maasv2.cfg	1970-01-01 00:00:00 +0000
+++ config/cloud-maasv2.cfg	2018-05-31 04:33:07 +0000
@@ -0,0 +1,10 @@
+#cloud-config
+#This is generic enough to build for both MAAS and general cloud images
+package_upgrade: true
+password: ubuntu
+packages:
+- bzr
+- qemu-utils
+- zerofree
+- gdisk
+- proot

=== added file 'config/cloud-maasv3.cfg'
--- config/cloud-maasv3.cfg	1970-01-01 00:00:00 +0000
+++ config/cloud-maasv3.cfg	2018-05-31 04:33:07 +0000
@@ -0,0 +1,10 @@
+#cloud-config
+#This is generic enough to build for both MAAS and general cloud images
+package_upgrade: true
+password: ubuntu
+packages:
+- bzr
+- qemu-utils
+- zerofree
+- gdisk
+- proot

=== added file 'config/cloud-precise.cfg'
--- config/cloud-precise.cfg	1970-01-01 00:00:00 +0000
+++ config/cloud-precise.cfg	2018-05-31 04:33:07 +0000
@@ -0,0 +1,24 @@
+#cloud-config
+package_upgrade: true
+password: ubuntu
+chpasswd: { expire: False }
+ssh_pwauth: True
+ssh_import_id:
+- daniel-thewatkins
+- philroche
+- rcj
+packages:
+- bzr
+- debootstrap
+- python-vm-builder
+- pastebinit
+- kpartx
+- qemu-kvm
+- qemu-kvm-extras
+- qemu-kvm-extras-static
+- debhelper
+- virtualbox
+- u-boot-tools
+- zerofree
+- gdisk
+- ubuntu-dev-tools

=== added file 'config/cloud-trusty-pp64el.cfg'
--- config/cloud-trusty-pp64el.cfg	1970-01-01 00:00:00 +0000
+++ config/cloud-trusty-pp64el.cfg	2018-05-31 04:33:07 +0000
@@ -0,0 +1,13 @@
+#cloud-config
+packages:
+- bzr
+- debootstrap
+- kpartx
+- debhelper
+- zerofree
+- gdisk
+- qemu-utils
+- ubuntu-dev-tools
+- gcc
+- make
+- zlib1g-dev

=== added file 'config/cloud-trusty.cfg'
--- config/cloud-trusty.cfg	1970-01-01 00:00:00 +0000
+++ config/cloud-trusty.cfg	2018-05-31 04:33:07 +0000
@@ -0,0 +1,26 @@
+#cloud-config
+#This is generic enough to build for both MAAS and general cloud images
+package_upgrade: true
+password: ubuntu
+chpasswd: { expire: False }
+ssh_pwauth: True
+ssh_import_id:
+- daniel-thewatkins
+- philroche
+- rcj
+apt_sources:
+- source: deb $MIRROR $RELEASE multiverse
+packages:
+- bzr
+- debootstrap
+- kpartx
+- qemu-kvm
+- qemu-user-static
+- debhelper
+- virtualbox
+- zerofree
+- gdisk
+- proot
+- u-boot-tools
+- ubuntu-dev-tools
+- zlib1g-dev

=== added file 'config/cloud-vps.cfg'
--- config/cloud-vps.cfg	1970-01-01 00:00:00 +0000
+++ config/cloud-vps.cfg	2018-05-31 04:33:07 +0000
@@ -0,0 +1,6 @@
+#cloud-config
+packages:
+- pastebinit
+- zerofree
+- btrfs-tools
+- ubuntu-dev-tools

=== added file 'config/cloud.cfg'
--- config/cloud.cfg	1970-01-01 00:00:00 +0000
+++ config/cloud.cfg	2018-05-31 04:33:07 +0000
@@ -0,0 +1,17 @@
+#cloud-config
+# Generic cloud-config for builder instance
+package_upgrade: true
+password: ubuntu
+chpasswd: { expire: False }
+ssh_pwauth: True
+apt_sources:
+- source: deb $MIRROR $RELEASE multiverse
+packages:
+- bzr
+- zerofree
+- gdisk
+- gcc
+- make
+- git
+- ubuntu-dev-tools
+- zlib1g-dev

=== added file 'copy_to_final.sh'
--- copy_to_final.sh	1970-01-01 00:00:00 +0000
+++ copy_to_final.sh	2018-05-31 04:33:07 +0000
@@ -0,0 +1,52 @@
+#!/bin/bash
+#
+# copies the files to their staging location
+
+DISTRO="${DISTRO:-$1}"
+WORKSPACE="${WORKSPACE:-$2}"
+SERIAL="${SERIAL:-$3}"
+BTYPE="${BTYPE:-$4}"
+BTYPE="${BTYPE:-server}"
+
+# Allow for legacy positional arguments
+test_build="${5:-0}"
+sandbox_build="${6:-0}"
+proposed_build="${7:-0}"
+
+# Allow for environment variable to control this
+TEST_BUILD="${TEST_BUILD:-$test_build}"
+SANDBOX_BUILD="${SANDBOX_BUILD:-$sandbox_build}"
+PROPOSED_BUILD="${PROPOSED_BUILD:-$proposed_build}"
+
+ROOT_D="${ROOT_D:-/srv/ec2-images}"
+base_d="${ROOT_D}/${DISTRO}/${SERIAL}"
+[ "${TEST_BUILD}" -eq 1 ] && base_d="${ROOT_D}/test_builds/${DISTRO}/${SERIAL}"
+[ "${SANDBOX_BUILD}" -eq 1 ] && base_d="${ROOT_D}/sandbox/${DISTRO}/${SERIAL}"
+[ "${PROPOSED_BUILD}" -eq 1 ] && base_d="${ROOT_D}/proposed/${DISTRO}/${SERIAL}"
+[ "${BTYPE}" = "desktop" ] && base_d="${ROOT_D}/desktop/${DISTRO}/${SERIAL}"
+
+# Make sure that the HWE directory is created
+if [[ "${BTYPE}" =~ server-hwe ]]; then
+    base_d="${base_d}/${BTYPE//server-/}"
+    [ ! -e "${base_d}" ] && mkdir -p "${base_d}"
+fi
+
+for roottar in $(find . -iname "*root.tar.gz"); do
+    echo "Generating file listing"
+
+    case ${roottar} in
+        *amd64*)    arch_name="amd64";;
+        *i386*)     arch_name="i386";;
+        *armel*)    arch_name="armel";;
+        *armhf*)    arch_name="armhf";;
+        *ppc64*)    arch_name="ppc64el";;
+        *arm64*)    arch_name="arm64";;
+        *)          arch_name="unknown-$(date +%s)";;
+    esac
+
+    tar -tzvf ${roottar} >> "${WORKSPACE}/file-list-${arch_name}.log" ||
+        echo "Non fatal error. Failed to gather file list for ${roottar}"
+done
+
+cp -au ${DISTRO}-*/* ${base_d} || exit 1
+exit 0

=== added file 'create-vhd.sh'
--- create-vhd.sh	1970-01-01 00:00:00 +0000
+++ create-vhd.sh	2018-05-31 04:33:07 +0000
@@ -0,0 +1,97 @@
+#!/bin/bash
+source "./functions/locker"
+
+usage() {
+cat << EOF
+This program is used to convert raw images to VHD files.
+
+    --suite:        the Ubuntu Code name to build against
+    --source_file:  the name of the raw image file to convert
+    --size:         the size of the converted image in G (defaults to 30G)
+EOF
+exit 1
+}
+
+# Defaults
+vhd_size=30
+
+# Command line parsing
+short_opts="h"
+long_opts="suite:,source_file:,size:"
+getopt_out=$(getopt --name "${0##*/}" --options "${short_opts}"\
+             --long "${long_opts}" -- "$@")
+if [ $? -eq 0 ]; then
+    eval set -- "${getopt_out}"
+else
+    usage
+    exit 1
+fi
+
+while [ $# -ne 0 ]; do
+    cur=${1}; next=${2};
+
+    case "${cur}" in
+        --size)         vhd_size="${2}"; shift;;
+        --source_file)  source_file="${2}"; shift;;
+        --suite)        suite="${2}"; shift;;
+        -h|--help)      usage; exit 0;;
+        ?)              usage; exit 1;;
+        --) shift; break;;
+    esac
+    shift;
+done
+
+if [ -z "$source_file" -o -z "$suite" ]; then
+    echo "--source_file and --suite required."
+    exit 1
+fi
+
+raw_name=$(readlink -f "$source_file")
+case ${suite} in
+    precise|trusty|wily|xenial)
+        vhd_name="${PWD}/${suite}-server-cloudimg-amd64-disk1.vhd"
+        ;;
+    *)
+        vhd_name="${PWD}/${suite}-server-cloudimg-amd64.vhd"
+        ;;
+esac
+
+# Copy the raw image to make it ready for VHD production
+cp  --sparse=always "${raw_name}" "${raw_name}.pre-vhd" &&
+    debug "Copied raw image VHD production" ||
+    fail "Failed to copy raw image to ${raw_name}.pre-vhd"
+
+# Resize the copied RAW image
+debug "Truncating image to ${vhd_size}G"
+truncate -s "${vhd_size}G" "${raw_name}.pre-vhd" &&
+    debug "Truncated image at ${vhd_size}G" ||
+    fail  "Failed to truncate disk image"
+
+# Convert to VHD first, step 1 of cheap hack
+# This is a cheap hack...half the time the next command
+# will fail with "VERR_INVALID_PARAMETER", so this is the,
+# er, workaround
+debug "Converting to VHD"
+_vbox_cmd convertfromraw --format VHD \
+    "${raw_name}.pre-vhd" \
+    "${vhd_name}.pre" &&
+    debug "Converted raw disk to VHD" ||
+    fail "Failed to convert raw image to VHD"
+
+# Clone the disk to fixed, VHD for Azure
+debug "Converting to VHD format from raw..."
+debug ".....this might take a while...."
+_vbox_cmd clonehd --format VHD --variant Fixed \
+    "${vhd_name}.pre" \
+    "${vhd_name}" &&
+    debug "Converted raw disk to VHD format using VirtualBox" ||
+    fail  "Failed to convert raw image to VHD disk!"
+
+# Remove the unneeded files
+rm "${vhd_name}.pre" "${raw_name}.pre-vhd"
+
+debug "Image Characteristics:"
+_vbox_cmd showhdinfo "${vhd_name}"
+
+
+debug "Raw image converted to VHD"

=== added file 'ec2_publisher.sh'
--- ec2_publisher.sh	1970-01-01 00:00:00 +0000
+++ ec2_publisher.sh	2018-05-31 04:33:07 +0000
@@ -0,0 +1,98 @@
+#!/bin/bash
+#
+# Simple execution wrapper for publishing images to EC2 from within Jenkins
+#
+suite="${1}"
+serial="${2}"
+btype="${3}"
+work_d="${4}"
+test_build="${5:-0}"
+sandbox_build="${6:-0}"
+allow_existing="${7:-0}"
+pub_type="daily"
+
+umask 022
+ec2_pub_scripts="${EC2_PUB_LOC:-${PWD}/ec2-publishing-scripts}"
+cronrun="/srv/builder/vmbuilder/bin/cronrun"
+
+# Override and set some home variables
+export HOME="/srv/builder/vmbuilder"
+export EC2_DAILY="${EC2_DAILY:-$HOME/ec2-daily}"
+export CDIMAGE_BIN="${CDIMAGE_BIN:-$HOME/cdimage/bin}"
+export CDIMAGE_ROOT="${CDIMAGE_ROOT:-$HOME/cdimage}"
+AUTO_BUILDS="${AUTO_BUILDS:-$EC2_DAILY/automated-ec2-builds}"
+PUBLISH_SCRIPTS="${PUBLISH_SCRIPTS:-$HOME/ec2-publishing-scripts}"
+XC2_PATH="${EC2_DAILY}/xc2"
+S3CMD_PATH="${S3CMD_PATH:-$EC2_DAILY/s3cmd}"
+MISC_PATH="${MISC_PATH:-$EC2_DAILY/misc}"
+VMBUILDER_PATH="${VMBUILDER_PATH:-$EC2_DAILY/vmbuilder}"
+( which euca-version >> /dev/null >&1 ) || EUCA2OOLS_PATH="${EC2_DAILY}/euca2ools"
+BOTO_PATH="${EC2_DAILY}/boto"
+
+export EC2_AMITOOL_HOME="${EC2_DAILY}/ec2-ami-tools"
+export LIVE_BUILD_PATH="${EC2_DAILY}/live-build"
+MYPATH=${VMBUILDER_PATH}:${XC2_PATH}:${S3CMD_PATH}:${PUBLISH_SCRIPTS}:${AUTO_BUILDS}:${VMBUILDER_PATH}:${EC2_AMITOOL_HOME}/bin:$HOME/bin:${CDIMAGE_BIN}
+
+[ -n "${EUCA2OOLS_PATH}" ] && MYPATH="${MYPATH}:${EUCA2OOLS_PATH}/bin"
+
+export PYTHONPATH="${BOTO_PATH}:${EUCA2OOLS_PATH}"
+export PATH=${MYPATH}:/usr/bin:/usr/sbin:/usr/bin:/sbin:/bin
+export JAVA_HOME=/usr
+export START_D=${EC2_DAILY}
+export PUBLISH_BASE=/srv/ec2-images
+export XC2_RETRY_ON="Server.InternalError Read.timeout Server.Unavailable Unable.to.connect"
+
+export PATH="/srv/builder/vmbuilder/cdimage/bin:${ec2_pub_scripts}:${PATH}"
+
+fail() { echo "${@}" 2>&1; exit 1;}
+
+[ -e "${ec2_pub_scripts}" ] ||
+    fail "Please make sure that ec2-publishing-scripts in the current path or define EC2_PUB_LOC"
+
+[ "$#" -eq 4 -o "$#" -eq 5 -o "$#" -eq 6 -o "$#" -eq 7 ] ||
+    fail "Incorrect number of parameters. Must invoke with: <suite> <serial> <build type> <directory>"
+
+[ "${test_build}" -eq 1 ] && {
+    echo "Build has been marked as a test build!";
+    echo "Publishing image to sandbox location";
+    pub_type="testing";
+}
+
+[ "${sandbox_build}" -eq 1 ] && {
+    echo "Build has been marked as a sandbox build!";
+    echo "Publishing image to Sandbox location";
+    pub_type="sandbox";
+}
+
+echo "Checksumming result directories"
+checksum-directory "${work_d}" &&
+    checksum-directory "${work_d}/unpacked" ||
+    fail "Failed to checksum result directories"
+
+# Drop ebs-standard and ebs-io1 from publication for xenial and after
+if [[ "${suite}" > "xenial" || "${suite}" == "xenial" ]] ; then
+    export OVERRIDE_ITEMS_EBS="i386:ebs-ssd amd64:ebs-ssd"
+    export OVERRIDE_ITEMS_HVM="amd64:hvm-ssd"
+fi
+
+echo "Publishing to EC2"
+pub_args=(--verbose)
+[ "${allow_existing}" -eq 1 ] && pub_args+=(--allow-existing)
+${cronrun} publish-build \
+    "${pub_args[@]}" \
+    "${suite}" \
+    "${btype}" \
+    "${pub_type}" \
+    "${work_d}" ||
+    fail "failed publish-build ${suite} ${btype} daily ${work_d}"
+
+# Update current
+base_d="${work_d%/*}"
+serial_d="${work_d##*/}"
+current_d="${base_d}/current"
+[ -e "${current_d}" ] && rm "${current_d}"
+( cd "${base_d}" && ln -s "${serial_d}" current ) ||
+    fail "failed to update current directory"
+
+exit 0
+

=== added directory 'functions'
=== added file 'functions/bzr_check.sh'
--- functions/bzr_check.sh	1970-01-01 00:00:00 +0000
+++ functions/bzr_check.sh	2018-05-31 04:33:07 +0000
@@ -0,0 +1,14 @@
+#!/bin/bash
+
+error() { echo "$@" 1>&2; }
+fail() { error "$@"; exit 1; }
+debug() { error "$(date -R):" "$@"; }
+
+check_branch() {
+    [ -e "${2}" ] && rm -rf "${2}"
+    debug "Checking out ${1} to ${2}"
+    bzr checkout --lightweight "${1}" "${2}" &&
+        debug "Checked out ${1}" ||
+        fail  "Failed to checkout ${1}"
+}
+

=== added file 'functions/bzr_commit.sh'
--- functions/bzr_commit.sh	1970-01-01 00:00:00 +0000
+++ functions/bzr_commit.sh	2018-05-31 04:33:07 +0000
@@ -0,0 +1,23 @@
+#!/bin/bash
+info_dir=${1}
+oargs=${*//${1}/}
+TEMP_D=""
+error() { echo "$@" 1>&2; }
+fail() { [ $# -eq 0 ] || error "$@"; exit 1; }
+
+echo "Commit comment is: ${oargs}"
+if [ ! -d "${info_dir}/.bzr" ]; then
+    ( cd "${info_dir}" && bzr init && bzr add --quiet . &&
+      bzr commit --quiet -m "initial state" ) >/dev/null &&
+        error "initialized bzr directory in ${info_dir}" ||
+        fail "failed to initialize bzr directory in ${info_dir}"
+fi
+
+bzr add "${info_dir}"
+if bzr diff "${info_dir}" >/dev/null; then
+    error "no changes were made to ${info_dir}"
+else
+    bzr commit -m "${oargs[*]}" "${info_dir}" ||
+        fail "failed to bzr commit in ${info_dir}"
+fi
+

=== added file 'functions/common'
--- functions/common	1970-01-01 00:00:00 +0000
+++ functions/common	2018-05-31 04:33:07 +0000
@@ -0,0 +1,37 @@
+# Common functions
+# vi: syntax=sh expandtab ts=4
+
+error() { echo "$@" 1>&2; }
+fail()  { error "$@"; exit 1; }
+debug() { echo "$(date -R): $@" 1>&2; }
+run()   { echo "$(date -R): running cmd: ${@}";
+            env ${@} && debug "Command successful: ${@}" ||
+                     fail  "failed to run cmd: ${@}"; }
+
+dist_ge() { [[ "$1" > "$2" || "$1" == "$2" ]]; }
+dist_le() { [[ "$1" < "$2" || "$1" == "$2" ]]; }
+
+map_version_to_suite() {
+    version=(${1//-LTS/ LTS})
+    awk '-F[, ]' \
+        '$2 ~ /LTS/ && $1 ==V {print $5}; $2 !~ /LTS/ && $1 == V {print $4}' \
+        V="${version[0]}" /usr/share/distro-info/ubuntu.csv
+}
+
+map_suite_to_version() {
+    suite=${1}
+    awk '-F[, ]' \
+        '$2 ~ /LTS/ && $5 == S {print $1"-"$2}; $2 !~ /LTS/ && $4 == S {print $1}' \
+        S="${suite}" /usr/share/distro-info/ubuntu.csv
+}
+
+# Look for common names
+[ -z "${kvm}" -a -n "${kvm_builder}" ] && kvm="${kvm_builder}"
+[ -z "${kvm_builder}" -a -n "${kvm}" ] && kvm_builder="${kvm}"
+
+[ -n "${kvm}" ] && scripts="${kvm}"
+[ -n "${kvm_builder}" ] && scripts="${kvm_builder}"
+
+export kvm="${scripts}"
+export kvm_builder="${scripts}"
+export scripts

=== added file 'functions/locker'
--- functions/locker	1970-01-01 00:00:00 +0000
+++ functions/locker	2018-05-31 04:33:07 +0000
@@ -0,0 +1,49 @@
+# This prevents concurrent commands from running.
+_script=$(readlink -f "${BASH_SOURCE[0]:?}")
+_my_dir=$(dirname "$_script")
+source "${_my_dir}/common"
+source "${_my_dir}/retry"
+
+cmd_lock() {
+        LOCKFILE="/tmp/wrapper-`basename $1`"
+        LOCKFD=99
+
+        _lock()             { flock -$1 $LOCKFD; }
+        _no_more_locking()  { _lock u; _lock xn && rm -f $LOCKFILE; }
+        _prepare_locking()  { eval "exec $LOCKFD>\"$LOCKFILE\""; trap _no_more_locking EXIT; }
+
+        _prepare_locking
+
+        exlock_now()        { _lock xn; }  # obtain an exclusive lock immediately or fail
+        exlock()            { _lock x; }   # obtain an exclusive lock
+        shlock()            { _lock s; }   # obtain a shared lock
+        unlock()            { _lock u; }   # drop a lock
+
+        count=0
+        max_count=60
+
+        while (! exlock_now );
+        do
+            let wait_time=$RANDOM%30
+            error "Waiting ${wait_time} seconds due to concurrent ${1} command"
+            sleep ${wait_time}
+
+            count=$(expr ${count} + 1)
+
+            if [ ${count} -gt ${max_count} ]; then
+                echo "Max wait expired. Failing."
+                exit 1
+            fi
+        done
+
+        error "Executing command, lock is free for: ${@}"
+        "${@}"
+        unlock
+}
+
+_vbox_cmd() {
+    # Virtual box is a real pain. This function uses the locker function above to
+    # wrap up vboxmanage to prevent its stupid issues with concurrency. 
+    cmd_lock vboxmanage ${@} ||
+        fail "Failed to execute locked command: vboxmange ${@}"
+}

=== added file 'functions/merge_templates'
--- functions/merge_templates	1970-01-01 00:00:00 +0000
+++ functions/merge_templates	2018-05-31 04:33:07 +0000
@@ -0,0 +1,53 @@
+#!/bin/bash
+# vi: ts=4 noexpandtab syntax=sh
+#
+# This is just like mk_template.sh, but differs in that it handles
+# an arbitrary number of templates being merged in.
+#
+# ARG1 - base template
+# ARG2 - final templates
+# ARG* - addin templates
+
+# This merges templates together
+merge_templates() {
+    local cur_dir=${PWD}
+    local args=(${@})
+    local main_template=${1}; args=("${args[@]:1}")
+    local new_template=${2};  args=("${args[@]:1}")
+    local addins=("${args[@]}")
+
+    if [ "${#addins[@]}" -ge 1 ]; then
+      ntmp_dir=$(mktemp -d template.XXXXX --tmpdir=${TMPDIR:-/tmp})
+      cd ${ntmp_dir}
+
+      # Split the base template "ADDIN_HERE"
+      awk '/ADDIN_HERE/{n++}{print >"template" n ".txt" }' \
+            ${main_template} ||
+                fail "failed to split template!"
+
+      # Combine the split template with the addin in the middle
+      cat template.txt \
+          ${addins[@]} \
+          template1.txt \
+            > ${new_template}
+
+      # Do some variable replacement
+      sed -e "s,ADDIN_HERE,# END Addins,g" \
+          -e "s,%%PPA%%,${PPA},g" \
+          -e "s,%%PROPOSED%%,${PROPOSED:-0},g" \
+          -i  ${new_template} ||
+            fail "Unable to finalize template!"
+
+    else
+
+        sed -e "s,ADDIN_HERE,# END Addins,g" \
+               "${main_template}" > "${new_template}"
+
+    fi
+
+    # Remove the temp directory if it exists
+    [ -n "${ntmp_dir}" ] && rm -rf "${ntmp_dir}"
+
+    # Get back to where we started
+    cd ${cur_dir}
+}

=== added file 'functions/mk_template.sh'
--- functions/mk_template.sh	1970-01-01 00:00:00 +0000
+++ functions/mk_template.sh	2018-05-31 04:33:07 +0000
@@ -0,0 +1,41 @@
+#!/bin/bash
+
+# This merges templates together
+merge_template() {
+    cur_dir=${PWD}
+    main_template=${1}
+    addin_template=${2}
+    new_template=${3}
+
+    if [ -n "${addin_template}" ]; then
+      ntmp_dir=$(mktemp -d template.XXXXX --tmpdir=${TMPDIR:-/tmp})
+      cd ${ntmp_dir}
+
+      # Split the base template "ADDIN_HERE"
+      awk '/ADDIN_HERE/{n++}{print >"template" n ".txt" }' \
+            ${main_template} ||
+                fail "failed to split template!"
+
+      # Combine the split template with the addin in the middle
+      cat template.txt \
+          ${addin_template} \
+          template1.txt \
+            > ${new_template}
+
+      # Do some variable replacement
+      sed -e "s,ADDIN_HERE,# END Addins,g" \
+          -e "s,%%PPA%%,${PPA},g" \
+          -e "s,%%PROPOSED%%,${PROPOSED:-0},g" \
+          -i  ${new_template} ||
+            fail "Unable to finalize template!"
+
+    else
+        "${main_template}" "${new_template}"
+    fi
+
+    # Remove the temp directory if it exists
+    [ -n "${ntmp_dir}" ] && rm -rf "${ntmp_dir}"
+
+    # Get back to where we started
+    cd ${cur_dir}
+}

=== added file 'functions/retry'
--- functions/retry	1970-01-01 00:00:00 +0000
+++ functions/retry	2018-05-31 04:33:07 +0000
@@ -0,0 +1,16 @@
+# Code for retrying commands
+
+retry() {
+    local trycount=${1} sleep=${2}
+    shift; shift;
+    local i=0 smsg=" sleeping ${sleep}: $*" ret=0
+    for((i=0;i<${trycount};i++)); do
+        "$@" && return 0
+        ret=$?
+        [ $(($i+1)) -eq ${trycount} ] && smsg=""
+        debug 1 "Warning: cmd failed [try $(($i+1))/${trycount}].${smsg}"
+        sleep $sleep
+    done
+    return $ret
+}
+

=== added file 'generate-ubuntu-lists.sh'
--- generate-ubuntu-lists.sh	1970-01-01 00:00:00 +0000
+++ generate-ubuntu-lists.sh	2018-05-31 04:33:07 +0000
@@ -0,0 +1,44 @@
+#!/bin/bash
+# Generate a list of Ubuntu releases
+
+final_d="${FIANL_D:-/srv/jenkins}"
+tmpd=$(mktemp -d)
+
+trap "rm -rf ${tmpd}" EXIT SIGINT
+
+# Get the regular info
+ubuntu-distro-info --supported \
+    > ${tmpd}/ubuntu-supported.txt
+
+ubuntu-distro-info --all \
+    > ${tmpd}/ubuntu-all.txt
+
+ubuntu-distro-info --unsupported \
+    > ${tmpd}/ubuntu-unsupported.txt
+
+ubuntu-distro-info --release --supported \
+    > ${tmpd}/ubuntu-versions.txt
+
+# Populate releases which may be missing
+for suite in vivid:15.04 wily:15.10 xenial:16.04;
+do
+    echo "${suite%%:*}" >> ${tmpd}/ubuntu-supported.txt
+    echo "${suite%%:*}" >> ${tmpd}/ubuntu-all.txt
+    echo "${suite##*:}" >> ${tmpd}/ubuntu-versions.txt
+done
+
+# Sort and make it pretty
+cat ${tmpd}/ubuntu-supported.txt \
+    | sort -r -u > ${final_d}/ubuntu-supported.txt
+
+cat ${tmpd}/ubuntu-all.txt \
+    | egrep -v warty \
+    | sort -r -u > ${final_d}/ubuntu-all.txt
+
+cat ${tmpd}/ubuntu-versions.txt \
+    | sed "s, ,-,g" \
+    | sort -r -u \
+    > ${final_d}/ubuntu-versions.txt
+
+sort -r -u ${tmpd}/ubuntu-unsupported.txt \
+    > ${final_d}/ubuntu-unsupported.txt

=== added file 'get_serial.sh'
--- get_serial.sh	1970-01-01 00:00:00 +0000
+++ get_serial.sh	2018-05-31 04:33:07 +0000
@@ -0,0 +1,157 @@
+#!/bin/bash -xe
+#
+# Determine the build serial and place files into the build serial location
+# Also, handle the unlikely race condition in case multiple builders arrive
+# At the same point.
+# copies the files to their staging location
+# Prevent race conditions for populating the aggregate build directory
+#
+# OUTPUT:
+# - serial.txt file in ${WORKSPACE}
+# - build_properties (OR ${BUILD_PROPERTIES}) file in ${PWD}
+# - build-info.txt in ${base_d}/unpacked (or ${base_nd}/unpacked})
+# NOTE: see code for how base_d and base_nd are computed
+
+# Required options to even do a build
+DISTRO="${DISTRO:-$1}"
+WORKSPACE="${WORKSPACE:-$2}" # where is the workspace
+BUILD_ID="${BUILD_ID:-$3}"   # build id
+
+# Convert hwe builds to regular for the sake of tooling
+btype="${4:-server}"         # server or something else
+if [[ "${btype}" =~ hwe ]]; then
+    hwe_btype="${btype}"
+    bytpe="server"; BTYPE="server"
+fi
+
+# Support the legacy broken positional stuff. This should have been
+# done with environmental variables or flags
+test_build="${5:-0}"         # test build?
+sandbox_build="${6:-0}"      # should it be a sandbox build
+allow_existing="${7:-1}"     # allow existing
+publish_image="${8:-0}"      # publish the image
+proposed_build="${9:-0}"     # build from proposed
+
+# Make this less confusing by allowing someone to use environmental
+# variables.
+# TODO: utlemming: convert this --flags
+BTYPE="${BTYPE:-$btype}"
+TEST_BUILD="${TEST_BUILD:-$test_build}"
+SANDBOX_BUILD="${SANDBOX_BUILD:-$sandbox_build}"
+PUBLISH_IMAGE="${PUBLISH_IMAGE:-$publish_image}"
+PROPOSED_BUILD="${PROPOSED_BUILD:-$proposed_build}"
+
+ROOT_D="${ROOT_D:-/srv/ec2-images}"
+base_d="${ROOT_D}/${DISTRO}"
+[ "${TEST_BUILD}" -eq 1 ]     && base_d="${ROOT_D}/test_builds/${DISTRO}"
+[ "${SANDBOX_BUILD}" -eq 1 ]  && base_d="${ROOT_D}/sandbox/${DISTRO}" && TEST_BUILD=0
+[ "${BTYPE}" = "desktop" ]    && base_d="${ROOT_D}/desktop/${DISTRO}"
+[ "${PROPOSED_BUILD}" -eq 1 ] && base_d="${ROOT_D}/proposed/${DISTRO}" &&
+    TEST_BUILD=0 && SANDBOX_BUILD=0
+
+let wait_time=$RANDOM%50
+sleep $wait_time # Make build collisions a bit harder
+
+make_hwe_meta() {
+    # Create a sub build-info.txt for HWE builds
+    serial="${1##*/}"
+    hwe_unpacked="${base_d}/${serial}/${hwe_btype//$BTYPE-/}/unpacked"
+    if [ -n "${hwe_btype}" ]; then
+        [ -d "${hwe_unpacked}" ] || mkdir -p "${hwe_unpacked}"
+        cat << EOF > "${hwe_unpacked}/build-info.txt"
+serial=${serial}
+orig_prefix=${DISTRO}-${hwe_btype}-cloudimg
+suite=${DISTRO}
+build_name=${hwe_btype}
+EOF
+    fi
+}
+
+make_meta() {
+   # Write the property file for publishing. This used
+   # to write trigger the EC2 publishing job
+   serial=${1##*/}
+   cat << EOM > "${BUILD_PROPERTIES:-$WORKSPACE/build_properties}"
+BUILD_TYPE=${BTYPE}
+SERIAL=${serial}
+SUITE=${DISTRO}
+TEST_BUILD=${TEST_BUILD}
+SANDBOX_BUILD=${SANDBOX_BUILD}
+PUBLISH_IMAGE=${PUBLISH_IMAGE}
+ALLOW_EXISTING=${ALLOW_EXISTING}
+PROPOSED_BUILD=${PROPOSED_BUILD}
+EOM
+
+   # Write the build-info.txt file. This is used in
+   # the publishing process
+    [ -d "${1}/unpacked" ] || mkdir -p "${1}/unpacked"
+    cat << EOF > "${1}/unpacked/build-info.txt"
+serial=${serial}
+orig_prefix=${DISTRO}-${BTYPE}-cloudimg
+suite=${DISTRO}
+build_name=${BTYPE}
+EOF
+    make_hwe_meta ${serial}
+    exit 0
+}
+
+$(stat /tmp/${DISTRO}-${BUILD_ID} > /dev/null 2>&1) && {
+   echo "Another builder is/has reserved this part of the build. Deferring..."
+   while [ -z "${destdir}" ]
+   do
+      sleep 5
+      finaldir=""
+
+      [ -e "${WORKSPACE}/serial.txt" ] && {
+         read serial < "${WORKSPACE}/serial.txt"
+         destdir="${base_d}/${serial}"
+      }
+
+      while read destdir
+      do
+         echo "Candidate serial found: ${destdir##*/}"
+         finaldir="${destdir}"
+      done < /tmp/${DISTRO}-${BUILD_ID}
+
+      if [ -n "${finaldir}" ]; then
+         echo "Aggregation directory reported as ${finaldir}"
+         echo "${finaldir##*/}" > "${WORKSPACE}/serial.txt"
+         make_hwe_meta "${finaldir##*/}"
+         exit 0
+      else
+         echo "destdir is not defined!" && exit 10
+      fi
+
+   done
+}
+
+# if we get here, then know that the build dir hasn't been created yet
+touch /tmp/${DISTRO}-$BUILD_ID
+test_base_d="${base_d}/$(date +%Y%m%d)"
+
+make_and_write() {
+   serial="${1##*/}"
+   echo "Creating aggregation directory ${1}"
+   echo "${serial}" > "${WORKSPACE}/serial.txt"
+   mkdir -p "${1}" &&
+      echo "${1}" >> /tmp/${DISTRO}-$BUILD_ID ||
+      exit 10
+
+   # Copy stuff to where it should go
+   make_meta "${1}"
+}
+
+if [ ! -d "${test_base_d}" ]; then
+   make_and_write "${test_base_d}"
+else
+   for bs in {1..30}
+   do
+     base_nd="${test_base_d}.${bs}"
+     serial="${base_nd##*/}"
+     echo "Checking on directory ${base_nd}"
+     [ ! -d "${base_nd}" ] && make_and_write "${base_nd}"
+     make_hwe_meta "${serial}"
+   done
+fi
+
+exit 0

=== added directory 'jenkins'
=== added file 'jenkins/CloudImages_Azure.sh'
--- jenkins/CloudImages_Azure.sh	1970-01-01 00:00:00 +0000
+++ jenkins/CloudImages_Azure.sh	2018-05-31 04:33:07 +0000
@@ -0,0 +1,162 @@
+#!/bin/bash
+fail() { [ $# -eq 0 ] || echo "$@"; exit 1; }
+
+# set default umask
+umask 022
+
+# Pre-setup: Read the build properties from the previous build
+# and discard what we don't want
+[ -e build.info ] && cp build.info build_properties
+source build_properties
+
+
+# Load up some libraries
+my_dir="$( cd "$( dirname "$0" )" && pwd )"
+base_dir=$(dirname ${my_dir})
+source "${base_dir}/functions/locker"
+source "${base_dir}/functions/common"
+source "${base_dir}/functions/retry"
+
+debug() { echo "${@}"; }
+fail() { echo "${@}" 2>&1; exit 1;}
+
+
+# Shuffle stuff around"
+[ -e build_properties ] && mv build_properties parent_build_properties
+[ -e build.log ] && mv build.log parent_build.log
+[ -e "${SUITE}-build.sh" ] && rm "${SUITE}-build.sh"
+
+echo "-------------------"
+echo " Distro: ${SUITE}"
+echo " Serial: ${SERIAL}"
+echo " Type:   ${BUILD_TYPE}"
+echo "-------------------"
+
+set -x
+# Variables
+case ${SUITE} in
+    precise|trusty|wily|xenial)
+        disk_name="${SUITE}-server-cloudimg-amd64-disk1.img"
+        raw_name="${PWD}/${SUITE}-server-cloudimg-amd64-disk1.raw"
+        vhd_name="${PWD}/${SUITE}-server-cloudimg-amd64-disk1.vhd"
+        ;;
+    *)
+        disk_name="${SUITE}-server-cloudimg-amd64.img"
+        raw_name="${PWD}/${SUITE}-server-cloudimg-amd64.raw"
+        vhd_name="${PWD}/${SUITE}-server-cloudimg-amd64.vhd"
+        ;;
+esac
+disk_root="${DISK_ROOT:-/srv/ec2-images/${SUITE}/${SERIAL}}"
+raw_disk="${PWD}/results.raw"
+launch_config="${PWD}/launch_config.sh"
+register_config="${PWD}/register_config.sh"
+pkg_tar="${PWD}/pkg.tar"
+pkg_tar_d="${PKG_TAR_D:-${kvm_builder}/azure_pkgs}"
+proposed="${PROPOSED:-false}"
+vhd_size=${VHD_SIZE:-30}
+
+# Covert image to a RAW disk to work with. The raw image is used
+# to populate the daily VHD in Azure
+debug "Converting QCow2 to Raw Disk"
+qemu-img \
+    convert -O raw \
+    "${disk_root}/${disk_name}" \
+    "${raw_name}" &&
+        debug "Converted QCow2 to Raw disk for manipulation" ||
+        fail  "Failed to convert QCow2 to Raw disk"
+
+config_opts=(${CONFIG_OPTS})
+config_opts+=(
+    --version $(${kvm_builder}/ubuntu-adj2version ${SUITE})
+    --serial "${SERIAL}"
+    --out "${launch_config}"
+    )
+
+# Turns on building from proposed
+[ "${proposed}" == "true" ] &&
+    config_opts+=(--proposed)
+
+# Setup the configuration
+${kvm_builder}/azure_config.sh \
+    ${config_opts[@]} ||
+        fail "Failed to configure instance runtime"
+
+# Full disk populate for 12.04
+root_size=2
+if [ "${SUITE}" == "precise" ]; then
+    root_size=29
+    truncate -s 29G "${raw_name}.pre-vhd" &&
+        debug "Resized 12.04 image to full size" ||
+        fail  "Failed to resize 12.04 to full size"
+fi
+
+case ${SUITE} in
+    precise|trusty|xenial)
+        builder_img=/srv/builder/images/precise-builder-latest.img
+        ;;
+    *)
+        builder_img=/srv/builder/images/artful-builder-latest.img
+        ;;
+esac
+
+# Launch KVM to do the work
+${kvm_builder}/launch_kvm.sh \
+    --id ${BUILD_ID} \
+    --user-data "${launch_config}" \
+    --cloud-config "${kvm_builder}/config/cloud-azure.cfg" \
+    --extra-disk "${raw_name}" \
+    --raw-disk "${WORKSPACE}/${SUITE}-output.raw" \
+    --raw-size ${root_size} \
+    --img-url ${builder_img} ||
+        fail "KVM instance failed to build image."
+
+rm "${WORKSPACE}/${SUITE}-output.raw"
+
+
+# Copy the raw image to make it ready for VHD production
+cp  --sparse=always "${raw_name}" "${raw_name}.pre-vhd" &&
+    debug "Copied raw image VHD production" ||
+    fail "Failed to copy raw image to ${raw_name}.pre-vhd"
+
+# Resize the copied RAW image
+debug "Truncating image to ${vhd_size}G"
+truncate -s "${vhd_size}G" "${raw_name}.pre-vhd" &&
+    debug "Truncated image at ${vhd_size}G" ||
+    fail  "Failed to truncate disk image"
+
+# Convert to VHD first, step 1 of cheap hack
+# This is a cheap hack...half the time the next command
+# will fail with "VERR_INVALID_PARAMETER", so this is the,
+# er, workaround
+debug "Converting to VHD"
+_vbox_cmd convertfromraw --format VHD \
+    "${raw_name}.pre-vhd" \
+    "${vhd_name}.pre" &&
+    debug "Converted raw disk to VHD" ||
+    fail "Failed to convert raw image to VHD"
+
+# Clone the disk to fixed, VHD for Azure
+debug "Converting to VHD format from raw..."
+debug ".....this might take a while...."
+_vbox_cmd clonehd --format VHD --variant Fixed \
+    "${vhd_name}.pre" \
+    "${vhd_name}" &&
+    debug "Converted raw disk to VHD format using VirtualBox" ||
+    fail  "Failed to convert raw image to VHD disk!"
+
+# Remove the unneeded files
+rm "${vhd_name}.pre" "${raw_name}.pre-vhd"
+
+debug "Image Characteristics:"
+_vbox_cmd showhdinfo "${vhd_name}"
+
+
+debug "Raw image converted to VHD"
+
+# Archive the bzip2 file
+#debug "Archiving the VHD image"
+#pbzip2 -f "${vhd_name}" &&
+#    debug "Created archive of the VHD image" ||
+#    fail "Failed to compress image"
+
+exit 0

=== added file 'jenkins/CloudImages_Base.sh'
--- jenkins/CloudImages_Base.sh	1970-01-01 00:00:00 +0000
+++ jenkins/CloudImages_Base.sh	2018-05-31 04:33:07 +0000
@@ -0,0 +1,96 @@
+#!/bin/bash
+
+# Set default umask
+umask 022
+
+DISTRO=${DISTRO:-$SUITE}
+DISTRO=${DISTRO:?Must define distro}
+build_config="${PWD}/${DISTRO}-build.sh"
+
+# Read in the common functions
+my_dir="$( cd "$( dirname "$0" )" && pwd )"
+base_dir=$(dirname ${my_dir})
+source "${base_dir}/functions/locker"
+source "${base_dir}/functions/common"
+source "${base_dir}/functions/retry"
+source "${my_dir}/build_lib.sh"
+select_build_config
+
+# Only block for serial if serial is unknown
+[ -z "${SERIAL}" ] && {
+    # Get the serial number
+    retry 3 10 \
+        "${base_dir}/get_serial.sh" \
+             "${DISTRO}" "${WORKSPACE}" "${BUILD_ID}" "${BTYPE}" 0 0 1 1 ||
+             fail "Failed to get serial for this build"
+
+    # Get the serial number
+    read SERIAL < serial.txt
+    [ -z ${SERIAL} ] && echo "NO SERIAL" && exit 10
+}
+
+# Create the configurations
+cmd=("${base_dir}/builder_config.sh"
+     --distro "${DISTRO}"
+     --build-type "${BTYPE}"
+     --arch "${ARCH_TYPE}"
+     --template ${base_dir}/templates/img-build.tmpl
+     --serial "${SERIAL}"
+     --out "${build_config}"
+)
+
+# Allow building from proposed
+[ "${PROPOSED_BUILD:-0}" -eq 1 ] && cmd+=("--proposed")
+[ "${USE_BUILDDS:-0}" -eq 1 ] && cmd+=("--bzr-automated-builds lp:~ubuntu-on-ec2/vmbuilder/automated-ec2-builds-buildd")
+[ -n "${BZR_AUTOMATED_EC2}" ] && cmd+=("--bzr-automated-builds ${BZR_AUTOMATED_EC2}")
+[ -n "${BZR_PUBSCRIPTS}" ] && cmd+=("--bzr-pubscripts ${BZR_PUBSCRIPTS}")
+[ -n "${BZR_LIVEBUILD}" ] && cmd+=("--bzr-livebuild ${BZR_LIVEBUILD}")
+[ -n "${BZR_VMBUILDER}" ] && cmd+=("--bzr-vmbuilder ${BZR_VMBUILDER}")
+
+# Do the build
+${cmd[@]} || fail "Failed to configure instance configuration"
+unset cmd
+
+# Exit after configuring for arm if so configured
+if [[ "${ARCH_TYPE}" =~ (arm|aarch64|arm64) ]]; then
+   echo "This is an ARM build. ARM rules will apply"
+   [ "${BUILD_ARM}" -eq 0 ] && exit 0
+fi
+
+# Launch the builder
+# Retry building the image twice, waiting five
+# minutes. This should buffer most failures caused
+# by bad mirrors.
+export MAX_CYCLES=2160
+retry 2 300 \
+    "${base_dir}/launch_kvm.sh" \
+         --id "${BUILD_ID}" \
+         --user-data "${build_config}" \
+         --cloud-config "${base_dir}/config/${cloud_init_cfg}" \
+         --img-url "${BUILDER_CLOUD_IMAGE}" \
+         --raw-disk "${WORKSPACE}/${DISTRO}.raw" \
+         --raw-size 20 ||
+         fail "KVM instance failed"
+
+tar -xvvf "${WORKSPACE}/${DISTRO}.raw" ||
+    fail "Result tar failed to unpack"
+
+rm "${WORKSPACE}/${DISTRO}.raw" ||
+    fail "Failed to remove unnecessary file"
+
+# Put the bits in place
+"${base_dir}/copy_to_final.sh" \
+    "${DISTRO}" \
+    "${WORKSPACE}" \
+    "${SERIAL}" \
+    "${BTYPE}" \
+    "${TEST_BUILD}" \
+    "${SANDBOX_BUILD}" \
+    "${PROPOSED_BUILD}" ||
+    fail "Failed to place final files to destination"
+
+# Copy the build properties into the workspace. This is set by get_serial.sh
+[ "${BUILD_PROPERTIES}" != "${WORKSPACE}/build_properties" ] &&
+    cp ${BUILD_PROPERTIES} ${WORKSPACE}/build_properties
+
+echo "ARCH=${ARCH_TYPE}" >> build_properties

=== added file 'jenkins/CloudImages_Base_Release_Delta.sh'
--- jenkins/CloudImages_Base_Release_Delta.sh	1970-01-01 00:00:00 +0000
+++ jenkins/CloudImages_Base_Release_Delta.sh	2018-05-31 04:33:07 +0000
@@ -0,0 +1,255 @@
+#!/bin/bash -x
+
+# Set default umask
+umask 022
+
+# Skip promotion if this file exists
+HOLIDAY_FILE=/srv/jenkins/HOLIDAY
+
+# Write the build properties file
+cat << EOF > "${WORKSPACE}/build_properties"
+SUITE=${SUITE}
+STREAM=${STREAM}
+SERIAL=${SERIAL}
+BUILD_TYPE=${BUILD_TYPE}
+
+EOF
+
+# Write the environmental variables to the run file
+env > ${SUITE}.run
+
+fail() { echo "$@"; exit 1;}
+dist_ge() { [[ "$1" > "$2" || "$1" == "$2" ]]; }
+
+arches=(i386 amd64 armel armhf arm64 ppc64el)
+exec_c="/srv/builder/vmbuilder/bin/cronrun"
+rel_base="/srv/ec2-images/releases/${SUITE}/release"
+rel_link=$(readlink ${rel_base})
+
+[ "${BUILD_TYPE}" = "desktop" ] &&
+    echo "Not valid for desktop builds" &&
+    exit 0
+
+# Find the existing manifest file
+old_manifest=$(find -L ${rel_base} -maxdepth 1 -iname '*amd64.manifest') ||
+    echo "Unable to find release manifest file"
+
+# Find the new manifest file
+new_manifest_d="/srv/ec2-images/${SUITE}/${SERIAL}"
+[ "${TEST_BUILD:-0}" -eq 1 ] && new_manifest_d="/srv/ec2-images/test_builds/${SUITE}/${SERIAL}"
+[ "${SANDBOX_BUILD:-0}" -eq 1 ] && new_manifest_d="/srv/ec2-images/sandbox/${SUITE}/${SERIAL}"
+new_manifest=$(find ${new_manifest_d} -maxdepth 1 -iname '*amd64.manifest') ||
+    fail "Unable to find new manifest file"
+
+# Find the previous serial if there was one
+previous_serial=$(find  /srv/ec2-images/${SUITE}/ -maxdepth 1 -type d |\
+    awk -F\/ '{print$NF}' | sort -rn | grep "." | grep -v "${SERIAL}" | head -n1) ||
+    echo "Unable to find prior daily manifest"
+
+previous_manifest=${new_manifest//$SERIAL/$previous_serial}
+
+# Generate the pure package diffs
+for arch in "${arches[@]}"
+do
+    nm=${new_manifest//amd64/$arch}
+    om=${old_manifest//amd64/$arch}
+    pm=${previous_manifest/amd64/$arch}
+
+    [ -e "${nm}" ] &&
+        cp "${nm}" "${WORKSPACE}/manifest-${arch}-daily-${SERIAL}.txt"
+
+    # Generate the diff from daily to release
+    if [ -e "${nm}" -a -e "${om}" ]; then
+        release_diff=${new_manifest##*/}
+        release_diff=${release_diff//.manifest/-$rel_link-to-daily_manifest.diff}
+        release_diff=${release_diff//amd64/$arch}
+        diff -u ${om} ${nm} > "${WORKSPACE}/${release_diff}"
+        cp ${om} "${WORKSPACE}/manifest-${arch}-release.txt"
+    fi
+
+    # Generate the diff from daily to old daily
+    if [ -e "${nm}" -a -e "${pm}" ]; then
+        daily_diff=${new_manifest##*/}
+        daily_diff=${daily_diff//.manifest/-$previous_serial-to-$SERIAL-manifest.diff}
+        daily_diff=${daily_diff//amd64/$arch}
+        diff -u ${pm} ${nm} > "${WORKSPACE}/${daily_diff}"
+        cp ${pm} "${WORKSPACE}/manifest-${arch}-previous_daily-${previous_serial}.txt"
+    fi
+done
+
+# Determine if there is a different version of a particular package.
+# If so, write out the package information to the specified trigger file
+package_differences() {
+    # If a pattern is used, it should match a single entry in the manifest
+    package_pattern=$1
+    trigger_file=$2
+
+    echo "Checking for differences in ${package_pattern}"
+
+    v1="$(awk "/${package_pattern}/ {print\$NF}" ${old_manifest})"
+    v2="$(awk "/${package_pattern}/ {print\$NF}" ${new_manifest})"
+
+    if [ "x${v1}" != "x${v2}" ]; then
+        echo "  Package changed old:${v1}, new:${v2}"
+        cat << PKGDIFF >> ${trigger_file}
+'${package_pattern}':
+- old: '${v1}'
+- new: '${v2}'
+
+PKGDIFF
+    else
+        echo "  No difference   old:${v1}, new:${v2}"
+    fi
+}
+
+# Set packages to trigger an automated promotion in this array
+# This list of packages is controlled through application of
+# lp:~cloudware/cpc-core/+git/cpc_policy:policies/0003_automated_daily_promotion.rst
+# trigger_set MUST NOT be modified without accompanying policy doc change
+declare -a trigger_set
+# This is an array of package names where the string is any awk-friendly
+# pattern supported by the expression in package_differences(), but it must
+# only match a single package
+#trigger_set=('example_package' 'example_package2-*')
+trigger_set=('pollinate')
+
+# Append the kernel package to the trigger_set array
+if dist_ge ${SUITE} quantal; then
+    trigger_set[${#trigger_set[@]}]='linux-image.*generic'
+else
+    trigger_set[${#trigger_set[@]}]='linux-image-virtual'
+fi
+
+# For legacy reasons the jenkins jobs use a "kernel" trigger file
+# for automated build promotion.  All package changes will
+# use this single trigger file until the need arises for more
+# granularity.
+trigger_file="${WORKSPACE}/${SUITE}-kernel-trigger"
+if [ -e "${trigger_file}" ] ; then
+    echo "Cleaning up old trigger file in workspace"
+    rm --verbose "${trigger_file}"
+fi
+
+# Check all packages in the trigger_set array
+for pkg in ${trigger_set[@]} ; do
+    package_differences "${pkg}" "${trigger_file}"
+done
+
+# If the trigger file exists, determine if it should be pushed to the
+# build trigger directory for action
+if [ -e "${trigger_file}" ] ; then
+    if [ ! -e "${rel_base}" ]; then
+        echo "${SUITE} not released, not triggering"
+        rm --verbose "${trigger_file}"
+    elif [ ! -f ${HOLIDAY_FILE} ]; then
+        echo "Creating trigger file with contents:"
+        cat "${trigger_file}"
+        cp --verbose "${trigger_file}" \
+           "${TRIGGER_LOCATION:-/srv/builder/triggers/kernel}/${PARENT_BUILDER_ID}.trigger"
+    else
+        echo "Not creating trigger, ${HOLIDAY_FILE} found"
+        if [ -e "${TRIGGER_LOCATION:-/srv/builder/triggers/kernel}/${PARENT_BUILDER_ID}.trigger" ]; then
+            echo "Removing existing trigger from old build with same parent ID."
+            rm --verbose "${TRIGGER_LOCATION:-/srv/builder/triggers/kernel}/${PARENT_BUILDER_ID}.trigger"
+        fi
+    fi
+else
+    echo "No trigger file found"
+fi
+
+# Copy the diffs into the current workspace
+cp ${WORKSPACE}/*.diff ${new_manifest_d}/unpacked
+
+# Generate the mfdiff between the dailies
+[ -e "${previous_manifest}" -a -e "${new_manifest}" ] &&
+    ${exec_c} mfdiff amd64 ${SUITE} ${previous_manifest} ${new_manifest} >\
+    "${WORKSPACE}/${SUITE}-daily.changelog"
+
+# Generate the diff between daily and the released image
+[ -e "${old_manifest}" -a -e "${new_manifest}" ] &&
+    ${exec_c} mfdiff amd64 ${SUITE} ${old_manifest} ${new_manifest} >\
+    "${WORKSPACE}/${SUITE}-${rel_link}-to-daily.changelog"
+
+# Copy the changelogs into the current workspace
+cp ${WORKSPACE}/*.changelog ${new_manifest_d}/unpacked
+
+# The rest of the operations are for released images only
+[ ! -e "${rel_base}" ] &&
+    echo "No current release, aborting comparison" &&
+    exit 0
+
+# Tar up the deltas
+tar -C ${WORKSPACE} -jcvf "${WORKSPACE}/${SUITE}-${SERIAL}.tar.bz2" \
+    *.changelog \
+    *.txt \
+    *.diff ||
+    fail "Failed to create tarball"
+
+# Start the email report work
+changed_pkgs=$(grep '=>' ${SUITE}-${rel_link}-to-daily.changelog | \
+    sed -e 's,====,,g' -e 's,^, *,g' | sort -k2)
+
+# Generate the email template
+VER=$(${kvm}/ubuntu-adj2version ${SUITE})
+
+case ${VER} in
+    *8.04*)  VER="${VER} LTS";
+             CODENAME="Hardy Heron";;
+    *10.04*) VER="${VER} LTS";
+             CODENAME="Lucid Lynx";;
+    *11.04*) CODENAME="Natty Narwhal";;
+    *11.10*) CODENAME="Oneiric Ocelot";;
+    *12.04*) VER="${VER} LTS";
+             CODENAME="Precise Pangolin";;
+    *12.10*) CODENAME="Quantal Queztal";;
+    *13.04*) CODENAME="Raring Ringtail";;
+    *13.10*) CODENAME="Saucy Salamander";;
+    *14.04*) VER="${VER} LTS";
+             CODENAME="Trusty Tahr";;
+    *14.10*) CODENAME="Utopic Unicorn";;
+    *15.04*) CODENAME="Vivid Vervet";;
+    *15.10*) CODENAME="Wily Werewolf";;
+esac
+
+email_name="${WORKSPACE}/${SUITE}-release_announcement.email"
+cat << EOF > "${email_name}"
+SUBJECT: Refreshed Cloud Images of ${VER} (${CODENAME}) [${SERIAL}]
+TO: [e-mail]; [e-mail]; [e-mail]
+
+A new release of the Ubuntu Cloud Images for stable Ubuntu release ${VER} (${CODENAME}) is available at [1]. These new images superseded the existing images [2]. Images are available for download or immediate use on EC2 via publish AMI ids. Users who wish to update their existing installations can do so with:
+   'sudo apt-get update && sudo apt-get dist-upgrade && sudo reboot'.
+
+EOF
+
+if [ "${old_linux_kernel}" != "${new_linux_kernel}" ]; then
+    cat << EOF >> "${email_name}"
+The Linux kernel was updated from ${old_linux_kernel} [3] to ${new_linux_kernel} [4]
+
+EOF
+fi
+
+cat << EOF >> "${email_name}"
+The following packages have been updated. Please see the full changelogs
+for a complete listing of changes:
+${changed_pkgs}
+
+
+The following is a complete changelog for this image.
+$(cat ${SUITE}-${rel_link}-to-daily.changelog)
+
+--
+[1] http://cloud-images.ubuntu.com/releases/${SUITE}/release-${SERIAL}/
+[2] http://cloud-images.ubuntu.com/releases/${SUITE}/${rel_link}/
+EOF
+
+if [ "${old_linux_kernel}" != "${new_linux_kernel}" ]; then
+cat << EOF >> "${email_name}"
+[3] http://changelogs.ubuntu.com/changelogs/pool/main/l/linux/linux_${old_linux_kernel}/changelog
+[4] http://changelogs.ubuntu.com/changelogs/pool/main/l/linux/linux_${new_linux_kernel}/changelog
+EOF
+fi
+
+# Create release notes
+lnc=$(wc -l ${email_name} | awk '{print$1}')
+tail -n `expr $lnc - 3` ${email_name} > "${WORKSPACE}/release_notes.txt"
+cp ${WORKSPACE}/release_notes.txt ${new_manifest_d}/unpacked

=== added file 'jenkins/CloudImages_Juju.sh'
--- jenkins/CloudImages_Juju.sh	1970-01-01 00:00:00 +0000
+++ jenkins/CloudImages_Juju.sh	2018-05-31 04:33:07 +0000
@@ -0,0 +1,253 @@
+#!/bin/bash
+
+# Set default umask
+umask 022
+
+# Read in the common files
+my_name=$(readlink -f ${0})
+my_dir=$(dirname ${my_name})
+my_pdir=$(dirname ${my_dir})
+
+# Source in the common functions
+source "${my_pdir}/functions/common"
+source "${my_pdir}/functions/retry"
+source "${my_pdir}/functions/locker"
+export HOME=${WORKSPACE}
+
+# needed for building on Jenkins
+[ -e "build_properties" ] && source build_properties
+
+# Copy the target disk imags
+ARCH_TYPE=${ARCH_TYPE:-$ARCH}
+disk_orig="${SUITE}-server-cloudimg-${ARCH_TYPE}-disk1.img"
+disk_cp="${disk_orig//$ARCH_TYPE/$ARCH_TYPE-juju-vagrant}"
+disk_root="${SRV_D:-/srv/ec2-images}/${SUITE}/${SERIAL:-current}"
+disk_working="${WORKSPACE}/${disk_cp}"
+final_disk="${WORKSPACE}/box-disk1.vdi"
+final_location="${OUTPUT_D:-/srv/ec2-images}/vagrant/${SUITE}/${SERIAL}"
+box_name="${disk_working//.img/.box}"
+raw_f="${WORKSPACE}/raw_f-$(date +%s).img"
+build_host_suite=$(lsb_release -c -s)
+
+jenkins_build() {
+    [ -e "build_properties" ] &&
+        source build_properties ||
+        fail "Failed to read build_properties. I don't know what I'm doing!"
+
+    # Bail if something isn't right
+    SUITE=${SUITE:?Suite must be defined}
+    SERIAL=${SERIAL:?Serial must be defined}
+
+    cp "${disk_root}/${disk_orig}" "${disk_working}" ||
+        fail "Unable to copy ${disk_orig} from ${disk_root}"
+
+    qemu-img resize ${disk_working} 40G
+
+    # Launch KVM to do the worK
+    ${my_pdir}/launch_kvm.sh \
+            --id "${ARCH_TYPE}-${BUILD_ID}" \
+            --user-data "${my_pdir}/config/cloud-vps.cfg" \
+            --cloud-config "${my_pdir}/templates/img-juju.tmpl" \
+            --extra-disk "${disk_working}" \
+            --disk-gb 1 \
+            --raw-disk "${raw_f}" \
+            --raw-size 1 \
+            --img-url /srv/builder/images/precise-builder-latest.img ||
+                fail "KVM instance failed to build image."
+}
+
+# Assume that we're building in Jenkins unless otherwise stated
+# What this allows us to do is to use the standalone builder for testing
+# and finish running the bits below
+[ "${LOCAL_BUILD:-0}" -eq 1 ] || jenkins_build
+
+# Covert to VMDK.
+qemu-img convert -O raw ${disk_working} ${disk_working//.img/.raw}
+
+_vbox_cmd convertfromraw \
+    --format vdi \
+    ${disk_working//.img/.raw} ${final_disk}
+
+# Create the VM
+vmname="ubuntu-cloudimg-${SUITE}-juju-vagrant-${ARCH_TYPE}"
+_vbox_cmd modifyhd --compact ${final_disk}
+
+dist_v="Ubuntu"
+[ "${ARCH_TYPE}" = "amd64" ] && dist_v="Ubuntu_64"
+_vbox_cmd createvm \
+    --name ${vmname} \
+    --ostype ${dist_v} \
+    --register
+
+_vbox_cmd modifyvm ${vmname} \
+    --memory 2048 \
+    --boot1 disk \
+    --boot2 none \
+    --boot3 none \
+    --boot4 none \
+    --vram 12 \
+    --pae off \
+    --acpi on \
+    --ioapic on \
+    --rtcuseutc on \
+    --bioslogodisplaytime 0 \
+    --nic1 nat \
+    --nictype1 virtio
+
+if [ "${ARCH_TYPE}" = "i386" ]; then
+    _vbox_cmd modifyvm ${vmname} \
+        --ioapic off \
+        --pae on
+fi
+
+
+_vbox_cmd modifyvm ${vmname} --natpf1 "guestssh,tcp,,2222,,22"
+
+storage_cmd=(
+    _vbox_cmd storagectl "${vmname}"
+    --name "SATAController"
+    --add sata
+    --controller IntelAhci
+    --hostiocache on
+    )
+
+if [ "$(lsb_release -r -s | sed 's/\.//')" -lt 1404 ]; then
+	storage_cmd+=(--sataportcount 1)
+else
+	storage_cmd+=(--portcount 1)
+fi
+
+${storage_cmd[@]}
+
+_vbox_cmd storageattach ${vmname} \
+    --storagectl "SATAController" \
+    --port 0 \
+    --device 0 \
+    --type hdd \
+    --medium ${final_disk}
+
+# Set extra-data
+_vbox_cmd setextradata ${vmname} installdate ${serial}
+_vbox_cmd setextradata ${vmname} supported false
+
+# Set the Guest information to get rid of error message
+[ -e vagrant_image.pkgs ] && {
+
+    vbox_version=""
+    while read -r line
+    do
+        line=( $(echo ${line}) )
+        [[ ${line[0]} =~ virtualbox-guest-utils ]] && vbox_version=${line[1]}
+    done < vagrant_image.pkgs
+    debug "Guest Additions version is ${vbox_version}"
+
+    # Set the revision to some arbitrary value
+    _vbox_cmd guestproperty set ${vmname} \
+        "/VirtualBox/GuestAdd/Revision" '8000'
+
+    # Set the Ubuntu packaged version correctly
+    _vbox_cmd guestproperty set ${vmname} \
+        "/VirtualBox/GuestAdd/VersionExt" \
+        "${vbox_version//-dfsg-*/_Ubuntu}"
+
+    # Set the version string appropriately
+    _vbox_cmd guestproperty set ${vmname} \
+        "/VirtualBox/GuestAdd/Version" \
+        "${vbox_version//-dfsg-*/}"
+}
+
+mkdir ${WORKSPACE}/box
+_vbox_cmd export ${vmname} --output ${WORKSPACE}/box/box.ovf
+
+# Create the Vagrant file
+#macaddr="02:$(openssl rand -hex 5)"
+macaddr=$(awk '-F"' '/<Adapter slot="0" enabled="true"/ {print$6}' ${WORKSPACE}/box/box.ovf)
+cat << EOF > ${WORKSPACE}/box/Vagrantfile
+\$script = <<SCRIPT
+bzr branch lp:jujuredirector /tmp/jujuredir
+
+if ! grep precise /etc/lsb-release > /dev/null; then
+    cat << EOM > "/etc/apt/apt.conf.d/90proxy"
+Acquire::http::Proxy "http://10.0.3.1:8000";
+EOM
+
+    for series in precise trusty; do
+        version=\$(grep \$series /usr/share/distro-info/ubuntu.csv | cut -d, -f1 | cut -d' ' -f1)
+        expected_filename=/var/cache/lxc/cloud-\${series}/ubuntu-\${version}-server-cloudimg-${ARCH_TYPE}-root.tar.gz
+        if [ ! -e \$expected_filename ]; then
+            mkdir -p "/var/cache/lxc/cloud-\${series}"
+            curl -o "\$expected_filename" \
+                http://cloud-images.ubuntu.com/releases/\${series}/release/ubuntu-\${version}-server-cloudimg-${ARCH_TYPE}-root.tar.gz
+        fi
+    done
+
+    # Set up squid in the LXC template
+    for lxc_template in \$(ls /var/cache/lxc/cloud-*/*-root.tar.gz); do
+        gunzip "\$lxc_template"
+        unwrapped_name=\$(dirname "\$lxc_template")/\$(basename "\$lxc_template" .gz)
+        mkdir -p etc/apt/apt.conf.d
+        echo 'Acquire::http::Proxy "http://10.0.3.1:8000";' > etc/apt/apt.conf.d/90proxy
+        tar rf "\$unwrapped_name" etc/apt/apt.conf.d/90proxy
+        gzip "\$unwrapped_name"
+        rm -rf etc
+    done
+fi
+
+bash /tmp/jujuredir/setup-juju.sh 6079
+echo "export JUJU_REPOSITORY=/charms" >> /home/vagrant/.bashrc
+SCRIPT
+
+system 'mkdir', '-p', 'charms'
+
+Vagrant.configure("2") do |config|
+  # This Vagrantfile is auto-generated by 'vagrant package' to contain
+  # the MAC address of the box. Custom configuration should be placed in
+  # the actual 'Vagrantfile' in this box.
+
+  config.vm.base_mac = "${macaddr}"
+  config.vm.network :forwarded_port, guest: 22, host: 2122, host_ip: "127.0.0.1"
+  config.vm.network :forwarded_port, guest: 80, host: 6080, host_ip: "127.0.0.1"
+  config.vm.network :forwarded_port, guest: 6079, host: 6079, host_ip: "127.0.0.1"
+  config.vm.network "private_network", ip: "172.16.250.15"
+  config.vm.provider "virtualbox" do |vb|
+       vb.customize ["modifyvm", :id, "--natdnshostresolver1", "on"]
+  end
+  config.vm.provision "shell", inline: \$script
+
+  config.vm.synced_folder "charms/", "/charms"
+end
+
+# Load include vagrant file if it exists after the auto-generated
+# so it can override any of the settings
+include_vagrantfile = File.expand_path("../include/_Vagrantfile", __FILE__)
+load include_vagrantfile if File.exist?(include_vagrantfile)
+EOF
+
+# Now pack it all up....
+tar -C ${WORKSPACE}/box -Scvf ${box_name} box.ovf Vagrantfile box-disk1.vmdk ||
+        fail "Unable to create box file"
+
+# Some minor cleanup
+rm ${disk_working} ${disk_working//.img/.raw} || /bin/true
+rm -rf ${WORKSPACE}/box *.vdi
+[ -e "${raw_f}" ] && rm "${raw_f}"
+
+# Bail here if this is a local build
+[ "${LOCAL_BUILD:-0}" -eq 1 ] && exit 0
+
+# Put the box in place
+mkdir -p "${final_location}" ||
+    fail "Unable to create the vagrant image location"
+
+cp ${box_name} ${final_location} ||
+    fail "Failed to place vagrant image in final home"
+
+# Now Checksum it all
+
+# Override and set some home variables
+export HOME="/srv/builder"
+export CDIMAGE_BIN="${HOME}/cdimage/bin"
+PUBLISH_SCRIPTS=${HOME}/ec2-publishing-scripts
+export CDIMAGE_ROOT="${HOME}/cdimage"
+export PATH="${PUBLISH_SCRIPTS}:${CDIMAGE_BIN}:${PATH}"
+checksum-directory ${final_location}

=== added file 'jenkins/CloudImages_Update_Builder.sh'
--- jenkins/CloudImages_Update_Builder.sh	1970-01-01 00:00:00 +0000
+++ jenkins/CloudImages_Update_Builder.sh	2018-05-31 04:33:07 +0000
@@ -0,0 +1,68 @@
+#!/bin/bash
+
+# Set default umask
+umask 022
+
+# Read in the common files
+source "${kvm}/functions/common"
+source "${kvm}/functions/retry"
+
+# Apply the build stuff
+find . -iname "*build_properties" | xargs -I FILE cp FILE .
+[ -e "build_properties" ] &&
+    source build_properties ||
+    fail "Failed to read build_properties. I don't know what I'm doing!"
+
+[ -e failed ] && rm
+[ -e success ] && rm
+
+# Copy the target disk image
+case ${SUITE} in
+    trusty|xenial)
+        disk_orig="${SUITE}-server-cloudimg-${ARCH}-disk1.img"
+        builder_img=/srv/builder/images/trusty-builder-latest.img
+        ;;
+    zesty)
+        # Zesty needs yakkety or newer due to ext4 tool changes
+        disk_orig="${SUITE}-server-cloudimg-${ARCH}.img"
+        builder_img=/srv/builder/images/zesty-builder-latest.img
+        ;;
+    *)
+        disk_orig="${SUITE}-server-cloudimg-${ARCH}.img"
+        builder_img=/srv/builder/images/artful-builder-latest.img
+        ;;
+esac
+
+disk_cp="${disk_orig//cloudimg/cloudimg-builder-$(date +%Y%m%d)}"
+disk_root="/srv/ec2-images/${SUITE}/${SERIAL:-current}"
+disk_working="${WORKSPACE}/${disk_cp}"
+raw_f="${WORKSPACE}/raw_f-$(date +%s).img"
+
+cp "${disk_root}/${disk_orig}" "${disk_working}" ||
+    fail "Unable to copy ${disk_orig} from ${disk_root}"
+
+qemu-img resize "${disk_working}" 5G ||
+    fail "unable to resize disk"
+
+# Launch KVM to do the work
+${kvm}/launch_kvm.sh \
+        --id "${ARCH}-${BUILD_ID}" \
+        --user-data "${kvm}/config/cloud-vps.cfg" \
+        --cloud-config "${kvm}/templates/img-update.tmpl" \
+        --extra-disk "${disk_working}" \
+        --disk-gb 5 \
+        --raw-disk "${raw_f}" \
+        --raw-size 1 \
+        --img-url ${builder_img} ||
+            fail "KVM instance failed to build image."
+
+# Remove the results
+rm "${raw_f}" || /bin/true
+
+# Compress it down...
+mv "${disk_working}" "${disk_working}.new"
+qemu-img convert "${disk_working}.new" -c -O qcow2 "${disk_working}" ||
+    fail "Failed to create compressed image"
+
+rm "${disk_working}.new"
+

=== added file 'jenkins/CloudImages_Vagrant.sh'
--- jenkins/CloudImages_Vagrant.sh	1970-01-01 00:00:00 +0000
+++ jenkins/CloudImages_Vagrant.sh	2018-05-31 04:33:07 +0000
@@ -0,0 +1,232 @@
+#!/bin/bash
+
+# Set default umask
+umask 022
+
+# Read in the common files
+source "${kvm}/functions/common"
+source "${kvm}/functions/retry"
+source "${kvm}/functions/locker"
+export HOME=${WORKSPACE}
+
+# Apply the build stuff
+[ -e "build_properties" ] &&
+    source build_properties ||
+    fail "Failed to read build_properties. I don't know what I'm doing!"
+
+rm {failed,success} || /bin/true
+
+# Copy the target disk image
+ARCH_TYPE=${ARCH_TYPE:-$ARCH}
+disk_orig="${SUITE}-server-cloudimg-${ARCH_TYPE}-disk1.img"
+disk_cp="${disk_orig//$ARCH_TYPE/$ARCH_TYPE-vagrant}"
+disk_root="${SRV_D:-/srv/ec2-images}/${SUITE}/${SERIAL:-current}"
+disk_working="${WORKSPACE}/${disk_cp}"
+final_disk="${WORKSPACE}/box-disk1.vdi"
+final_location="${OUTPUT_D:-/srv/ec2-images}/vagrant/${SUITE}/${SERIAL}"
+box_name="${disk_working//.img/.box}"
+raw_f="${WORKSPACE}/raw_f-$(date +%s).img"
+
+[ -e "${final_location}/${box_name}" -a "${REBUILD}" != "true" ] && exit 0
+
+cp "${disk_root}/${disk_orig}" "${disk_working}" ||
+    fail "Unable to copy ${disk_orig} from ${disk_root}"
+
+# Resize it to 4G, but not the full 40G because we want it sparse
+qemu-img resize ${disk_working} 4G
+
+# Launch KVM to do the work
+${kvm}/launch_kvm.sh \
+        --id "${ARCH_TYPE}-${BUILD_ID}" \
+        --user-data "${kvm}/config/cloud-vps.cfg" \
+        --cloud-config "${kvm}/templates/img-vagrant.tmpl" \
+        --extra-disk "${disk_working}" \
+        --disk-gb 1 \
+        --raw-disk "${raw_f}" \
+        --raw-size 1 \
+        --img-url /srv/builder/images/precise-builder-latest.img ||
+            fail "KVM instance failed to build image."
+
+# Covert to VMDK.
+qemu-img convert -O raw ${disk_working} ${disk_working//.img/.raw}
+truncate -s 40G ${disk_working//.img/.raw}
+
+_vbox_cmd convertfromraw \
+    --format vdi \
+    ${disk_working//.img/.raw} ${final_disk}
+
+# Create the VM
+vmname="ubuntu-cloudimg-${SUITE}-vagrant-${ARCH_TYPE}"
+_vbox_cmd modifyhd --compact ${final_disk}
+
+dist_v="Ubuntu"
+[ "${ARCH_TYPE}" = "amd64" ] && dist_v="Ubuntu_64"
+_vbox_cmd createvm \
+    --name ${vmname} \
+    --ostype ${dist_v} \
+    --register
+
+_vbox_cmd modifyvm ${vmname} \
+    --memory 512 \
+    --boot1 disk \
+    --boot2 none \
+    --boot3 none \
+    --boot4 none \
+    --vram 12 \
+    --pae off \
+    --acpi on \
+    --ioapic on \
+    --rtcuseutc on
+#    --natnet1 default \
+
+if [ "${ARCH_TYPE}" = "i386" ]; then
+    _vbox_cmd modifyvm ${vmname} \
+        --ioapic off \
+        --pae on
+fi
+
+
+_vbox_cmd modifyvm ${vmname} --natpf1 "guestssh,tcp,,2222,,22"
+
+_vbox_cmd storagectl "${vmname}" \
+    --name "SATAController" \
+    --add sata \
+    --controller IntelAhci \
+    --sataportcount 1 \
+    --hostiocache on
+
+_vbox_cmd storageattach ${vmname} \
+    --storagectl "SATAController" \
+    --port 0 \
+    --device 0 \
+    --type hdd \
+    --medium ${final_disk}
+
+# Set extra-data
+_vbox_cmd setextradata ${vmname} installdate ${serial}
+_vbox_cmd setextradata ${vmname} supported false
+
+# Set the Guest information to get rid of error message
+[ -e vagrant_image.pkgs ] && {
+
+    vbox_version=""
+    while read -r line
+    do
+        line=( $(echo ${line}) )
+        [[ ${line[0]} =~ virtualbox-guest-utils ]] && vbox_version=${line[1]}
+    done < vagrant_image.pkgs
+    debug "Guest Additions version is ${vbox_version}"
+
+    # Set the revision to some arbitrary value
+    _vbox_cmd guestproperty set ${vmname} \
+        "/VirtualBox/GuestAdd/Revision" '8000'
+
+    # Set the Ubuntu packaged version correctly
+    _vbox_cmd guestproperty set ${vmname} \
+        "/VirtualBox/GuestAdd/VersionExt" \
+        "${vbox_version//-dfsg-*/_Ubuntu}"
+
+    # Set the version string appropriately
+    _vbox_cmd guestproperty set ${vmname} \
+        "/VirtualBox/GuestAdd/Version" \
+        "${vbox_version//-dfsg-*/}"
+}
+
+mkdir box
+_vbox_cmd export ${vmname} --output box/box.ovf
+
+# Create the Vagrant file
+#macaddr="02:$(openssl rand -hex 5)"
+macaddr=$(awk '-F"' '/<Adapter slot="0" enabled="true"/ {print$6}' ${WORKSPACE}/box/box.ovf)
+cat << EOF > ${WORKSPACE}/box/Vagrantfile
+Vagrant::Config.run do |config|
+  # This Vagrantfile is auto-generated by 'vagrant package' to contain
+  # the MAC address of the box. Custom configuration should be placed in
+  # the actual 'Vagrantfile' in this box.
+  config.vm.base_mac = "${macaddr}"
+end
+
+# Load include vagrant file if it exists after the auto-generated
+# so it can override any of the settings
+include_vagrantfile = File.expand_path("../include/_Vagrantfile", __FILE__)
+load include_vagrantfile if File.exist?(include_vagrantfile)
+EOF
+
+# Now pack it all up....
+tar -C ${WORKSPACE}/box -Scvf ${box_name} box.ovf Vagrantfile box-disk1.vmdk ||
+        fail "Unable to create box file"
+
+# Some minor cleanup
+rm ${disk_working} ${disk_working//.img/.raw} || /bin/true
+rm -rf ${WORKSPACE}/box *.vdi
+rm "${raw_f}" || /bin/true
+
+# Put the box in place
+mkdir -p "${final_location}" ||
+    fail "Unable to create the vagrant image location"
+
+cp ${box_name} ${final_location} ||
+    fail "Failed to place vagrant image in final home"
+
+# box_d is where the boxes are stored
+box_d="${OUTPUT_D:-/srv/ec2-images}/vagrant/${SUITE}"
+
+# Only proceed if the required boxes exist
+boxes=($(find ${box_d}/${SERIAL} -regextype posix-extended -regex ".*(amd64|i386)-vagrant-disk1.box"))
+if [ "${#boxes[@]}" -ne 2 ]; then
+    echo "Not updating current, required boxes are missing"
+    [[ ! "${boxes[@]}" =~ "amd64" ]] && echo "Missing build for amd64"
+    [[ ! "${boxes[@]}" =~ "i386" ]] && echo "Missing build for i386"
+
+    # We don't want to fail here.
+    exit 0
+else
+    echo "Updating current links; all builds are present"
+fi
+
+# Update the link to current
+current_l="${box_d}/current"
+[ -e "${current_l}" ] && rm "${current_l}"
+( cd "${box_d}" && ln -s "${SERIAL}" current )
+
+# Cleanup old builds
+builds=($(find ${box_d} -mindepth 1 -maxdepth 1 -type d | sort -r))
+build_count="${#builds[@]}"
+
+echo "------------------------"
+echo "Clean-up for prior builds"
+echo "Found ${build_count} builds for consideration"
+
+for b in ${builds[@]}
+do
+    echo " - found build ${b}"
+done
+echo ""
+
+[ "${build_count}" -gt 4 ] && {
+    for item in $(seq 4 ${build_count})
+    do
+        [ -e "${builds[$item]}" ] && {
+            echo "Removing build ${builds[$item]} for deletion"
+            rm -rf ${builds[$item]} ||
+                echo "Failed to remove build ${builds[$item]}"
+        }
+    done
+
+    for item in $(seq 0 3)
+    do
+        [ -e "${builds[$item]}" ] &&
+            echo "Preserving build ${builds[$item]}"
+    done
+
+} || echo "No builds marked for removal"
+
+
+# Override and set some home variables
+export HOME="/srv/builder"
+export CDIMAGE_BIN="${HOME}/cdimage/bin"
+PUBLISH_SCRIPTS=${HOME}/ec2-publishing-scripts
+export CDIMAGE_ROOT="${HOME}/cdimage"
+export PATH="${PUBLISH_SCRIPTS}:${CDIMAGE_BIN}:${PATH}"
+checksum-directory ${final_location}
+

=== added file 'jenkins/MAAS_Builder.sh'
--- jenkins/MAAS_Builder.sh	1970-01-01 00:00:00 +0000
+++ jenkins/MAAS_Builder.sh	2018-05-31 04:33:07 +0000
@@ -0,0 +1,171 @@
+#!/bin/bash
+set -x
+fail() { [ $# -eq 0 ] || echo "$@"; exit 1; }
+
+find . -iname "*build_properties" -exec cp {} . \; ||
+    echo "Unable to copy build properties, this might be v2"
+
+[ -z "${SERIAL}" -a -z "${SUITE}" -a -e "build_properties" ] && {
+    source build_properties ||
+    fail "Failed to read build_properties. I don't know what I'm doing!";
+}
+
+# Read in the common functions
+my_dir="$( cd "$( dirname "$0" )" && pwd )"
+base_dir=$(dirname ${my_dir})
+export PATH="${base_dir}:${my_dir}:${PATH}"
+source "${base_dir}/functions/locker"
+source "${base_dir}/functions/common"
+source "${base_dir}/functions/retry"
+source ${my_dir}/build_lib.sh
+select_build_config
+
+export WORKSPACE="${WORKSPACE:-$WORKSPACE_R}"
+out_f="${WORKSPACE}/maas-${SUITE}-${STREAM}-config.sh"
+raw_f="${WORKSPACE}/${SUITE}-output.raw"
+query_t="${WORKSPACE}/cloud-images-query.tar"
+base_name="${SUITE}-server-cloudimg"
+rel_base_name="ubuntu-$(ubuntu-adj2version ${SUITE})-${stream//-/}-server-cloudimg"
+
+export maas_branch="${MAAS_BRANCH:-http://bazaar.launchpad.net/~smoser/maas/maas.ubuntu.com.images-ephemeral}"
+
+case "${STREAM}" in
+    release)  build_f="/srv/ec2-images/releases/${SUITE}/release-${SERIAL}";
+              base_name=${rel_base_name};
+              out_d="/srv/maas-images/ephemeral/releases/${SUITE}/release-${SERIAL}"
+              ;;
+    daily)    build_f="/srv/ec2-images/${SUITE}/${SERIAL}";
+              out_d="/srv/maas-images/ephemeral/daily/${SUITE}/${SERIAL}";
+              ;;
+    alpha*|beta*)  build_f="/srv/ec2-images/releases/${SUITE}/${STREAM}";
+              base_name=${rel_base_name};
+              out_d="/srv/maas-images/ephemeral/release/${SUITE}/${STREAM}";
+              ;;
+    *)        fail "Unknown stream ${STREAM}.";;
+esac
+
+final_out_d="${out_d}"
+
+[ -e "${final_out_d}" -a "${REBUILD:-false}" = "false" ] &&
+    fail "Build already exists. Rebuild is set to false. Failing this build"
+
+# Tar up query for use in-image
+[ ! -e "${query_t}" ] && {
+    tar cvf ${query_t} \
+        ${QUERY_D:-/srv/ec2-images/query} \
+        ${build_f}  \
+        --exclude "*img" --exclude "*azure*" --exclude "*html" \
+        --exclude "*armel*" --exclude "*root.tar.gz" \
+        --exclude "*floppy" ||
+        fail "Failed to pack up build elements for MAAS builder"; }
+
+# Generate the template file
+ci_cfg="${kvm_builder}/config/cloud-maas.cfg"
+template="${kvm_builder}/templates/img-maas.tmpl"
+[ "${IS_MAAS_V2:-0}" -eq 1 ] && {
+     template="${kvm_builder}/templates/img-maasv2.tmpl"
+     ci_cfg="${kvm_builder}/config/cloud-maasv2.cfg"
+}
+
+maas_config.sh \
+    --distro "${SUITE}" \
+    --stream "${STREAM}" \
+    --template "${template}" \
+    --base-name "${base_name}" \
+    --local "${build_f}" \
+    --serial "${SERIAL}" \
+    --out "${out_f}" \
+    --out_d "${out_d}" ||
+    fail "Failed to configure KVM instance for building"
+
+[ -n "${cloud_init_cfg}" ] && ci_cfg="${kvm_builder}/config/${cloud_init_cfg}"
+
+# Launch KVM to do the work
+launch_kvm.sh \
+    --id ${BUILD_ID} \
+    --user-data "${out_f}" \
+    --cloud-config "${ci_cfg}" \
+    --extra-disk "${query_t}" \
+    --disk-gb 50 \
+    --raw-disk "${raw_f}" \
+    --raw-size 20 \
+    --img-url ${BUILDER_CLOUD_IMAGE} ||
+        fail "KVM instance failed to build image."
+
+# Extract the result set
+tar -xvvf "${raw_f}" ||
+    fail "Failed to extract information from instance"
+
+# Useful for off-host builds, like ppc64el. Just make sure that any-off host
+# builds are done before the on-hosts builds.
+[ "${BUILD_ONLY:-0}" -eq 1 ] && exit 0
+
+# Extracted reslts should be here
+[ ! -e "${WORKSPACE}/${out_d}" ] && fail "Expected result directory is missing: ${WORKSPACE}/${out_d}"
+
+# Checksum the results (and sign 'em)
+export CDIMAGE_ROOT="/srv/builder/vmbuilder/cdimage"
+/srv/builder/vmbuilder/bin/cronrun checksum-directory "${WORKSPACE}/${out_d}" ||
+    fail "Failed to create checksums and GPG signatures"
+
+set -x
+# Put the bits where they go...
+mkdir -p "${final_out_d}" &&
+    cp -a ${WORKSPACE}${out_d}/* "${final_out_d}" &&
+    echo "Copied bits to final location ${final_out_d}" ||
+    fail "Unable to copy build bits to final location"
+
+# Produce build-info
+cat << EOF > "${final_out_d}/build-info.txt"
+serial=${SERIAL}
+orig_prefix=${SUITE}-ephemeral-maas
+suite=${SUITE}
+build_name=ephemeral
+EOF
+
+# Clean up the dailies
+if [ "${STREAM}" = "daily" ]; then
+    base_d="${out_d%/*}"
+    builds=( $(find ${base_d} -maxdepth 1 -mindepth 1 -type d | sort -r) )
+    build_count=${#builds[@]}
+
+    # Delete all but the 
+    if [ ${build_count} -gt 6 ]; then
+        for item in $(seq 6 ${build_count})
+        do
+            [ -e "${builds[$item]}" ] && {
+                rm -rf ${builds[$item]};
+                echo "Build ${SUITE} ${builds[$item]##*/} has been deleted";
+            }
+        done
+
+        for item in $(seq 0 5)
+        do
+                echo "Preserving ${SUITE} ${builds[$item]##*/}"
+        done
+    else
+        echo "No builds marked for deletion"
+    fi
+fi
+
+# Generate the Query2 tree
+src_tree="${WORKSPACE}/maas_src"
+bzr branch "${maas_branch}" "${src_tree}"
+${src_tree}/tree2query \
+    --commit-msg "Build ${BUILD_ID}" \
+    --namespace maas \
+    /srv/maas-images
+
+# Update current
+if [ "${STREAM}" = "daily" ]; then
+    cur_d="/srv/maas-images/ephemeral/daily/${SUITE}/current"
+    [ -e "${cur_d}" ] && rm "${cur_d}"
+    ln -s "${final_out_d}" "${cur_d}" ||
+        echo "Failed to update ${cur_d}"
+fi
+
+
+# Remove the results
+rm "${raw_f}" ||
+    fail "Failed to clean up files!"
+

=== added file 'jenkins/MAAS_Promotion.sh'
--- jenkins/MAAS_Promotion.sh	1970-01-01 00:00:00 +0000
+++ jenkins/MAAS_Promotion.sh	2018-05-31 04:33:07 +0000
@@ -0,0 +1,31 @@
+#!/bin/bash
+
+fail() { [ $# -eq 0 ] || echo "$@"; exit 1; }
+
+if [ "${TAG}" == "release" ]; then
+    TAG="release-${SERIAL}"
+fi
+
+src_d="/srv/maas-images/ephemeral/daily/${SUITE}/${SERIAL}"
+final_out_d="/srv/maas-images/ephemeral/releases/${SUITE}/${TAG}"
+
+[ -e ${src_d} ] ||
+    fail "Source ${src_d} does not exist"
+
+[ -e ${final_out_d} ] &&
+    fail "Serial has already been promoted"
+
+mkdir -p "${final_out_d}" &&
+    rsync -a ${src_d}/ ${final_out_d} &&
+    echo "Copied bits to final location ${final_out_d}" ||
+    fail "Unable to copy build bits to final location"
+
+# Generate the Query2 tree
+export maas_branch="${MAAS_BRANCH:-http://bazaar.launchpad.net/~smoser/maas/maas.ubuntu.com.images-ephemeral}"
+src_tree="${WORKSPACE}/maas_src"
+bzr branch "${maas_branch}" "${src_tree}"
+${src_tree}/tree2query \
+    --commit-msg "Build ${BUILD_ID}" \
+    --namespace maas \
+    /srv/maas-images
+

=== added file 'jenkins/MAASv2_Builder.sh'
--- jenkins/MAASv2_Builder.sh	1970-01-01 00:00:00 +0000
+++ jenkins/MAASv2_Builder.sh	2018-05-31 04:33:07 +0000
@@ -0,0 +1,191 @@
+#!/bin/bash
+set -x
+fail() { [ $# -eq 0 ] || echo "$@"; exit 1; }
+
+find . -iname "*build_properties" -exec cp {} . \; ||
+    echo "Unable to copy build properties, this might be v2"
+
+[ -z "${SERIAL}" -a -z "${SUITE}" -a -e "build_properties" ] && {
+    source build_properties ||
+    fail "Failed to read build_properties. I don't know what I'm doing!";
+}
+
+STREAM="${STREAM:-daily}"
+# Read in the common functions
+my_dir="$( cd "$( dirname "$0" )" && pwd )"
+base_dir=$(dirname ${my_dir})
+export PATH="${base_dir}:${my_dir}:${PATH}"
+source "${base_dir}/functions/locker"
+source "${base_dir}/functions/common"
+source "${base_dir}/functions/retry"
+source ${my_dir}/build_lib.sh
+select_build_config
+
+export WORKSPACE="${WORKSPACE:-$WORKSPACE_R}"
+out_f="${WORKSPACE}/maas-${SUITE}-${STREAM}-config.sh"
+raw_f="${WORKSPACE}/${SUITE}-output.raw"
+query_t="${WORKSPACE}/cloud-images-query.tar"
+base_name="${SUITE}-server-cloudimg"
+rel_base_name="ubuntu-${VERSION:-$(ubuntu-adj2version ${SUITE})}-${stream//-/}-server-cloudimg"
+
+export maas_branch_v1="http://bazaar.launchpad.net/~smoser/maas/maas.ubuntu.com.images-ephemeral"
+export maas_branch="${MAAS_BRANCH:-$maas_branch_v1}"
+
+case "${STREAM}" in
+    release)  build_f="/srv/ec2-images/releases/${SUITE}/release-${SERIAL}";
+              base_name=${rel_base_name};
+              out_d="/srv/maas-images/ephemeral/releases/${SUITE}/release-${SERIAL}"
+              ;;
+    daily)    build_f="/srv/ec2-images/${SUITE}/${SERIAL}";
+              out_d="/srv/maas-images/ephemeral/daily/${SUITE}/${SERIAL}";
+              ;;
+    alpha*|beta*)  build_f="/srv/ec2-images/releases/${SUITE}/${STREAM}";
+              base_name=${rel_base_name};
+              out_d="/srv/maas-images/ephemeral/releases/${SUITE}/${STREAM}";
+              ;;
+    *)        fail "Unknown stream ${STREAM}.";;
+esac
+
+final_out_d="${out_d}"
+
+[ -e "${final_out_d}" -a "${REBUILD:-false}" = "false" ] &&
+    fail "Build already exists. Rebuild is set to false. Failing this build"
+
+if [ ! -e "${query_t}" ]; then
+
+    if  [ "${MAASv2:-0}" -eq 1 ]; then
+        # MAAS v2 doesn't need this information
+        out_d="/tmp/maas_final"
+        touch ${WORKSPACE}/maasv2
+        tar cvf ${query_t} ${WORKSPACE}/maasv2
+
+        if  [ -e "${WORKSPACE}/tmp/maas-final" ]; then
+            tar cvf ${query_t} maas-final ||
+                fail "Failed to create tarball of MAAS images"
+        fi
+
+    else
+        # MAAS v1 need information
+        tar cvf ${query_t} \
+            ${QUERY_D:-/srv/ec2-images/query} \
+            ${build_f}  \
+            --exclude "*img" --exclude "*azure*" --exclude "*html" \
+            --exclude "*armel*" --exclude "*root.tar.gz" \
+            --exclude "*floppy" ||
+                fail "Failed to pack up build elements for MAAS builder";
+    fi
+fi
+
+# Select the right template
+tmpl="${kvm_builder}/templates/img-maas.tmpl"
+[ "${MAASv2:-0}" -eq 1 ] && tmpl="${tmpl//maas.tmpl/maasv2.tmpl}"
+
+# Construct the right template
+maas_config.sh \
+    --distro "${SUITE}" \
+    --stream "${STREAM}" \
+    --template "${tmpl}" \
+    --base-name "${base_name}" \
+    --local "${build_f}" \
+    --serial "${SERIAL}" \
+    --out "${out_f}" \
+    --maas-branch "${maas_branch}" \
+    --out_d "${out_d}" ||
+    fail "Failed to configure KVM instance for building"
+set +x
+
+ci_cfg="${kvm_builder}/config/cloud-maasv2.cfg"
+[ "$(uname -m)" == "ppc64" ] && ci_cfg="${kvm_builder}/config/cloud-trusty-pp64el.cfg"
+
+# Launch KVM to do the work
+launch_kvm.sh \
+    --id ${BUILD_ID} \
+    --user-data "${out_f}" \
+    --cloud-config "${ci_cfg}" \
+    --extra-disk "${query_t}" \
+    --disk-gb 50 \
+    --raw-disk "${raw_f}" \
+    --raw-size 20 \
+    --mem 1G \
+    --img-url ${BUILDER_CLOUD_IMAGE} ||
+        fail "KVM instance failed to build image."
+
+# Extract the result set
+tar -xvvf "${raw_f}" ||
+    fail "Failed to extract information from instance"
+
+# Useful for off-host builds, like ppc64el. Just make sure that any-off host
+# builds are done before the on-hosts builds.
+
+[ "${BUILD_ONLY:-0}" -eq 1 ] && exit 0
+[ "${MAASv2:-0}" -eq 1 ] && exit 0
+
+# Extracted reslts should be here
+[ ! -e "${WORKSPACE}/${out_d}" ] && fail "Expected result directory is missing: ${WORKSPACE}/${out_d}"
+
+# Checksum the results (and sign 'em)
+export CDIMAGE_ROOT="/srv/builder/vmbuilder/cdimage"
+/srv/builder/vmbuilder/bin/cronrun checksum-directory "${WORKSPACE}/${out_d}" ||
+    fail "Failed to create checksums and GPG signatures"
+
+set -x
+# Put the bits where they go...
+mkdir -p "${final_out_d}" &&
+    cp -a ${WORKSPACE}${out_d}/* "${final_out_d}" &&
+    echo "Copied bits to final location ${final_out_d}" ||
+    fail "Unable to copy build bits to final location"
+
+# Produce build-info
+cat << EOF > "${final_out_d}/build-info.txt"
+serial=${SERIAL}
+orig_prefix=${SUITE}-ephemeral-maas
+suite=${SUITE}
+build_name=ephemeral
+EOF
+
+# Clean up the dailies
+if [ "${STREAM}" = "daily" ]; then
+    base_d="${out_d%/*}"
+    builds=( $(find ${base_d} -maxdepth 1 -mindepth 1 -type d | sort -r) )
+    build_count=${#builds[@]}
+
+    # Delete all but the
+    if [ ${build_count} -gt 6 ]; then
+        for item in $(seq 6 ${build_count})
+        do
+            [ -e "${builds[$item]}" ] && {
+                rm -rf ${builds[$item]};
+                echo "Build ${SUITE} ${builds[$item]##*/} has been deleted";
+            }
+        done
+
+        for item in $(seq 0 5)
+        do
+                echo "Preserving ${SUITE} ${builds[$item]##*/}"
+        done
+    else
+        echo "No builds marked for deletion"
+    fi
+fi
+
+# Generate the Query2 tree
+src_tree="${WORKSPACE}/maas_src"
+bzr branch "${maas_branch_v1}" "${src_tree}"
+${src_tree}/tree2query \
+    --commit-msg "Build ${BUILD_ID}" \
+    --namespace maas \
+    /srv/maas-images
+
+# Update current
+if [ "${STREAM}" = "daily" ]; then
+    cur_d="/srv/maas-images/ephemeral/daily/${SUITE}/current"
+    [ -e "${cur_d}" ] && rm "${cur_d}"
+    ln -s "${final_out_d}" "${cur_d}" ||
+        echo "Failed to update ${cur_d}"
+fi
+
+
+# Remove the results
+rm "${raw_f}" ||
+    fail "Failed to clean up files!"
+

=== added file 'jenkins/MAASv2_Cleaner.sh'
--- jenkins/MAASv2_Cleaner.sh	1970-01-01 00:00:00 +0000
+++ jenkins/MAASv2_Cleaner.sh	2018-05-31 04:33:07 +0000
@@ -0,0 +1,55 @@
+#!/bin/bash
+#
+# Clean up MAAS2 v2/v3 builds/streams
+#
+my_dir="$(dirname $0)"
+my_p_dir="$(dirname $my_dir)"
+source ${my_p_dir}/functions/common
+source ${my_p_dir}/functions/bzr_check.sh
+
+# Number of builds to publish in the stream
+MAX_BUILDS=${MAX_BUILDS:-3}
+# Number of days to keep files not referenced in the stream data
+REAP_AGE=${REAP_AGE:-2d}
+
+WORKSPACE=${WORKSPACE:-$PWD}
+OUTDIR=${JENKINS_HOME:?}/.config/MAASv2_Cleaner/
+DAILY_ROOT=/srv/maas-images/ephemeral-v2/daily/
+RELEASE_ROOT=/srv/maas-images/ephemeral-v2/releases/
+INDEX_PATH=streams/v1/index.json
+
+# Local checkouts
+sstreams=${WORKSPACE}/sstreams
+maasv2=${WORKSPACE}/maasv2
+check_branch ${BZR_SIMPLESTREAMS:-lp:simplestreams} ${sstreams}
+check_branch ${BZR_MAASv2:-lp:maas-images} ${maasv2}
+
+for METADATA_ROOT in /srv/maas-images/ephemeral-v2/daily/ \
+                     /srv/maas-images/ephemeral-v3/daily/;  do
+    case $METADATA_ROOT in
+        *v2*)
+            orphan_json="${OUTDIR}/daily.json"
+            ;;
+        *v3*)
+            orphan_json="${OUTDIR}/daily-v3.json"
+            ;;
+        *)
+            echo "Unexpected METADATA_ROOT"
+            exit 1
+            ;;
+    esac
+    run PYTHONPATH=${sstreams}:${maasv2} \
+        ${maasv2}/bin/meph2-util clean-md \
+            ${MAX_BUILDS} ${METADATA_ROOT}/${INDEX_PATH}
+
+    run PYTHONPATH=${sstreams}:${maasv2} \
+        ${maasv2}/bin/meph2-util find-orphans \
+            "${orphan_json}" \
+            ${METADATA_ROOT} ${METADATA_ROOT}/${INDEX_PATH}
+
+    run PYTHONPATH=${sstreams}:${maasv2} \
+        ${maasv2}/bin/meph2-util reap-orphans \
+            --older ${REAP_AGE} \
+            "${orphan_json}" \
+            ${METADATA_ROOT}
+done

=== added file 'jenkins/MAASv3_Builder.sh'
--- jenkins/MAASv3_Builder.sh	1970-01-01 00:00:00 +0000
+++ jenkins/MAASv3_Builder.sh	2018-05-31 04:33:07 +0000
@@ -0,0 +1,67 @@
+#!/bin/bash -x
+
+fail() { [ $# -eq 0 ] || echo "$@"; exit 1; }
+
+[ -z "${SERIAL}" -a -z "${SUITE}" -a -e "build_properties" ] && {
+    source build_properties ||
+    fail "Failed to read build_properties.";
+}
+
+# Read in the common functions
+my_dir="$( cd "$( dirname "$0" )" && pwd )"
+base_dir=$(dirname ${my_dir})
+export PATH="${base_dir}:${my_dir}:${PATH}"
+source "${base_dir}/functions/locker"
+source "${base_dir}/functions/common"
+source "${base_dir}/functions/retry"
+source ${my_dir}/build_lib.sh
+select_build_config
+
+case "${STREAM:?}" in
+    daily)    build_f="/srv/ec2-images/${SUITE}/${SERIAL}";
+              base_name="${SUITE}-server-cloudimg"
+              ;;
+    *)        fail "Unknown/unsupported stream ${STREAM}.";;
+esac
+
+export WORKSPACE="${WORKSPACE:-$WORKSPACE_R}"
+out_f="${WORKSPACE:?}/maas-${SUITE}-${STREAM}-config.sh"
+raw_f="${WORKSPACE}/${SUITE}-output.raw"
+
+export maas_branch="${MAAS_BRANCH:?}"
+
+touch ${WORKSPACE}/maasv3
+tar cvf ${query_t} ${WORKSPACE}/maasv3
+
+# Construct the right template
+maas_config.sh \
+    --distro "${SUITE}" \
+    --stream "${STREAM}" \
+    --template "${kvm_builder}/templates/img-maasv3.tmpl" \
+    --base-name "${base_name}" \
+    --local "${build_f}" \
+    --serial "${SERIAL}" \
+    --out "${out_f}" \
+    --maas-branch "${maas_branch}" \
+    --out_d "/tmp/maas_final" ||
+    fail "Failed to configure KVM instance for building"
+
+ci_cfg="${kvm_builder}/config/cloud-maasv3.cfg"
+[ "$(uname -m)" == "ppc64" ] && ci_cfg="${kvm_builder}/config/cloud-trusty-pp64el.cfg"
+
+# Launch KVM to do the work
+launch_kvm.sh \
+    --id ${BUILD_ID} \
+    --user-data "${out_f}" \
+    --cloud-config "${ci_cfg}" \
+    --extra-disk "${query_t}" \
+    --disk-gb 50 \
+    --raw-disk "${raw_f}" \
+    --raw-size 20 \
+    --mem 1G \
+    --img-url ${BUILDER_CLOUD_IMAGE} ||
+        fail "KVM instance failed to build image."
+
+# Extract the result set
+tar -xvvf "${raw_f}" ||
+    fail "Failed to extract information from instance"

=== added file 'jenkins/Promote_Daily.sh'
--- jenkins/Promote_Daily.sh	1970-01-01 00:00:00 +0000
+++ jenkins/Promote_Daily.sh	2018-05-31 04:33:07 +0000
@@ -0,0 +1,55 @@
+#!/bin/bash
+echo "---------------------------------------------------"
+echo "Instructed to Promote Daily job:
+echo "       Suite: ${SUITE}"
+echo "      Serial: ${SERIAL}"
+echo "   Milestone: ${MILESTONE_LABEL}"
+echo "      Stream: ${BTYPE}"
+echo "      Public: ${MAKE_PUBLIC}"
+echo "  PrePublish: ${PREPUBLISH}"
+echo "
+echo "---------------------------------------------------"
+
+cat << EOF > "${WORKSPACE}/build_properties"
+SUITE=${SUITE}
+SERIAL=${SERIAL}
+MILESTONE=${MILESTONE_LABEL}
+STREAM=${BTYPE}
+PUBLIC=${MAKE_PUBLIC}
+PREPUBLISH=${PREPUBLISH}
+EOF
+
+export HOME="/srv/builder/vmbuilder"
+
+cmd=(
+   '/srv/builder/vmbuilder/bin/cronrun'
+   'promote-daily'
+   '--verbose'
+   '--allow-existing' )
+
+if [ "${PREPUBLISH}" == "true" ]; then
+    echo "Pre-publishing rules, will not make public"
+else
+    [ "${MAKE_PUBLIC}" == "true" ] && cmd+=('--make-public')
+fi
+
+case ${BTYPE} in
+    *server*hwe*) pub_path="/srv/ec2-images/server/${SUITE}/${SERIAL}/${BTYPE//server-/}";;
+               *) pub_path="/srv/ec2-images/${BTYPE}/${SUITE}/${SERIAL}"
+            ;;
+esac
+
+if [ "${REPUBLISH}" == "true" ]; then
+    cmd+=('--republish')
+    if [ "${MILESTONE_LABEL}" == "release" ]; then
+        pub_path="/srv/ec2-images/releases/${SUITE}/release-${SERIAL}"
+    else
+        pub_path="/srv/ec2-images/releases/${SUITE}/${MILESTONE_LABEL}"
+    fi
+    [[ "${BTYPE}" =~ server-hwe ]] && pub_path="${pub_path}/${BTYPE//server-/}"
+fi
+
+cmd+=("${MILESTONE_LABEL}" ${pub_path})
+
+echo "Executing command: ${cmd[@]}"
+exec ${cmd[@]}

=== added file 'jenkins/Promote_MAAS_Daily.sh'
--- jenkins/Promote_MAAS_Daily.sh	1970-01-01 00:00:00 +0000
+++ jenkins/Promote_MAAS_Daily.sh	2018-05-31 04:33:07 +0000
@@ -0,0 +1,48 @@
+#!/bin/bash
+
+export maas_branch="${MAAS_BRANCH:-http://bazaar.launchpad.net/~smoser/maas/maas.ubuntu.com.images-ephemeral}"
+
+fail() { [ $# -eq 0 ] || echo "$@"; exit 1; }
+
+[ -z "${SERIAL}" ] && fail "Serial must be defined"
+[ -z "${SUITE}" ] && fail "Suite must be defined"
+[ -z "${MILESTONE}" ] && fail "Milestone must be defined"
+
+cp_d="/srv/maas-images/ephemeral/daily/${SUITE}/${SERIAL}"
+finald="/srv/maas-images/ephemeral/releases/${SUITE}"
+
+case "${MILESTONE}" in
+    release)    final_d="${finald}/release-${SERIAL}"
+    alpha|beta) final_d="${finald}/${milestone}"
+esac
+
+# Sanity check
+[ ! -e "${cp_d}" ] && fail "Serial ${SERIAL} for ${SUITE} does not exist"
+[ -e "${final_d}" ] && fail "Already released ${SERIAL} for ${SUITE} as ${MILESTONE}"
+
+# Make the home directory
+mkdir -p "${final_d}" ||
+    fail "Unable to create final destination"
+
+# Put the files in final destination
+cp -au ${cp_d}/* "${final_d}" ||
+    fail "Failed to copy source files for promotion"
+
+# Generate the Query2 tree
+src_tree="${WORKSPACE}/maas_src"
+bzr branch "${maas_branch}" "${src_tree}"
+${src_tree}/tree2query \
+    --commit-msg "Build ${BUILD_ID}" \
+    --namespace maas \
+    /srv/maas-images
+
+# Update the "release" link
+if [ "${MILESTONE}" = "release" ]; then
+    cur_d="/srv/maas-images/ephemeral/releases/${SUITE}/release"
+    [ -e "${cur_d}" ] && rm "${cur_d}"
+    ln -s "${final_d}" "${cur_d}" ||
+        echo "Failed to update ${cur_d}"
+fi
+
+# Sync the stuff
+KEY=maas /srv/builder/vmbuilder/bin/trigger-sync

=== added file 'jenkins/Publish_EC2.sh'
--- jenkins/Publish_EC2.sh	1970-01-01 00:00:00 +0000
+++ jenkins/Publish_EC2.sh	2018-05-31 04:33:07 +0000
@@ -0,0 +1,64 @@
+#!/bin/bash -x
+
+# Add in the retry stub
+source "${kvm}/functions/retry"
+source "${kvm}/functions/common"
+
+# Exit if trigger job does not want this published
+[ "${PUBLISH_IMAGE}" -eq 0 ] && exit 0
+
+# Set the build directories
+WORK_D="/srv/ec2-images/${BUILD_TYPE}/${SUITE}/${SERIAL}"
+[ "${TEST_BUILD}" -eq 1 ] && WORK_D="/srv/ec2-images/test_builds/${BUILD_TYPE}/${SUITE}/${SERIAL}"
+[ "${SANDBOX_BUILD}" -eq 1 ] && WORK_D="/srv/ec2-images/sandbox/${BUILD_TYPE}/${SUITE}/${SERIAL}"
+
+# Handle the special case of HWE builds. Otherwise they get the names of server
+# and things don't go well.
+[[ "${HWE_SUFFIX}" =~ hwe ]] &&
+    WORK_D="${WORK_D}/${HWE_SUFFIX}" &&
+    BUILD_TYPE="${BUILD_TYPE}-${HWE_SUFFIX}"
+
+
+echo "Using ${WORK_D} as the directory"
+[ -e "${WORK_D}" ] || { echo "Working directory does not exist!"; exit 1; }
+
+ec2_pub="${PWD}/ec2-publishing-scripts"
+
+# Check out the scripts needed
+[ -e "${ec2_pub}" ] && rm -rf "${ec2_pub}"
+bzr branch "${EC2_PUB_SCRIPTS}" "${ec2_pub}"
+
+# Add some elements to the path
+VMBUILDER_PATH="${VMBUILDER_PATH:-/srv/builder/vmbuilder}"
+VMBUILDER_BIN="${VMBUILDER_PATH}/bin"
+XC2_PATH="${VMBUILDER_PATH}/ec2-daily/xc2"
+export PUBLISH_SCRIPTS="${PUBLISH_SCRIPTS:-$VMBUILDER_PATH/ec2-publishing-scripts}"
+export PATH="${VMBUILDER_BIN}:${VMBUILDER_BIN}:${VMBUILDER_PATH}:${XC2_PATH}:${PATH}"
+export HOME="/srv/builder/vmbuilder"
+export CDIMAGE_ROOT="${CDIMAGE_ROOT:-/srv/builder/cdimage}"
+export EC2_PUB_LOC="${ec2_pub}"
+
+ec2publish() {
+    # Run the publisher job
+    ${kvm}/ec2_publisher.sh \
+           ${SUITE} \
+           ${SERIAL} \
+           ${BUILD_TYPE} \
+           ${WORK_D} \
+           ${TEST_BUILD} \
+           ${SANDBOX_BUILD} \
+           ${ALLOW_EXISTING}
+}
+
+# Retry the publishing up to 3 times
+retry 6 120 ec2publish ||
+    fail "Failed three attempts to publish EC2 images!"
+
+# Add the new daily to the tracker
+#exec_tracker=${ADD_TO_TRACKER:-0}
+#[ "${exec_tracker}" -eq 1 ] && {
+#    ${kvm}/tracker.sh daily ${SUITE} ${SERIAL} &&
+#         exit $? || fail "Unable to execute tracker!"
+#    }
+#
+#exit 0

=== added file 'jenkins/Publish_Results_to_Tracker.sh'
--- jenkins/Publish_Results_to_Tracker.sh	1970-01-01 00:00:00 +0000
+++ jenkins/Publish_Results_to_Tracker.sh	2018-05-31 04:33:07 +0000
@@ -0,0 +1,34 @@
+#!/bin/bash
+
+# Environmental variables:
+#  HOST: the Jenkins host URL to poll from
+#  SUITE: Ubuntu codename
+#  MILESTONE: i.e. Alpha 2
+#  SERIAL: What is the build serial, i.e 20130213
+#  OUT: File to execute
+
+set -x
+
+# Setup the QA tracker code
+bzr branch http://bazaar.launchpad.net/~jibel/+junk/qatracker
+cd qatracker
+sed -i "s/iso.qa.ubuntu.com/cloud.qa.ubuntu.com/g" tracker_update_result
+
+bzr branch http://bazaar.launchpad.net/~ubuntu-qa-website-devel/ubuntu-qa-website/python-qatracker
+ln -s python-qatracker/qatracker.py .
+export PATH="${PATH}:${WORKSPACE}/qatracker"
+
+# Get the actual working script
+${scripts}/tests/tracker.py \
+     --host ${HOST} \
+     --suite ${SUITE} \
+     --test ${TEST} \
+     --milestone "${MILESTONE}" \
+     --serial ${SERIAL} \
+     --out "${WORKSPACE}/script.sh"
+
+# Execute the script
+env API_USER="${API_USER}" \
+    API_KEY="${API_KEY}" \
+    bash ${WORKSPACE}/script.sh ||
+    exit 1 | tee publish.log

=== added file 'jenkins/README.txt'
--- jenkins/README.txt	1970-01-01 00:00:00 +0000
+++ jenkins/README.txt	2018-05-31 04:33:07 +0000
@@ -0,0 +1,1 @@
+This directory contains the jobs that Jenkins executes. Most of the jobs just setup up an environmental component and then call another script, usually one directory below

=== added file 'jenkins/Test_Azure.sh'
--- jenkins/Test_Azure.sh	1970-01-01 00:00:00 +0000
+++ jenkins/Test_Azure.sh	2018-05-31 04:33:07 +0000
@@ -0,0 +1,17 @@
+#!/bin/bash
+fail() { [ $# -eq 0 ] || echo "$@"; exit 1; }
+
+umask 022
+set -x
+source watch_properties || fail "Failed to read watch properties"
+
+echo "-------------------"
+echo "Image for testing:"
+cat watch_properties
+echo "-------------------"
+
+
+my_dir="$( cd "$( dirname "$0" )" && pwd )"
+base_dir=$(dirname ${my_dir})
+
+${my_dir}/tests/azure.sh ${1}

=== added file 'jenkins/build_lib.sh'
--- jenkins/build_lib.sh	1970-01-01 00:00:00 +0000
+++ jenkins/build_lib.sh	2018-05-31 04:33:07 +0000
@@ -0,0 +1,33 @@
+#!/bin/bash
+
+# set default umask
+umask 022
+
+# Read in the common functions
+my_dir="$( cd "$( dirname "$0" )" && pwd )"
+base_dir=$(dirname ${my_dir})
+export PATH="${base_dir}:${my_dir}:${PATH}"
+source "${base_dir}/functions/locker"
+source "${base_dir}/functions/common"
+source "${base_dir}/functions/retry"
+
+dist_ge() { [[ "$1" > "$2" || "$1" == "$2" ]]; }
+
+[ -z "${DISTRO}" -a -n "${SUITE}" ] && DISTRO="${SUITE}"
+
+select_build_config() {
+
+    [ -z "${BUILDER_CLOUD_IMAGE}" ] && {
+        # Use the latest 14.04 LTS image to do the build.
+        BUILDER_CLOUD_IMAGE="http://cloud-images.ubuntu.com/releases/trusty/release/ubuntu-14.04-server-cloudimg-amd64-uefi1.img"
+        export cloud_init_cfg="cloud-trusty.cfg"
+       }
+
+    # For ppc64el, we use ppc64el images
+    [ "${ARCH_TYPE}" == "ppc64el" ] && {
+            export cloud_init_cfg="cloud-trusty-pp64el.cfg"
+            BUILDER_CLOUD_IMAGE="${BUILDER_CLOUD_IMAGE//amd64/ppc64el}"
+            export BUILDER_CLOUD_IMAGE="${BUILDER_CLOUD_IMAGE//uefi1/disk1}"
+        }
+    echo "Using ${BUILDER_CLOUD_IMAGE} to do the build"
+}

=== added file 'jenkins/env-test.sh'
--- jenkins/env-test.sh	1970-01-01 00:00:00 +0000
+++ jenkins/env-test.sh	2018-05-31 04:33:07 +0000
@@ -0,0 +1,2 @@
+#!/bin/bash
+env

=== added file 'launch_kvm.sh'
--- launch_kvm.sh	1970-01-01 00:00:00 +0000
+++ launch_kvm.sh	2018-05-31 04:33:07 +0000
@@ -0,0 +1,222 @@
+#!/bin/bash
+usage() {
+cat << EOF
+This program is a KVM wrapper for performing tasks inside a KVM Environment.
+Its primary goal is to help developers do dangerous tasks that their IS/IT
+deparment won't allow them to do on an existing machine.
+    --id <ARG>           The ID you want to use to identify the KVM image
+                         this is used to name the image
+    --disk-gb <ARG>      Disk size you want to resize the image too
+                         Default it to _add_ 30GB
+    --smp <ARG>          KVM SMP options, defaults to:
+                         ${smp_opt}
+    --mem <ARG>          How much RAM do you want to use
+    --user-data <ARG>    Cloud-Init user-data file
+    --cloud-config <ARG> Cloud-Init cloud-config file
+    --img-url <ARG>      Location of the image file.
+    --raw-disk <ARG>     Name of RAW disk to create and attach.
+    --raw-size <ARG>     Size of RAW disk in GB.
+    --extra-disk <ARG>   Add an extra disk, starting with /dev/vdd
+    --cloud-init-file <ARG> Additional file for the cloud-init data
+EOF
+exit 1
+}
+
+short_opts="h"
+long_opts="id:,ssh_port,disk-gb:,mem:,bzr-automated-ec2-builds:,cloud-config:,user-data:,kernel-url:,img-url:,raw-disk:,raw-size:,smp:,extra-disk:,cloud-init-file:,help"
+getopt_out=$(getopt --name "${0##*/}" \
+    --options "${short_opts}" --long "${long_opts}" -- "$@") &&
+    eval set -- "${getopt_out}" ||
+    usage
+
+builder_id=$(uuidgen)
+uuid=${builder_id}
+bname="server"
+size_gb=15
+mem=512
+smp_opt="4"
+ud=""
+cloud_config=""
+img_loc="${BUILDER_CLOUD_IMAGE:-http://cloud-images.ubuntu.com/releases/precise/release/ubuntu-12.04-server-cloudimg-amd64-disk1.img}"
+KVM_PID=""
+
+while [ $# -ne 0 ]; do
+    cur=${1}; next=${2};
+    case "$cur" in
+    --id)                       id="$2"; shift;;
+    --disk-gb)                  size_gb="$2"; shift;;
+    --mem)                      mem="$2"; shift;;
+    --cloud-config)             ccloud="$2"; shift;;
+    --user-data)                ud="$2"; shift;;
+    --img-url)                  img_loc="$2"; shift;;
+    --raw-disk)                 raw_disk="$2"; shift;;
+    --raw-size)                 raw_size="$2"; shift;;
+    --smp)                      smp_opts="$2"; shift;;
+    --extra-disk)               [ -z "${extra_disk}" ] && extra_disk=$2 || extra_disk="${extra_disk} $2"; shift;;
+    --cloud-init-file)          [ -z "${cloud_init_files}" ] && cloud_init_files=$2 || cloud_init_files="${cloud_init_files} $2"; shift;;
+    -h|--help)                  usage; exit 0;;
+    --) shift; break;;
+  esac
+  shift;
+done
+
+work_d="$(mktemp -d /tmp/kvm-builder.XXXX)"
+kvm_pidfile="$(mktemp --tmpdir=${work_d})"
+
+error() { echo "$@" 1>&2; }
+cleanup() {
+        [ -n "${KVM_PID}" ] && kill -9 ${KVM_PID};
+        [ -n "${TAIL_PID}" ] && kill -9 ${TAIL_PID};
+        rm -rf "${work_d}";
+}
+fail() { error "$@"; cleanup;  exit 1; }
+debug() { error "$(date -R):" "$@"; }
+sysfail() { fail "Failure in commands detected; purging "; }
+
+# Make sure that we kill everything
+trap sysfail SIGINT SIGTERM
+
+[ -z "${ud}" ] && fail "Must define user-data script via --user-data"
+[ -z "${ccloud}" ] && fail "Must define cloud-config script via --cloud-config"
+
+debug "Creating Cloud-Init configuration..."
+write_mime_args=(
+    -o "${work_d}/user-data.txt"
+    "${ccloud}"
+    "${ud}")
+write_mime_args+=(${cloud_init_files[@]})
+write_mime_location="$(which write-mime-multipart)"
+if which python3 > /dev/null; then
+    "${write_mime_location}" ${write_mime_args[@]} || fail "Unable to create user-data"
+else
+    python "${write_mime_location}" ${write_mime_args[@]} || fail "Unable to create user-data"
+fi
+
+echo "instance-id: $(uuidgen)" > "${work_d}/meta-data"
+echo "local-hostname: builder" >> "${work_d}/meta-data"
+
+debug "Creating Seed for Cloud-Init..."
+"${0%/*}/make-seed.sh" "${work_d}/seed.img" "${work_d}/user-data.txt" "${work_d}/meta-data" ||
+    fail "Failed to create Configruation ISO"
+
+# Place the image in place
+debug "Build image location is ${img_loc}"
+if [[ "${img_loc}" =~ "http" ]]; then
+    debug "Fetching cloud image from ${img_loc}"
+    curl -s -o "${work_d}/img-${builder_id}" "${img_loc}" ||
+        fail "Unable to fetch pristine image from '${img_loc}'"
+else
+    cp "${img_loc}" "${work_d}/img-${builder_id}" ||
+        fail "Unable to copy '${img_loc}'"
+fi
+
+debug "Adding ${size_gb}G to image size"
+qemu-img resize "${work_d}/img-${builder_id}" +"${size_gb}G" ||
+    fail "Unable to resize image to ${size_gb}G"
+
+if [ -n "${raw_disk}" -a ! -e "${raw_disk}" ]; then
+    if [ -n "${raw_size}" ]; then
+        dd if=/dev/zero of=${raw_disk} bs=1k count=1 seek=$((${raw_size} * 1024000)) &&
+            debug "Create new raw disk" ||
+            fail "Unable to create raw disk"
+    else
+        fail "Undefined raw disk size"
+    fi
+else
+    debug "Using existing raw disk."
+fi
+
+
+debug "________________________________________________"
+debug "Launching instance..."
+kvm_cmd=(
+   ${QEMU_COMMAND:-kvm}
+   -name ${uuid}
+   -drive file=${work_d}/img-${builder_id},if=virtio,bus=0,cache=unsafe,unit=0
+   -drive file=${raw_disk},if=virtio,format=raw,bus=0,unit=1
+   -drive file=${work_d}/seed.img,if=virtio,media=cdrom,bus=0,cache=unsafe,unit=2
+   -net nic,model=virtio
+   -net user
+   -no-reboot
+   -display none
+   -daemonize
+   -serial file:${work_d}/console.log
+   -pidfile ${kvm_pidfile}
+   )
+kvm_cmd+=(${QEMU_ARGS[@]})
+
+# Arch independant stuff
+if [[ "$(uname -p)" =~ "ppc64" ]]; then
+    # Use more memory for building on PPC64
+    kvm_cmd+=(-m 4G)
+else
+    kvm_cmd+=(-smp ${smp_opt} -m ${mem})
+fi
+
+# Allow for kernel and append
+if [ -n "${QEMU_KERNEL}" ]; then
+     root="/dev/vda1"
+     if [[ "$(uname -p)" =~ "ppc64" ]]; then
+         root="/dev/vda"
+     fi
+     kvm_cmd+=(-kernel ${QEMU_KERNEL}
+               -append "earlyprintk root=${root} console=hvc0"
+              )
+fi
+
+unit_c=3
+for disk in ${extra_disk}
+do
+    if [[ $(file ${disk}) =~ (disk|qcow|QCOW|vmdk|VMDK|vdi|VDI) ]]; then
+        debug "Adding extra disk $disk to KVM configuration"
+        kvm_cmd+=(-drive file=${extra_disk},if=virtio,bus=1,unit=${unit_c})
+    else
+        debug "Adding extra disk as a raw formated disk"
+        kvm_cmd+=(-drive file=${extra_disk},if=virtio,format=raw,bus=1,unit=${unit_c})
+    fi
+    unit_c=$((unit_c+1))
+done
+
+debug "KVM command is: ${kvm_cmd[@]}"
+"${kvm_cmd[@]}" ||
+    fail "Failed to launch KVM image\n${kvm_out}"
+
+read KVM_PID < ${kvm_pidfile}
+debug "KVM PID is: ${KVM_PID}"
+
+tail -f "${work_d}/console.log" &
+TAIL_PID=$!
+
+# Wait on the pid until the max timeout
+count=0
+max_count=${MAX_CYCLES:-720}
+while $(ps ${KVM_PID} > /dev/null 2>&1)
+do
+    sleep 10
+    count=$((count + 1))
+    if [ "${count}" -gt "${max_count}" ]; then
+        kill -15 ${KVM_PID}
+        debug "Build timed out...killing PID ${KVM_PID}"
+    fi
+done
+
+debug "________________________________________________"
+debug "KVM PID has ended. Work is done"
+kill -15 ${TAIL_PID}
+
+unset KVM_PID
+unset TAIL_PID
+
+[ -n "${raw_disk}" ] &&
+    debug "Extracting raw tarball" &&
+    { tar xvvf "${raw_disk}" || /bin/true; }
+
+[ ! -e success ] &&
+    fail "Tarball contents reported failure"
+
+cp "${work_d}/console.log" .
+
+# Wait for Cloud-Init to finish any work
+debug "Cleaning up..."
+cleanup
+exit 0

=== added file 'maas_config.sh'
--- maas_config.sh	1970-01-01 00:00:00 +0000
+++ maas_config.sh	2018-05-31 04:33:07 +0000
@@ -0,0 +1,75 @@
+#!/bin/bash
+short_opts="h"
+long_opts="distro:,stream:,maas-branch:,out:,template:,serial:,local:,base-name:,out_d:"
+getopt_out=$(getopt --name "${0##*/}" \
+    --options "${short_opts}" --long "${long_opts}" -- "$@") &&
+    eval set -- "${getopt_out}" || { echo "BAD INVOCATION!"; usage; exit 1; }
+
+usage() {
+    cat <<EOM
+${0##/} - Populated values in build temple.
+
+    Required:
+    --distro        Distro code name, i.e. precise
+    --template      Template file
+    --stream        Stream, i.e. daily, release
+    --base-name     The name of the file to work on
+    --serial        The build serial
+    --out           The output file
+    --out_d         Where to stuff the output files
+
+    Optional:
+    --maas-branch   bzr branch for maas image code
+EOM
+}
+
+
+fail() { echo "${@}" 2>&1; exit 1;}
+
+serial="${serial:-$(date +%Y%m%d)}"
+maas_branch="${maas_branch:-http://bazaar.launchpad.net/~smoser/maas/maas.ubuntu.com.images-ephemeral}"
+template_f="${PWD}/img-maas.tmpl"
+
+while [ $# -ne 0 ]; do
+  cur=${1}; next=${2};
+  case "$cur" in
+    --distro)                   distro=$2;  shift;;
+    --stream)                   stream=$2;  shift;;
+    --local)                    local_d=$2; shift;;
+    --maas-branch)              maas_branch=$2; shift;;
+    --base-name)                base_name=$2; shift;;
+    --template)                 template_f=$2; shift;;
+    --out)                      out_f=$2; shift;;
+    --out_d)                    out_d=$2; shift;;
+    --) shift; break;;
+  esac
+  shift;
+done
+
+fail_usage() { fail "Must define $@"; }
+
+[ -z "${distro}" ] && fail_usage "--distro"
+[ -z "${stream}" ] && fail_usage "--stream"
+[ -z "${local_d}" ] && fail_usage "--local"
+[ -z "${out_f}" ] && fail_usage "--out"
+[ -z "${out_d}" ] && fail_usage "--out_d"
+[ -z "${base_name}" ] && fail_usage "--base-name"
+
+case "$distro" in
+   trusty) arches="${ARCH_TYPE:-i386 amd64 armhf}";
+           [[ "$(uname -m)" =~ ppc64 ]] && arches="ppc64el";;
+   *) arches="${ARCH_TYPE:-i386 amd64 armhf}";;
+esac
+
+sed -e "s,%d,${distro},g" \
+    -e "s,%S,${stream},g" \
+    -e "s,%M,${maas_branch},g" \
+    -e "s,%D,${local_d},g" \
+    -e "s,%B,${base_name},g" \
+    -e "s,%s,${serial},g" \
+    -e "s,%O,${out_d},g" \
+    -e "s,%A,${arches},g" \
+    ${template_f} > ${out_f} ||
+        fail "Unable to write template file"
+
+exit 0

=== added file 'make-seed.sh'
--- make-seed.sh	1970-01-01 00:00:00 +0000
+++ make-seed.sh	2018-05-31 04:33:07 +0000
@@ -0,0 +1,147 @@
+#!/bin/bash
+
+VERBOSITY=0
+TEMP_D=""
+DEF_DISK_FORMAT="raw"
+DEF_FILESYSTEM="iso9660"
+
+error() { echo "$@" 1>&2; }
+errorp() { printf "$@" 1>&2; }
+fail() { [ $# -eq 0 ] || error "$@"; exit 1; }
+failp() { [ $# -eq 0 ] || errorp "$@"; exit 1; }
+
+Usage() {
+    cat <<EOF
+Usage: ${0##*/} [ options ] output user-data [meta-data]
+
+   Create a disk for cloud-init to utilize nocloud
+
+   options:
+     -h | --help            show usage
+     -d | --disk-format D   disk format to output. default: raw
+     -f | --filesystem  F   filesystem format (vfat or iso), default: iso9660
+
+     -i | --interfaces  F   write network interfaces file into metadata
+     -m | --dsmode      M   add 'dsmode' ('local' or 'net') to the metadata
+                            default in cloud-init is 'net', meaning network is
+                            required.
+
+   Example:
+    * cat my-user-data
+      #cloud-config
+      password: passw0rd
+      chpasswd: { expire: False }
+      ssh_pwauth: True
+    * echo "instance-id: \$(uuidgen || echo i-abcdefg)" > my-meta-data
+    * ${0##*/} my-seed.img my-user-data my-meta-data
+EOF
+}
+
+bad_Usage() { Usage 1>&2; [ $# -eq 0 ] || error "$@"; exit 1; }
+cleanup() {
+    [ -z "${TEMP_D}" -o ! -d "${TEMP_D}" ] || rm -Rf "${TEMP_D}"
+}
+
+debug() {
+    local level=${1}; shift;
+    [ "${level}" -gt "${VERBOSITY}" ] && return
+    error "${@}"
+}
+
+short_opts="hi:d:f:m:o:v"
+long_opts="disk-format:,dsmode:,filesystem:,help,interfaces:,output:,verbose"
+getopt_out=$(getopt --name "${0##*/}" \
+    --options "${short_opts}" --long "${long_opts}" -- "$@") &&
+    eval set -- "${getopt_out}" ||
+    bad_Usage
+
+## <<insert default variables here>>
+output=""
+userdata=""
+metadata=""
+filesystem=$DEF_FILESYSTEM
+diskformat=$DEF_DISK_FORMAT
+interfaces=_unset
+dsmode=""
+
+
+while [ $# -ne 0 ]; do
+    cur=${1}; next=${2};
+    case "$cur" in
+        -h|--help) Usage ; exit 0;;
+        -v|--verbose) VERBOSITY=$((${VERBOSITY}+1));;
+        -d|--disk-format) diskformat=$next; shift;;
+        -f|--filesystem) filesystem=$next; shift;;
+        -m|--dsmode) dsmode=$next; shift;;
+        -i|--interfaces) interfaces=$next; shift;;
+        --) shift; break;;
+    esac
+    shift;
+done
+
+## check arguments here
+## how many args do you expect?
+[ $# -ge 1 ] || bad_Usage "must provide output, userdata"
+[ $# -le 3 ] || bad_Usage "confused by additional args"
+
+output=$1
+userdata=$2
+metadata=$3
+
+[ -n "$metadata" -a "${interfaces}" != "_unset" ] &&
+    fail "metadata and --interfaces are incompatible"
+[ -n "$metadata" -a -n "$dsmode" ] &&
+    fail "metadata and dsmode are incompatible"
+[ "$interfaces" = "_unset" -o -r "$interfaces" ] ||
+    fail "$interfaces: not a readable file"
+
+TEMP_D=$(mktemp -d "${TMPDIR:-/tmp}/${0##*/}.XXXXXX") ||
+    fail "failed to make tempdir"
+trap cleanup EXIT
+
+if [ -n "$metadata" ]; then
+    cp "$metadata" "$TEMP_D/meta-data" || fail "$metadata: failed to copy"
+else
+    {
+    echo "instance-id: iid-local01"
+    [ -n "$dsmode" ] && echo "dsmode: $dsmode"
+    [ -n "$interfaces" ] && echo "interfaces: |" &&
+        sed 's,^,  ,' "$interfaces"
+    } > "$TEMP_D/meta-data"
+fi
+
+if [ "$userdata" = "-" ]; then
+    cat > "$TEMP_D/user-data" || fail "failed to read from stdin"
+else
+    cp "$userdata" "$TEMP_D/user-data" || fail "$userdata: failed to copy"
+fi
+
+## alternatively, create a vfat filesystem with same files
+img="$TEMP_D/seed.img"
+truncate --size 100K "$img" || fail "failed truncate image"
+
+case "$filesystem" in
+    iso9660|iso)
+        genisoimage  -output "$img" -volid cidata \
+            -joliet -rock "$TEMP_D/user-data" "$TEMP_D/meta-data" \
+            > "$TEMP_D/err" 2>&1 ||
+            { cat "$TEMP_D/err" 1>&2; fail "failed to genisoimage"; }
+        ;;
+    vfat)
+        mkfs.vfat -n cidata "$img" || fail "failed mkfs.vfat"
+        mcopy -oi "$img" "$TEMP_D/user-data" "$TEMP_D/meta-data" :: ||
+            fail "failed to copy user-data, meta-data to img"
+        ;;
+    *) fail "unknown filesystem $filesystem";;
+esac
+
+[ "$output" = "-" ] && output="$TEMP_D/final"
+qemu-img convert -f raw -O "$diskformat" "$img" "$output" ||
+    fail "failed to convert to disk format $diskformat"
+
+[ "$output" != "$TEMP_D/final" ] || { cat "$output" && output="-"; } ||
+    fail "failed to write to -"
+
+error "wrote ${output} with filesystem=$filesystem and diskformat=$diskformat"
+# vi: ts=4 noexpandtab
+

=== added file 'overlay.sh'
--- overlay.sh	1970-01-01 00:00:00 +0000
+++ overlay.sh	2018-05-31 04:33:07 +0000
@@ -0,0 +1,23 @@
+#!/bin/bash
+# Overlays a new branch over this branch. This allows you to reuse code
+# from this branch against a development branch or a private branch.
+
+my_script_path=$(readlink -f ${0})
+my_s_dir=$(dirname ${my_script_path})
+
+source "${my_s_dir}/functions/bzr_check.sh"
+source "${my_s_dir}/functions/common.sh"
+
+rsync_merge() {
+    debug "Merging ${1} with private"
+    [ -d ${1} ] || fail "no such directory ${1} for merging!"
+    rsync -av ${1}/* ${my_s_dir} ||
+        fail "failed to merge ${1}"
+}
+
+for i in ${@}
+do
+    check_tmp=$(mktemp --directory --tmpdir=${my_s_dir} bzrbranch.XXX)
+    check_branch ${i} ${check_tmp}
+    rsync_merge ${check_tmp}
+done

=== added directory 'pylib'
=== added directory 'pylib/changelogger'
=== added file 'pylib/changelogger.py'
--- pylib/changelogger.py	1970-01-01 00:00:00 +0000
+++ pylib/changelogger.py	2018-05-31 04:33:07 +0000
@@ -0,0 +1,222 @@
+from __future__ import print_function
+
+import logging
+import re
+import requests
+import subprocess
+from debian.changelog import (Changelog)
+
+# http://changelogs.ubuntu.com/changelogs/pool/main/l/linux-meta/linux-meta_4.2.0.17.19/changelog
+changelog_url_base = \
+    "http://changelogs.ubuntu.com/changelogs/pool/{}/{}/{}/{}_{}/changelog"
+
+
+class ChangeDelta(object):
+
+    def __init__(self, pkg, changelog=None):
+        self._pkg = pkg
+        self._changelogs = changelog
+
+    def _set_pkg(self, pkg):
+        if pkg is not None:
+            self._pkg = str(pkg)
+
+    def _get_pkg(self):
+        return self._pkg
+
+    pkg = property(_get_pkg, _set_pkg)
+
+    def _get_changelogs(self):
+        try:
+            return self._changelogs
+        except KeyError:
+            return []
+
+    def _set_changelogs(self, changelogs):
+        self._changelogs = changelogs
+
+    changelogs = property(_get_changelogs, _set_changelogs)
+
+    def iter_changelogs(self, text=False):
+        for block in self.changelogs:
+            if text:
+                yield str(block)
+            else:
+                yield block
+
+    def _get_bug_cves(self):
+        """extract information from the changelog block"""
+        mappings, bugs, cves = ({}, [], [])
+        cve_re = re.compile("CVE-\d+-\d+")
+        bug_re = re.compile("\(LP:.#(\d+)\)")
+
+        for block in self.iter_changelogs():
+            _block = str(block)
+            cves_in_block = cve_re.findall(_block)
+            cves.extend(cves_in_block)
+
+            bugs_in_block = bug_re.findall(_block)
+            bugs.extend(bugs_in_block)
+
+            ver = str(block.version)
+            mappings[ver] = {'cves': cves_in_block,
+                             'bugs': bugs_in_block}
+
+        self.mappings = mappings
+        self.cves = cves
+        self.bugs = bugs
+
+    def _get_cves(self):
+        self._get_bug_cves()
+        return self._cves
+
+    def _set_cves(self, cves):
+        self._cves = cves
+
+    cves = property(_get_cves, _set_cves)
+
+    def cve_in_delta(self):
+        if len(self.cves) > 0:
+            return True
+        return False
+
+    def _get_bugs(self):
+        self._get_bug_cves()
+        return self._bugs
+
+    def _set_bugs(self, bugs):
+        self._bugs = bugs
+
+    bugs = property(_get_bugs, _set_bugs)
+
+    def _get_min_version(self):
+        if self.changelogs:
+            return self.changelogs[-1].version
+
+    min_version = property(_get_min_version)
+
+    def _get_max_version(self):
+        if self.changelogs:
+            return self.changelogs[0].version
+
+    max_version = property(_get_max_version)
+
+    def _get_mappings(self):
+        try:
+            return self._mappings
+        except KeyError:
+            return {}
+
+    def _set_mappings(self, mapping):
+        self._mappings = mapping
+
+    mappings = property(_get_mappings, _set_mappings)
+
+    def format_changelogs(self):
+        changeblocks = "\n".join(self.iter_changelogs(text=True))
+        return changeblocks
+
+    def __str__():
+        return self.format_changelogs()
+
+
+class ReadChangeLog(Changelog):
+
+    def __init__(self, pkg, version):
+        self.logger = logging.getLogger("__changelog_{}__".format(pkg))
+        logging.basicConfig(format=
+                        '%(asctime)s  %(levelname)s - [PARSING {}] %(message)s'.format(
+                            pkg))
+        self.logger.setLevel(logging.DEBUG)
+        self.logger.debug("Parsing changelog for {}".format(version))
+
+        ch_url = self.get_changelog_url(pkg, version)
+        self.logger.debug("URL: {}".format(ch_url))
+        try:
+            raw_changelog = self.get_changelog_from_url(ch_url)
+            Changelog.__init__(self, raw_changelog)
+        except Exception as e:
+            self.logger.debug("Failed to parse changelog!\n{}".format(e))
+
+        self.min_version = self._blocks[-1].version
+        self.max_version = self.version
+
+        self.logger.debug("Opened changelog:")
+        self.logger.debug("   Versions {} through {}".format(self.min_version,
+                                                        self.max_version))
+
+    def get_changelog_url(self, pkg, version, pocket='main', url=None):
+        """Return the changelog URL"""
+
+        url = url or changelog_url_base
+        pdir = pkg[0]
+        if pkg.startswith("lib"):
+            pdir = pkg[:4]
+        return url.format(pocket, pdir, pkg, pkg, version)
+
+    def get_changelog_from_url(self, url):
+        """Fetch the change log"""
+        try:
+            chlog = requests.get(url)
+            if chlog.status_code == requests.codes.ok:
+                return chlog.text
+            else:
+                chlog.raise_for_status()
+
+        except requests.exceptions.HTTPError as e:
+            self.logger.critical("Failed to fetch changelog at {}:\n{}".format(
+                            url, e))
+
+    def compare_versions(self, v1, operator, v2):
+        """Dirty, slow hack to compare versions"""
+        cmd = ['/usr/bin/dpkg', '--compare-versions', str(v1), str(operator),
+                str(v2)]
+        try:
+            subprocess.check_call(cmd)
+        except subprocess.CalledProcessError as e:
+            return False
+
+        return True
+
+    def iter_changeblocks(self):
+        """Iterate over the change logs"""
+        for block in self._blocks:
+            yield block
+
+    def get_changes_between(self, minv=None, maxv=None, commits=None):
+        """Get the changes between two versions"""
+        blocks = []
+        # Don't waste CPU time if we are getting the whole log
+        if minv is None and maxv is None:
+            for block in self.iter_changeblocks():
+                blocks.append(block)
+
+        # Now deal with changes between
+        minver = minv or self.min_version
+        maxver = maxv or self.max_version
+
+        # Allow for comparing the latest version against arbitrary counts
+        # i.e. you don't have to know the prior version
+        if minver <= -1:
+            minver = self.versions[(abs(minver) - 1)]
+
+        if minv or maxv:
+            for block in self.iter_changeblocks():
+                bver = block.version
+                if minv:
+                    if not self.compare_versions(bver, 'ge', minver):
+                        continue
+                    if not maxv:
+                        blocks.append(block)
+                    elif self.compare_versions(bver, 'le', maxver):
+                        blocks.append(block)
+                elif maxv:
+                    if not self.compare_versions(bver, 'le', maxver):
+                        continue
+                    if not minv:
+                        blocks.append(block)
+                    elif self.compare_versions(bver, 'ge', minver):
+                        blocks.append(block)
+
+        ret = ChangeDelta(self.package, blocks)
+        return ret

=== added file 'pylib/changelogger/ChangeLogger.py'
--- pylib/changelogger/ChangeLogger.py	1970-01-01 00:00:00 +0000
+++ pylib/changelogger/ChangeLogger.py	2018-05-31 04:33:07 +0000
@@ -0,0 +1,222 @@
+from __future__ import print_function
+
+import logging
+import re
+import requests
+import subprocess
+from debian.changelog import (Changelog)
+
+# http://changelogs.ubuntu.com/changelogs/pool/main/l/linux-meta/linux-meta_4.2.0.17.19/changelog
+changelog_url_base = \
+    "http://changelogs.ubuntu.com/changelogs/pool/{}/{}/{}/{}_{}/changelog"
+
+
+class ChangeDelta(object):
+
+    def __init__(self, pkg, changelog=None):
+        self._pkg = pkg
+        self._changelogs = changelog
+
+    def _set_pkg(self, pkg):
+        if pkg is not None:
+            self._pkg = str(pkg)
+
+    def _get_pkg(self):
+        return self._pkg
+
+    pkg = property(_get_pkg, _set_pkg)
+
+    def _get_changelogs(self):
+        try:
+            return self._changelogs
+        except KeyError:
+            return []
+
+    def _set_changelogs(self, changelogs):
+        self._changelogs = changelogs
+
+    changelogs = property(_get_changelogs, _set_changelogs)
+
+    def iter_changelogs(self, text=False):
+        for block in self.changelogs:
+            if text:
+                yield str(block)
+            else:
+                yield block
+
+    def _get_bug_cves(self):
+        """extract information from the changelog block"""
+        mappings, bugs, cves = ({}, [], [])
+        cve_re = re.compile("CVE-\d+-\d+")
+        bug_re = re.compile("\(LP:.#(\d+)\)")
+
+        for block in self.iter_changelogs():
+            _block = str(block)
+            cves_in_block = cve_re.findall(_block)
+            cves.extend(cves_in_block)
+
+            bugs_in_block = bug_re.findall(_block)
+            bugs.extend(bugs_in_block)
+
+            ver = str(block.version)
+            mappings[ver] = {'cves': cves_in_block,
+                             'bugs': bugs_in_block}
+
+        self.mappings = mappings
+        self.cves = cves
+        self.bugs = bugs
+
+    def _get_cves(self):
+        self._get_bug_cves()
+        return self._cves
+
+    def _set_cves(self, cves):
+        self._cves = cves
+
+    cves = property(_get_cves, _set_cves)
+
+    def cve_in_delta(self):
+        if len(self.cves) > 0:
+            return True
+        return False
+
+    def _get_bugs(self):
+        self._get_bug_cves()
+        return self._bugs
+
+    def _set_bugs(self, bugs):
+        self._bugs = bugs
+
+    bugs = property(_get_bugs, _set_bugs)
+
+    def _get_min_version(self):
+        if self.changelogs:
+            return self.changelogs[-1].version
+
+    min_version = property(_get_min_version)
+
+    def _get_max_version(self):
+        if self.changelogs:
+            return self.changelogs[0].version
+
+    max_version = property(_get_max_version)
+
+    def _get_mappings(self):
+        try:
+            return self._mappings
+        except KeyError:
+            return {}
+
+    def _set_mappings(self, mapping):
+        self._mappings = mapping
+
+    mappings = property(_get_mappings, _set_mappings)
+
+    def format_changelogs(self):
+        changeblocks = "\n".join(self.iter_changelogs(text=True))
+        return changeblocks
+
+    def __str__():
+        return self.format_changelogs()
+
+
+class ReadChangelog(Changelog):
+
+    def __init__(self, pkg, version):
+        self.logger = logging.getLogger("__changelog_{}__".format(pkg))
+        logging.basicConfig(format=
+                        '%(asctime)s  %(levelname)s - [PARSING {}] %(message)s'.format(
+                            pkg))
+        self.logger.setLevel(logging.DEBUG)
+        self.logger.debug("Parsing changelog for {}".format(version))
+
+        ch_url = self.get_changelog_url(pkg, version)
+        self.logger.debug("URL: {}".format(ch_url))
+        try:
+            raw_changelog = self.get_changelog_from_url(ch_url)
+            Changelog.__init__(self, raw_changelog)
+        except Exception as e:
+            self.logger.debug("Failed to parse changelog!\n{}".format(e))
+
+        self.min_version = self._blocks[-1].version
+        self.max_version = self.version
+
+        self.logger.debug("Opened changelog:")
+        self.logger.debug("   Versions {} through {}".format(self.min_version,
+                                                        self.max_version))
+
+    def get_changelog_url(self, pkg, version, pocket='main', url=None):
+        """Return the changelog URL"""
+
+        url = url or changelog_url_base
+        pdir = pkg[0]
+        if pkg.startswith("lib"):
+            pdir = pkg[:4]
+        return url.format(pocket, pdir, pkg, pkg, version)
+
+    def get_changelog_from_url(self, url):
+        """Fetch the change log"""
+        try:
+            chlog = requests.get(url)
+            if chlog.status_code == requests.codes.ok:
+                return chlog.text
+            else:
+                chlog.raise_for_status()
+
+        except requests.exceptions.HTTPError as e:
+            self.logger.critical("Failed to fetch changelog at {}:\n{}".format(
+                            url, e))
+
+    def compare_versions(self, v1, operator, v2):
+        """Dirty, slow hack to compare versions"""
+        cmd = ['/usr/bin/dpkg', '--compare-versions', str(v1), str(operator),
+                str(v2)]
+        try:
+            subprocess.check_call(cmd)
+        except subprocess.CalledProcessError as e:
+            return False
+
+        return True
+
+    def iter_changeblocks(self):
+        """Iterate over the change logs"""
+        for block in self._blocks:
+            yield block
+
+    def get_changes_between(self, minv=None, maxv=None, commits=None):
+        """Get the changes between two versions"""
+        blocks = []
+        # Don't waste CPU time if we are getting the whole log
+        if minv is None and maxv is None:
+            for block in self.iter_changeblocks():
+                blocks.append(block)
+
+        # Now deal with changes between
+        minver = minv or self.min_version
+        maxver = maxv or self.max_version
+
+        # Allow for comparing the latest version against arbitrary counts
+        # i.e. you don't have to know the prior version
+        if minver <= -1:
+            minver = self.versions[(abs(minver) - 1)]
+
+        if minv or maxv:
+            for block in self.iter_changeblocks():
+                bver = block.version
+                if minv:
+                    if not self.compare_versions(bver, 'ge', minver):
+                        continue
+                    if not maxv:
+                        blocks.append(block)
+                    elif self.compare_versions(bver, 'le', maxver):
+                        blocks.append(block)
+                elif maxv:
+                    if not self.compare_versions(bver, 'le', maxver):
+                        continue
+                    if not minv:
+                        blocks.append(block)
+                    elif self.compare_versions(bver, 'ge', minver):
+                        blocks.append(block)
+
+        ret = ChangeDelta(self.package, blocks)
+        return ret

=== added file 'pylib/changelogger/__init__.py'
=== added directory 'pylib/requests'
=== added file 'pylib/requests/__init__.py'
--- pylib/requests/__init__.py	1970-01-01 00:00:00 +0000
+++ pylib/requests/__init__.py	2018-05-31 04:33:07 +0000
@@ -0,0 +1,77 @@
+# -*- coding: utf-8 -*-
+
+#   __
+#  /__)  _  _     _   _ _/   _
+# / (   (- (/ (/ (- _)  /  _)
+#          /
+
+"""
+requests HTTP library
+~~~~~~~~~~~~~~~~~~~~~
+
+Requests is an HTTP library, written in Python, for human beings. Basic GET
+usage:
+
+   >>> import requests
+   >>> r = requests.get('http://python.org')
+   >>> r.status_code
+   200
+   >>> 'Python is a programming language' in r.content
+   True
+
+... or POST:
+
+   >>> payload = dict(key1='value1', key2='value2')
+   >>> r = requests.post("http://httpbin.org/post", data=payload)
+   >>> print(r.text)
+   {
+     ...
+     "form": {
+       "key2": "value2",
+       "key1": "value1"
+     },
+     ...
+   }
+
+The other HTTP methods are supported - see `requests.api`. Full documentation
+is at <http://python-requests.org>.
+
+:copyright: (c) 2014 by Kenneth Reitz.
+:license: Apache 2.0, see LICENSE for more details.
+
+"""
+
+__title__ = 'requests'
+__version__ = '2.3.0'
+__build__ = 0x020300
+__author__ = 'Kenneth Reitz'
+__license__ = 'Apache 2.0'
+__copyright__ = 'Copyright 2014 Kenneth Reitz'
+
+# Attempt to enable urllib3's SNI support, if possible
+try:
+    from .packages.urllib3.contrib import pyopenssl
+    pyopenssl.inject_into_urllib3()
+except ImportError:
+    pass
+
+from . import utils
+from .models import Request, Response, PreparedRequest
+from .api import request, get, head, post, patch, put, delete, options
+from .sessions import session, Session
+from .status_codes import codes
+from .exceptions import (
+    RequestException, Timeout, URLRequired,
+    TooManyRedirects, HTTPError, ConnectionError
+)
+
+# Set default logging handler to avoid "No handler found" warnings.
+import logging
+try:  # Python 2.7+
+    from logging import NullHandler
+except ImportError:
+    class NullHandler(logging.Handler):
+        def emit(self, record):
+            pass
+
+logging.getLogger(__name__).addHandler(NullHandler())

=== added file 'pylib/requests/adapters.py'
--- pylib/requests/adapters.py	1970-01-01 00:00:00 +0000
+++ pylib/requests/adapters.py	2018-05-31 04:33:07 +0000
@@ -0,0 +1,388 @@
+# -*- coding: utf-8 -*-
+
+"""
+requests.adapters
+~~~~~~~~~~~~~~~~~
+
+This module contains the transport adapters that Requests uses to define
+and maintain connections.
+"""
+
+import socket
+
+from .models import Response
+from .packages.urllib3.poolmanager import PoolManager, proxy_from_url
+from .packages.urllib3.response import HTTPResponse
+from .packages.urllib3.util import Timeout as TimeoutSauce
+from .compat import urlparse, basestring, urldefrag, unquote
+from .utils import (DEFAULT_CA_BUNDLE_PATH, get_encoding_from_headers,
+                    prepend_scheme_if_needed, get_auth_from_url)
+from .structures import CaseInsensitiveDict
+from .packages.urllib3.exceptions import MaxRetryError
+from .packages.urllib3.exceptions import TimeoutError
+from .packages.urllib3.exceptions import SSLError as _SSLError
+from .packages.urllib3.exceptions import HTTPError as _HTTPError
+from .packages.urllib3.exceptions import ProxyError as _ProxyError
+from .cookies import extract_cookies_to_jar
+from .exceptions import ConnectionError, Timeout, SSLError, ProxyError
+from .auth import _basic_auth_str
+
+DEFAULT_POOLBLOCK = False
+DEFAULT_POOLSIZE = 10
+DEFAULT_RETRIES = 0
+
+
+class BaseAdapter(object):
+    """The Base Transport Adapter"""
+
+    def __init__(self):
+        super(BaseAdapter, self).__init__()
+
+    def send(self):
+        raise NotImplementedError
+
+    def close(self):
+        raise NotImplementedError
+
+
+class HTTPAdapter(BaseAdapter):
+    """The built-in HTTP Adapter for urllib3.
+
+    Provides a general-case interface for Requests sessions to contact HTTP and
+    HTTPS urls by implementing the Transport Adapter interface. This class will
+    usually be created by the :class:`Session <Session>` class under the
+    covers.
+
+    :param pool_connections: The number of urllib3 connection pools to cache.
+    :param pool_maxsize: The maximum number of connections to save in the pool.
+    :param int max_retries: The maximum number of retries each connection
+        should attempt. Note, this applies only to failed connections and
+        timeouts, never to requests where the server returns a response.
+    :param pool_block: Whether the connection pool should block for connections.
+
+    Usage::
+
+      >>> import requests
+      >>> s = requests.Session()
+      >>> a = requests.adapters.HTTPAdapter(max_retries=3)
+      >>> s.mount('http://', a)
+    """
+    __attrs__ = ['max_retries', 'config', '_pool_connections', '_pool_maxsize',
+                 '_pool_block']
+
+    def __init__(self, pool_connections=DEFAULT_POOLSIZE,
+                 pool_maxsize=DEFAULT_POOLSIZE, max_retries=DEFAULT_RETRIES,
+                 pool_block=DEFAULT_POOLBLOCK):
+        self.max_retries = max_retries
+        self.config = {}
+        self.proxy_manager = {}
+
+        super(HTTPAdapter, self).__init__()
+
+        self._pool_connections = pool_connections
+        self._pool_maxsize = pool_maxsize
+        self._pool_block = pool_block
+
+        self.init_poolmanager(pool_connections, pool_maxsize, block=pool_block)
+
+    def __getstate__(self):
+        return dict((attr, getattr(self, attr, None)) for attr in
+                    self.__attrs__)
+
+    def __setstate__(self, state):
+        # Can't handle by adding 'proxy_manager' to self.__attrs__ because
+        # because self.poolmanager uses a lambda function, which isn't pickleable.
+        self.proxy_manager = {}
+        self.config = {}
+
+        for attr, value in state.items():
+            setattr(self, attr, value)
+
+        self.init_poolmanager(self._pool_connections, self._pool_maxsize,
+                              block=self._pool_block)
+
+    def init_poolmanager(self, connections, maxsize, block=DEFAULT_POOLBLOCK):
+        """Initializes a urllib3 PoolManager. This method should not be called
+        from user code, and is only exposed for use when subclassing the
+        :class:`HTTPAdapter <requests.adapters.HTTPAdapter>`.
+
+        :param connections: The number of urllib3 connection pools to cache.
+        :param maxsize: The maximum number of connections to save in the pool.
+        :param block: Block when no free connections are available.
+        """
+        # save these values for pickling
+        self._pool_connections = connections
+        self._pool_maxsize = maxsize
+        self._pool_block = block
+
+        self.poolmanager = PoolManager(num_pools=connections, maxsize=maxsize,
+                                       block=block)
+
+    def cert_verify(self, conn, url, verify, cert):
+        """Verify a SSL certificate. This method should not be called from user
+        code, and is only exposed for use when subclassing the
+        :class:`HTTPAdapter <requests.adapters.HTTPAdapter>`.
+
+        :param conn: The urllib3 connection object associated with the cert.
+        :param url: The requested URL.
+        :param verify: Whether we should actually verify the certificate.
+        :param cert: The SSL certificate to verify.
+        """
+        if url.lower().startswith('https') and verify:
+
+            cert_loc = None
+
+            # Allow self-specified cert location.
+            if verify is not True:
+                cert_loc = verify
+
+            if not cert_loc:
+                cert_loc = DEFAULT_CA_BUNDLE_PATH
+
+            if not cert_loc:
+                raise Exception("Could not find a suitable SSL CA certificate bundle.")
+
+            conn.cert_reqs = 'CERT_REQUIRED'
+            conn.ca_certs = cert_loc
+        else:
+            conn.cert_reqs = 'CERT_NONE'
+            conn.ca_certs = None
+
+        if cert:
+            if not isinstance(cert, basestring):
+                conn.cert_file = cert[0]
+                conn.key_file = cert[1]
+            else:
+                conn.cert_file = cert
+
+    def build_response(self, req, resp):
+        """Builds a :class:`Response <requests.Response>` object from a urllib3
+        response. This should not be called from user code, and is only exposed
+        for use when subclassing the
+        :class:`HTTPAdapter <requests.adapters.HTTPAdapter>`
+
+        :param req: The :class:`PreparedRequest <PreparedRequest>` used to generate the response.
+        :param resp: The urllib3 response object.
+        """
+        response = Response()
+
+        # Fallback to None if there's no status_code, for whatever reason.
+        response.status_code = getattr(resp, 'status', None)
+
+        # Make headers case-insensitive.
+        response.headers = CaseInsensitiveDict(getattr(resp, 'headers', {}))
+
+        # Set encoding.
+        response.encoding = get_encoding_from_headers(response.headers)
+        response.raw = resp
+        response.reason = response.raw.reason
+
+        if isinstance(req.url, bytes):
+            response.url = req.url.decode('utf-8')
+        else:
+            response.url = req.url
+
+        # Add new cookies from the server.
+        extract_cookies_to_jar(response.cookies, req, resp)
+
+        # Give the Response some context.
+        response.request = req
+        response.connection = self
+
+        return response
+
+    def get_connection(self, url, proxies=None):
+        """Returns a urllib3 connection for the given URL. This should not be
+        called from user code, and is only exposed for use when subclassing the
+        :class:`HTTPAdapter <requests.adapters.HTTPAdapter>`.
+
+        :param url: The URL to connect to.
+        :param proxies: (optional) A Requests-style dictionary of proxies used on this request.
+        """
+        proxies = proxies or {}
+        proxy = proxies.get(urlparse(url.lower()).scheme)
+
+        if proxy:
+            proxy = prepend_scheme_if_needed(proxy, 'http')
+            proxy_headers = self.proxy_headers(proxy)
+
+            if not proxy in self.proxy_manager:
+                self.proxy_manager[proxy] = proxy_from_url(
+                                                proxy,
+                                                proxy_headers=proxy_headers,
+                                                num_pools=self._pool_connections,
+                                                maxsize=self._pool_maxsize,
+                                                block=self._pool_block)
+
+            conn = self.proxy_manager[proxy].connection_from_url(url)
+        else:
+            # Only scheme should be lower case
+            parsed = urlparse(url)
+            url = parsed.geturl()
+            conn = self.poolmanager.connection_from_url(url)
+
+        return conn
+
+    def close(self):
+        """Disposes of any internal state.
+
+        Currently, this just closes the PoolManager, which closes pooled
+        connections.
+        """
+        self.poolmanager.clear()
+
+    def request_url(self, request, proxies):
+        """Obtain the url to use when making the final request.
+
+        If the message is being sent through a HTTP proxy, the full URL has to
+        be used. Otherwise, we should only use the path portion of the URL.
+
+        This should not be called from user code, and is only exposed for use
+        when subclassing the
+        :class:`HTTPAdapter <requests.adapters.HTTPAdapter>`.
+
+        :param request: The :class:`PreparedRequest <PreparedRequest>` being sent.
+        :param proxies: A dictionary of schemes to proxy URLs.
+        """
+        proxies = proxies or {}
+        scheme = urlparse(request.url).scheme
+        proxy = proxies.get(scheme)
+
+        if proxy and scheme != 'https':
+            url, _ = urldefrag(request.url)
+        else:
+            url = request.path_url
+
+        return url
+
+    def add_headers(self, request, **kwargs):
+        """Add any headers needed by the connection. As of v2.0 this does
+        nothing by default, but is left for overriding by users that subclass
+        the :class:`HTTPAdapter <requests.adapters.HTTPAdapter>`.
+
+        This should not be called from user code, and is only exposed for use
+        when subclassing the
+        :class:`HTTPAdapter <requests.adapters.HTTPAdapter>`.
+
+        :param request: The :class:`PreparedRequest <PreparedRequest>` to add headers to.
+        :param kwargs: The keyword arguments from the call to send().
+        """
+        pass
+
+    def proxy_headers(self, proxy):
+        """Returns a dictionary of the headers to add to any request sent
+        through a proxy. This works with urllib3 magic to ensure that they are
+        correctly sent to the proxy, rather than in a tunnelled request if
+        CONNECT is being used.
+
+        This should not be called from user code, and is only exposed for use
+        when subclassing the
+        :class:`HTTPAdapter <requests.adapters.HTTPAdapter>`.
+
+        :param proxies: The url of the proxy being used for this request.
+        :param kwargs: Optional additional keyword arguments.
+        """
+        headers = {}
+        username, password = get_auth_from_url(proxy)
+
+        if username and password:
+            headers['Proxy-Authorization'] = _basic_auth_str(username,
+                                                             password)
+
+        return headers
+
+    def send(self, request, stream=False, timeout=None, verify=True, cert=None, proxies=None):
+        """Sends PreparedRequest object. Returns Response object.
+
+        :param request: The :class:`PreparedRequest <PreparedRequest>` being sent.
+        :param stream: (optional) Whether to stream the request content.
+        :param timeout: (optional) The timeout on the request.
+        :param verify: (optional) Whether to verify SSL certificates.
+        :param cert: (optional) Any user-provided SSL certificate to be trusted.
+        :param proxies: (optional) The proxies dictionary to apply to the request.
+        """
+
+        conn = self.get_connection(request.url, proxies)
+
+        self.cert_verify(conn, request.url, verify, cert)
+        url = self.request_url(request, proxies)
+        self.add_headers(request)
+
+        chunked = not (request.body is None or 'Content-Length' in request.headers)
+
+        timeout = TimeoutSauce(connect=timeout, read=timeout)
+
+        try:
+            if not chunked:
+                resp = conn.urlopen(
+                    method=request.method,
+                    url=url,
+                    body=request.body,
+                    headers=request.headers,
+                    redirect=False,
+                    assert_same_host=False,
+                    preload_content=False,
+                    decode_content=False,
+                    retries=self.max_retries,
+                    timeout=timeout
+                )
+
+            # Send the request.
+            else:
+                if hasattr(conn, 'proxy_pool'):
+                    conn = conn.proxy_pool
+
+                low_conn = conn._get_conn(timeout=timeout)
+
+                try:
+                    low_conn.putrequest(request.method,
+                                        url,
+                                        skip_accept_encoding=True)
+
+                    for header, value in request.headers.items():
+                        low_conn.putheader(header, value)
+
+                    low_conn.endheaders()
+
+                    for i in request.body:
+                        low_conn.send(hex(len(i))[2:].encode('utf-8'))
+                        low_conn.send(b'\r\n')
+                        low_conn.send(i)
+                        low_conn.send(b'\r\n')
+                    low_conn.send(b'0\r\n\r\n')
+
+                    r = low_conn.getresponse()
+                    resp = HTTPResponse.from_httplib(
+                        r,
+                        pool=conn,
+                        connection=low_conn,
+                        preload_content=False,
+                        decode_content=False
+                    )
+                except:
+                    # If we hit any problems here, clean up the connection.
+                    # Then, reraise so that we can handle the actual exception.
+                    low_conn.close()
+                    raise
+                else:
+                    # All is well, return the connection to the pool.
+                    conn._put_conn(low_conn)
+
+        except socket.error as sockerr:
+            raise ConnectionError(sockerr, request=request)
+
+        except MaxRetryError as e:
+            raise ConnectionError(e, request=request)
+
+        except _ProxyError as e:
+            raise ProxyError(e)
+
+        except (_SSLError, _HTTPError) as e:
+            if isinstance(e, _SSLError):
+                raise SSLError(e, request=request)
+            elif isinstance(e, TimeoutError):
+                raise Timeout(e, request=request)
+            else:
+                raise
+
+        return self.build_response(request, resp)

=== added file 'pylib/requests/api.py'
--- pylib/requests/api.py	1970-01-01 00:00:00 +0000
+++ pylib/requests/api.py	2018-05-31 04:33:07 +0000
@@ -0,0 +1,120 @@
+# -*- coding: utf-8 -*-
+
+"""
+requests.api
+~~~~~~~~~~~~
+
+This module implements the Requests API.
+
+:copyright: (c) 2012 by Kenneth Reitz.
+:license: Apache2, see LICENSE for more details.
+
+"""
+
+from . import sessions
+
+
+def request(method, url, **kwargs):
+    """Constructs and sends a :class:`Request <Request>`.
+    Returns :class:`Response <Response>` object.
+
+    :param method: method for the new :class:`Request` object.
+    :param url: URL for the new :class:`Request` object.
+    :param params: (optional) Dictionary or bytes to be sent in the query string for the :class:`Request`.
+    :param data: (optional) Dictionary, bytes, or file-like object to send in the body of the :class:`Request`.
+    :param headers: (optional) Dictionary of HTTP Headers to send with the :class:`Request`.
+    :param cookies: (optional) Dict or CookieJar object to send with the :class:`Request`.
+    :param files: (optional) Dictionary of 'name': file-like-objects (or {'name': ('filename', fileobj)}) for multipart encoding upload.
+    :param auth: (optional) Auth tuple to enable Basic/Digest/Custom HTTP Auth.
+    :param timeout: (optional) Float describing the timeout of the request in seconds.
+    :param allow_redirects: (optional) Boolean. Set to True if POST/PUT/DELETE redirect following is allowed.
+    :param proxies: (optional) Dictionary mapping protocol to the URL of the proxy.
+    :param verify: (optional) if ``True``, the SSL cert will be verified. A CA_BUNDLE path can also be provided.
+    :param stream: (optional) if ``False``, the response content will be immediately downloaded.
+    :param cert: (optional) if String, path to ssl client cert file (.pem). If Tuple, ('cert', 'key') pair.
+
+    Usage::
+
+      >>> import requests
+      >>> req = requests.request('GET', 'http://httpbin.org/get')
+      <Response [200]>
+    """
+
+    session = sessions.Session()
+    return session.request(method=method, url=url, **kwargs)
+
+
+def get(url, **kwargs):
+    """Sends a GET request. Returns :class:`Response` object.
+
+    :param url: URL for the new :class:`Request` object.
+    :param \*\*kwargs: Optional arguments that ``request`` takes.
+    """
+
+    kwargs.setdefault('allow_redirects', True)
+    return request('get', url, **kwargs)
+
+
+def options(url, **kwargs):
+    """Sends a OPTIONS request. Returns :class:`Response` object.
+
+    :param url: URL for the new :class:`Request` object.
+    :param \*\*kwargs: Optional arguments that ``request`` takes.
+    """
+
+    kwargs.setdefault('allow_redirects', True)
+    return request('options', url, **kwargs)
+
+
+def head(url, **kwargs):
+    """Sends a HEAD request. Returns :class:`Response` object.
+
+    :param url: URL for the new :class:`Request` object.
+    :param \*\*kwargs: Optional arguments that ``request`` takes.
+    """
+
+    kwargs.setdefault('allow_redirects', False)
+    return request('head', url, **kwargs)
+
+
+def post(url, data=None, **kwargs):
+    """Sends a POST request. Returns :class:`Response` object.
+
+    :param url: URL for the new :class:`Request` object.
+    :param data: (optional) Dictionary, bytes, or file-like object to send in the body of the :class:`Request`.
+    :param \*\*kwargs: Optional arguments that ``request`` takes.
+    """
+
+    return request('post', url, data=data, **kwargs)
+
+
+def put(url, data=None, **kwargs):
+    """Sends a PUT request. Returns :class:`Response` object.
+
+    :param url: URL for the new :class:`Request` object.
+    :param data: (optional) Dictionary, bytes, or file-like object to send in the body of the :class:`Request`.
+    :param \*\*kwargs: Optional arguments that ``request`` takes.
+    """
+
+    return request('put', url, data=data, **kwargs)
+
+
+def patch(url, data=None, **kwargs):
+    """Sends a PATCH request. Returns :class:`Response` object.
+
+    :param url: URL for the new :class:`Request` object.
+    :param data: (optional) Dictionary, bytes, or file-like object to send in the body of the :class:`Request`.
+    :param \*\*kwargs: Optional arguments that ``request`` takes.
+    """
+
+    return request('patch', url,  data=data, **kwargs)
+
+
+def delete(url, **kwargs):
+    """Sends a DELETE request. Returns :class:`Response` object.
+
+    :param url: URL for the new :class:`Request` object.
+    :param \*\*kwargs: Optional arguments that ``request`` takes.
+    """
+
+    return request('delete', url, **kwargs)

=== added file 'pylib/requests/auth.py'
--- pylib/requests/auth.py	1970-01-01 00:00:00 +0000
+++ pylib/requests/auth.py	2018-05-31 04:33:07 +0000
@@ -0,0 +1,193 @@
+# -*- coding: utf-8 -*-
+
+"""
+requests.auth
+~~~~~~~~~~~~~
+
+This module contains the authentication handlers for Requests.
+"""
+
+import os
+import re
+import time
+import hashlib
+
+from base64 import b64encode
+
+from .compat import urlparse, str
+from .cookies import extract_cookies_to_jar
+from .utils import parse_dict_header
+
+CONTENT_TYPE_FORM_URLENCODED = 'application/x-www-form-urlencoded'
+CONTENT_TYPE_MULTI_PART = 'multipart/form-data'
+
+
+def _basic_auth_str(username, password):
+    """Returns a Basic Auth string."""
+
+    return 'Basic ' + b64encode(('%s:%s' % (username, password)).encode('latin1')).strip().decode('latin1')
+
+
+class AuthBase(object):
+    """Base class that all auth implementations derive from"""
+
+    def __call__(self, r):
+        raise NotImplementedError('Auth hooks must be callable.')
+
+
+class HTTPBasicAuth(AuthBase):
+    """Attaches HTTP Basic Authentication to the given Request object."""
+    def __init__(self, username, password):
+        self.username = username
+        self.password = password
+
+    def __call__(self, r):
+        r.headers['Authorization'] = _basic_auth_str(self.username, self.password)
+        return r
+
+
+class HTTPProxyAuth(HTTPBasicAuth):
+    """Attaches HTTP Proxy Authentication to a given Request object."""
+    def __call__(self, r):
+        r.headers['Proxy-Authorization'] = _basic_auth_str(self.username, self.password)
+        return r
+
+
+class HTTPDigestAuth(AuthBase):
+    """Attaches HTTP Digest Authentication to the given Request object."""
+    def __init__(self, username, password):
+        self.username = username
+        self.password = password
+        self.last_nonce = ''
+        self.nonce_count = 0
+        self.chal = {}
+        self.pos = None
+
+    def build_digest_header(self, method, url):
+
+        realm = self.chal['realm']
+        nonce = self.chal['nonce']
+        qop = self.chal.get('qop')
+        algorithm = self.chal.get('algorithm')
+        opaque = self.chal.get('opaque')
+
+        if algorithm is None:
+            _algorithm = 'MD5'
+        else:
+            _algorithm = algorithm.upper()
+        # lambdas assume digest modules are imported at the top level
+        if _algorithm == 'MD5' or _algorithm == 'MD5-SESS':
+            def md5_utf8(x):
+                if isinstance(x, str):
+                    x = x.encode('utf-8')
+                return hashlib.md5(x).hexdigest()
+            hash_utf8 = md5_utf8
+        elif _algorithm == 'SHA':
+            def sha_utf8(x):
+                if isinstance(x, str):
+                    x = x.encode('utf-8')
+                return hashlib.sha1(x).hexdigest()
+            hash_utf8 = sha_utf8
+
+        KD = lambda s, d: hash_utf8("%s:%s" % (s, d))
+
+        if hash_utf8 is None:
+            return None
+
+        # XXX not implemented yet
+        entdig = None
+        p_parsed = urlparse(url)
+        path = p_parsed.path
+        if p_parsed.query:
+            path += '?' + p_parsed.query
+
+        A1 = '%s:%s:%s' % (self.username, realm, self.password)
+        A2 = '%s:%s' % (method, path)
+
+        HA1 = hash_utf8(A1)
+        HA2 = hash_utf8(A2)
+
+        if nonce == self.last_nonce:
+            self.nonce_count += 1
+        else:
+            self.nonce_count = 1
+        ncvalue = '%08x' % self.nonce_count
+        s = str(self.nonce_count).encode('utf-8')
+        s += nonce.encode('utf-8')
+        s += time.ctime().encode('utf-8')
+        s += os.urandom(8)
+
+        cnonce = (hashlib.sha1(s).hexdigest()[:16])
+        noncebit = "%s:%s:%s:%s:%s" % (nonce, ncvalue, cnonce, qop, HA2)
+        if _algorithm == 'MD5-SESS':
+            HA1 = hash_utf8('%s:%s:%s' % (HA1, nonce, cnonce))
+
+        if qop is None:
+            respdig = KD(HA1, "%s:%s" % (nonce, HA2))
+        elif qop == 'auth' or 'auth' in qop.split(','):
+            respdig = KD(HA1, noncebit)
+        else:
+            # XXX handle auth-int.
+            return None
+
+        self.last_nonce = nonce
+
+        # XXX should the partial digests be encoded too?
+        base = 'username="%s", realm="%s", nonce="%s", uri="%s", ' \
+               'response="%s"' % (self.username, realm, nonce, path, respdig)
+        if opaque:
+            base += ', opaque="%s"' % opaque
+        if algorithm:
+            base += ', algorithm="%s"' % algorithm
+        if entdig:
+            base += ', digest="%s"' % entdig
+        if qop:
+            base += ', qop="auth", nc=%s, cnonce="%s"' % (ncvalue, cnonce)
+
+        return 'Digest %s' % (base)
+
+    def handle_401(self, r, **kwargs):
+        """Takes the given response and tries digest-auth, if needed."""
+
+        if self.pos is not None:
+            # Rewind the file position indicator of the body to where
+            # it was to resend the request.
+            r.request.body.seek(self.pos)
+        num_401_calls = getattr(self, 'num_401_calls', 1)
+        s_auth = r.headers.get('www-authenticate', '')
+
+        if 'digest' in s_auth.lower() and num_401_calls < 2:
+
+            setattr(self, 'num_401_calls', num_401_calls + 1)
+            pat = re.compile(r'digest ', flags=re.IGNORECASE)
+            self.chal = parse_dict_header(pat.sub('', s_auth, count=1))
+
+            # Consume content and release the original connection
+            # to allow our new request to reuse the same one.
+            r.content
+            r.raw.release_conn()
+            prep = r.request.copy()
+            extract_cookies_to_jar(prep._cookies, r.request, r.raw)
+            prep.prepare_cookies(prep._cookies)
+
+            prep.headers['Authorization'] = self.build_digest_header(
+                prep.method, prep.url)
+            _r = r.connection.send(prep, **kwargs)
+            _r.history.append(r)
+            _r.request = prep
+
+            return _r
+
+        setattr(self, 'num_401_calls', 1)
+        return r
+
+    def __call__(self, r):
+        # If we have a saved nonce, skip the 401
+        if self.last_nonce:
+            r.headers['Authorization'] = self.build_digest_header(r.method, r.url)
+        try:
+            self.pos = r.body.tell()
+        except AttributeError:
+            pass
+        r.register_hook('response', self.handle_401)
+        return r

=== added file 'pylib/requests/cacert.pem'
--- pylib/requests/cacert.pem	1970-01-01 00:00:00 +0000
+++ pylib/requests/cacert.pem	2018-05-31 04:33:07 +0000
@@ -0,0 +1,5026 @@
+# This Source Code Form is subject to the terms of the Mozilla Public
+# License, v. 2.0. If a copy of the MPL was not distributed with this
+# file, You can obtain one at http://mozilla.org/MPL/2.0/.
+
+# Issuer: CN=GTE CyberTrust Global Root O=GTE Corporation OU=GTE CyberTrust Solutions, Inc.
+# Subject: CN=GTE CyberTrust Global Root O=GTE Corporation OU=GTE CyberTrust Solutions, Inc.
+# Label: "GTE CyberTrust Global Root"
+# Serial: 421
+# MD5 Fingerprint: ca:3d:d3:68:f1:03:5c:d0:32:fa:b8:2b:59:e8:5a:db
+# SHA1 Fingerprint: 97:81:79:50:d8:1c:96:70:cc:34:d8:09:cf:79:44:31:36:7e:f4:74
+# SHA256 Fingerprint: a5:31:25:18:8d:21:10:aa:96:4b:02:c7:b7:c6:da:32:03:17:08:94:e5:fb:71:ff:fb:66:67:d5:e6:81:0a:36
+-----BEGIN CERTIFICATE-----
+MIICWjCCAcMCAgGlMA0GCSqGSIb3DQEBBAUAMHUxCzAJBgNVBAYTAlVTMRgwFgYD
+VQQKEw9HVEUgQ29ycG9yYXRpb24xJzAlBgNVBAsTHkdURSBDeWJlclRydXN0IFNv
+bHV0aW9ucywgSW5jLjEjMCEGA1UEAxMaR1RFIEN5YmVyVHJ1c3QgR2xvYmFsIFJv
+b3QwHhcNOTgwODEzMDAyOTAwWhcNMTgwODEzMjM1OTAwWjB1MQswCQYDVQQGEwJV
+UzEYMBYGA1UEChMPR1RFIENvcnBvcmF0aW9uMScwJQYDVQQLEx5HVEUgQ3liZXJU
+cnVzdCBTb2x1dGlvbnMsIEluYy4xIzAhBgNVBAMTGkdURSBDeWJlclRydXN0IEds
+b2JhbCBSb290MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQCVD6C28FCc6HrH
+iM3dFw4usJTQGz0O9pTAipTHBsiQl8i4ZBp6fmw8U+E3KHNgf7KXUwefU/ltWJTS
+r41tiGeA5u2ylc9yMcqlHHK6XALnZELn+aks1joNrI1CqiQBOeacPwGFVw1Yh0X4
+04Wqk2kmhXBIgD8SFcd5tB8FLztimQIDAQABMA0GCSqGSIb3DQEBBAUAA4GBAG3r
+GwnpXtlR22ciYaQqPEh346B8pt5zohQDhT37qw4wxYMWM4ETCJ57NE7fQMh017l9
+3PR2VX2bY1QY6fDq81yx2YtCHrnAlU66+tXifPVoYb+O7AWXX1uw16OFNMQkpw0P
+lZPvy5TYnh+dXIVtx6quTx8itc2VrbqnzPmrC3p/
+-----END CERTIFICATE-----
+
+# Issuer: CN=Thawte Server CA O=Thawte Consulting cc OU=Certification Services Division
+# Subject: CN=Thawte Server CA O=Thawte Consulting cc OU=Certification Services Division
+# Label: "Thawte Server CA"
+# Serial: 1
+# MD5 Fingerprint: c5:70:c4:a2:ed:53:78:0c:c8:10:53:81:64:cb:d0:1d
+# SHA1 Fingerprint: 23:e5:94:94:51:95:f2:41:48:03:b4:d5:64:d2:a3:a3:f5:d8:8b:8c
+# SHA256 Fingerprint: b4:41:0b:73:e2:e6:ea:ca:47:fb:c4:2f:8f:a4:01:8a:f4:38:1d:c5:4c:fa:a8:44:50:46:1e:ed:09:45:4d:e9
+-----BEGIN CERTIFICATE-----
+MIIDEzCCAnygAwIBAgIBATANBgkqhkiG9w0BAQQFADCBxDELMAkGA1UEBhMCWkEx
+FTATBgNVBAgTDFdlc3Rlcm4gQ2FwZTESMBAGA1UEBxMJQ2FwZSBUb3duMR0wGwYD
+VQQKExRUaGF3dGUgQ29uc3VsdGluZyBjYzEoMCYGA1UECxMfQ2VydGlmaWNhdGlv
+biBTZXJ2aWNlcyBEaXZpc2lvbjEZMBcGA1UEAxMQVGhhd3RlIFNlcnZlciBDQTEm
+MCQGCSqGSIb3DQEJARYXc2VydmVyLWNlcnRzQHRoYXd0ZS5jb20wHhcNOTYwODAx
+MDAwMDAwWhcNMjAxMjMxMjM1OTU5WjCBxDELMAkGA1UEBhMCWkExFTATBgNVBAgT
+DFdlc3Rlcm4gQ2FwZTESMBAGA1UEBxMJQ2FwZSBUb3duMR0wGwYDVQQKExRUaGF3
+dGUgQ29uc3VsdGluZyBjYzEoMCYGA1UECxMfQ2VydGlmaWNhdGlvbiBTZXJ2aWNl
+cyBEaXZpc2lvbjEZMBcGA1UEAxMQVGhhd3RlIFNlcnZlciBDQTEmMCQGCSqGSIb3
+DQEJARYXc2VydmVyLWNlcnRzQHRoYXd0ZS5jb20wgZ8wDQYJKoZIhvcNAQEBBQAD
+gY0AMIGJAoGBANOkUG7I/1Zr5s9dtuoMaHVHoqrC2oQl/Kj0R1HahbUgdJSGHg91
+yekIYfUGbTBuFRkC6VLAYttNmZ7iagxEOM3+vuNkCXDF/rFrKbYvScg71CcEJRCX