Merge lp:~1chb1n/ubuntu-openstack-ci/arches-1705a into lp:ubuntu-openstack-ci

[Ryan Beisner]

Merge the live operating branch of OSCI back into trunk.

Add retries to charm-test-infra juju commands to work around
LP 1709324. Also clean up the checkout dir to prevent disk
from filling on jenkins slaves.

Update mojo_runner for Juju 2.x

Set MOJO_SERIES for legacy specs

Calculate GATEWAY based on CIDR_EXT

Announce MOJO_* env vars

Correct VIP_RANGE usage

Use venv for lolo_logpuller

Update to use MODEL_NAME

Actually delete ext-ports again

Use bc venv for env-render tool too

Raise charm store api timeout from 10s default to 60s

Export USER env var for charm-tools compat

Update tmp cleanup

Use master of charm-test-infra

Switch charm-single to xenial and bionic

Enable gerrit account whitelist

Move bot-control back to master

Update artifact junk cleanup dates

Timeout swift commands

Update mojo runner for git repo and branch

Merge the live operating branch of OSCI back into trunk.

Add retries to charm-test-infra juju commands to work around
LP 1709324. Also clean up the checkout dir to prevent disk
from filling on jenkins slaves.

Update mojo_runner for Juju 2.x

Set MOJO_SERIES for legacy specs

Calculate GATEWAY based on CIDR_EXT

Announce MOJO_* env vars

Correct VIP_RANGE usage

Use venv for lolo_logpuller

Update to use MODEL_NAME

Actually delete ext-ports again

Use bc venv for env-render tool too

Raise charm store api timeout from 10s default to 60s

Export USER env var for charm-tools compat

Update tmp cleanup

Use master of charm-test-infra

Switch charm-single to xenial and bionic

Enable gerrit account whitelist

Move bot-control back to master

Update artifact junk cleanup dates

Timeout swift commands

Update mojo runner for git repo and branch

[lp:~1chb1n/ubuntu-openstack-ci/arches-1705a]

394. By Ryan Beisner on 2017-06-02

Temporarily use WIP branch

395. By Ryan Beisner on 2017-06-02

Add initial openstack-bundles runner

396. By Ryan Beisner on 2017-06-02

Use CTI temp wip branch

397. By Ryan Beisner on 2017-06-02

Add maas bootstrap wrapper

398. By Ryan Beisner on 2017-06-04

Fix path trailing slash

399. By Ryan Beisner on 2017-07-06

Update osci master ip

400. By Ryan Beisner on 2017-07-06

Update other hosts

401. By Ryan Beisner on 2017-07-06

Set known hosts

402. By Ryan Beisner on 2017-08-01

Remove Yakkety from charm-single test

403. By Ryan Beisner on 2017-08-22

Add retries to charm-test-infra juju commands to work around
LP 1709324. Also clean up the checkout dir to prevent disk
from filling on jenkins slaves.

404. By Ryan Beisner on 2017-08-30

Update mojo_runner for Juju 2.x

405. By Ryan Beisner on 2017-08-30

Set MOJO_SERIES for legacy specs

406. By Ryan Beisner on 2017-08-30

Calculate GATEWAY based on CIDR_EXT

407. By Ryan Beisner on 2017-08-30

Announce MOJO_* env vars

408. By Ryan Beisner on 2017-08-30

Correct VIP_RANGE usage

409. By Ryan Beisner on 2017-11-01

Use venv for lolo_logpuller

410. By Ryan Beisner on 2017-11-01

Update to use MODEL_NAME

411. By Ryan Beisner on 2017-11-01

Actually delete ext-ports again

412. By Ryan Beisner on 2017-11-02

Use bc venv for env-render tool too

413. By Ryan Beisner on 2017-11-05

Raise charm store api timeout from 10s default to 60s

414. By Ryan Beisner on 2017-11-06

Export USER env var for charm-tools compat

415. By Ryan Beisner on 2017-11-09

Update tmp cleanup

416. By Ryan Beisner on 2017-11-28

Use master of charm-test-infra

417. By Ryan Beisner on 2017-12-05

Switch charm-single to xenial and bionic

418. By Ryan Beisner on 2018-01-10

Enable gerrit account whitelist

419. By Ryan Beisner on 2018-01-17

Move bot-control back to master

420. By Ryan Beisner on 2018-02-28

Update artifact junk cleanup dates

421. By Ryan Beisner on 2018-02-28

Update artifact junk cleanup dates

422. By Ryan Beisner on 2018-02-28

Timeout swift commands

423. By Ryan Beisner on 2018-03-21

Update mojo runner for git repo and branch

[uosci-testing-bot (uosci-testing-bot)]

review: Approve

=== modified file 'deploy/README.md'
--- deploy/README.md	2017-05-22 22:11:26 +0000
+++ deploy/README.md	2018-03-21 18:25:26 +0000
@@ -6,6 +6,6 @@
 * osci-model-deploy.sh
 * May have to `juju resolved osci-master/0` to trigger a hook retry.  The jenkins charm as of 2017 Apr 13 has a race condition which consistently requires one re-run of the storage-attached hook.
 * osci-float-all-instances.sh
-* osci-nets-and-models.py
+* osci-nets-and-models.py - idempotent script to create/recreate/update osci networks and distribute the oscirc file to all slaves
 * osci-set-subnets-dns.sh
 * post-deploy/*  # Look first, use with intent and caution.

=== modified file 'deploy/assets/gerrit_host_keys'
--- deploy/assets/gerrit_host_keys	2016-02-24 19:45:46 +0000
+++ deploy/assets/gerrit_host_keys	2018-03-21 18:25:26 +0000
@@ -1,3 +1,2 @@
 |1|yRJCBMbG5INu/6HIArsnDPkaz74=|u2cF49fVES1y6Akk6+RS8ogkgBg= ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfsIj/jqpI+2CFdjCL6kOiqdORWvxQ2sQbCzSzzmLXic8yVhCCbwarkvEpfUOHG4eyB0vqVZfMffxf0Yy3qjURrsroBCiuJ8GdiAcGdfYwHNfBI0cR6kydBZL537YDasIk0Z3ILzhwf7474LmkVzS7V2tMTb4ZiBS/jUeiHsVp88FZhIBkyhlb/awAGcUxT5U4QBXCAmerYXeB47FPuz9JFOVyF08LzH9JRe9tfXtqaCNhlSdRe/2pPRvn2EIhn5uHWwATACG9MBdrK8xv8LqPOik2w1JkgLWyBj11vDd5I3IjrmREGw8dqImqp0r6MD8rxqADlc1elfDIXYsy+TVH
 |1|Q9MKxCi+c8lj5lqOlX+gg9H6Xs4=|WiU87JswA/sA8P0BLG2sEvagDQY= ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfsIj/jqpI+2CFdjCL6kOiqdORWvxQ2sQbCzSzzmLXic8yVhCCbwarkvEpfUOHG4eyB0vqVZfMffxf0Yy3qjURrsroBCiuJ8GdiAcGdfYwHNfBI0cR6kydBZL537YDasIk0Z3ILzhwf7474LmkVzS7V2tMTb4ZiBS/jUeiHsVp88FZhIBkyhlb/awAGcUxT5U4QBXCAmerYXeB47FPuz9JFOVyF08LzH9JRe9tfXtqaCNhlSdRe/2pPRvn2EIhn5uHWwATACG9MBdrK8xv8LqPOik2w1JkgLWyBj11vDd5I3IjrmREGw8dqImqp0r6MD8rxqADlc1elfDIXYsy+TVH
-

=== added file 'deploy/assets/link_secrets.sh'
--- deploy/assets/link_secrets.sh	1970-01-01 00:00:00 +0000
+++ deploy/assets/link_secrets.sh	2018-03-21 18:25:26 +0000
@@ -0,0 +1,7 @@
+#!/bin/bash -uex
+# Files should be manually placed in a secrets dir 1 level down
+# from the uosci repo checkout.  Then link them here.
+ln -s ../../../secrets/id_rsa .
+ln -s ../../../secrets/id_rsa.pub  .
+ln -s ../../../secrets/novarc-artifactcloud .
+ln -s ../../../secrets/uosci-bot-go-cookies.tar.bz2 .

=== modified file 'deploy/assets/other_host_keys'
--- deploy/assets/other_host_keys	2017-04-21 21:34:05 +0000
+++ deploy/assets/other_host_keys	2018-03-21 18:25:26 +0000
@@ -3,4 +3,5 @@
 |1|G7bqYoGs0/7WcgVONKheHbvjbW0=|EWaUmfiDVX5psurof0gbxBW6djs= ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHfg+sp98jxYz9zhxM33ouLNx0rDhXrTLVWU2fZQiMmOCc91H376aU9VzDYd+GRjSewqyz67zpSqsDHd1t6e+J0=
 |1|/uNVERTIWkRm/NIA2HQGM/e+WSs=|584GneqB0FnxjW4yPzufWpoZ3Xw= ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHfg+sp98jxYz9zhxM33ouLNx0rDhXrTLVWU2fZQiMmOCc91H376aU9VzDYd+GRjSewqyz67zpSqsDHd1t6e+J0=
 |1|pJss/g21mdSvAv7PCtXEl6ikX8s=|lu9UENRQqyzTsGJdMPRJ3YPb4gQ= ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDIYipSuaupaQ0PVZiaGWY8oc+r1SLI2PtVyXJA3WcfqrJ79Of7ZOSoHHwwNDkKnzOGj2yu7gohyWBJY2HP83WU=
+|1|rBkrkPfcDAT/MLb0T6o/YeBvmAQ=|49UQeV5BQ/6IFUR2LnixyMktzTU= ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBM1zv4lS+da0WaBu6mnxGEjFAjGzmKez02OQb9yYQwmp6NGlkKTr421YFyf+kHK83Lip+K0YSEhhyv0pl2oiy0A=
 

=== modified file 'deploy/post-deploy/102-ssh-creds.sh'
--- deploy/post-deploy/102-ssh-creds.sh	2017-04-14 05:08:42 +0000
+++ deploy/post-deploy/102-ssh-creds.sh	2018-03-21 18:25:26 +0000
@@ -11,7 +11,7 @@
 f_tmp="${U_HOME}/.ssh"
 f_src="${PUB_KEY} ${PRIV_KEY} ${SSH_CONF}"
 base_src="assets"
-lp_hk=$(cat ${base_src}/launchpad_host_keys)
+hostkeys=$(cat ${base_src}/*_host_keys)
 d_str=$(date +'%H%M%S')
 
 for srcs in $f_src; do
@@ -25,7 +25,7 @@
 for unit in $UNITS; do
     echo "Copying SSH key files to ${unit}..."
     juju ssh ${unit} "sudo mkdir -vp ${f_dst} && sudo mkdir -p ${f_tmp} &&\
-        sudo echo '${lp_hk}' > /tmp/hk${d_str} &&\
+        sudo echo '${hostkeys}' > /tmp/hk${d_str} &&\
         sudo cp -fv /tmp/hk${d_str} /root/.ssh/known_hosts &&\
         sudo mv -fv /tmp/hk${d_str} ${f_dst}/known_hosts"
 

=== modified file 'deploy/templates/oscirc'
--- deploy/templates/oscirc	2017-05-23 21:50:25 +0000
+++ deploy/templates/oscirc	2018-03-21 18:25:26 +0000
@@ -7,8 +7,9 @@
 export AMULET_HTTP_PROXY="http://squid.internal:3128/"
 export BAK_REMOTE_BASE_DIR="/home/ubuntu/osci/bak-osci-master"
 export BAK_USER_HOST="ubuntu@10.245.168.2"
-export OSCI_MASTER_IP="10.245.162.9"
-export OSCI_REPO="lp:ubuntu-openstack-ci"
+export OSCI_MASTER_IP="10.245.162.49"
+#export OSCI_REPO="lp:ubuntu-openstack-ci"
+export OSCI_REPO="lp:~1chb1n/ubuntu-openstack-ci/arches-1705a"  # WIP
 
 : ${EXECUTOR_NUMBER:="z"}
 export OSCI_ROOT="/var/lib/jenkins/tools/$EXECUTOR_NUMBER/ubuntu-openstack-ci"

=== modified file 'run/job-parts/build_bucket_junk_cleanup.sh'
--- run/job-parts/build_bucket_junk_cleanup.sh	2017-04-19 17:54:10 +0000
+++ run/job-parts/build_bucket_junk_cleanup.sh	2018-03-21 18:25:26 +0000
@@ -5,9 +5,9 @@
 f_gerrit_set_blob_vars
 f_novarc_artifactcloud
 
-set -uex
+set -ex
 
-junk_pattern="///|nosuch|sandbox|/_/|2016-01|2016-02|2016-03|2016-04|2016-05|2016-06|2016-07|2016-08|2016-09|2016-10|2015-|2014-"
+junk_pattern="///|nosuch|sandbox|/_/|2016-|2015-|2014-"
 bucket_list_tmp="$(mktemp)"
 junk_list_tmp="$(mktemp)"
 

=== modified file 'run/job-parts/build_charm_single.sh'
--- run/job-parts/build_charm_single.sh	2017-04-27 12:58:23 +0000
+++ run/job-parts/build_charm_single.sh	2018-03-21 18:25:26 +0000
@@ -10,8 +10,8 @@
 f_layerci_build_if_necessary
 
 # Series to exercise
-DEVEL_SERIES_EXERCISE="xenial yakkety zesty"
-STABLE_SERIES_EXERCISE="xenial yakkety"
+DEVEL_SERIES_EXERCISE="xenial bionic"
+STABLE_SERIES_EXERCISE="xenial"
 
 if [[ -z "$SERIES_EXERCISE" ]] && [[ "$GERRIT_BRANCH" != *stable* ]] && [[ "$GIT_BRANCH" != *stable* ]]; then
     # Master (dev)
@@ -67,6 +67,7 @@
         cp $CHARM_SINGLE_YAML $YAML_TMP
         echo " + Rendering file: $CHARM_SINGLE_YAML to $YAML_TMP"
 
+        f_activate_botcontrol_tools_virtualenv
         ${UOSCI_BOT_CONTROL_ROOT}/tools/env-render -vy $YAML_TMP
         sed -e "s/^$BASE_NAME:/$APP_NAME:/" -i $YAML_TMP
         cp $YAML_TMP $WORKSPACE/$APP_NAME.yaml.txt

=== modified file 'run/job-parts/build_mojo_runner.sh'
--- run/job-parts/build_mojo_runner.sh	2017-04-26 22:43:21 +0000
+++ run/job-parts/build_mojo_runner.sh	2018-03-21 18:25:26 +0000
@@ -1,27 +1,15 @@
 #!/bin/bash -e
-#  Mojo runner for UOSCI
-
-# /!\ Temporary override for spec dev
-# [[ -z "$MOJO_OPENSTACK_SPECS_REPO" ]] && export MOJO_OPENSTACK_SPECS_REPO="lp:~ost-maintainers/openstack-mojo-specs/mojo-openstack-specs"
-
-[[ -z "$MOJO_SERIES" ]] && export MOJO_SERIES="trusty"
-[[ -z "$MOJO_SPEC" ]] && export MOJO_SPEC="specs/full_stack/next_deploy_baremetal/icehouse"
-
-# Source the common functions
 . ${OSCI_ROOT}/run/job-parts/osci_job_common.sh
+f_fresh_clean_start
+f_tool_refresh_bot_control
+f_openstack_uos_env_var_normalize
+f_activate_clients
 
-# Set common, constant mojo workspace and project names
+# Set mojo workspace and project names
 export MOJO_WKSP="osci-mojo"
 export MOJO_PROJ="mojo-openstack-specs"
 export MOJO_ROOT="$(mktemp -d)"
 
-# Prep environment
-f_fresh_clean_start
-f_env_no_pass
-
-f_tool_refresh_bot_control
-f_mojo_create_venv
-
 f_line "validating mojo runner env vars"
 f_mojo_handle_env_vars
 
@@ -31,22 +19,21 @@
 f_line "defining new mojo workspace"
 f_mojo_new_workspace
 
-f_line "check that mojo spec manifest exists"
-MOJO_MANIFEST_FILE=${MOJO_ROOT}/${MOJO_PROJ}/${MOJO_SERIES}/${MOJO_WKSP}/spec/${MOJO_SPEC}/manifest
-ls -alh $MOJO_MANIFEST_FILE
-
-f_line "bootstrapping"
+MOJO_MANIFEST_FILE=${MOJO_ROOT}/${MOJO_PROJ}/${UBUNTU_RELEASE}/${MOJO_WKSP}/spec/${MOJO_SPEC}/manifest
 
 if [[ "$MOJO_SPEC" == *metal* ]]; then
   # Handle baremetal specs
-  export UOSCI_JUJU_TAGS="arch=amd64 tags=uosci"
-  juju bootstrap --upload-tools --constraints="${UOSCI_JUJU_TAGS}" 2>&1 | tee $WORKSPACE/juju-bootstrap.$BUILD_NUMBER.txt
-  if [[ "${PIPESTATUS[0]}" != "0" ]]; then
-    echo " ! Bootstrap failed."
-    touch $WORKSPACE/fyi-${test:0:4}.${test}-bootstrap.failed
-  fi
+  exit 1  # Not yet re-implemented for Juju 2.x
+  # Legacy Juju 1.x:
+  #   export UOSCI_JUJU_TAGS="arch=amd64 tags=uosci"
+  #   juju bootstrap --upload-tools --constraints="${UOSCI_JUJU_TAGS}" 2>&1 | tee $WORKSPACE/juju-bootstrap.$BUILD_NUMBER.txt
+  #   if [[ "${PIPESTATUS[0]}" != "0" ]]; then
+  #     echo " ! Bootstrap failed."
+  #     touch $WORKSPACE/fyi-${test:0:4}.${test}-bootstrap.failed
+  #   fi
 else
   # Not baremetal
+  # Bootstrap Controller and add Model if necessary
   f_juju_bootstrap
 fi
 
@@ -54,12 +41,10 @@
 f_mojo_run
 f_line "end of mojo's run"
 
-# DEBUG: list all running instances and all neutron ports
-f_list_nova_instances
-f_list_neutron_ports
+f_juju_stat
+f_lolo_log_puller
 
-# Wrap up
-# XXX: f_collect_juju_logs
 f_set_build_descr
 f_nice_finish
 f_check_fail
+

=== added file 'run/job-parts/build_openstack_bundles_runner.sh'
--- run/job-parts/build_openstack_bundles_runner.sh	1970-01-01 00:00:00 +0000
+++ run/job-parts/build_openstack_bundles_runner.sh	2018-03-21 18:25:26 +0000
@@ -0,0 +1,21 @@
+#!/bin/bash -e
+. ${OSCI_ROOT}/run/job-parts/osci_job_common.sh
+
+f_fresh_clean_start
+f_amulet_set_network_env_vars
+f_tool_refresh_cti
+f_juju_bootstrap_maas
+f_line
+
+cp -fv $HOME/maas/maas-temp.yaml $CTI_CODIR/juju-configs/credentials.yaml
+
+set -u
+stat -t $CTI_CODIR/runners/$BUNDLE_NAME/run.sh
+cd $CTI_CODIR/runners/$BUNDLE_NAME
+./run.sh
+
+f_line
+f_set_build_descr
+f_nice_finish
+f_check_fail
+

=== modified file 'run/job-parts/build_prescribe.sh'
--- run/job-parts/build_prescribe.sh	2017-04-18 19:39:32 +0000
+++ run/job-parts/build_prescribe.sh	2018-03-21 18:25:26 +0000
@@ -9,6 +9,7 @@
 f_line "Prescribing build and/or deploy needs"
 f_skip_juju_foo
 f_tool_refresh_bot_control
+f_gerrit_trigger_whitelist_check
 f_check_out_universal
 
 f_layerci_get_asset_type $CO_DIR

=== renamed file 'run/job-parts/osci_juju1_common.sh' => 'run/job-parts/deprecated/osci_juju1_common.sh'
--- run/job-parts/osci_juju1_common.sh	2017-04-26 22:43:21 +0000
+++ run/job-parts/deprecated/osci_juju1_common.sh	2018-03-21 18:25:26 +0000
@@ -30,15 +30,15 @@
     case "$JUJU_SOURCE" in
       $JUJU_STABLE)
         echo " . Juju stable"
-        JENV_NAME="$(grep "maas-${MOJO_SERIES}:$" ${juju_enviro_file} | head -n1 | cut -f1 -d ':' | xargs)"
+        JENV_NAME="$(grep "maas-${UBUNTU_RELEASE}:$" ${juju_enviro_file} | head -n1 | cut -f1 -d ':' | xargs)"
         ;;
       $JUJU_PROPOSED)
         echo " . Juju proposed"
-        JENV_NAME="$(grep .jprop:$ ${juju_enviro_file} | grep "maas-${MOJO_SERIES}" | head -n1 | cut -f1 -d ':' | xargs)"
+        JENV_NAME="$(grep .jprop:$ ${juju_enviro_file} | grep "maas-${UBUNTU_RELEASE}" | head -n1 | cut -f1 -d ':' | xargs)"
         ;;
       $JUJU_DEVEL)
         echo " . Juju devel"
-        JENV_NAME="$(grep .jdev:$ ${juju_enviro_file} | grep "maas-${MOJO_SERIES}" | head -n1 | cut -f1 -d ':' | xargs)"
+        JENV_NAME="$(grep .jdev:$ ${juju_enviro_file} | grep "maas-${UBUNTU_RELEASE}" | head -n1 | cut -f1 -d ':' | xargs)"
         ;;
       *)
         echo " ! Unknown JUJU_SOURCE value while determining jenv name."

=== modified file 'run/job-parts/osci_gerrit_common.sh'
--- run/job-parts/osci_gerrit_common.sh	2017-04-20 12:36:39 +0000
+++ run/job-parts/osci_gerrit_common.sh	2018-03-21 18:25:26 +0000
@@ -181,7 +181,7 @@
   f_gerrit_create_artifact_index
   echo "Uploading..."
   f_novarc_artifactcloud
-  swift upload $UOSCI_BUCKET_NAME --object-name $UOSCI_BLOB_PATH ${WORKSPACE}
+  timeout 600 swift upload $UOSCI_BUCKET_NAME --object-name $UOSCI_BLOB_PATH ${WORKSPACE}
 
   # Set content-type for txt and html files
   export OBJECT_MATCH_PATTERN="$UOSCI_BLOB_PATH"
@@ -197,9 +197,9 @@
   # The Liberty swift client does not set headers on upload (with radosgw
   # object storage), hence the post after upload.
   set -x
-  swift upload -c -H "content-type:text/html" $UOSCI_BUCKET_NAME index.html
-  swift post -H "content-type:text/html" $UOSCI_BUCKET_NAME index.html
-  swift stat $UOSCI_BUCKET_NAME index.html
+  timeout 600 swift upload -c -H "content-type:text/html" $UOSCI_BUCKET_NAME index.html
+  timeout 600 swift post -H "content-type:text/html" $UOSCI_BUCKET_NAME index.html
+  timeout 600 swift stat $UOSCI_BUCKET_NAME index.html
   set +x
 }
 
@@ -280,7 +280,7 @@
       add_line "  <table>"
 
   # Get index string from key 4, reverse sort by key 2 (date)
-  INDICES=$(swift list uosci-artifacts -l | grep -e 'index.html$' | grep -v ' index.html' | sort -rk 2 | awk '{ print $4 }')
+  INDICES=$(timeout 600 swift list uosci-artifacts -l | grep -e 'index.html$' | grep -v ' index.html' | sort -rk 2 | awk '{ print $4 }')
 
   for i in $INDICES; do
     # link="<a href=\"${i}\">VIEW</a>"
@@ -397,7 +397,7 @@
 
   echo " . Selecting .txt and .html files in $UOSCI_BUCKET_NAME..."
   echo " . OBJECT_MATCH_PATTERN:  $OBJECT_MATCH_PATTERN"
-  swift list $UOSCI_BUCKET_NAME | egrep ".txt$|.html$" | grep "$OBJECT_MATCH_PATTERN" > $tmp_object_list ||:
+  timeout 600 swift list $UOSCI_BUCKET_NAME | egrep ".txt$|.html$" | grep "$OBJECT_MATCH_PATTERN" > $tmp_object_list ||:
 
   if [[ "$(wc -l $tmp_object_list)" == "0 $tmp_object_list" ]]; then
     echo " . No objects found to update"
@@ -413,9 +413,9 @@
     if [[ -n "$line" ]]; then
       echo " - Updating object:  $line"
       if [[ "$line" == *.html ]]; then
-        swift post $UOSCI_BUCKET_NAME $line -H "content-type:text/html"
+        timeout 600 swift post $UOSCI_BUCKET_NAME $line -H "content-type:text/html"
       elif [[ "$line" == *.txt ]]; then
-        swift post $UOSCI_BUCKET_NAME $line -H "content-type:text/plain"
+        timeout 600 swift post $UOSCI_BUCKET_NAME $line -H "content-type:text/plain"
       else
         echo " ! Ignoring:  $line"
       fi
@@ -434,3 +434,9 @@
 function add_line(){
   echo "$1" >> $OUT_FILE
 }
+
+
+function f_gerrit_trigger_whitelist_check(){
+    export GERRIT_ACCOUNT_WHITELIST="$UOSCI_BOT_CONTROL_ROOT/tactical/gerrit-trigger-whitelist.txt"
+    $UOSCI_BOT_CONTROL_ROOT/tactical/gerrit-trigger-whitelist-check.sh
+}

=== modified file 'run/job-parts/osci_job_common.sh'
--- run/job-parts/osci_job_common.sh	2017-04-26 22:43:21 +0000
+++ run/job-parts/osci_job_common.sh	2018-03-21 18:25:26 +0000
@@ -6,13 +6,25 @@
 #  set by the build parameters or the calling script(s).
 
 . ${OSCI_ROOT}/run/job-parts/osci_repos_common.sh
-. ${OSCI_ROOT}/run/job-parts/osci_juju1_common.sh
 . ${OSCI_ROOT}/run/job-parts/osci_openstack_common.sh
 . ${OSCI_ROOT}/run/job-parts/osci_mojo_common.sh
 . ${OSCI_ROOT}/run/job-parts/osci_gerrit_common.sh
 . ${OSCI_ROOT}/run/job-parts/osci_layerci_common.sh
 
 
+function retry_command() {
+# Work-around wrapper to deal with things like https://bugs.launchpad.net/juju/+bug/1709324
+    command=$@
+    i=0
+    attempts=3
+    while [ $i -lt $attempts ]; do
+        $command && break
+        let "i+=1"
+        sleep 10
+    done
+}
+
+
 function f_juju_wait() {
     f_line $FUNCNAME
     sleep 30
@@ -62,6 +74,7 @@
   f_line $FUNCNAME
   f_workspace_cleanup
   f_ssh_known_hosts_clean
+  f_co_cleanup
   if [[ "${SKIP_JUJU_FOO^^}" != "TRUE" ]]; then
       f_tool_refresh_cti
       f_tool_refresh_juju_wait
@@ -113,8 +126,8 @@
   # CLEANUP: delete neutron ext-net ports, commonly added by mojo or other
   # post-deployment configuration for the network node's 2nd nic.
   f_novarc_undercloud
-  if [[ -z "${JENV_NAME}" ]]; then
-    echo " ! Missing required env var JENV_NAME"
+  if [[ -z "${MODEL_NAME}" ]]; then
+    echo " ! Missing required env var MODEL_NAME"
     exit 1
   fi
   if [[ ! -f "${UOSCI_BOT_CONTROL_ROOT}/tools/juju-sym-switch" ]]; then
@@ -122,14 +135,10 @@
   fi
   if [[ "${JENV_NAME^^}" != *MAAS* ]]; then
     f_line "Deleting undercloud tenant's neutron ext-net ports if they exist"
-    pwd
-    cd ${UOSCI_BOT_CONTROL_ROOT}/tools
-    deactivate ||:
-    tox -re port-cleanup
-    . .tox/port-cleanup/bin/activate
-    echo NOT DELETING: ${UOSCI_BOT_CONTROL_ROOT}/tools/port-cleanup -pn "juju-${JENV_NAME}.*-machine.*ext-port" -ps "DOWN" --delete
-    cd $WORKSPACE
-    pwd
+    f_novarc_undercloud
+    f_activate_botcontrol_tools_virtualenv
+    echo "DELETING ext-ports:"
+    ${UOSCI_BOT_CONTROL_ROOT}/tools/port-cleanup -pn "juju-.*-${MODEL_NAME}.*_ext-port" -ps "DOWN" --delete
   fi
 }
 
@@ -192,14 +201,14 @@
 # Generally run at the end of every build
   if [[ "${NO_POST_DESTROY^^}" != "TRUE" ]]; then
     f_tmp_cleanup
+    f_co_cleanup
   fi
   if [[ "${NO_POST_DESTROY^^}" != "TRUE" ]] && [[ "${SKIP_JUJU_FOO^^}" != "TRUE" ]]; then
     f_line "Destroy"
     f_juju_destroy_model
     f_post_destroy_info_dump
 
-    # XXX: revisit ext port delete only if still needed
-    # f_delete_neutron_ext_net_ports
+    f_delete_neutron_ext_net_ports
 
     # XXX: Disable cleanup until determined if still necessary; needs updated if so.
     # f_line "Post-Destroy Additional Clean-up"
@@ -271,6 +280,7 @@
 
 
 function f_set_co_dir() {
+# Set the checkout dir
   mkdir -p $JENKINS_HOME/checkout/$EXECUTOR_NUMBER
   export CO_DIR=$JENKINS_HOME/checkout/$EXECUTOR_NUMBER/$BASE_NAME
 }
@@ -461,7 +471,23 @@
   export BOOTSTRAP_CONSTRAINTS="virt-type=kvm cores=4 mem=3G"
   (
     cd $CTI_CODIR
-    ./juju-openstack-controller-example.sh
+    retry_command ./juju-openstack-controller-example.sh
+  )
+}
+
+
+function f_juju_bootstrap_maas() {
+  # Repo should be present before now, but if it's not, get it!
+  stat $CTI_CODIR/tox.ini ||\
+      f_tool_refresh_cti
+
+  export CLOUD_NAME="ruxton-maas"
+  export CONTROLLER_NAME="ruxton-maas-${NODE_NAME}"
+  export BOOTSTRAP_CONSTRAINTS="arch=$ARCH tags=uosci"
+  export MODEL_CONSTRAINTS="$BOOTSTRAP_CONSTRAINTS"
+  (
+    cd $CTI_CODIR
+    retry_command ./juju-maas-controller-example.sh
   )
 }
 
@@ -479,9 +505,7 @@
 
 function f_lolo_log_puller() {
     f_line $FUNCNAME
-    stat $UOSCI_BOT_CONTROL_ROOT/tools/lolo-log-puller ||\
-        f_tool_refresh_bot_control
-
+    f_activate_botcontrol_tools_virtualenv
     $UOSCI_BOT_CONTROL_ROOT/tools/lolo-log-puller $WORKSPACE/logs
 }
 
@@ -494,7 +518,7 @@
   f_novarc_undercloud
   (
     cd $CTI_CODIR
-    ./juju-destroy-controller-and-models-example.sh -y
+    retry_command ./juju-destroy-controller-and-models-example.sh -y
   )
 }
 
@@ -507,7 +531,7 @@
   f_novarc_undercloud
   (
     cd $CTI_CODIR
-    ./juju-destroy-model-example.sh -y
+    retry_command ./juju-destroy-model-example.sh -y
   )
 }
 
@@ -529,10 +553,37 @@
 }
 
 
+function f_activate_botcontrol_tools_virtualenv() {
+  deactivate ||:
+
+  # Tool should be present before now, but if it's not, get it!
+  stat $UOSCI_BOT_CONTROL_ROOT/tox.ini ||\
+      f_tool_refresh_bot_control
+
+  # Venv may have already been built, but if not, build it!
+  stat $UOSCI_BOT_CONTROL_ROOT/.tox/port-cleanup ||\
+  (
+      cd $UOSCI_BOT_CONTROL_ROOT/tools
+      tox -e port-cleanup
+  )
+  . $UOSCI_BOT_CONTROL_ROOT/tools/.tox/port-cleanup/bin/activate
+}
+
+
 function f_tmp_cleanup(){
 # Clean up tmp dir items as the jenkins user
   f_line "Cleaning up /tmp (Safe to ignore file/dir permission errors)"
-  find /tmp -type f -user jenkins -writable -readable -mmin +60 | xargs rm -rf ||:
+  time find /tmp -type f -user jenkins -writable -readable -mmin +60 | xargs rm -rf ||:
+  time find /tmp -type d -user jenkins -writable -readable -mmin +60 | xargs rm -rf ||:
+}
+
+
+function f_co_cleanup(){
+# Clean up checkout dir items as the jenkins user
+    if [[ -d "$CO_DIR" ]] && [[ -n "$(echo $CO_DIR | grep '/var/lib/jenkins')" ]]; then
+        f_line "Cleaning up checkout dir $CO_DIR"
+        rm -rf $CO_DIR
+    fi
 }
 
 
@@ -552,4 +603,10 @@
     export NO_POST_DESTROY="true"
 fi
 
+# Charm store timeout value, consumed by libcharmstore
+export CS_API_TIMEOUT=60
+
+# Charm-tools work-around
+export USER="$(whoami)"
+
 f_line "osci_job_common complete"

=== modified file 'run/job-parts/osci_mojo_common.sh'
--- run/job-parts/osci_mojo_common.sh	2017-04-19 18:51:47 +0000
+++ run/job-parts/osci_mojo_common.sh	2018-03-21 18:25:26 +0000
@@ -12,15 +12,6 @@
 }
 
 
-function f_mojo_create_venv() {
-  cd $UOSCI_BOT_CONTROL_ROOT/tools/mojo-openstack
-  deactivate ||:
-  tox
-  . .tox/mojo/bin/activate
-  cd $WORKSPACE
-}
-
-
 function f_mojo_new_project() {
 # Add mojo project.
   if [[ -n "$(ls $WORKSPACE -1 | egrep '.failed|.bad')" ]]; then
@@ -28,7 +19,7 @@
     touch $WORKSPACE/fyi-${test:0:4}.mojo-project-new.skipped
   else
     set -x
-    mojo-project-new --series ${MOJO_SERIES} \
+    mojo-project-new --series ${UBUNTU_RELEASE} \
         --container containerless ${MOJO_PROJ} --mojo-root ${MOJO_ROOT} 2>&1 | tee $WORKSPACE/mojo-project-new.txt
     # Error trap
     if [[ "${PIPESTATUS[0]}" != "0" ]]; then
@@ -48,8 +39,9 @@
     touch $WORKSPACE/fyi-${test:0:4}.mojo-workspace-new.skipped
   else
     set -x
-    mojo --mojo-root ${MOJO_ROOT} workspace-new --project ${MOJO_PROJ} --series ${MOJO_SERIES} \
-        --stage ${MOJO_SPEC} ${MOJO_OPENSTACK_SPECS_REPO} ${MOJO_WKSP} \
+    f_activate_clients
+    mojo --mojo-root ${MOJO_ROOT} workspace-new --project ${MOJO_PROJ} --series ${UBUNTU_RELEASE} \
+        --stage ${MOJO_SPEC}/${OPENSTACK_RELEASE} ${MOJO_OPENSTACK_SPECS_REPO} ${MOJO_WKSP} \
         2>&1 | tee $WORKSPACE/mojo-workspace-new.txt
 
     # Error trap
@@ -69,12 +61,11 @@
     echo " ! Skipping mojo run, one or more preceding steps are failed or bad."
     touch $WORKSPACE/fyi-${test:0:4}.mojo-run.skipped
   else
-    # Construct network env vars based on bootstrap node 0
     f_openstack_set_network_env_vars
-    #f_amulet_set_network_env_vars
     set -x
-    mojo --mojo-root ${MOJO_ROOT} run --project ${MOJO_PROJ} --series ${MOJO_SERIES} \
-        --stage ${MOJO_SPEC} ${MOJO_OPENSTACK_SPECS_REPO} ${MOJO_WKSP} \
+    f_activate_clients
+    mojo --mojo-root ${MOJO_ROOT} run --project ${MOJO_PROJ} --series ${UBUNTU_RELEASE} \
+        --stage ${MOJO_SPEC}/${OPENSTACK_RELEASE} "${MOJO_OPENSTACK_SPECS_REPO};revno=${MOJO_OPENSTACK_SPECS_BRANCH}" ${MOJO_WKSP} \
         2>&1 | tee $WORKSPACE/mojo-run.txt
 
     # Error trap
@@ -89,9 +80,8 @@
 
 function f_mojo_handle_env_vars() {
 # Validate and process mojo runner environment variables.
-
   # Basic confirmation of non-null values.
-  options="MOJO_OPENSTACK_SPECS_REPO MOJO_SPEC MOJO_SERIES"
+  options="MOJO_OPENSTACK_SPECS_REPO MOJO_OPENSTACK_SPECS_BRANCH MOJO_SPEC UBUNTU_RELEASE OPENSTACK_RELEASE MOJO_SERIES"
   for option in $options; do
     if [[ -z "${!option}" ]]; then
       echo " ! Failing on null option: ${option}"
@@ -101,17 +91,6 @@
     fi
   done
 
-  # Check for custom mojo spec value.
-  if [[ "$MOJO_SPEC" == "MOJO_SPEC_CUSTOM" ]]; then
-    if [[ -n "$MOJO_SPEC_CUSTOM" ]]; then
-      MOJO_SPEC="${MOJO_SPEC_CUSTOM}"
-      echo "Using custom mojo spec: ${MOJO_SPEC}"
-    else
-      echo " ! Instructed to use a custom spec but did not provide a custom spec value!"
-      touch $WORKSPACE/fyi-${test:0:4}.${test}-mojo-env-vars.failed
-    fi
-  fi
-
   # Fatal if error.  Aborts job build.
   f_set_build_descr
   f_check_fail

=== modified file 'run/job-parts/osci_openstack_common.sh'
--- run/job-parts/osci_openstack_common.sh	2017-04-25 21:57:48 +0000
+++ run/job-parts/osci_openstack_common.sh	2018-03-21 18:25:26 +0000
@@ -42,23 +42,20 @@
 function f_openstack_uos_env_var_normalize() {
 # Ensure that both representations of the Ubuntu Release x OpenStack Release
 # environment variables are set.
-  if [[ -z "$UBUNTU_RELEASE" ]] && [[ -z "$OPENSTACK_RELEASE" ]] && [[ -n "$UOS_COMBO" ]]; then
-    echo " . Using UOS_COMBO to determine UBUNTU_RELEASE and OPENSTACK_RELEASE."
-    export UBUNTU_RELEASE=$(echo $UOS_COMBO | cut -f 1 -d "-")
-    export OPENSTACK_RELEASE=$(echo $UOS_COMBO | cut -f 2 -d "-")
-
-  elif [[ -n "$UBUNTU_RELEASE" ]] && [[ -n "$OPENSTACK_RELEASE" ]] && [[ -z "$UOS_COMBO" ]]; then
-    echo " . Using UBUNTU_RELEASE and OPENSTACK_RELEASE to determine UOS_COMBO."
-    export UOS_COMBO="${UBUNTU_RELEASE}-${OPENSTACK_RELEASE}"
-
-  elif [[ -n "$UBUNTU_RELEASE" ]] && [[ -n "$OPENSTACK_RELEASE" ]] && [[ -n "$UOS_COMBO" ]]; then
-    echo " . UBUNTU_RELEASE, OPENSTACK_RELEASE and UOS_COMBO are all set."
-
-  # XXX: No longer have this safety net;  expect values.
-  # else
-  #   export UBUNTU_RELEASE="xenial"
-  #   export OPENSTACK_RELEASE="mitaka"
-  #   export UOS_COMBO="${UBUNTU_RELEASE}-${OPENSTACK_RELEASE}"
+  if [[ -z "$UBUNTU_RELEASE" ]] && [[ -z "$OPENSTACK_RELEASE" ]] && [[ -n "$U_OS" ]]; then
+    echo " . Using U_OS to determine UBUNTU_RELEASE and OPENSTACK_RELEASE."
+    export UBUNTU_RELEASE=$(echo $U_OS | cut -f 1 -d "-")
+    export MOJO_SERIES=$UBUNTU_RELEASE
+    export OPENSTACK_RELEASE=$(echo $U_OS | cut -f 2 -d "-")
+
+  elif [[ -n "$UBUNTU_RELEASE" ]] && [[ -n "$OPENSTACK_RELEASE" ]] && [[ -z "$U_OS" ]]; then
+    echo " . Using UBUNTU_RELEASE and OPENSTACK_RELEASE to determine U_OS."
+    export U_OS="${UBUNTU_RELEASE}-${OPENSTACK_RELEASE}"
+    export MOJO_SERIES=$UBUNTU_RELEASE
+
+  elif [[ -n "$UBUNTU_RELEASE" ]] && [[ -n "$OPENSTACK_RELEASE" ]] && [[ -n "$U_OS" ]]; then
+    echo " . UBUNTU_RELEASE, OPENSTACK_RELEASE and U_OS are all set."
+    export MOJO_SERIES=$UBUNTU_RELEASE
   fi
 }
 
@@ -225,8 +222,8 @@
     export NAME_SERVER="$(awk '/^nameserver /{ print $2 }' /etc/resolv.conf | head -n 1 | xargs)"
     export NET_ID="$NETWORK_ID"
     export CIDR_EXT="$NETWORK_CIDR"
-    export GATEWAY="$(/sbin/route -n | awk '/^0.0.0.0 /{ print $2 }')"
-
+    # export GATEWAY="$(/sbin/route -n | awk '/^0.0.0.0 /{ print $2 }')"
+    export GATEWAY="$(ipcalc $CIDR_EXT | awk '/HostMin:/{ print $2 }')"
     CIDR_END="$(ipcalc $CIDR_EXT | awk '/^HostMax: /{ print $2 }')"
     FIP_END="$(echo $CIDR_END | sed 's/254/239/g')"
     # OSCI subnet dhcp pools end at the highest .199, so the
@@ -272,6 +269,7 @@
     echo "    AMULET_OS_VIP11: $AMULET_OS_VIP11"
     echo "    AMULET_OS_VIP12: $AMULET_OS_VIP12"
     echo "    AMULET_OS_VIP13: $AMULET_OS_VIP13"
+    echo "    VIP_RANGE: $VIP_RANGE"
     f_amulet_set_odl_location
     f_amulet_to_mojo_env_vars
     f_amulet_set_ldap_location
@@ -306,7 +304,30 @@
     export MOJO_OS_VIP12=$AMULET_OS_VIP12
     export MOJO_OS_VIP13=$AMULET_OS_VIP13
     export MOJO_OS_VIP=$AMULET_OS_VIP
-    export MOJO_VIP_RANGE=$AMULET_VIP_RANGE
+    export MOJO_VIP_RANGE=$VIP_RANGE
+    echo "  + Setting mojo network env vars:"
+    echo "    MOJO_NAME_SERVER: $MOJO_NAME_SERVER"
+    echo "    MOJO_NET_ID: $MOJO_NET_ID"
+    echo "    MOJO_SUBNET_ID: $MOJO_SUBNET_ID"
+    echo "    MOJO_CIDR_EXT: $MOJO_CIDR_EXT"
+    echo "    MOJO_GATEWAY: $MOJO_GATEWAY"
+    echo "    MOJO_FIP_RANGE: $MOJO_FIP_RANGE"
+    echo "    MOJO_OS_VIP00: $MOJO_OS_VIP00"
+    echo "    MOJO_OS_VIP01: $MOJO_OS_VIP01"
+    echo "    MOJO_OS_VIP02: $MOJO_OS_VIP02"
+    echo "    MOJO_OS_VIP03: $MOJO_OS_VIP03"
+    echo "    MOJO_OS_VIP04: $MOJO_OS_VIP04"
+    echo "    MOJO_OS_VIP05: $MOJO_OS_VIP05"
+    echo "    MOJO_OS_VIP06: $MOJO_OS_VIP06"
+    echo "    MOJO_OS_VIP07: $MOJO_OS_VIP07"
+    echo "    MOJO_OS_VIP08: $MOJO_OS_VIP08"
+    echo "    MOJO_OS_VIP09: $MOJO_OS_VIP09"
+    echo "    MOJO_OS_VIP10: $MOJO_OS_VIP10"
+    echo "    MOJO_OS_VIP11: $MOJO_OS_VIP11"
+    echo "    MOJO_OS_VIP12: $MOJO_OS_VIP12"
+    echo "    MOJO_OS_VIP13: $MOJO_OS_VIP13"
+    echo "    MOJO_OS_VIP: $MOJO_OS_VIP"
+    echo "    MOJO_VIP_RANGE: $MOJO_VIP_RANGE"
 }
 
 

=== modified file 'run/job-parts/osci_repos_common.sh'
--- run/job-parts/osci_repos_common.sh	2017-05-27 00:22:19 +0000
+++ run/job-parts/osci_repos_common.sh	2018-03-21 18:25:26 +0000
@@ -3,7 +3,7 @@
 #  Common functions for UOSCI jobs, specific to refreshing tool repos/branches.
 
 : ${EXECUTOR_NUMBER:="z"}
-: ${BASE_CODIR:="${HOME}/tools/$EXECUTOR_NUMBER/"}
+: ${BASE_CODIR:="${HOME}/tools/$EXECUTOR_NUMBER"}
 
 : ${CTI_REPO:="https://github.com/openstack-charmers/charm-test-infra"}
 : ${CTI_REPO_BRANCH:="master"}