Merge lp:~1chb1n/ubuntu-openstack-ci/use-port-cleanup-tool into lp:ubuntu-openstack-ci

[Ryan Beisner]

Update cleanup; use mojo from a venv; add ocata mapping

 - Use port-cleanup tool to delete neutron ext-ports left over from mojo runs. [Bug 1656306]
 - Update and re-enable undercloud instance reaping when error conditions are known. [Bug 1668778]
 - Block forever when destroy fails and cleanup fails, where human intervention is required anyway. [Bug 1668778]
 - Use mojo from a python virtualenv. [Bug 1666746]
 - Use python virtualenv for juju_status_health_check to work around paramiki/crypto IV CTR issues upstream. [Bug 1665598]

Update cleanup; use mojo from a venv; add ocata mapping

 - Use port-cleanup tool to delete neutron ext-ports left over from mojo runs. [Bug 1656306]
 - Update and re-enable undercloud instance reaping when error conditions are known. [Bug 1668778]
 - Block forever when destroy fails and cleanup fails, where human intervention is required anyway. [Bug 1668778]
 - Use mojo from a python virtualenv. [Bug 1666746]
 - Use python virtualenv for juju_status_health_check to work around paramiki/crypto IV CTR issues upstream. [Bug 1665598]

[Ryan Beisner (1chb1n)]

Here is this in action in OSCI:
http://pastebin.ubuntu.com/23931679/

[Ryan Beisner (1chb1n)]

Example of blocking forever w/o cleanup:
http://pastebin.ubuntu.com/24086467/

Example of automatically cleaning up that same type of error condition:
http://pastebin.ubuntu.com/24086472/

[Ryan Beisner (1chb1n)]

# Example of automatic ERROR instance cleanup:
http://pastebin.ubuntu.com/24086476/

[Ryan Beisner (1chb1n)]

NOTE: UOSCI is running live on this branch and has been running on it since Feb 10.

[lp:~1chb1n/ubuntu-openstack-ci/use-port-cleanup-tool]

406. By Ryan Beisner on 2017-03-01

Check that mojo manifest file exists before bootstrapping

407. By Ryan Beisner on 2017-03-01

Express or-true consistently in juju1* tools

[Andrew McLeod (admcleod)]

[Bug 1656306] - approve this MR on basis that a fix is released for this bug and confirmed functional
[Bug 1668778] - approve although this bug has not been 'fix released' it has been tested OK
[Bug 1666746] - approve, fix released and workaround validated
[Bug 1665598] - approve, fix released and validated by another user

On the basis of the notes for these bugs and a review of the diff below I give my +1 for this MR.

review: Approve

=== modified file '.bzrignore'
--- .bzrignore	2016-08-31 01:07:46 +0000
+++ .bzrignore	2017-03-01 03:45:48 +0000
@@ -17,3 +17,4 @@
 temp.py
 .coverage
 deployment/deploy-files/novarc-artifactcloud
+*/.tox

=== modified file 'common/osci_utils.py'
--- common/osci_utils.py	2016-03-02 00:10:08 +0000
+++ common/osci_utils.py	2017-03-01 03:45:48 +0000
@@ -586,7 +586,7 @@
     # Resolve if possible
     try:
         _host = socket.gethostbyname(host)
-    except socket.error, e:
+    except socket.error as e:
         logging.debug('Unable to resolve address: '
                       '{} ({}) Trying anyway!'.format(host, e))
         _host = host
@@ -606,7 +606,7 @@
         logging.debug('Socket connect OK for host '
                       '{} on port {}.'.format(host_str, port))
         return True
-    except socket.error, e:
+    except socket.error as e:
         logging.debug('Socket connect FAIL for'
                       ' {} port {} ({})'.format(host_str, port, e))
         return False
@@ -642,13 +642,13 @@
         logging.debug('SSH expected: {}'.format(expected))
         logging.debug('SSH stdout: {}'.format(stdout_lines))
         logging.debug('SSH stderr: {}'.format(stderr_lines))
-    except paramiko.ssh_exception.SSHException, e:
+    except paramiko.ssh_exception.SSHException as e:
         logging.error('SSH session connect FAILED for host '
                       '{} ({})'.format(host, e))
         if mpq:
             mpq.put(False)
         return False
-    except socket.error, e:
+    except socket.error as e:
         logging.error('SSH socket connect FAIL for'
                       ' {} ({})'.format(host, e))
         if mpq:

=== modified file 'job-parts/build_deploy_test.sh'
--- job-parts/build_deploy_test.sh	2016-06-13 10:21:46 +0000
+++ job-parts/build_deploy_test.sh	2017-03-01 03:45:48 +0000
@@ -26,7 +26,7 @@
 
 # Confirm deployment, configure it, run tempest smoke
 f_line "Confirm deployment"
-juju_ok="$(${OSCI_ROOT}/job-parts/juju_deploy_ok.sh ||:)"
+juju_ok="$(${OSCI_ROOT}/job-parts/juju1_deploy_ok.sh ||:)"
 echo $juju_ok
 f_juju_status_health_check "-post-deploy"
 

=== modified file 'job-parts/build_deployer_quickcheck.sh'
--- job-parts/build_deployer_quickcheck.sh	2016-06-13 10:21:46 +0000
+++ job-parts/build_deployer_quickcheck.sh	2017-03-01 03:45:48 +0000
@@ -20,7 +20,7 @@
 
 # Confirm deployment
 f_line "Confirm deployment"
-juju_ok="$(${OSCI_ROOT}/job-parts/juju_deploy_ok.sh ||:)"
+juju_ok="$(${OSCI_ROOT}/job-parts/juju1_deploy_ok.sh ||:)"
 echo $juju_ok
 f_juju_status_health_check "-post-deploy"
 

=== modified file 'job-parts/build_git_deploy_test.sh'
--- job-parts/build_git_deploy_test.sh	2016-06-13 10:21:46 +0000
+++ job-parts/build_git_deploy_test.sh	2017-03-01 03:45:48 +0000
@@ -30,7 +30,7 @@
 
 # Confirm deployment, configure it, run tempest smoke
 f_line "Confirm deployment"
-juju_ok="$(${OSCI_ROOT}/job-parts/juju_deploy_ok.sh ||:)"
+juju_ok="$(${OSCI_ROOT}/job-parts/juju1_deploy_ok.sh ||:)"
 echo $juju_ok
 f_juju_status_health_check "-post-deploy"
 

=== modified file 'job-parts/build_juju_deploy_single_charm.sh'
--- job-parts/build_juju_deploy_single_charm.sh	2016-06-13 10:21:46 +0000
+++ job-parts/build_juju_deploy_single_charm.sh	2017-03-01 03:45:48 +0000
@@ -17,14 +17,14 @@
 f_juju_wait
 
 # Wait and check
-ready="$(${OSCI_ROOT}/job-parts/juju_deploy_ok.sh ||:)"
+ready="$(${OSCI_ROOT}/job-parts/juju1_deploy_ok.sh ||:)"
 tries=0
 while [[ "$ready" != *DEPLOYED:* ]] && [ $tries -le 20 ]
 do
   echo "Waiting for environment state to become started/ACTIVE..."
   echo "${ready}"
   sleep 30
-  ready="$(${OSCI_ROOT}/job-parts/juju_deploy_ok.sh ||:)"
+  ready="$(${OSCI_ROOT}/job-parts/juju1_deploy_ok.sh ||:)"
   tries=$(( $tries + 1 ))
 done
 

=== modified file 'job-parts/build_mojo_runner.sh'
--- job-parts/build_mojo_runner.sh	2016-12-09 18:14:03 +0000
+++ job-parts/build_mojo_runner.sh	2017-03-01 03:45:48 +0000
@@ -25,18 +25,18 @@
 # Source the common functions
 . ${OSCI_ROOT}/job-parts/osci_job_common.sh
 
-# Confirm executable works and announce version
-echo "Mojo version:"
-mojo --version
-
 # Set common, constant mojo workspace and project names
-export MOJO_WORKSPACE_NAME="osci-mojo"
+export MOJO_WKSP="osci-mojo"
 export MOJO_PROJ="mojo-openstack-specs"
+export MOJO_ROOT="$(mktemp -d)"
 
 # Prep environment
 f_fresh_clean_start
 f_env_no_pass
 
+f_tool_refresh_bot_control
+f_mojo_create_venv
+
 f_line "validating mojo runner env vars"
 f_mojo_handle_env_vars
 
@@ -46,6 +46,10 @@
 f_line "defining new mojo workspace"
 f_mojo_new_workspace
 
+f_line "check that mojo spec manifest exists"
+MOJO_MANIFEST_FILE=${MOJO_ROOT}/${MOJO_PROJ}/${MOJO_SERIES}/${MOJO_WKSP}/spec/${MOJO_SPEC}/manifest
+ls -alh $MOJO_MANIFEST_FILE
+
 f_line "bootstrapping"
 f_juju_get_pkg_info juju-pkg-info.txt
 

=== modified file 'job-parts/build_osi.sh'
--- job-parts/build_osi.sh	2016-08-30 01:52:10 +0000
+++ job-parts/build_osi.sh	2017-03-01 03:45:48 +0000
@@ -53,7 +53,7 @@
 # Confirm deployment, configure it, run tests
 f_line "Confirm deployment"
 export JUJU_HOME=$OSI_JUJU_HOME
-juju_ok="$(${OSCI_ROOT}/job-parts/juju_deploy_ok.sh ||:)"
+juju_ok="$(${OSCI_ROOT}/job-parts/juju1_deploy_ok.sh ||:)"
 echo $juju_ok
 f_juju_status_health_check "-post-deploy"
 

=== added file 'job-parts/deprecated/.keep'
=== removed file 'job-parts/deprecated/Z_DEPRECATE_build_mojo_runner_baremetal.sh'
--- job-parts/deprecated/Z_DEPRECATE_build_mojo_runner_baremetal.sh	2016-02-10 22:46:53 +0000
+++ job-parts/deprecated/Z_DEPRECATE_build_mojo_runner_baremetal.sh	1970-01-01 00:00:00 +0000
@@ -1,69 +0,0 @@
-#!/bin/bash -e
-#  Mojo runner for OSCI
-test="mojo-baremetal"
-
-# Define defaults for manual runs outside of a Jenkins environment
-#[[ -z "$" ]] && export =""
-[[ -z "$JENKINS_HOME" ]] && export JENKINS_HOME="$HOME"
-[[ -z "$WORKSPACE" ]] && export WORKSPACE="$(mktemp -d)"
-[[ -z "$BUILD_NUMBER" ]] && export BUILD_NUMBER="manual"
-
-[[ -z "$OSCI_REPO" ]] && export OSCI_REPO="lp:ubuntu-openstack-ci"
-[[ -z "$OSCI_ROOT" ]] && export OSCI_ROOT="${HOME}/tools/ubuntu-openstack-ci"
-
-[[ -z "$OPENSTACK_CHARM_TESTING_REPO" ]] && export OPENSTACK_CHARM_TESTING_REPO="lp:openstack-charm-testing"
-[[ -z "$OPENSTACK_CHARM_TESTING_ROOT" ]] && export OPENSTACK_CHARM_TESTING_ROOT="${HOME}/tools/openstack-charm-testing"
-
-[[ -z "$MOJO_OPENSTACK_SPECS_REPO" ]] && export MOJO_OPENSTACK_SPECS_REPO="lp:~ost-maintainers/openstack-mojo-specs/mojo-openstack-specs"
-
-[[ -z "$MOJO_SERIES" ]] && export MOJO_SERIES="trusty"
-[[ -z "$MOJO_SPEC" ]] && export MOJO_SPEC="specs/full_stack/next_deploy_baremetal/icehouse"
-
-# Avoid environment teardown if run outside of a Jenkins environment
-[[ -z "$BUILD_URL" ]] && export NO_POST_DESTROY="true"
-
-# Source the common functions
-. ${OSCI_ROOT}/job-parts/osci_job_common.sh
-
-# Need to work in something like this
-#export BAREMETAL=True
-
-# Set common, constant mojo workspace and project names
-export MOJO_WORKSPACE_NAME="osci-mojo"
-export MOJO_PROJ="mojo-openstack-specs"
-export UOSCI_JUJU_TAGS="arch=amd64 tags=uosci"
-
-# Prep environment
-f_fresh_clean_start
-f_env_no_pass
-f_mojo_install
-
-# Mojo Go!
-f_line "validating mojo runner env vars"
-f_mojo_handle_env_vars
-
-f_line "bootstrapping"
-f_juju_get_pkg_info juju-pkg-info.txt
-juju switch maas-${MOJO_SERIES}
-juju bootstrap --upload-tools --constraints="${UOSCI_JUJU_TAGS}" 2>&1 | tee $WORKSPACE/juju-bootstrap.$BUILD_NUMBER
-if [[ "${PIPESTATUS[0]}" != "0" ]]; then
-  echo " ! Bootstrap failed."
-  touch $WORKSPACE/fyi-${test:0:4}.${test}-bootstrap.failed
-fi
-
-f_line "defining new mojo project"
-f_mojo_new_project
-
-f_line "defining new mojo workspace"
-f_mojo_perms_workaround
-f_mojo_new_workspace
-
-f_line "run mojo, run!"
-f_mojo_run
-f_line "end of mojo's run"
-
-# Wrap up
-f_collect_juju_logs
-f_set_build_descr
-f_nice_finish
-f_check_fail

=== removed file 'job-parts/deprecated/part_juju_cleanup.sh'
--- job-parts/deprecated/part_juju_cleanup.sh	2016-08-30 02:59:59 +0000
+++ job-parts/deprecated/part_juju_cleanup.sh	1970-01-01 00:00:00 +0000
@@ -1,22 +0,0 @@
-#!/bin/bash -e
-echo $0
-
-# Source the common functions
-#. ${OSCI_ROOT}/job-parts/osci_job_common.sh
-
-#echo "Making sure juju environment is clear for use on ${NODE_NAME}..."
-#juju_ok="$(${OSCI_ROOT}/job-parts/juju_deploy_ok.sh || true)"
-#echo $juju_ok
-
-#if [[ "$juju_ok" == *CLEAR:* ]]; then
-#  echo "Juju environment is clear on ${NODE_NAME}."
-#else
-#  echo "An environment exists on ${NODE_NAME}!"
-#  echo "Attempting juju enviro destroy of ${JENV_NAME} on ${NODE_NAME}..."
-#  timeout 90 juju destroy-environment $env_name --force -y || true
-#  timeout 90 juju destroy-environment ${env_name}-jdev --force -y || true
-#  sleep 30
-#fi
-
-echo " ! WARNING: deprecated, using the bigger hammer..."
-${OSCI_ROOT}/job-parts/juju1_bighammer_destroy.sh

=== modified file 'job-parts/juju1_bighammer_destroy.sh'
--- job-parts/juju1_bighammer_destroy.sh	2016-12-03 16:14:58 +0000
+++ job-parts/juju1_bighammer_destroy.sh	2017-03-01 03:45:48 +0000
@@ -33,12 +33,18 @@
 . ${OSCI_ROOT}/job-parts/osci_job_common.sh
 f_novarc_undercloud
 
+# Extra safety measure, as nova delete might be called.
+if [[ $OS_USERNAME != osci ]] && [[ "${JENV_NAME^^}" != *MAAS* ]]; then
+  echo " ! novarc env vars do not look like OSCI, halting here."
+  exit 1
+fi
+
 env_names="$(juju switch -l)"
 j_env_dir="${HOME}/.juju/environments"
 f_get_env_files
 
 if [[ ! -d "$j_env_dir" ]]; then
-  echo " ! Never bootstrapped here before? Juju environment dir not found: ${j_env_dir}"
+  echo " ! Juju environment dir not found: ${j_env_dir}"
 fi
 
 # Always attempt a destroy on all enviros/models if ANY are found
@@ -69,40 +75,37 @@
 # Delete nova instances if matches are found
 f_get_env_files
 f_get_my_nova_instances
+
 if [[ -n "$my_instances" ]]; then
   echo " ! Nova instances unexpectedly remain after juju destroy.  This is likely an undercloud issue."
-  echo "_:destroy.failed"
 
-# XXX: nova instance cleanup is disabled, as it should not be necessary.
-# If it becomes necessary, either the undercloud is failing to destroy
-# instances, or the juju tooling has a bug.
-#
-#  echo -e " - These nova instances will now be deleted:"
+  echo -e " - These nova instances will be deleted:"
   # Use filter on my_instances for added safety
-#  my_instances_filtered=$(echo "$my_instances" | grep "$nova_instance_safety_substring")
-#  echo "$my_instances_filtered"
+  my_instances_filtered=$(echo "$my_instances" | egrep "$nova_instance_safety_substring")
+  echo "$my_instances_filtered"
 
-#  while read -r line; do
-#    instance_id=$(echo $line | awk '{ print $2 }')
-#    echo " - Deleting nova instance: ${instance_id}"
-#    timeout 120 nova delete $instance_id
-#  done <<< "$my_instances_filtered"
-#  sleep 30
+  while read -r line; do
+    instance_id=$(echo $line | awk '{ print $2 }')
+    echo " - Deleting nova instance: ${instance_id}"
+    timeout 120 openstack server delete --wait $instance_id ||:
+  done <<< "$my_instances_filtered"
 
   # Re-check nova instances
-#  f_get_my_nova_instances
-#  if [[ -n "$my_instances" ]]; then
-#    echo "$my_instances"
-#    echo " ! Nova instance cleanup failed.  One or more still exist.  Manual intervention required."
-#    exit 1
-#  else
-#    echo " . Nova instances appear to be adequately cleaned up."
-#  fi
+  f_get_my_nova_instances
+  if [[ -n "$my_instances" ]]; then
+    echo "$my_instances"
+    echo " ! Nova instance cleanup failed.  One or more still exist.  Manual intervention required."
+    echo "_:destroy.failed"
+    # Never return if destroy failed; block to require human intervention.
+    sleep infinity
+  else
+    echo " . Nova instances appear to be adequately cleaned up."
+  fi
 fi
 
 # Delete juju .env files if they remain, after backing them up
 if [[ -n "$env_files" ]]; then
-  echo " ! Juju environment file(s) remain after juju destroy:"
+  echo " ! Juju1 env file(s) remain after juju destroy:"
   echo "$env_files"
   d_str=$(date +"%Y%m%d%H%M%S%N")
   loc_dir="${HOME}/tmp.juju-env-files"
@@ -124,7 +127,7 @@
 
 for env_name in $env_names; do
   export JENV_NAME="${env_name}"
-  juju_ok="$(timeout 30 ${OSCI_ROOT}/job-parts/juju_deploy_ok.sh || true)"
+  juju_ok="$(timeout 30 ${OSCI_ROOT}/job-parts/juju1_deploy_ok.sh ||:)"
 
   if [[ "$juju_ok" != *CLEAR:* ]]; then
     echo " ! ${JENV_NAME} on ${NODE_NAME} is not clear."
@@ -143,7 +146,9 @@
   echo "$env_files"
   echo " ! Instances:"
   echo "$my_instances"
-  echo " ! Juju destroy failed.  Manual intervention required."
+  echo " ! juju1_bighammer_destroy failed.  Manual intervention required."
   echo "_:destroy.failed"
-  exit 1
+
+  # Never return if destroy failed; block to require human intervention.
+  sleep infinity
 fi

=== renamed file 'job-parts/juju_deploy_ok.sh' => 'job-parts/juju1_deploy_ok.sh'
--- job-parts/juju_deploy_ok.sh	2016-09-06 15:57:19 +0000
+++ job-parts/juju1_deploy_ok.sh	2017-03-01 03:45:48 +0000
@@ -2,11 +2,12 @@
 # Check current juju environment for all good agent and instance states,
 # or for an environment which is clean and ready to deploy to.
 #
-# Return 0 if enviro is clear to deploy to, or if deployment is nova/juju sane.
+# Return 0 if enviro is clear to deploy to, or if deployment appears to be
+# sane from a nova undercloud and juju perspective.
 
-clear_tmps() {
-  rm -f $instances
-  rm -f $jstat
+clear_tmp_files() {
+  rm -f $instances_tmp_file
+  rm -f $jstat_tmp_file
 }
 
 # Handle manual runs.
@@ -26,34 +27,43 @@
 echo "Juju env name:  ${JENV_NAME}"
 
 # Check for bad agent/machine states in juju status yaml output
-jstat=$(mktemp)
-juju stat --format=yaml &> $jstat || true
-bad_states="$(cat $jstat | grep 'agent\-state:' | egrep -v 'started' || true)"
+jstat_tmp_file=$(mktemp)
+timeout 120 juju stat --format=yaml &> $jstat_tmp_file ||:
+bad_states="$(cat $jstat_tmp_file | grep 'agent\-state:' | egrep -v 'started' ||:)"
 
 # Check for juju environment files
 echo "Checking for jenv files..."
 if [[ -n "$OSI_JUJU_HOME" ]]; then
   # openstack-installer is in play
-  jenv_files="$(ls ${HOME}/.juju/environments/*jenv ${OSI_JUJU_HOME}/environments/*jenv -alh 2> /dev/null || true)"
+  jenv_files="$(ls ${HOME}/.juju/environments/*jenv ${OSI_JUJU_HOME}/environments/*jenv -alh 2> /dev/null ||:)"
 else
   # juju deployment (not openstack-installer)
-  jenv_files="$(ls ${HOME}/.juju/environments/*jenv -alh 2> /dev/null || true)"
-fi
-
-# Check for good and bad nova instances where instance name
-# contains the juju environment name.
+  jenv_files="$(ls ${HOME}/.juju/environments/*jenv -alh 2> /dev/null ||:)"
+fi
+
+# Check that juju status output looks like juju status output if env file exists
+if ! grep "environment:" $jstat_tmp_file && [[ -n "$jenv_files" ]]; then
+  msg="! MODEL-ERROR:  Juju1 env file(s) exist but juju status not OK for ${JENV_NAME}."
+  echo $bad_states
+  clear_tmp_files
+  echo "ERROR 6 - ${msg}"
+  exit 6
+fi
+
+# Check for good and bad nova instances where instance name contains the
+# juju model/environment name.
 if [[ "${JENV_NAME^^}" != *MAAS* ]]; then
   echo "Checking for nova instances..."
-  instances=$(mktemp)
-  nova list | grep "\-${JENV_NAME}\-machine" > $instances || true
-  good_instances=$(grep 'ACTIVE' $instances || true)
-  bad_instances=$(grep -v 'ACTIVE' $instances || true)
+  instances_tmp_file=$(mktemp)
+  openstack server list | grep "\-${JENV_NAME}\-machine" > $instances_tmp_file ||:
+  good_instances=$(grep 'ACTIVE' $instances_tmp_file ||:)
+  bad_instances=$(grep -v 'ACTIVE' $instances_tmp_file ||:)
 
   if [[ -n $good_instances ]]; then
     if [[ -n $bad_instances ]]; then
       echo "NON-ACTIVE:  Nova list found non-ACTIVE instances for ${JENV_NAME}." \
            "Perhaps a build is in progress or incomplete?"
-      clear_tmps
+      clear_tmp_files
       exit 1
     fi
   fi
@@ -64,44 +74,44 @@
        "active/STARTED, and Juju jenv file(s) exist for enviro: ${JENV_NAME}."
   echo $bad_states
   echo $jenv_files
-  cp $jstat $WORKSPACE/juju-stat-bad-states.$BUILD_NUMBER || true
-  clear_tmps
+  cp $jstat_tmp_file $WORKSPACE/juju-stat-bad-states.$BUILD_NUMBER ||:
+  clear_tmp_files
   exit 2
 elif [[ -n $bad_states ]] && [[ -z "$jenv_files" ]]; then
   msg="! TEARDOWN-NEEDED:  Juju environment, agent and/or instance states are bad, and Juju jenv file(s) do not exist for ${JENV_NAME}."
   echo $bad_states
-  clear_tmps
-  while true; do echo "ERROR 3 - Manual intervention required.  Undercloud issue?  ${msg}"; sleep 120; done
-#  exit 3
+  clear_tmp_files
+  echo "ERROR 3 - Manual intervention required.  Undercloud issue?  ${msg}"
+  exit 3
 elif [[ -z $bad_states ]] && [[ -n "$jenv_files" ]] && [[ -z $bad_instances ]] && [[ -n $good_instances ]]; then
   echo " . DEPLOYED:  Juju environment, agent and instance states are good," \
        "and Juju jenv file(s) exist for ${JENV_NAME}."
-  clear_tmps
+  clear_tmp_files
   exit 0
 elif [[ -z $bad_states ]] && [[ -n "$jenv_files" ]] && [[ "${JENV_NAME^^}" == *MAAS* ]]; then
-  echo " . DEPLOYED:  Juju environment (maas substrate), agent states are good," \
+  echo " . DEPLOYED:  Juju environment (maas provider), agent states are good," \
        "and Juju jenv file(s) exist for ${JENV_NAME}."
-  clear_tmps
+  clear_tmp_files
   exit 0
 elif [[ -z $bad_states ]] && [[ -z "$jenv_files" ]] && [[ -z $bad_instances ]] && [[ -z $good_instances ]]; then
   echo " . CLEAR:  Juju environment, agent and instance states not found," \
        "and Juju jenv file(s) do not exist for ${JENV_NAME}."
-  clear_tmps
+  clear_tmp_files
   exit 0
 elif [[ -z $good_instances ]] && [[ -z $bad_instances ]] && [[ -n "$jenv_files" ]] && [[ "${JENV_NAME^^}" != *MAAS* ]]; then
-  msg=" ! TEARDOWN-NEEDED:  Funky state!  Juju jenv file(s) exist, but there are no nova instances for:  ${JENV_NAME}."
+  msg=" ! TEARDOWN-NEEDED:  Juju jenv file(s) exist, but there are no nova instances for:  ${JENV_NAME}."
   echo "jenv_files: ${jenv_files}"
-  clear_tmps
-  while true; do echo "ERROR 4 - Manual intervention required.  Undercloud issue?  ${msg}"; sleep 120; done
-#  exit 4
+  clear_tmp_files
+  echo "ERROR 4 - Manual intervention required.  Undercloud issue?  ${msg}"
+  exit 4
 else
-  msg=" ! FAIL:  Unknown state, need to debug or troubleshoot this node.  Env name: ${JENV_NAME}.  Possibly a non-default juju environment deployed?"
+  msg=" ! FAIL:  Unknown state, need to debug or troubleshoot this node.  Env name: ${JENV_NAME}."
   echo "1: ${JENV_NAME}"
   echo "2: ${bad_instances}"
   echo "3: ${good_instances}"
   echo "4: ${bad_states}"
   echo "5: ${jenv_files}"
-  clear_tmps
-  while true; do echo "ERROR 5 - Manual intervention required.  Undercloud issue?  ${msg}"; sleep 120; done
-#  exit 5
+  clear_tmp_files
+  echo "ERROR 5 - Manual intervention required.  Undercloud issue?  ${msg}"
+  exit 5
 fi

=== modified file 'job-parts/osci_job_common.sh'
--- job-parts/osci_job_common.sh	2016-11-22 19:38:19 +0000
+++ job-parts/osci_job_common.sh	2017-03-01 03:45:48 +0000
@@ -112,7 +112,13 @@
   [[ -n "$1" ]] && PARAM1="$1" || PARAM1=""
   f_line "Checking juju stat health: $PARAM1"
 
-  ${OSCI_ROOT}/tools/juju_status_health_check.py --Xb 2>&1 | tee $WORKSPACE/juju-stat-health-check${PARAM1}.txt; ret_code=${PIPESTATUS[0]}
+  # Run jshc from a venv as a work-around to:
+  #   https://bugs.launchpad.net/ubuntu/+source/paramiko/+bug/1665565
+  #   https://bugs.launchpad.net/charm-test-infra/+bug/1665598
+  # ${OSCI_ROOT}/tools/juju_status_health_check.py --Xb 2>&1 | tee $WORKSPACE/juju-stat-health-check${PARAM1}.txt; ret_code=${PIPESTATUS[0]}
+  cd ${OSCI_ROOT}/tools
+  deactivate ||:
+  tox -re jshc | tee $WORKSPACE/juju-stat-health-check${PARAM1}.txt; ret_code=${PIPESTATUS[0]}
   echo "Returned:  ${ret_code}"
   case "$ret_code" in
     0)
@@ -185,7 +191,7 @@
 
   if [[ "${SKIP_JUJU_LXC^^}" != "TRUE" ]]; then
     ${OSCI_ROOT}/job-parts/juju1_bighammer_destroy.sh YES
-#    f_delete_neutron_ext_net_ports ||:
+    f_delete_neutron_ext_net_ports
     f_mojo_broom
     f_osi_broom
     f_juju_get_pkg_info tools-pkg-info.txt
@@ -232,20 +238,29 @@
 }
 
 
-# Deprecated - neutron port-list no longer contains a juju human name to grep.
-#function f_delete_neutron_ext_net_ports() {
+function f_delete_neutron_ext_net_ports() {
   # CLEANUP: delete neutron ext-net ports, commonly added by mojo or other
   # post-deployment configuration for the network node's 2nd nic.
-#  f_novarc_undercloud
-#  if [[ -z "${JENV_NAME}" ]]; then
-#    echo " ! Missing required env var JENV_NAME"
-#    exit 1
-#  fi
-#  if [[ "${JENV_NAME^^}" != *MAAS* ]]; then
-#    f_line "Deleting undercloud tenant's neutron ext-net ports if they exist"
-#    for i in $(neutron port-list | grep juju-$JENV_NAME | grep ext-port | awk '{ print $2 }'); do echo $i; neutron port-delete $i; done
-#  fi
-#}
+  f_novarc_undercloud
+  if [[ -z "${JENV_NAME}" ]]; then
+    echo " ! Missing required env var JENV_NAME"
+    exit 1
+  fi
+  if [[ ! -f "${UOSCI_BOT_CONTROL_ROOT}/tools/juju-sym-switch" ]]; then
+    f_tool_refresh_bot_control
+  fi
+  if [[ "${JENV_NAME^^}" != *MAAS* ]]; then
+    f_line "Deleting undercloud tenant's neutron ext-net ports if they exist"
+    pwd
+    cd ${UOSCI_BOT_CONTROL_ROOT}/tools
+    deactivate ||:
+    tox -re port-cleanup
+    . .tox/port-cleanup/bin/activate
+    ${UOSCI_BOT_CONTROL_ROOT}/tools/port-cleanup -pn "juju-${JENV_NAME}.*-machine.*ext-port" -ps "DOWN" --delete
+    cd $WORKSPACE
+    pwd
+  fi
+}
 
 
 function f_delete_secgroups() {
@@ -275,7 +290,7 @@
   if [[ "${NO_POST_DESTROY^^}" != "TRUE" ]] && [[ "${SKIP_JUJU_LXC^^}" != "TRUE" ]]; then
     f_line "Destroy"
     ${OSCI_ROOT}/job-parts/juju1_bighammer_destroy.sh YES
-#    f_delete_neutron_ext_net_ports ||:
+    f_delete_neutron_ext_net_ports
 
     f_line "Post-Destroy Info Dump"
     export SUBNET_ID=$(neutron net-show $(cat $JENKINS_HOME/.juju/environments.yaml | grep network: | head -n1 | awk '{ print $2 }') | grep subnets | awk '{ print $4 }')
@@ -690,7 +705,9 @@
 
 function f_get_my_nova_instances() {
 # Get nova instances for the current juju environment
-  export my_instances=$(timeout 30 nova list | grep "juju\-${JENV_NAME}\-machine" ||:)
+  if [[ "${JENV_NAME^^}" != *MAAS* ]]; then
+    export my_instances=$(timeout 60 nova list | grep "juju\-${JENV_NAME}\-machine" ||:)
+  fi
 }
 
 

=== modified file 'job-parts/osci_mojo_common.sh'
--- job-parts/osci_mojo_common.sh	2016-12-09 18:22:55 +0000
+++ job-parts/osci_mojo_common.sh	2017-03-01 03:45:48 +0000
@@ -4,12 +4,23 @@
 
 function f_mojo_broom() {
   if [[ -d "/srv/mojo/mojo-openstack-specs/" ]]; then
-    echo " - Purging /srv/mojo/mojo-openstack-specs ..."
-    rm -rfv /srv/mojo/mojo-openstack-specs/*
+    echo " - Purging /srv/mojo/mojo-openstack-specs"
+    rm -rf /srv/mojo/mojo-openstack-specs/*
+    echo " - Purging $MOJO_ROOT"
+    rm -rf ${MOJO_ROOT}
   fi
 }
 
 
+function f_mojo_create_venv() {
+  cd $UOSCI_BOT_CONTROL_ROOT/tools/mojo-openstack
+  deactivate ||:
+  tox
+  . .tox/mojo/bin/activate
+  cd $WORKSPACE
+}
+
+
 function f_mojo_new_project() {
 # Add mojo project.
   if [[ -n "$(ls $WORKSPACE -1 | egrep '.failed|.bad')" ]]; then
@@ -17,8 +28,8 @@
     touch $WORKSPACE/fyi-${test:0:4}.mojo-project-new.skipped
   else
     set -x
-    mojo project-new --series ${MOJO_SERIES} \
-        --container containerless ${MOJO_PROJ} 2>&1 | tee $WORKSPACE/mojo-project-new.txt
+    mojo-project-new --series ${MOJO_SERIES} \
+        --container containerless ${MOJO_PROJ} --mojo-root ${MOJO_ROOT} 2>&1 | tee $WORKSPACE/mojo-project-new.txt
     # Error trap
     if [[ "${PIPESTATUS[0]}" != "0" ]]; then
       echo " ! Failed to create new mojo project."
@@ -30,16 +41,6 @@
 }
 
 
-# No longer necessary, as we run mojo "containerless"
-#function f_mojo_lxc_perms_workaround() {
-# Workaround for bug https://bugs.launchpad.net/mojo/+bug/1346873
-#  set -x
-#  sudo chmod 755 /var/lib/lxc/${MOJO_PROJ}.${MOJO_SERIES} || true
-#  sudo chmod 755 /var/lib/lxc || true
-#  set +x
-#}
-
-
 function f_mojo_new_workspace() {
 # Add mojo workspace.
   if [[ -n "$(ls $WORKSPACE -1 | egrep '.failed|.bad')" ]]; then
@@ -47,9 +48,10 @@
     touch $WORKSPACE/fyi-${test:0:4}.mojo-workspace-new.skipped
   else
     set -x
-    mojo workspace-new --project ${MOJO_PROJ} --series ${MOJO_SERIES} \
-        --stage ${MOJO_SPEC} ${MOJO_OPENSTACK_SPECS_REPO} ${MOJO_WORKSPACE_NAME} \
+    mojo --mojo-root ${MOJO_ROOT} workspace-new --project ${MOJO_PROJ} --series ${MOJO_SERIES} \
+        --stage ${MOJO_SPEC} ${MOJO_OPENSTACK_SPECS_REPO} ${MOJO_WKSP} \
         2>&1 | tee $WORKSPACE/mojo-workspace-new.txt
+
     # Error trap
     if [[ "${PIPESTATUS[0]}" != "0" ]]; then
       echo " ! Failed to create new mojo workspace."
@@ -71,9 +73,10 @@
     f_openstack_set_network_env_vars
     #f_amulet_set_network_env_vars
     set -x
-    mojo run --project ${MOJO_PROJ} --series ${MOJO_SERIES} \
-        --stage ${MOJO_SPEC} ${MOJO_OPENSTACK_SPECS_REPO} ${MOJO_WORKSPACE_NAME} \
+    mojo --mojo-root ${MOJO_ROOT} run --project ${MOJO_PROJ} --series ${MOJO_SERIES} \
+        --stage ${MOJO_SPEC} ${MOJO_OPENSTACK_SPECS_REPO} ${MOJO_WKSP} \
         2>&1 | tee $WORKSPACE/mojo-run.txt
+
     # Error trap
     if [[ "${PIPESTATUS[0]}" != "0" ]]; then
       echo " ! Mojo run manifest failed."
@@ -84,23 +87,6 @@
 }
 
 
-# DEPRECATED
-#function f_lxc_destroy_all() {
-# Destroy all lxc containser if any exist
-#  lxc_ls="$(sudo lxc-ls -1)"
-#  if [[ -z "$lxc_ls" ]]; then
-#    echo "OK: no lxc containers to destroy."
-#  else
-#    echo " - Destroying all lxc containers:"
-#    for container in "$(sudo lxc-ls -1)"; do
-#      echo " -   ${container}"
-#      sudo lxc-destroy --name ${container} --force || true
-#      sudo lxc-destroy --name ${container} --force || true
-#    done
-#  fi
-#}
-
-
 function f_mojo_handle_env_vars() {
 # Validate and process mojo runner environment variables.
 

=== modified file 'job-parts/trigger_by_mojo_group.sh'
--- job-parts/trigger_by_mojo_group.sh	2016-09-27 05:03:27 +0000
+++ job-parts/trigger_by_mojo_group.sh	2017-03-01 03:45:48 +0000
@@ -76,6 +76,9 @@
   NEWTON_ALL)
     tools/trigger_mojo_group.py -o newton
     ;;
+  OCATA_ALL)
+    tools/trigger_mojo_group.py -o ocata
+    ;;
 
   PRECISE_ALL)
     tools/trigger_mojo_group.py -u precise
@@ -89,6 +92,9 @@
   YAKKETY_ALL)
     tools/trigger_mojo_group.py -u yakkety
     ;;
+  ZESTY_ALL)
+    tools/trigger_mojo_group.py -u zesty
+    ;;
 
   CHARM_UPGRADE)
     tools/trigger_mojo_group.py -s stable_to_next

=== modified file 'populate/mappings.yaml'
--- populate/mappings.yaml	2016-12-09 20:05:45 +0000
+++ populate/mappings.yaml	2017-03-01 03:45:48 +0000
@@ -32,8 +32,11 @@
   xenial:
   - mitaka
   - newton
+  - ocata
   yakkety:
   - newton
+  zesty:
+  - ocata
 
 # the things we care most about right now
 hot:
@@ -48,8 +51,11 @@
   xenial:
   - mitaka
   - newton
+  - ocata
   yakkety:
   - newton
+  zesty:
+  - ocata
 
 # openstack installation sources
 inst-sources:
@@ -144,11 +150,8 @@
     http://docs.openstack.org/infra/manual/developers.html
 trigger-deployments:
   default.yaml:
-  - trusty-icehouse
-  - trusty-kilo
-  - trusty-liberty
-#  - trusty-mitaka
-#  - xenial-mitaka
+  - trusty-mitaka
+  - xenial-mitaka
   next.yaml:
   - trusty-icehouse
   - trusty-kilo
@@ -156,14 +159,13 @@
   - trusty-mitaka
   - xenial-mitaka
   - xenial-newton
+  - xenial-ocata
 
 # deployment set to trigger based on merge proposals
 # the merge-proposed branch will be substituted for the target branch
 mp-trigger-deployments:
   next.yaml:
   - trusty-icehouse
-  - trusty-kilo
-  - trusty-liberty
   - trusty-mitaka
   - xenial-mitaka
   - xenial-newton

=== modified file 'tools/juju_status_health_check.py'
--- tools/juju_status_health_check.py	2016-06-03 17:20:08 +0000
+++ tools/juju_status_health_check.py	2017-03-01 03:45:48 +0000
@@ -1,4 +1,4 @@
-#!/usr/bin/python
+#!/usr/bin/env python
 '''
   juju status health check (not to be confused with juju health status)
 

=== added file 'tools/requirements.txt'
--- tools/requirements.txt	1970-01-01 00:00:00 +0000
+++ tools/requirements.txt	2017-03-01 03:45:48 +0000
@@ -0,0 +1,10 @@
+# Work-arounds for:
+#   https://bugs.launchpad.net/ubuntu/+source/paramiko/+bug/1665565
+#   https://bugs.launchpad.net/charm-test-infra/+bug/1665598
+paramiko==2.1.1
+crypto==1.4.1
+distro-info
+launchpadlib==1.10.5
+bzr==2.7.0
+jinja2==2.9.5
+simplejson==3.10.0

=== added file 'tools/tox.ini'
--- tools/tox.ini	1970-01-01 00:00:00 +0000
+++ tools/tox.ini	2017-03-01 03:45:48 +0000
@@ -0,0 +1,19 @@
+[tox]
+envlist = noop
+skipsdist = True
+
+[testenv]
+setenv = VIRTUAL_ENV={envdir}
+         PYTHONHASHSEED=0
+passenv = HOME TERM
+install_command =
+  pip install --allow-unverified python-apt {opts} {packages}
+
+[testenv:noop]
+basepython = python2.7
+
+[testenv:jshc]
+basepython = python2.7
+deps = -r{toxinidir}/requirements.txt
+whitelist_externals = juju_status_health_check.py
+commands = ./juju_status_health_check.py