Merge lp:~abentley/juju-ci-tools/hammer-time-timeout into lp:juju-ci-tools

[Aaron Bentley]

Allow hammer-time timeout to be configured.

This branch adds an optional TIMEOUT for hammer-time.

Looking at the recent AWS failures of hammer-time, they are timing out at 30 minutes. I think we need to increase the timeout. But there's no way to increase the timeout used by hammer-time-job.

hammer-time-job allows the number of actions to be controlled. The more actions, the longer the duration. But it doesn't allow the maximum duration to be controlled, so it can't adapt as the number of actions is configured.

I therefore think it makes sense to add a TIMEOUT.

[Curtis Hovey (sinzui)]

This is fine for now. Maybe we want a base timeout (possibly by substrate) that is multiplied by the number of actions. I have pondered something similar for bundles in azure and gce.

review: Approve (code)

=== modified file 'hammer-time-job.bash'
--- hammer-time-job.bash	2017-04-03 19:25:50 +0000
+++ hammer-time-job.bash	2017-04-04 15:42:21 +0000
@@ -12,12 +12,15 @@
 #
 # Optional:
 # replay_build_number The number of a previous build to replay.
+# TIMEOUT The timeout for the operation.  Should be a value acceptable to
+#         /usr/bin/timeout, e.g. 35m for 35 minutes.  Default: 30m
 set -eu
 export ARTIFACTS=$WORKSPACE/artifacts
 export MODEL_NAME=$JOB_NAME
 export DATA_DIR=$JUJU_HOME/jes-homes/$MODEL_NAME
 export PLAN=$ARTIFACTS/plan.yaml
 export HAMMER_DIR=$(dirname $(dirname $HAMMER_TIME))
+: ${TIMEOUT=30m}
 set -x
 s3ci.py get-summary $revision_build $base_config
 jujuci.py -v setup-workspace $WORKSPACE
@@ -28,7 +31,7 @@
 fi
 export JUJU_BIN=$(s3ci.py get-juju-bin $revision_build $WORKSPACE)
 set +e
-timeout 30m bash <<"EOT"
+timeout $TIMEOUT bash <<"EOT"
 set -eux
 deploy_job.py $base_config $JUJU_BIN $ARTIFACTS $MODEL_NAME \
   --series $series --agent-stream=revision-build-$revision_build \