bootstrap doesn't know when to give up

Bug #1257427 reported by Aaron Bentley
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
juju-core
Fix Released
High
John A Meinel

Bug Description

Attempting to bootstrap hpcloud on 1.17 is neither succeeding or failing, but hanging. For CI, we need operations to either succeed or fail. Waiting forever is no good.

We see this only in 1.17. On 1.16.4, hpcloud deploys fine.

This was seen in http://162.213.35.54:8080/job/hp-upgrade-and-deploy/72/ which ran for more than an hour before I terminated it. Normal runs take ~16 minutes on HP Cloud.

Here is the log:

Started by upstream project "prepare-new-version" build number 89
[workspace] $ /bin/sh -xe /tmp/hudson1086249507721591764.sh
+ export SCRIPTS=/var/lib/jenkins/ci-cd-scripts2
+ ENVS=test-release-hp /var/lib/jenkins/ci-cd-scripts2/upgrade-and-deploy-job
+ rm buildvars.bash extracted-bin new-version.deb -rf
+ artifact=localhost:8080/job/prepare-new-version/lastSuccessfulBuild/artifact
+ wget -q localhost:8080/job/prepare-new-version/lastSuccessfulBuild/artifact/new-version.deb
+ wget -q localhost:8080/job/prepare-new-version/lastSuccessfulBuild/artifact/buildvars.bash
+ source buildvars.bash
++ export BRANCH=lp:juju-core
++ BRANCH=lp:juju-core
++ export REVNO=2116
++ REVNO=2116
+ echo 'Testing lp:juju-core 2116 on test-release-hp'
Testing lp:juju-core 2116 on test-release-hp
+ dpkg-deb -x /var/lib/jenkins/jobs/hp-upgrade-and-deploy/workspace/new-version.deb extracted-bin
+++ find extracted-bin -name juju
++ dirname extracted-bin/usr/lib/juju-1.17.0/bin/juju
+ export NEW_PATH=extracted-bin/usr/lib/juju-1.17.0/bin:/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/sbin:/sbin:/bin
+ NEW_PATH=extracted-bin/usr/lib/juju-1.17.0/bin:/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/sbin:/sbin:/bin
+ /var/lib/jenkins/ci-cd-scripts2/destroy-environment test-release-hp
juju destroy-environment -e test-release-hp -y
+ /var/lib/jenkins/ci-cd-scripts2/deploy_stack.py test-release-hp
juju bootstrap -e test-release-hp --constraints mem=2G --show-log
2013-12-03 15:20:40 INFO juju.provider.openstack provider.go:116 opening environment "test-release-hp"
2013-12-03 15:20:42 INFO juju.environs.tools tools.go:181 filtering tools by released version
2013-12-03 15:20:42 INFO juju.environs.tools tools.go:85 reading tools with major.minor version 1.16
2013-12-03 15:20:42 INFO juju.environs.tools tools.go:96 filtering tools by series: precise
2013-12-03 15:20:46 INFO juju.environs.boostrap bootstrap.go:50 bootstrapping environment "test-release-hp"
2013-12-03 15:20:46 INFO juju.environs.tools tools.go:181 filtering tools by released version
2013-12-03 15:20:46 INFO juju.environs.tools tools.go:85 reading tools with major.minor version 1.16
2013-12-03 15:20:46 INFO juju.environs.tools tools.go:96 filtering tools by series: precise
2013-12-03 15:20:48 INFO juju.environs.boostrap bootstrap.go:71 environs: picked newest version: 1.16.3
2013-12-03 15:21:04 INFO juju.provider.openstack provider.go:714 started instance "2618321"
2013-12-03 15:21:04 INFO juju supercommand.go:286 command finished
juju deploy -e test-release-hp wordpress
juju deploy -e test-release-hp mysql
juju add-relation -e test-release-hp mysql wordpress
juju expose -e test-release-hp wordpress
<test-release-hp> pending: 1, 2, mysql/0, wordpress/0
<test-release-hp> pending: 1, 2, mysql/0, wordpress/0
<test-release-hp> pending: 1, 2, mysql/0, wordpress/0
<test-release-hp> pending: 1, 2, mysql/0, wordpress/0
<test-release-hp> pending: 1, 2, mysql/0, wordpress/0
<test-release-hp> pending: 1, 2, mysql/0, wordpress/0
<test-release-hp> pending: 1, 2, mysql/0, wordpress/0
<test-release-hp> pending: 2, mysql/0, wordpress/0
<test-release-hp> pending: 2, mysql/0, wordpress/0
<test-release-hp> pending: 2, mysql/0, wordpress/0
<test-release-hp> down: 2 | pending: mysql/0, wordpress/0
<test-release-hp> installed: mysql/0
<test-release-hp> installed: mysql/0
+ EXIT_STATUS=0
+ PATH=extracted-bin/usr/lib/juju-1.17.0/bin:/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/sbin:/sbin:/bin
+ PACKAGE=/var/lib/jenkins/jobs/hp-upgrade-and-deploy/workspace/new-version.deb
+ /var/lib/jenkins/ci-cd-scripts2/test-new-version
+ which juju
extracted-bin/usr/lib/juju-1.17.0/bin/juju
++ juju --version
++ sed -r 's/([^-]*).*/\1/'
+ export version=1.17.0
+ version=1.17.0
+ for env in '$ENVS'
+ '[' test-release-hp == local ']'
+ extra_args=
+ juju upgrade-juju --show-log -e test-release-hp --version 1.17.0
2013-12-03 15:28:11 WARNING juju.environs.config config.go:279 Config attribute "tools-url" (https://region-a.geo-1.objects.hpcloudsvc.com/v1/60502529753910/juju-dist/testing/tools) is deprecated.
The location to find tools is now specified using the "tools-metadata-url" attribute.
Your configuration should be updated to set "tools-metadata-url" as follows
tools-metadata-url: https://region-a.geo-1.objects.hpcloudsvc.com/v1/60502529753910/juju-dist/testing/tools.
2013-12-03 15:28:11 INFO juju.provider.openstack provider.go:156 opening environment "test-release-hp"
2013-12-03 15:28:12 WARNING juju.environs.config config.go:279 Config attribute "tools-url" (https://region-a.geo-1.objects.hpcloudsvc.com/v1/60502529753910/juju-dist/testing/tools) is deprecated and will be ignored since
the new tools URL attribute "tools-metadata-url" has also been used.
The attribute "tools-url" should be removed from your configuration.
2013-12-03 15:28:12 WARNING juju.environs.config config.go:673 unknown config field "default-image-id"
2013-12-03 15:28:12 WARNING juju.environs.config config.go:673 unknown config field "default-instance-type"
2013-12-03 15:28:12 WARNING juju.environs.config config.go:673 unknown config field "public-bucket"
2013-12-03 15:28:12 WARNING juju.environs.config config.go:673 unknown config field "public-bucket-url"
2013-12-03 15:28:13 WARNING juju.environs.config config.go:279 Config attribute "tools-url" (https://region-a.geo-1.objects.hpcloudsvc.com/v1/60502529753910/juju-dist/testing/tools) is deprecated and will be ignored since
the new tools URL attribute "tools-metadata-url" has also been used.
The attribute "tools-url" should be removed from your configuration.
2013-12-03 15:28:16 INFO juju apiclient.go:111 state/api: dialing "wss://15.185.251.141:17070/"
2013-12-03 15:28:16 INFO juju apiclient.go:121 state/api: connection established
2013-12-03 15:28:17 WARNING juju.environs.config config.go:279 Config attribute "tools-url" (https://region-a.geo-1.objects.hpcloudsvc.com/v1/60502529753910/juju-dist/testing/tools) is deprecated and will be ignored since
the new tools URL attribute "tools-metadata-url" has also been used.
The attribute "tools-url" should be removed from your configuration.
2013-12-03 15:28:17 WARNING juju.environs.config config.go:673 unknown config field "default-instance-type"
2013-12-03 15:28:17 WARNING juju.environs.config config.go:673 unknown config field "public-bucket"
2013-12-03 15:28:17 WARNING juju.environs.config config.go:673 unknown config field "public-bucket-url"
2013-12-03 15:28:17 WARNING juju.environs.config config.go:673 unknown config field "default-image-id"
2013-12-03 15:28:18 WARNING juju.environs.config config.go:279 Config attribute "tools-url" (https://region-a.geo-1.objects.hpcloudsvc.com/v1/60502529753910/juju-dist/testing/tools) is deprecated and will be ignored since
the new tools URL attribute "tools-metadata-url" has also been used.
The attribute "tools-url" should be removed from your configuration.
2013-12-03 15:28:18 WARNING juju api.go:83 running in 1.16 compatibility mode; connection may fail if environment is just bootstrapped
2013-12-03 15:28:19 WARNING juju upgradejuju.go:169 running in 1.16 compatibility mode
2013-12-03 15:28:19 WARNING juju.environs.config config.go:279 Config attribute "tools-url" (https://region-a.geo-1.objects.hpcloudsvc.com/v1/60502529753910/juju-dist/testing/tools) is deprecated.
The location to find tools is now specified using the "tools-metadata-url" attribute.
Your configuration should be updated to set "tools-metadata-url" as follows
tools-metadata-url: https://region-a.geo-1.objects.hpcloudsvc.com/v1/60502529753910/juju-dist/testing/tools.
2013-12-03 15:28:19 INFO juju.provider.openstack provider.go:156 opening environment "test-release-hp"
2013-12-03 15:28:20 WARNING juju.environs.config config.go:279 Config attribute "tools-url" (https://region-a.geo-1.objects.hpcloudsvc.com/v1/60502529753910/juju-dist/testing/tools) is deprecated and will be ignored since
the new tools URL attribute "tools-metadata-url" has also been used.
The attribute "tools-url" should be removed from your configuration.
2013-12-03 15:28:20 WARNING juju.environs.config config.go:673 unknown config field "default-image-id"
2013-12-03 15:28:20 WARNING juju.environs.config config.go:673 unknown config field "default-instance-type"
2013-12-03 15:28:20 WARNING juju.environs.config config.go:673 unknown config field "public-bucket"
2013-12-03 15:28:20 WARNING juju.environs.config config.go:673 unknown config field "public-bucket-url"
2013-12-03 15:28:20 WARNING juju.environs.config config.go:279 Config attribute "tools-url" (https://region-a.geo-1.objects.hpcloudsvc.com/v1/60502529753910/juju-dist/testing/tools) is deprecated and will be ignored since
the new tools URL attribute "tools-metadata-url" has also been used.
The attribute "tools-url" should be removed from your configuration.
2013-12-03 15:28:24 INFO juju.state open.go:67 opening state; mongo addresses: ["15.185.251.141:37017"]; entity ""
2013-12-03 15:28:24 INFO juju.state open.go:105 connection established
2013-12-03 15:28:28 WARNING juju.environs.config config.go:279 Config attribute "tools-url" (https://region-a.geo-1.objects.hpcloudsvc.com/v1/60502529753910/juju-dist/testing/tools) is deprecated and will be ignored since
the new tools URL attribute "tools-metadata-url" has also been used.
The attribute "tools-url" should be removed from your configuration.
2013-12-03 15:28:28 WARNING juju.environs.config config.go:673 unknown config field "default-image-id"
2013-12-03 15:28:28 WARNING juju.environs.config config.go:673 unknown config field "default-instance-type"
2013-12-03 15:28:28 WARNING juju.environs.config config.go:673 unknown config field "public-bucket"
2013-12-03 15:28:28 WARNING juju.environs.config config.go:673 unknown config field "public-bucket-url"
2013-12-03 15:28:28 WARNING juju.environs.config config.go:279 Config attribute "tools-url" (https://region-a.geo-1.objects.hpcloudsvc.com/v1/60502529753910/juju-dist/testing/tools) is deprecated and will be ignored since
the new tools URL attribute "tools-metadata-url" has also been used.
The attribute "tools-url" should be removed from your configuration.
2013-12-03 15:28:29 WARNING juju.environs.config config.go:279 Config attribute "tools-url" (https://region-a.geo-1.objects.hpcloudsvc.com/v1/60502529753910/juju-dist/testing/tools) is deprecated.
The location to find tools is now specified using the "tools-metadata-url" attribute.
Your configuration should be updated to set "tools-metadata-url" as follows
tools-metadata-url: https://region-a.geo-1.objects.hpcloudsvc.com/v1/60502529753910/juju-dist/testing/tools.
2013-12-03 15:28:29 WARNING juju.environs.config config.go:279 Config attribute "tools-url" (https://region-a.geo-1.objects.hpcloudsvc.com/v1/60502529753910/juju-dist/testing/tools) is deprecated.
The location to find tools is now specified using the "tools-metadata-url" attribute.
Your configuration should be updated to set "tools-metadata-url" as follows
tools-metadata-url: https://region-a.geo-1.objects.hpcloudsvc.com/v1/60502529753910/juju-dist/testing/tools.
2013-12-03 15:28:29 INFO juju.environs.tools tools.go:87 reading tools with major version 1
2013-12-03 15:28:31 INFO juju upgradejuju.go:201 upgrade version chosen: 1.17.0
2013-12-03 15:28:31 INFO juju upgradejuju.go:203 available tools: 1.17.0-precise-amd64;1.17.0-saucy-amd64
2013-12-03 15:28:33 INFO juju upgradejuju.go:208 started upgrade to 1.17.0
2013-12-03 15:28:33 INFO juju supercommand.go:286 command finished
+ for env in '$ENVS'
+ /var/lib/jenkins/ci-cd-scripts2/wait_for_agent_update.py test-release-hp
WARNING Config attribute "tools-url" (https://region-a.geo-1.objects.hpcloudsvc.com/v1/60502529753910/juju-dist/testing/tools) is deprecated.
The location to find tools is now specified using the "tools-metadata-url" attribute.
Your configuration should be updated to set "tools-metadata-url" as follows
tools-metadata-url: https://region-a.geo-1.objects.hpcloudsvc.com/v1/60502529753910/juju-dist/testing/tools.
WARNING Config attribute "tools-url" (https://region-a.geo-1.objects.hpcloudsvc.com/v1/60502529753910/juju-dist/testing/tools) is deprecated and will be ignored since
the new tools URL attribute "tools-metadata-url" has also been used.
The attribute "tools-url" should be removed from your configuration.
WARNING unknown config field "default-image-id"
WARNING unknown config field "default-instance-type"
WARNING unknown config field "public-bucket-url"
WARNING unknown config field "public-bucket"
WARNING Config attribute "tools-url" (https://region-a.geo-1.objects.hpcloudsvc.com/v1/60502529753910/juju-dist/testing/tools) is deprecated and will be ignored since
the new tools URL attribute "tools-metadata-url" has also been used.
The attribute "tools-url" should be removed from your configuration.
WARNING Config attribute "tools-url" (https://region-a.geo-1.objects.hpcloudsvc.com/v1/60502529753910/juju-dist/testing/tools) is deprecated and will be ignored since
the new tools URL attribute "tools-metadata-url" has also been used.
The attribute "tools-url" should be removed from your configuration.
WARNING unknown config field "default-image-id"
WARNING unknown config field "public-bucket-url"
WARNING unknown config field "public-bucket"
WARNING unknown config field "default-instance-type"
WARNING Config attribute "tools-url" (https://region-a.geo-1.objects.hpcloudsvc.com/v1/60502529753910/juju-dist/testing/tools) is deprecated and will be ignored since
the new tools URL attribute "tools-metadata-url" has also been used.
The attribute "tools-url" should be removed from your configuration.
WARNING Config attribute "tools-url" (https://region-a.geo-1.objects.hpcloudsvc.com/v1/60502529753910/juju-dist/testing/tools) is deprecated.
The location to find tools is now specified using the "tools-metadata-url" attribute.
Your configuration should be updated to set "tools-metadata-url" as follows
tools-metadata-url: https://region-a.geo-1.objects.hpcloudsvc.com/v1/60502529753910/juju-dist/testing/tools.
<test-release-hp> 1.16.3: wordpress/0
WARNING Config attribute "tools-url" (https://region-a.geo-1.objects.hpcloudsvc.com/v1/60502529753910/juju-dist/testing/tools) is deprecated.
The location to find tools is now specified using the "tools-metadata-url" attribute.
Your configuration should be updated to set "tools-metadata-url" as follows
tools-metadata-url: https://region-a.geo-1.objects.hpcloudsvc.com/v1/60502529753910/juju-dist/testing/tools.
WARNING Config attribute "tools-url" (https://region-a.geo-1.objects.hpcloudsvc.com/v1/60502529753910/juju-dist/testing/tools) is deprecated and will be ignored since
the new tools URL attribute "tools-metadata-url" has also been used.
The attribute "tools-url" should be removed from your configuration.
WARNING unknown config field "default-image-id"
WARNING unknown config field "default-instance-type"
WARNING unknown config field "public-bucket-url"
WARNING unknown config field "public-bucket"
WARNING Config attribute "tools-url" (https://region-a.geo-1.objects.hpcloudsvc.com/v1/60502529753910/juju-dist/testing/tools) is deprecated and will be ignored since
the new tools URL attribute "tools-metadata-url" has also been used.
The attribute "tools-url" should be removed from your configuration.
WARNING Config attribute "tools-url" (https://region-a.geo-1.objects.hpcloudsvc.com/v1/60502529753910/juju-dist/testing/tools) is deprecated and will be ignored since
the new tools URL attribute "tools-metadata-url" has also been used.
The attribute "tools-url" should be removed from your configuration.
WARNING unknown config field "default-image-id"
WARNING unknown config field "default-instance-type"
WARNING unknown config field "public-bucket"
WARNING unknown config field "public-bucket-url"
WARNING Config attribute "tools-url" (https://region-a.geo-1.objects.hpcloudsvc.com/v1/60502529753910/juju-dist/testing/tools) is deprecated and will be ignored since
the new tools URL attribute "tools-metadata-url" has also been used.
The attribute "tools-url" should be removed from your configuration.
WARNING Config attribute "tools-url" (https://region-a.geo-1.objects.hpcloudsvc.com/v1/60502529753910/juju-dist/testing/tools) is deprecated.
The location to find tools is now specified using the "tools-metadata-url" attribute.
Your configuration should be updated to set "tools-metadata-url" as follows
tools-metadata-url: https://region-a.geo-1.objects.hpcloudsvc.com/v1/60502529753910/juju-dist/testing/tools.
+ /var/lib/jenkins/ci-cd-scripts2/destroy-environment test-release-hp
juju destroy-environment test-release-hp -y
WARNING Config attribute "tools-url" (https://region-a.geo-1.objects.hpcloudsvc.com/v1/60502529753910/juju-dist/testing/tools) is deprecated.
The location to find tools is now specified using the "tools-metadata-url" attribute.
Your configuration should be updated to set "tools-metadata-url" as follows
tools-metadata-url: https://region-a.geo-1.objects.hpcloudsvc.com/v1/60502529753910/juju-dist/testing/tools.
WARNING Config attribute "tools-url" (https://region-a.geo-1.objects.hpcloudsvc.com/v1/60502529753910/juju-dist/testing/tools) is deprecated and will be ignored since
the new tools URL attribute "tools-metadata-url" has also been used.
The attribute "tools-url" should be removed from your configuration.
WARNING unknown config field "default-image-id"
WARNING unknown config field "public-bucket"
WARNING unknown config field "public-bucket-url"
WARNING unknown config field "default-instance-type"
WARNING Config attribute "tools-url" (https://region-a.geo-1.objects.hpcloudsvc.com/v1/60502529753910/juju-dist/testing/tools) is deprecated and will be ignored since
the new tools URL attribute "tools-metadata-url" has also been used.
The attribute "tools-url" should be removed from your configuration.
+ sleep 5
+ /var/lib/jenkins/ci-cd-scripts2/deploy_stack.py test-release-hp
juju bootstrap -e test-release-hp --constraints mem=2G --show-log
2013-12-03 15:29:16 WARNING juju.environs.config config.go:279 Config attribute "tools-url" (https://region-a.geo-1.objects.hpcloudsvc.com/v1/60502529753910/juju-dist/testing/tools) is deprecated.
The location to find tools is now specified using the "tools-metadata-url" attribute.
Your configuration should be updated to set "tools-metadata-url" as follows
tools-metadata-url: https://region-a.geo-1.objects.hpcloudsvc.com/v1/60502529753910/juju-dist/testing/tools.
2013-12-03 15:29:20 WARNING juju.environs.config config.go:279 Config attribute "tools-url" (https://region-a.geo-1.objects.hpcloudsvc.com/v1/60502529753910/juju-dist/testing/tools) is deprecated and will be ignored since
the new tools URL attribute "tools-metadata-url" has also been used.
The attribute "tools-url" should be removed from your configuration.
2013-12-03 15:29:21 WARNING juju.environs.config config.go:279 Config attribute "tools-url" (https://region-a.geo-1.objects.hpcloudsvc.com/v1/60502529753910/juju-dist/testing/tools) is deprecated and will be ignored since
the new tools URL attribute "tools-metadata-url" has also been used.
The attribute "tools-url" should be removed from your configuration.
2013-12-03 15:29:21 INFO juju.provider.openstack provider.go:156 opening environment "test-release-hp"
2013-12-03 15:29:21 WARNING juju.environs.config config.go:279 Config attribute "tools-url" (https://region-a.geo-1.objects.hpcloudsvc.com/v1/60502529753910/juju-dist/testing/tools) is deprecated and will be ignored since
the new tools URL attribute "tools-metadata-url" has also been used.
The attribute "tools-url" should be removed from your configuration.
2013-12-03 15:29:22 WARNING juju.environs.config config.go:279 Config attribute "tools-url" (https://region-a.geo-1.objects.hpcloudsvc.com/v1/60502529753910/juju-dist/testing/tools) is deprecated and will be ignored since
the new tools URL attribute "tools-metadata-url" has also been used.
The attribute "tools-url" should be removed from your configuration.
2013-12-03 15:29:26 INFO juju.environs.bootstrap bootstrap.go:45 bootstrapping environment "test-release-hp"
2013-12-03 15:29:27 INFO juju.environs.tools tools.go:85 reading tools with major.minor version 1.17
2013-12-03 15:29:27 INFO juju.environs.tools tools.go:96 filtering tools by series: precise
2013-12-03 15:29:30 INFO juju.environs.bootstrap bootstrap.go:57 picked newest version: 1.17.0
2013-12-03 15:29:30 WARNING juju.environs.config config.go:279 Config attribute "tools-url" (https://region-a.geo-1.objects.hpcloudsvc.com/v1/60502529753910/juju-dist/testing/tools) is deprecated and will be ignored since
the new tools URL attribute "tools-metadata-url" has also been used.
The attribute "tools-url" should be removed from your configuration.
2013-12-03 15:29:31 WARNING juju.environs.config config.go:279 Config attribute "tools-url" (https://region-a.geo-1.objects.hpcloudsvc.com/v1/60502529753910/juju-dist/testing/tools) is deprecated and will be ignored since
the new tools URL attribute "tools-metadata-url" has also been used.
The attribute "tools-url" should be removed from your configuration.
2013-12-03 15:29:32 WARNING juju.environs.config config.go:279 Config attribute "tools-url" (https://region-a.geo-1.objects.hpcloudsvc.com/v1/60502529753910/juju-dist/testing/tools) is deprecated and will be ignored since
the new tools URL attribute "tools-metadata-url" has also been used.
The attribute "tools-url" should be removed from your configuration.
Launching instance
2013-12-03 15:29:36 WARNING juju.environs.config config.go:279 Config attribute "tools-url" (https://region-a.geo-1.objects.hpcloudsvc.com/v1/60502529753910/juju-dist/testing/tools) is deprecated and will be ignored since
the new tools URL attribute "tools-metadata-url" has also been used.
The attribute "tools-url" should be removed from your configuration.
2013-12-03 15:29:37 WARNING juju.environs.config config.go:279 Config attribute "tools-url" (https://region-a.geo-1.objects.hpcloudsvc.com/v1/60502529753910/juju-dist/testing/tools) is deprecated and will be ignored since
the new tools URL attribute "tools-metadata-url" has also been used.
The attribute "tools-url" should be removed from your configuration.
2013-12-03 15:29:37 WARNING juju.environs.config config.go:279 Config attribute "tools-url" (https://region-a.geo-1.objects.hpcloudsvc.com/v1/60502529753910/juju-dist/testing/tools) is deprecated and will be ignored since
the new tools URL attribute "tools-metadata-url" has also been used.
The attribute "tools-url" should be removed from your configuration.
2013-12-03 15:29:50 INFO juju.provider.openstack provider.go:733 started instance "2618373"
 - 2618373
Waiting for DNS name.
 - 10.2.53.125
Attempting to connect to 10.2.53.125:22........................................................................................................................................................................................................................................................................................................................................................Command '('juju', 'bootstrap', '-e', 'test-release-hp', '--constraints', 'mem=2G', '--show-log')' returned non-zero exit status 2
/var/lib/jenkins/ci-cd-scripts2/upgrade-and-deploy-job: line 36: 13650 Terminated PATH=$NEW_PATH PACKAGE=$WORKSPACE/new-version.deb $SCRIPTS/test-new-version
+ EXIT_STATUS=143
+ '[' 143 -ne 0 ']'
+ dump_logs
+ artifacts_path=/var/lib/jenkins/jobs/hp-upgrade-and-deploy/workspace/artifacts
+ mkdir -p /var/lib/jenkins/jobs/hp-upgrade-and-deploy/workspace/artifacts
Terminated
Build was aborted
Description set: lp:juju-core r2116
[BFA] Scanning build for known causes...

[BFA] Done. 0s
Finished: ABORTED

Related branches

Revision history for this message
John A Meinel (jameinel) wrote : Re: [Bug 1257427] [NEW] bootstrap doesn't know when to give up

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 2013-12-03 22:58, Aaron Bentley wrote:
> Public bug reported:
>
> Attempting to bootstrap hpcloud on 1.17 is neither succeeding or
> failing, but hanging. For CI, we need operations to either succeed
> or fail. Waiting forever is no good.

...

> Attempting to connect to 10.2.53.125:22 ......Command '('juju',
> 'bootstrap', '-e', 'test-release-hp', '--constraints', 'mem=2G',
> '--show-log')' returned non-zero exit status 2

By my count there are 345 dots there.

- From looking at the loop, it polls 1/second to wait for a DNS name,
and then a 'try to connect with a 5s timeout' and 'sleep 5s' before
trying again. So that could be 10s between dots, which would be ~1hr.

Given we are waiting for an instance to come up, I think our old
timeout was around 10min (which is generous, but not out of scope for
clouds like Azure).

As currently implemented "waitSSH" just uses a raw for{} loop that
waits until signalled, so we probably do want an overall timeout
there. I'll see if we have an easy 'wait no more than 10 minutes'
policy, and then implement it.

John
=:->

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.13 (Cygwin)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iEYEARECAAYFAlKe7g8ACgkQJdeBCYSNAAPltwCgiWOf1YFtoN0ECtaXBQeSQf8N
FBMAnjKh032jxBGFyrMoS/oS9ki9kuj0
=sP+0
-----END PGP SIGNATURE-----

Revision history for this message
John A Meinel (jameinel) wrote :

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 2013-12-03 22:58, Aaron Bentley wrote:
> Public bug reported:
>
> Attempting to bootstrap hpcloud on 1.17 is neither succeeding or
> failing, but hanging. For CI, we need operations to either succeed
> or fail. Waiting forever is no good.
>

DefaultDialOpts (which is how long something like Status would wait
for the server to be up before giving up) is 10min timeout with a 2s
wait delay.

I'll look to do something with that here.

John
=:->

  status: triaged
  importance: high
  assignee: jameinel

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.13 (Cygwin)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iEYEARECAAYFAlKe7ngACgkQJdeBCYSNAANkHwCgwecwCiYr4OqS3jVf6Dnl9HFd
N/0AoKfnCoQ9xM4GqhMM4f0nIrtboVI0
=qBjn
-----END PGP SIGNATURE-----

Changed in juju-core:
assignee: nobody → John A Meinel (jameinel)
importance: Critical → High
Go Bot (go-bot)
Changed in juju-core:
status: Triaged → Fix Committed
Curtis Hovey (sinzui)
Changed in juju-core:
milestone: none → 1.17.0
Curtis Hovey (sinzui)
Changed in juju-core:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.