bootstrap doesn't know when to give up
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
juju-core |
Fix Released
|
High
|
John A Meinel |
Bug Description
Attempting to bootstrap hpcloud on 1.17 is neither succeeding or failing, but hanging. For CI, we need operations to either succeed or fail. Waiting forever is no good.
We see this only in 1.17. On 1.16.4, hpcloud deploys fine.
This was seen in http://
Here is the log:
Started by upstream project "prepare-
[workspace] $ /bin/sh -xe /tmp/hudson1086
+ export SCRIPTS=
+ ENVS=test-
+ rm buildvars.bash extracted-bin new-version.deb -rf
+ artifact=
+ wget -q localhost:
+ wget -q localhost:
+ source buildvars.bash
++ export BRANCH=lp:juju-core
++ BRANCH=lp:juju-core
++ export REVNO=2116
++ REVNO=2116
+ echo 'Testing lp:juju-core 2116 on test-release-hp'
Testing lp:juju-core 2116 on test-release-hp
+ dpkg-deb -x /var/lib/
+++ find extracted-bin -name juju
++ dirname extracted-
+ export NEW_PATH=
+ NEW_PATH=
+ /var/lib/
juju destroy-environment -e test-release-hp -y
+ /var/lib/
juju bootstrap -e test-release-hp --constraints mem=2G --show-log
2013-12-03 15:20:40 INFO juju.provider.
2013-12-03 15:20:42 INFO juju.environs.tools tools.go:181 filtering tools by released version
2013-12-03 15:20:42 INFO juju.environs.tools tools.go:85 reading tools with major.minor version 1.16
2013-12-03 15:20:42 INFO juju.environs.tools tools.go:96 filtering tools by series: precise
2013-12-03 15:20:46 INFO juju.environs.
2013-12-03 15:20:46 INFO juju.environs.tools tools.go:181 filtering tools by released version
2013-12-03 15:20:46 INFO juju.environs.tools tools.go:85 reading tools with major.minor version 1.16
2013-12-03 15:20:46 INFO juju.environs.tools tools.go:96 filtering tools by series: precise
2013-12-03 15:20:48 INFO juju.environs.
2013-12-03 15:21:04 INFO juju.provider.
2013-12-03 15:21:04 INFO juju supercommand.go:286 command finished
juju deploy -e test-release-hp wordpress
juju deploy -e test-release-hp mysql
juju add-relation -e test-release-hp mysql wordpress
juju expose -e test-release-hp wordpress
<test-release-hp> pending: 1, 2, mysql/0, wordpress/0
<test-release-hp> pending: 1, 2, mysql/0, wordpress/0
<test-release-hp> pending: 1, 2, mysql/0, wordpress/0
<test-release-hp> pending: 1, 2, mysql/0, wordpress/0
<test-release-hp> pending: 1, 2, mysql/0, wordpress/0
<test-release-hp> pending: 1, 2, mysql/0, wordpress/0
<test-release-hp> pending: 1, 2, mysql/0, wordpress/0
<test-release-hp> pending: 2, mysql/0, wordpress/0
<test-release-hp> pending: 2, mysql/0, wordpress/0
<test-release-hp> pending: 2, mysql/0, wordpress/0
<test-release-hp> down: 2 | pending: mysql/0, wordpress/0
<test-release-hp> installed: mysql/0
<test-release-hp> installed: mysql/0
+ EXIT_STATUS=0
+ PATH=extracted-
+ PACKAGE=
+ /var/lib/
+ which juju
extracted-
++ juju --version
++ sed -r 's/([^-]*).*/\1/'
+ export version=1.17.0
+ version=1.17.0
+ for env in '$ENVS'
+ '[' test-release-hp == local ']'
+ extra_args=
+ juju upgrade-juju --show-log -e test-release-hp --version 1.17.0
2013-12-03 15:28:11 WARNING juju.environs.
The location to find tools is now specified using the "tools-
Your configuration should be updated to set "tools-
tools-metadata-url: https:/
2013-12-03 15:28:11 INFO juju.provider.
2013-12-03 15:28:12 WARNING juju.environs.
the new tools URL attribute "tools-
The attribute "tools-url" should be removed from your configuration.
2013-12-03 15:28:12 WARNING juju.environs.
2013-12-03 15:28:12 WARNING juju.environs.
2013-12-03 15:28:12 WARNING juju.environs.
2013-12-03 15:28:12 WARNING juju.environs.
2013-12-03 15:28:13 WARNING juju.environs.
the new tools URL attribute "tools-
The attribute "tools-url" should be removed from your configuration.
2013-12-03 15:28:16 INFO juju apiclient.go:111 state/api: dialing "wss://
2013-12-03 15:28:16 INFO juju apiclient.go:121 state/api: connection established
2013-12-03 15:28:17 WARNING juju.environs.
the new tools URL attribute "tools-
The attribute "tools-url" should be removed from your configuration.
2013-12-03 15:28:17 WARNING juju.environs.
2013-12-03 15:28:17 WARNING juju.environs.
2013-12-03 15:28:17 WARNING juju.environs.
2013-12-03 15:28:17 WARNING juju.environs.
2013-12-03 15:28:18 WARNING juju.environs.
the new tools URL attribute "tools-
The attribute "tools-url" should be removed from your configuration.
2013-12-03 15:28:18 WARNING juju api.go:83 running in 1.16 compatibility mode; connection may fail if environment is just bootstrapped
2013-12-03 15:28:19 WARNING juju upgradejuju.go:169 running in 1.16 compatibility mode
2013-12-03 15:28:19 WARNING juju.environs.
The location to find tools is now specified using the "tools-
Your configuration should be updated to set "tools-
tools-metadata-url: https:/
2013-12-03 15:28:19 INFO juju.provider.
2013-12-03 15:28:20 WARNING juju.environs.
the new tools URL attribute "tools-
The attribute "tools-url" should be removed from your configuration.
2013-12-03 15:28:20 WARNING juju.environs.
2013-12-03 15:28:20 WARNING juju.environs.
2013-12-03 15:28:20 WARNING juju.environs.
2013-12-03 15:28:20 WARNING juju.environs.
2013-12-03 15:28:20 WARNING juju.environs.
the new tools URL attribute "tools-
The attribute "tools-url" should be removed from your configuration.
2013-12-03 15:28:24 INFO juju.state open.go:67 opening state; mongo addresses: ["15.185.
2013-12-03 15:28:24 INFO juju.state open.go:105 connection established
2013-12-03 15:28:28 WARNING juju.environs.
the new tools URL attribute "tools-
The attribute "tools-url" should be removed from your configuration.
2013-12-03 15:28:28 WARNING juju.environs.
2013-12-03 15:28:28 WARNING juju.environs.
2013-12-03 15:28:28 WARNING juju.environs.
2013-12-03 15:28:28 WARNING juju.environs.
2013-12-03 15:28:28 WARNING juju.environs.
the new tools URL attribute "tools-
The attribute "tools-url" should be removed from your configuration.
2013-12-03 15:28:29 WARNING juju.environs.
The location to find tools is now specified using the "tools-
Your configuration should be updated to set "tools-
tools-metadata-url: https:/
2013-12-03 15:28:29 WARNING juju.environs.
The location to find tools is now specified using the "tools-
Your configuration should be updated to set "tools-
tools-metadata-url: https:/
2013-12-03 15:28:29 INFO juju.environs.tools tools.go:87 reading tools with major version 1
2013-12-03 15:28:31 INFO juju upgradejuju.go:201 upgrade version chosen: 1.17.0
2013-12-03 15:28:31 INFO juju upgradejuju.go:203 available tools: 1.17.0-
2013-12-03 15:28:33 INFO juju upgradejuju.go:208 started upgrade to 1.17.0
2013-12-03 15:28:33 INFO juju supercommand.go:286 command finished
+ for env in '$ENVS'
+ /var/lib/
WARNING Config attribute "tools-url" (https:/
The location to find tools is now specified using the "tools-
Your configuration should be updated to set "tools-
tools-metadata-url: https:/
WARNING Config attribute "tools-url" (https:/
the new tools URL attribute "tools-
The attribute "tools-url" should be removed from your configuration.
WARNING unknown config field "default-image-id"
WARNING unknown config field "default-
WARNING unknown config field "public-bucket-url"
WARNING unknown config field "public-bucket"
WARNING Config attribute "tools-url" (https:/
the new tools URL attribute "tools-
The attribute "tools-url" should be removed from your configuration.
WARNING Config attribute "tools-url" (https:/
the new tools URL attribute "tools-
The attribute "tools-url" should be removed from your configuration.
WARNING unknown config field "default-image-id"
WARNING unknown config field "public-bucket-url"
WARNING unknown config field "public-bucket"
WARNING unknown config field "default-
WARNING Config attribute "tools-url" (https:/
the new tools URL attribute "tools-
The attribute "tools-url" should be removed from your configuration.
WARNING Config attribute "tools-url" (https:/
The location to find tools is now specified using the "tools-
Your configuration should be updated to set "tools-
tools-metadata-url: https:/
<test-release-hp> 1.16.3: wordpress/0
WARNING Config attribute "tools-url" (https:/
The location to find tools is now specified using the "tools-
Your configuration should be updated to set "tools-
tools-metadata-url: https:/
WARNING Config attribute "tools-url" (https:/
the new tools URL attribute "tools-
The attribute "tools-url" should be removed from your configuration.
WARNING unknown config field "default-image-id"
WARNING unknown config field "default-
WARNING unknown config field "public-bucket-url"
WARNING unknown config field "public-bucket"
WARNING Config attribute "tools-url" (https:/
the new tools URL attribute "tools-
The attribute "tools-url" should be removed from your configuration.
WARNING Config attribute "tools-url" (https:/
the new tools URL attribute "tools-
The attribute "tools-url" should be removed from your configuration.
WARNING unknown config field "default-image-id"
WARNING unknown config field "default-
WARNING unknown config field "public-bucket"
WARNING unknown config field "public-bucket-url"
WARNING Config attribute "tools-url" (https:/
the new tools URL attribute "tools-
The attribute "tools-url" should be removed from your configuration.
WARNING Config attribute "tools-url" (https:/
The location to find tools is now specified using the "tools-
Your configuration should be updated to set "tools-
tools-metadata-url: https:/
+ /var/lib/
juju destroy-environment test-release-hp -y
WARNING Config attribute "tools-url" (https:/
The location to find tools is now specified using the "tools-
Your configuration should be updated to set "tools-
tools-metadata-url: https:/
WARNING Config attribute "tools-url" (https:/
the new tools URL attribute "tools-
The attribute "tools-url" should be removed from your configuration.
WARNING unknown config field "default-image-id"
WARNING unknown config field "public-bucket"
WARNING unknown config field "public-bucket-url"
WARNING unknown config field "default-
WARNING Config attribute "tools-url" (https:/
the new tools URL attribute "tools-
The attribute "tools-url" should be removed from your configuration.
+ sleep 5
+ /var/lib/
juju bootstrap -e test-release-hp --constraints mem=2G --show-log
2013-12-03 15:29:16 WARNING juju.environs.
The location to find tools is now specified using the "tools-
Your configuration should be updated to set "tools-
tools-metadata-url: https:/
2013-12-03 15:29:20 WARNING juju.environs.
the new tools URL attribute "tools-
The attribute "tools-url" should be removed from your configuration.
2013-12-03 15:29:21 WARNING juju.environs.
the new tools URL attribute "tools-
The attribute "tools-url" should be removed from your configuration.
2013-12-03 15:29:21 INFO juju.provider.
2013-12-03 15:29:21 WARNING juju.environs.
the new tools URL attribute "tools-
The attribute "tools-url" should be removed from your configuration.
2013-12-03 15:29:22 WARNING juju.environs.
the new tools URL attribute "tools-
The attribute "tools-url" should be removed from your configuration.
2013-12-03 15:29:26 INFO juju.environs.
2013-12-03 15:29:27 INFO juju.environs.tools tools.go:85 reading tools with major.minor version 1.17
2013-12-03 15:29:27 INFO juju.environs.tools tools.go:96 filtering tools by series: precise
2013-12-03 15:29:30 INFO juju.environs.
2013-12-03 15:29:30 WARNING juju.environs.
the new tools URL attribute "tools-
The attribute "tools-url" should be removed from your configuration.
2013-12-03 15:29:31 WARNING juju.environs.
the new tools URL attribute "tools-
The attribute "tools-url" should be removed from your configuration.
2013-12-03 15:29:32 WARNING juju.environs.
the new tools URL attribute "tools-
The attribute "tools-url" should be removed from your configuration.
Launching instance
2013-12-03 15:29:36 WARNING juju.environs.
the new tools URL attribute "tools-
The attribute "tools-url" should be removed from your configuration.
2013-12-03 15:29:37 WARNING juju.environs.
the new tools URL attribute "tools-
The attribute "tools-url" should be removed from your configuration.
2013-12-03 15:29:37 WARNING juju.environs.
the new tools URL attribute "tools-
The attribute "tools-url" should be removed from your configuration.
2013-12-03 15:29:50 INFO juju.provider.
- 2618373
Waiting for DNS name.
- 10.2.53.125
Attempting to connect to 10.2.53.
/var/lib/
+ EXIT_STATUS=143
+ '[' 143 -ne 0 ']'
+ dump_logs
+ artifacts_
+ mkdir -p /var/lib/
Terminated
Build was aborted
Description set: lp:juju-core r2116
[BFA] Scanning build for known causes...
[BFA] Done. 0s
Finished: ABORTED
Related branches
- Juju Engineering: Pending requested
-
Diff: 251 lines (+157/-7)3 files modifiedprovider/common/bootstrap.go (+55/-7)
provider/common/bootstrap_test.go (+101/-0)
provider/common/export_test.go (+1/-0)
Changed in juju-core: | |
status: | Triaged → Fix Committed |
Changed in juju-core: | |
milestone: | none → 1.17.0 |
Changed in juju-core: | |
status: | Fix Committed → Fix Released |
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
On 2013-12-03 22:58, Aaron Bentley wrote:
> Public bug reported:
>
> Attempting to bootstrap hpcloud on 1.17 is neither succeeding or
> failing, but hanging. For CI, we need operations to either succeed
> or fail. Waiting forever is no good.
...
> Attempting to connect to 10.2.53.125:22 ......Command '('juju',
> 'bootstrap', '-e', 'test-release-hp', '--constraints', 'mem=2G',
> '--show-log')' returned non-zero exit status 2
By my count there are 345 dots there.
- From looking at the loop, it polls 1/second to wait for a DNS name,
and then a 'try to connect with a 5s timeout' and 'sleep 5s' before
trying again. So that could be 10s between dots, which would be ~1hr.
Given we are waiting for an instance to come up, I think our old
timeout was around 10min (which is generous, but not out of scope for
clouds like Azure).
As currently implemented "waitSSH" just uses a raw for{} loop that
waits until signalled, so we probably do want an overall timeout
there. I'll see if we have an easy 'wait no more than 10 minutes'
policy, and then implement it.
John
=:->
-----BEGIN PGP SIGNATURE----- www.enigmail. net/
e7g8ACgkQJdeBCY SNAAPltwCgiWOf1 YFtoN0ECtaXBQeS Qf8N FyrMoS/ oS9ki9kuj0
Version: GnuPG v1.4.13 (Cygwin)
Comment: Using GnuPG with Thunderbird - http://
iEYEARECAAYFAlK
FBMAnjKh032jxBG
=sP+0
-----END PGP SIGNATURE-----