Merge lp:~evarlast/charms/trusty/mongodb/fix-dump-actions into lp:charms/trusty/mongodb

[Jay R. Wren]

Dump wasn't working. Now it works and if args has spaces, they can be escaped with quotes.

e.g.

$ juju action do mongodb/0 dump args='-q "{a:1, b:2}"'
Action queued with id: 7b627526-8b0c-4f2e-8494-bdb13dabe23d
$ juju action fetch 7b
results:
  command: mongodump -q "{a:1, b:2}"

[uosci-testing-bot (uosci-testing-bot)]

charm_lint_check #13430 mongodb for evarlast mp277191
    LINT OK: passed

Build: http://10.245.162.77:8080/job/charm_lint_check/13430/

[uosci-testing-bot (uosci-testing-bot)]

charm_unit_test #12564 mongodb for evarlast mp277191
    UNIT OK: passed

Build: http://10.245.162.77:8080/job/charm_unit_test/12564/

[uosci-testing-bot (uosci-testing-bot)]

charm_amulet_test #7839 mongodb for evarlast mp277191
    AMULET OK: passed

Build: http://10.245.162.77:8080/job/charm_amulet_test/7839/

[Tim Van Steenburgh (tvansteenburgh)]

Charm CI tests failed due to hardcoded unit numbers in the tests. Jenkins log: http://juju-ci.vapour.ws:8080/job/charm-bundle-test-lxc/1432/console

To fix, replace occurrences of `d.sentry.unit['mongodb/0']` with `d.sentry['mongodb'][0]` in the test files (tests/*).

review: Needs Fixing

[lp:~evarlast/charms/trusty/mongodb/fix-dump-actions]

80. By Jay R. Wren on 2015-11-12

fix hardcoded unit numbers in the tests

[uosci-testing-bot (uosci-testing-bot)]

charm_unit_test #12615 mongodb for evarlast mp277191
    UNIT OK: passed

Build: http://10.245.162.77:8080/job/charm_unit_test/12615/

[uosci-testing-bot (uosci-testing-bot)]

charm_lint_check #13484 mongodb for evarlast mp277191
    LINT OK: passed

Build: http://10.245.162.77:8080/job/charm_lint_check/13484/

[Matthew Williams (mattyw)]

LGTM

review: Approve

[uosci-testing-bot (uosci-testing-bot)]

charm_amulet_test #7845 mongodb for evarlast mp277191
    AMULET FAIL: amulet-test failed

AMULET Results (max last 2 lines):
make: *** [functional_test] Error 1
ERROR:root:Make target returned non-zero.

Full amulet test output: http://paste.ubuntu.com/13240036/
Build: http://10.245.162.77:8080/job/charm_amulet_test/7845/

[Review Queue (review-queue)]

This item has failed automated testing! Results available here http://juju-ci.vapour.ws:8080/job/charm-bundle-test-aws/1421/

review: Needs Fixing (automated testing)

[lp:~evarlast/charms/trusty/mongodb/fix-dump-actions]

81. By Jay R. Wren on 2015-11-13

use sentry[service][N] instead of sentry.unit[unit]

82. By Jay R. Wren on 2015-11-13

categories is now tags.

config key now defaults to ""

[uosci-testing-bot (uosci-testing-bot)]

charm_lint_check #13701 mongodb for evarlast mp277191
    LINT OK: passed

Build: http://10.245.162.77:8080/job/charm_lint_check/13701/

[uosci-testing-bot (uosci-testing-bot)]

charm_unit_test #12771 mongodb for evarlast mp277191
    UNIT OK: passed

Build: http://10.245.162.77:8080/job/charm_unit_test/12771/

[uosci-testing-bot (uosci-testing-bot)]

charm_amulet_test #7858 mongodb for evarlast mp277191
    AMULET FAIL: amulet-test failed

AMULET Results (max last 2 lines):
make: *** [functional_test] Error 1
ERROR:root:Make target returned non-zero.

Full amulet test output: http://paste.ubuntu.com/13249011/
Build: http://10.245.162.77:8080/job/charm_amulet_test/7858/

[lp:~evarlast/charms/trusty/mongodb/fix-dump-actions]

83. By Jay R. Wren on 2015-11-17

some ipv6 support

[uosci-testing-bot (uosci-testing-bot)]

charm_lint_check #13928 mongodb for evarlast mp277191
    LINT FAIL: lint-test failed

LINT Results (max last 2 lines):
make: *** [lint] Error 1
ERROR:root:Make target returned non-zero.

Full lint test output: http://paste.ubuntu.com/13318610/
Build: http://10.245.162.77:8080/job/charm_lint_check/13928/

[uosci-testing-bot (uosci-testing-bot)]

charm_unit_test #12984 mongodb for evarlast mp277191
    UNIT OK: passed

Build: http://10.245.162.77:8080/job/charm_unit_test/12984/

[uosci-testing-bot (uosci-testing-bot)]

charm_amulet_test #7911 mongodb for evarlast mp277191
    AMULET FAIL: amulet-test failed

AMULET Results (max last 2 lines):
make: *** [functional_test] Error 1
ERROR:root:Make target returned non-zero.

Full amulet test output: http://paste.ubuntu.com/13318703/
Build: http://10.245.162.77:8080/job/charm_amulet_test/7911/

[lp:~evarlast/charms/trusty/mongodb/fix-dump-actions]

84. By Jay R. Wren on 2015-11-17

fix ipv6 url

85. By Jay R. Wren on 2015-11-17

fix ipv6 url

86. By Jay R. Wren on 2015-11-18

fix lint

[uosci-testing-bot (uosci-testing-bot)]

charm_lint_check #14010 mongodb for evarlast mp277191
    LINT OK: passed

Build: http://10.245.162.77:8080/job/charm_lint_check/14010/

[uosci-testing-bot (uosci-testing-bot)]

charm_unit_test #13061 mongodb for evarlast mp277191
    UNIT OK: passed

Build: http://10.245.162.77:8080/job/charm_unit_test/13061/

[uosci-testing-bot (uosci-testing-bot)]

charm_amulet_test #7939 mongodb for evarlast mp277191
    AMULET OK: passed

Build: http://10.245.162.77:8080/job/charm_amulet_test/7939/

[lp:~evarlast/charms/trusty/mongodb/fix-dump-actions]

87. By Jay R. Wren on 2015-11-19

fix broken amulet tests

we do not know which is master, cannot assume it is /0, so try 0 and if it fails try /1.

enable ipv6 in mongodb. We cannot know which provider is being used and if the public-address may be ipv6, so we must enable ipv6

[uosci-testing-bot (uosci-testing-bot)]

charm_lint_check #14066 mongodb for evarlast mp277191
    LINT FAIL: lint-test failed

LINT Results (max last 2 lines):
make: *** [lint] Error 1
ERROR:root:Make target returned non-zero.

Full lint test output: http://paste.ubuntu.com/13340748/
Build: http://10.245.162.77:8080/job/charm_lint_check/14066/

[uosci-testing-bot (uosci-testing-bot)]

charm_unit_test #13112 mongodb for evarlast mp277191
    UNIT OK: passed

Build: http://10.245.162.77:8080/job/charm_unit_test/13112/

[uosci-testing-bot (uosci-testing-bot)]

charm_amulet_test #7941 mongodb for evarlast mp277191
    AMULET FAIL: amulet-test failed

AMULET Results (max last 2 lines):
make: *** [functional_test] Error 1
ERROR:root:Make target returned non-zero.

Full amulet test output: http://paste.ubuntu.com/13341016/
Build: http://10.245.162.77:8080/job/charm_amulet_test/7941/

[lp:~evarlast/charms/trusty/mongodb/fix-dump-actions]

88. By Jay R. Wren on 2015-11-20

use sentry.wait() instead of time.sleep - much faster.

[uosci-testing-bot (uosci-testing-bot)]

charm_lint_check #14136 mongodb for evarlast mp277191
    LINT FAIL: lint-test failed

LINT Results (max last 2 lines):
make: *** [lint] Error 1
ERROR:root:Make target returned non-zero.

Full lint test output: http://paste.ubuntu.com/13367320/
Build: http://10.245.162.77:8080/job/charm_lint_check/14136/

[uosci-testing-bot (uosci-testing-bot)]

charm_unit_test #13176 mongodb for evarlast mp277191
    UNIT OK: passed

Build: http://10.245.162.77:8080/job/charm_unit_test/13176/

[lp:~evarlast/charms/trusty/mongodb/fix-dump-actions]

89. By Jay R. Wren on 2015-11-20

fix lint

[uosci-testing-bot (uosci-testing-bot)]

charm_lint_check #14137 mongodb for evarlast mp277191
    LINT OK: passed

Build: http://10.245.162.77:8080/job/charm_lint_check/14137/

[uosci-testing-bot (uosci-testing-bot)]

charm_unit_test #13177 mongodb for evarlast mp277191
    UNIT OK: passed

Build: http://10.245.162.77:8080/job/charm_unit_test/13177/

[uosci-testing-bot (uosci-testing-bot)]

charm_amulet_test #7957 mongodb for evarlast mp277191
    AMULET FAIL: amulet-test failed

AMULET Results (max last 2 lines):
make: *** [functional_test] Error 1
ERROR:root:Make target returned non-zero.

Full amulet test output: http://paste.ubuntu.com/13368568/
Build: http://10.245.162.77:8080/job/charm_amulet_test/7957/

[uosci-testing-bot (uosci-testing-bot)]

charm_amulet_test #7956 mongodb for evarlast mp277191
    AMULET OK: passed

Build: http://10.245.162.77:8080/job/charm_amulet_test/7956/

[lp:~evarlast/charms/trusty/mongodb/fix-dump-actions]

90. By Jay R. Wren on 2015-11-20

Use hostname used in relation.

Can't assume relation hostname is public-address. It isn't in many environments.

[uosci-testing-bot (uosci-testing-bot)]

charm_lint_check #14138 mongodb for evarlast mp277191
    LINT OK: passed

Build: http://10.245.162.77:8080/job/charm_lint_check/14138/

[uosci-testing-bot (uosci-testing-bot)]

charm_unit_test #13178 mongodb for evarlast mp277191
    UNIT OK: passed

Build: http://10.245.162.77:8080/job/charm_unit_test/13178/

[lp:~evarlast/charms/trusty/mongodb/fix-dump-actions]

91. By Jay R. Wren on 2015-11-20

use sentry.wait not time.sleep

[uosci-testing-bot (uosci-testing-bot)]

charm_amulet_test #7958 mongodb for evarlast mp277191
    AMULET FAIL: amulet-test failed

AMULET Results (max last 2 lines):
make: *** [functional_test] Error 1
ERROR:root:Make target returned non-zero.

Full amulet test output: http://paste.ubuntu.com/13370168/
Build: http://10.245.162.77:8080/job/charm_amulet_test/7958/

[uosci-testing-bot (uosci-testing-bot)]

charm_unit_test #13180 mongodb for evarlast mp277191
    UNIT OK: passed

Build: http://10.245.162.77:8080/job/charm_unit_test/13180/

[uosci-testing-bot (uosci-testing-bot)]

charm_lint_check #14140 mongodb for evarlast mp277191
    LINT OK: passed

Build: http://10.245.162.77:8080/job/charm_lint_check/14140/

[uosci-testing-bot (uosci-testing-bot)]

charm_amulet_test #7960 mongodb for evarlast mp277191
    AMULET FAIL: amulet-test failed

AMULET Results (max last 2 lines):
make: *** [functional_test] Error 1
ERROR:root:Make target returned non-zero.

Full amulet test output: http://paste.ubuntu.com/13370662/
Build: http://10.245.162.77:8080/job/charm_amulet_test/7960/

[lp:~evarlast/charms/trusty/mongodb/fix-dump-actions]

92. By Jay R. Wren on 2015-11-20

wait longer for replset

Don't retry on unit 1 when unit 0 is not primary.
Additional logging on fail.

[uosci-testing-bot (uosci-testing-bot)]

charm_lint_check #14141 mongodb for evarlast mp277191
    LINT FAIL: lint-test failed

LINT Results (max last 2 lines):
make: *** [lint] Error 1
ERROR:root:Make target returned non-zero.

Full lint test output: http://paste.ubuntu.com/13372704/
Build: http://10.245.162.77:8080/job/charm_lint_check/14141/

[uosci-testing-bot (uosci-testing-bot)]

charm_unit_test #13181 mongodb for evarlast mp277191
    UNIT OK: passed

Build: http://10.245.162.77:8080/job/charm_unit_test/13181/

[uosci-testing-bot (uosci-testing-bot)]

charm_amulet_test #7961 mongodb for evarlast mp277191
    AMULET FAIL: amulet-test failed

AMULET Results (max last 2 lines):
make: *** [functional_test] Error 1
ERROR:root:Make target returned non-zero.

Full amulet test output: http://paste.ubuntu.com/13373290/
Build: http://10.245.162.77:8080/job/charm_amulet_test/7961/

[lp:~evarlast/charms/trusty/mongodb/fix-dump-actions]

93. By Jay R. Wren on 2015-11-20

fix lint

[uosci-testing-bot (uosci-testing-bot)]

charm_unit_test #13182 mongodb for evarlast mp277191
    UNIT OK: passed

Build: http://10.245.162.77:8080/job/charm_unit_test/13182/

[uosci-testing-bot (uosci-testing-bot)]

charm_lint_check #14142 mongodb for evarlast mp277191
    LINT OK: passed

Build: http://10.245.162.77:8080/job/charm_lint_check/14142/

[uosci-testing-bot (uosci-testing-bot)]

charm_amulet_test #7962 mongodb for evarlast mp277191
    AMULET FAIL: amulet-test failed

AMULET Results (max last 2 lines):
make: *** [functional_test] Error 1
ERROR:root:Make target returned non-zero.

Full amulet test output: http://paste.ubuntu.com/13374149/
Build: http://10.245.162.77:8080/job/charm_amulet_test/7962/

[lp:~evarlast/charms/trusty/mongodb/fix-dump-actions]

94. By Jay R. Wren on 2015-11-23

increase wait time for CI

[uosci-testing-bot (uosci-testing-bot)]

charm_lint_check #14253 mongodb for evarlast mp277191
    LINT OK: passed

Build: http://10.245.162.77:8080/job/charm_lint_check/14253/

[uosci-testing-bot (uosci-testing-bot)]

charm_unit_test #13284 mongodb for evarlast mp277191
    UNIT OK: passed

Build: http://10.245.162.77:8080/job/charm_unit_test/13284/

[uosci-testing-bot (uosci-testing-bot)]

charm_amulet_test #8016 mongodb for evarlast mp277191
    AMULET FAIL: amulet-test failed

AMULET Results (max last 2 lines):
make: *** [functional_test] Error 1
ERROR:root:Make target returned non-zero.

Full amulet test output: http://paste.ubuntu.com/13479215/
Build: http://10.245.162.77:8080/job/charm_amulet_test/8016/

[lp:~evarlast/charms/trusty/mongodb/fix-dump-actions]

95. By Jay R. Wren on 2015-11-23

increase wait time for CI

[uosci-testing-bot (uosci-testing-bot)]

charm_lint_check #14254 mongodb for evarlast mp277191
    LINT OK: passed

Build: http://10.245.162.77:8080/job/charm_lint_check/14254/

[uosci-testing-bot (uosci-testing-bot)]

charm_unit_test #13285 mongodb for evarlast mp277191
    UNIT OK: passed

Build: http://10.245.162.77:8080/job/charm_unit_test/13285/

[uosci-testing-bot (uosci-testing-bot)]

charm_amulet_test #8017 mongodb for evarlast mp277191
    AMULET FAIL: amulet-test failed

AMULET Results (max last 2 lines):
make: *** [functional_test] Error 1
ERROR:root:Make target returned non-zero.

Full amulet test output: http://paste.ubuntu.com/13479787/
Build: http://10.245.162.77:8080/job/charm_amulet_test/8017/

[Ryan Beisner (1chb1n)]

Thank you for your work on this.

While I realize that the test race wasn't introduced by your proposed changes, please see:
https://bugs.launchpad.net/charms/+source/mongodb/+bug/1518468

It would be awesome to resolve that test issue along with this, as it's definitely presenting a pain point.

review: Needs Fixing

[lp:~evarlast/charms/trusty/mongodb/fix-dump-actions]

96. By Jay R. Wren on 2015-11-24

retry every 10s for replset validation

[Jay R. Wren (evarlast)]

This should address https://bugs.launchpad.net/charms/+source/mongodb/+bug/1518468 now.

[uosci-testing-bot (uosci-testing-bot)]

charm_lint_check #14310 mongodb for evarlast mp277191
    LINT OK: passed

Build: http://10.245.162.77:8080/job/charm_lint_check/14310/

[uosci-testing-bot (uosci-testing-bot)]

charm_unit_test #13338 mongodb for evarlast mp277191
    UNIT OK: passed

Build: http://10.245.162.77:8080/job/charm_unit_test/13338/

[uosci-testing-bot (uosci-testing-bot)]

charm_amulet_test #8028 mongodb for evarlast mp277191
    AMULET OK: passed

Build: http://10.245.162.77:8080/job/charm_amulet_test/8028/

[Review Queue (review-queue)]

This item has failed automated testing! Results available here http://juju-ci.vapour.ws:8080/job/charm-bundle-test-lxc/1534/

review: Needs Fixing (automated testing)

[Review Queue (review-queue)]

This item has failed automated testing! Results available here http://juju-ci.vapour.ws:8080/job/charm-bundle-test-aws/1517/

review: Needs Fixing (automated testing)

[Review Queue (review-queue)]

This item has failed automated testing! Results available here http://juju-ci.vapour.ws:8080/job/charm-bundle-test-lxc/1536/

review: Needs Fixing (automated testing)

[uosci-testing-bot (uosci-testing-bot)]

charm_lint_check #14635 mongodb for evarlast mp277191
    LINT OK: passed

Build: http://10.245.162.77:8080/job/charm_lint_check/14635/

[Review Queue (review-queue)]

This item has failed automated testing! Results available here http://juju-ci.vapour.ws:8080/job/charm-bundle-test-lxc/1615/

review: Needs Fixing (automated testing)

[lp:~evarlast/charms/trusty/mongodb/fix-dump-actions]

97. By Jay R. Wren on 2015-12-02

retry replSetGetStatus and use minimum unit number for master

[uosci-testing-bot (uosci-testing-bot)]

charm_lint_check #14756 mongodb for evarlast mp277191
    LINT FAIL: lint-test failed

LINT Results (max last 2 lines):
make: *** [lint] Error 1
ERROR:root:Make target returned non-zero.

Full lint test output: http://paste.ubuntu.com/13631511/
Build: http://10.245.162.77:8080/job/charm_lint_check/14756/

[uosci-testing-bot (uosci-testing-bot)]

charm_unit_test #13759 mongodb for evarlast mp277191
    UNIT OK: passed

Build: http://10.245.162.77:8080/job/charm_unit_test/13759/

[lp:~evarlast/charms/trusty/mongodb/fix-dump-actions]

98. By Jay R. Wren on 2015-12-02

fix lint

[uosci-testing-bot (uosci-testing-bot)]

charm_lint_check #14757 mongodb for evarlast mp277191
    LINT OK: passed

Build: http://10.245.162.77:8080/job/charm_lint_check/14757/

[uosci-testing-bot (uosci-testing-bot)]

charm_unit_test #13761 mongodb for evarlast mp277191
    UNIT OK: passed

Build: http://10.245.162.77:8080/job/charm_unit_test/13761/

[uosci-testing-bot (uosci-testing-bot)]

charm_amulet_test #8078 mongodb for evarlast mp277191
    AMULET OK: passed

Build: http://10.245.162.77:8080/job/charm_amulet_test/8078/

[uosci-testing-bot (uosci-testing-bot)]

charm_amulet_test #8079 mongodb for evarlast mp277191
    AMULET FAIL: amulet-test failed

AMULET Results (max last 2 lines):
make: *** [functional_test] Error 124
ERROR:root:Make target returned non-zero.

Full amulet test output: http://paste.ubuntu.com/13632412/
Build: http://10.245.162.77:8080/job/charm_amulet_test/8079/

[uosci-testing-bot (uosci-testing-bot)]

charm_amulet_test #8081 mongodb for evarlast mp277191
    AMULET OK: passed

Build: http://10.245.162.77:8080/job/charm_amulet_test/8081/

[lp:~evarlast/charms/trusty/mongodb/fix-dump-actions]

99. By Jay R. Wren on 2015-12-03

retry shard status interface test
sort units and use them based on order in 04_deploy_with_storage test

[uosci-testing-bot (uosci-testing-bot)]

charm_lint_check #14813 mongodb for evarlast mp277191
    LINT OK: passed

Build: http://10.245.162.77:8080/job/charm_lint_check/14813/

[uosci-testing-bot (uosci-testing-bot)]

charm_unit_test #13813 mongodb for evarlast mp277191
    UNIT OK: passed

Build: http://10.245.162.77:8080/job/charm_unit_test/13813/

[Review Queue (review-queue)]

This item has failed automated testing! Results available here http://juju-ci.vapour.ws:8080/job/charm-bundle-test-lxc/1652/

review: Needs Fixing (automated testing)

[uosci-testing-bot (uosci-testing-bot)]

charm_amulet_test #8134 mongodb for evarlast mp277191
    AMULET FAIL: amulet-test failed

AMULET Results (max last 2 lines):
make: *** [functional_test] Error 1
ERROR:root:Make target returned non-zero.

Full amulet test output: http://paste.ubuntu.com/13658235/
Build: http://10.245.162.77:8080/job/charm_amulet_test/8134/

[Review Queue (review-queue)]

This item has failed automated testing! Results available here http://juju-ci.vapour.ws:8080/job/charm-bundle-test-lxc/1654/

review: Needs Fixing (automated testing)

[Review Queue (review-queue)]

This item has failed automated testing! Results available here http://juju-ci.vapour.ws:8080/job/charm-bundle-test-aws/1634/

review: Needs Fixing (automated testing)

[Review Queue (review-queue)]

This item has failed automated testing! Results available here http://juju-ci.vapour.ws:8080/job/charm-bundle-test-lxc/1666/

review: Needs Fixing (automated testing)

[Review Queue (review-queue)]

This item has failed automated testing! Results available here http://juju-ci.vapour.ws:8080/job/charm-bundle-test-aws/1646/

review: Needs Fixing (automated testing)

[lp:~evarlast/charms/trusty/mongodb/fix-dump-actions]

100. By Jay R. Wren on 2015-12-07

use constraints for storage test to prevent filling root volume

[uosci-testing-bot (uosci-testing-bot)]

charm_lint_check #15041 mongodb for evarlast mp277191
    LINT OK: passed

Build: http://10.245.162.77:8080/job/charm_lint_check/15041/

[uosci-testing-bot (uosci-testing-bot)]

charm_unit_test #14029 mongodb for evarlast mp277191
    UNIT OK: passed

Build: http://10.245.162.77:8080/job/charm_unit_test/14029/

[uosci-testing-bot (uosci-testing-bot)]

charm_amulet_test #8205 mongodb for evarlast mp277191
    AMULET OK: passed

Build: http://10.245.162.77:8080/job/charm_amulet_test/8205/

[Review Queue (review-queue)]

This item has failed automated testing! Results available here http://juju-ci.vapour.ws:8080/job/charm-bundle-test-lxc/1716/

review: Needs Fixing (automated testing)

[Review Queue (review-queue)]

This item has failed automated testing! Results available here http://juju-ci.vapour.ws:8080/job/charm-bundle-test-aws/1694/

review: Needs Fixing (automated testing)

[Review Queue (review-queue)]

This item has failed automated testing! Results available here http://juju-ci.vapour.ws:8080/job/charm-bundle-test-lxc/1725/

review: Needs Fixing (automated testing)

[Review Queue (review-queue)]

This item has failed automated testing! Results available here http://juju-ci.vapour.ws:8080/job/charm-bundle-test-aws/1702/

review: Needs Fixing (automated testing)

[Jay R. Wren (evarlast)]

The only fail remaining is from a known bug, which I am not attempting to fix.

From the README:
Deploying three or more units at start can sometimes lead to unexpected race-conditions so it's best to start with two nodes.

http://review.juju.solutions/review/2357 Shows the AWS test passing and the LXC test fails because of the above known bug.

Please consider this for merging.

review: Needs Resubmitting

[Matt Bruzek (mbruzek)]

Hello Jay,

While reviewing the queue today and noticed more than one mongodb charm merge proposals. Since we review the oldest first this proposal did not get reviewed first.

As Ryan points out in a bug 1518468 ( https://bugs.launchpad.net/charms/+source/mongodb/+bug/1518468 ) using sleep funtions are not a deterministic way to build tests that run on multiple environments as the times can be very long on some (azure) and lower on others (aws). This merge proposal changes some sleep() calls to sentry.wait() but introduces several more time.sleep() commands and the latter seems to be the wrong way to go.

## 03_deploy_replicaset.py
The new code contains the comment “# We assume minimum unit number is master.” There should be a better way to get the master address in the test. Again you may be able to run commands on the units to determine their master relationship, or use a mongo python library to determine the master reliably.

The tests passed for me on AWS and I am going to merge this code. I would encourage you to change the test code to detect the state of the mongodb deployment from reliable sources that do not involve sleeping for arbitrary amounts of time. Such as running a command on the unit to tell when it is ready, or using `juju status-history unit/0` to determine ready states.

review: Approve

[Review Queue (review-queue)]

This item has failed automated testing! Results available here http://juju-ci.vapour.ws:8080/job/charm-bundle-test-lxc/1754/

review: Needs Fixing (automated testing)

[Review Queue (review-queue)]

The results (PASS) are in and available here: http://juju-ci.vapour.ws:8080/job/charm-bundle-test-aws/1731/

review: Approve (automated testing)

=== modified file 'actions/backup.py'
--- actions/backup.py	2015-10-08 08:12:35 +0000
+++ actions/backup.py	2015-12-07 22:34:32 +0000
@@ -1,3 +1,4 @@
+import shlex
 import subprocess
 import os
 try:
@@ -14,7 +15,7 @@
 
 def execute(command, current_working_directory):
     return subprocess.check_output(
-        " ".join(command),
+        shlex.split(command),
         cwd=current_working_directory,
     )
 
@@ -38,7 +39,7 @@
         pass  # Ignoring, the directory already exists
     except Exception, e:
         action_set({"directory creation exception": e})
-        action_fail(e)
+        action_fail(str(e))
         return
 
     command = cmd + " " + args
@@ -50,7 +51,7 @@
     except subprocess.CalledProcessError, e:
         action_set({"error_code": e.returncode,
                     "exception": e, "output": e.output})
-        action_fail(e)
+        action_fail(str(e))
     except Exception, e:
         action_set({"exception": e})
-        action_fail(e)
+        action_fail(str(e))

=== modified file 'config.yaml'
--- config.yaml	2015-02-25 00:24:56 +0000
+++ config.yaml	2015-12-07 22:34:32 +0000
@@ -211,7 +211,7 @@
       option.
   key:
     type: string
-    default:
+    default: ""
     description: >
       Key ID to import to the apt keyring to support use with arbitary source
       configuration from outside of Launchpad archives or PPA's.

=== modified file 'hooks/hooks.py'
--- hooks/hooks.py	2015-09-09 12:59:18 +0000
+++ hooks/hooks.py	2015-12-07 22:34:32 +0000
@@ -254,6 +254,7 @@
     subprocess.call(['chown', '-R', 'mongodb:mongodb', config_data['dbpath']])
     config.append("dbpath=%s" % config_data['dbpath'])
     config.append("")
+    config.append("ipv6=true")
 
     # logpath
     # Create the directory if not there already

=== modified file 'metadata.yaml'
--- metadata.yaml	2015-05-19 13:21:16 +0000
+++ metadata.yaml	2015-12-07 22:34:32 +0000
@@ -18,7 +18,7 @@
   - Auto-sharding for cloud-level scalability (Q209)
     High performance, scalability, and reasonable depth of functionality
     are the goals for the project.
-categories:
+tags:
     - databases
 provides:
   nrpe-external-master:

=== modified file 'tests/01_deploy_single.py'
--- tests/01_deploy_single.py	2014-12-10 15:36:14 +0000
+++ tests/01_deploy_single.py	2015-12-07 22:34:32 +0000
@@ -24,7 +24,10 @@
 # Validate connectivity from $WORLD
 #############################################################
 def validate_world_connectivity():
-    client = MongoClient(d.sentry.unit['mongodb/0'].info['public-address'])
+    addy = d.sentry['mongodb'][0].info['public-address']
+    if ":" in addy:
+        addy = "[{}]".format(addy)
+    client = MongoClient(addy)
 
     db = client['test']
     # Can we successfully insert?

=== modified file 'tests/02_deploy_shard_test.py'
--- tests/02_deploy_shard_test.py	2014-12-09 23:45:14 +0000
+++ tests/02_deploy_shard_test.py	2015-12-07 22:34:32 +0000
@@ -2,6 +2,9 @@
 
 import amulet
 import requests
+import sys
+import time
+import traceback
 from pymongo import MongoClient
 
 #########################################################
@@ -49,10 +52,10 @@
     raise
 
 sentry_dict = {
-    'config-sentry': d.sentry.unit['configsvr/0'],
-    'mongos-sentry': d.sentry.unit['mongos/0'],
-    'shard1-sentry': d.sentry.unit['shard1/0'],
-    'shard2-sentry': d.sentry.unit['shard2/0']
+    'config-sentry': d.sentry['configsvr'][0],
+    'mongos-sentry': d.sentry['mongos'][0],
+    'shard1-sentry': d.sentry['shard1'][0],
+    'shard2-sentry': d.sentry['shard2'][0]
 }
 
 
@@ -60,10 +63,27 @@
 # Check presence of MongoDB GUI HEALTH Status
 #############################################################
 def validate_status_interface():
-    r = requests.get("http://{}:28017".format(
-        sentry_dict['config-sentry'].info['public-address']),
-        verify=False)
-    r.raise_for_status
+    pubaddy = sentry_dict['config-sentry'].info['public-address']
+    fmt = "http://{}:28017"
+    if ":" in pubaddy:
+        fmt = "http://[{}]:28017"
+    time_between = 10
+    tries = seconds / time_between
+    try:
+        r = requests.get(fmt.format(pubaddy), verify=False)
+        r.raise_for_status()
+    except requests.exception.ConnectionError as ex:
+        sys.stderr.write(
+            'Connection error, sleep and retry... to {}: {}\n'.
+            format(pubaddy, ex))
+        tb_lines = traceback.format_exception(ex.__class__,
+                                              ex, ex.__traceback__)
+        tb_text = ''.join(tb_lines)
+        sys.stderr.write(tb_text)
+        tries = tries - 1
+        if tries < 0:
+            sys.stderr.write('retry limit caught, failing...\n')
+        time.sleep(time_between)
 
 
 #############################################################
@@ -83,7 +103,10 @@
 # Validate connectivity from $WORLD
 #############################################################
 def validate_world_connectivity():
-    client = MongoClient(d.sentry.unit['mongos/0'].info['public-address'])
+    pubaddy = d.sentry['mongos'][0].info['public-address']
+    if ":" in pubaddy:
+        pubaddy = "[{}]".format(pubaddy)
+    client = MongoClient(pubaddy)
 
     db = client['test']
     # Can we successfully insert?
@@ -101,23 +124,28 @@
 #############################################################
 # broken pending 1273312
 def validate_relationships():
-    d.sentry.unit['configsvr/0'].relation('configsvr', 'mongos:mongos-cfg')
-    d.sentry.unit['shard1/0'].relation('database', 'mongos:mongos')
-    d.sentry.unit['shard2/0'].relation('database', 'mongos:mongos')
-    print(d.sentry.unit['shard1/0'].relation('database', 'mongos:mongos'))
+    d.sentry['configsvr'][0].relation('configsvr', 'mongos:mongos-cfg')
+    d.sentry['shard1'][0].relation('database', 'mongos:mongos')
+    d.sentry['shard2'][0].relation('database', 'mongos:mongos')
+    print(d.sentry['shard1'][0].relation('database', 'mongos:mongos'))
 
 
 def validate_manual_connection():
-    output, code = d.sentry.unit['shard1/0'].run("mongo {}".format(
-        d.sentry.unit['mongos/0'].info['public-address']))
+    fmt = "mongo {}"
+    addy = d.sentry['mongos'][0].info['public-address']
+    if ":" in addy:
+        fmt = "mongo --ipv6 {}:27017"
+    jujuruncmd = fmt.format(addy)
+    output, code = d.sentry['shard1'][0].run(jujuruncmd)
     if code != 0:
-        message = "Manual Connection failed for unit shard1"
+        message = ("Manual Connection failed for unit shard1:{} code:{} cmd:{}"
+                   .format(output, code, jujuruncmd))
         amulet.raise_status(amulet.SKIP, msg=message)
 
-    output, code = d.sentry.unit['shard2/0'].run("mongo {}".format(
-        d.sentry.unit['mongos/0'].info['public-address']))
+    output, code = d.sentry['shard2'][0].run(jujuruncmd)
     if code != 0:
-        message = "Manual Connection failed for unit shard2"
+        message = ("Manual Connection failed for unit shard2:{} code:{} cmd:{}"
+                   .format(output, code, jujuruncmd))
         amulet.raise_status(amulet.SKIP, msg=message)
 
 

=== modified file 'tests/03_deploy_replicaset.py'
--- tests/03_deploy_replicaset.py	2015-02-25 01:49:08 +0000
+++ tests/03_deploy_replicaset.py	2015-12-07 22:34:32 +0000
@@ -1,8 +1,11 @@
 #!/usr/bin/env python3
 
 import amulet
+import sys
+import time
+import traceback
 import requests
-import time
+import pymongo
 from pymongo import MongoClient
 from collections import Counter
 
@@ -13,9 +16,9 @@
 scale = 3
 seconds = 1800
 
-# amount of time to wait before testing for replicaset
+# max amount of time to wait before testing for replicaset
 # status
-wait_for_replicaset = 15
+wait_for_replicaset = 600
 
 #########################################################
 # 3shard cluster configuration
@@ -36,9 +39,9 @@
     raise
 
 sentry_dict = {
-    'mongodb0-sentry': d.sentry.unit['mongodb/0'],
-    'mongodb1-sentry': d.sentry.unit['mongodb/1'],
-    'mongodb2-sentry': d.sentry.unit['mongodb/2'],
+    'mongodb0-sentry': d.sentry['mongodb'][0],
+    'mongodb1-sentry': d.sentry['mongodb'][1],
+    'mongodb2-sentry': d.sentry['mongodb'][2],
 }
 
 
@@ -46,9 +49,11 @@
 # Check presence of MongoDB GUI HEALTH Status
 #############################################################
 def validate_status_interface():
-    r = requests.get("http://{}:28017".format(
-        d.sentry.unit['mongodb/0'].info['public-address']),
-        verify=False)
+    pubaddy = d.sentry['mongodb'][0].info['public-address']
+    fmt = "http://{}:28017"
+    if ":" in pubaddy:
+        fmt = "http://[{}]:28017"
+    r = requests.get(fmt.format(pubaddy), verify=False)
     r.raise_for_status
 
 
@@ -70,24 +75,47 @@
 #############################################################
 def validate_replicaset_setup():
 
-    time.sleep(wait_for_replicaset)
+    d.sentry.wait(seconds)
 
     unit_status = []
+    time_between = 10
+    tries = wait_for_replicaset / time_between
 
     for service in sentry_dict:
-        client = MongoClient(sentry_dict[service].info['public-address'])
-        r = client.admin.command('replSetGetStatus')
+        addy = sentry_dict[service].info['public-address']
+        if ":" in addy:
+            addy = "[{}]".format(addy)
+        while True:
+            try:
+                client = MongoClient(addy)
+                r = client.admin.command('replSetGetStatus')
+                break
+            except pymongo.errors.OperationFailure as ex:
+                sys.stderr.write(
+                    'OperationFailure, sleep and retry... to {}: {}\n'.
+                    format(addy, ex))
+                tb_lines = traceback.format_exception(ex.__class__,
+                                                      ex, ex.__traceback__)
+                tb_text = ''.join(tb_lines)
+                sys.stderr.write(tb_text)
+                tries = tries - 1
+                if tries < 0:
+                    sys.stderr.write('retry limit caught, failing...\n')
+                    break
+                time.sleep(time_between)
         unit_status.append(r['myState'])
         client.close()
 
     primaries = Counter(unit_status)[1]
     if primaries != 1:
-        message = "Only one PRIMARY unit allowed! Found: %s" % (primaries)
+        message = "Only one PRIMARY unit allowed! Found: %s %s" % (primaries,
+                                                                   unit_status)
         amulet.raise_status(amulet.FAIL, message)
 
     secondrs = Counter(unit_status)[2]
     if secondrs != 2:
-        message = "Only two SECONDARY units allowed! (Found %s)" % (secondrs)
+        message = ("Only two SECONDARY units allowed! (Found %s) %s" %
+                   (secondrs, unit_status))
         amulet.raise_status(amulet.FAIL, message)
 
 
@@ -95,11 +123,37 @@
 # Validate connectivity from $WORLD
 #############################################################
 def validate_world_connectivity():
-    client = MongoClient(d.sentry.unit['mongodb/0'].info['public-address'])
-
-    db = client['test']
-    # Can we successfully insert?
-    insert_id = db.amulet.insert({'assert': True})
+    d.sentry['mongodb']
+    ordered_units = sorted(d.sentry['mongodb'], key=lambda u: u.info['unit'])
+    # We assume minimum unit number is master.
+    addy = ordered_units[0].info['public-address']
+    if ":" in addy:
+        addy = "[{}]".format(addy)
+
+    time_between = 10
+    tries = wait_for_replicaset / time_between
+    insert_id = None
+    while True:
+        try:
+            client = MongoClient(addy)
+            db = client.test
+            # Can we successfully insert?
+            insert_id = db.amulet.insert({'assert': True})
+            break
+        except pymongo.errors.AutoReconnect as ex:
+            sys.stderr.write(
+                'AutoReconnect error, sleep and retry... to {}: {}\n'.
+                format(addy, ex))
+            tb_lines = traceback.format_exception(ex.__class__,
+                                                  ex, ex.__traceback__)
+            tb_text = ''.join(tb_lines)
+            sys.stderr.write(tb_text)
+            tries = tries - 1
+            if tries < 0:
+                sys.stderr.write('retry limit caught, failing...\n')
+                break
+            time.sleep(time_between)
+
     if insert_id is None:
         amulet.raise_status(amulet.FAIL, msg="Failed to insert test data")
     # Can we delete from a shard using the Mongos hub?

=== modified file 'tests/04_deploy_with_storage.py'
--- tests/04_deploy_with_storage.py	2015-06-02 05:33:09 +0000
+++ tests/04_deploy_with_storage.py	2015-12-07 22:34:32 +0000
@@ -1,7 +1,6 @@
 #!/usr/bin/env python3
 
 import amulet
-import time
 from pymongo import MongoClient
 from collections import Counter
 
@@ -14,16 +13,17 @@
 
 # amount of time to wait before testing for replicaset
 # status
-wait_for_replicaset = 30
+wait_for_replicaset = 60*5
 # amount of time to wait for the data relation
-wait_for_relation = 60*2
+wait_for_relation = 60*5
 
 #########################################################
 # 3shard cluster configuration
 #########################################################
 d = amulet.Deployment(series='trusty')
 
-d.add('mongodb', units=scale, series='trusty')
+d.add('mongodb', units=scale, series='trusty',
+      constraints={'root-disk': '20480M'})
 d.add('storage', charm='cs:~chris-gondolin/trusty/storage-5', series='trusty')
 d.configure('storage', {'provider': 'local'})
 
@@ -32,16 +32,17 @@
 # Perform the setup for the deployment.
 try:
     d.setup(seconds)
-    d.sentry.wait(seconds)
+    d.sentry.wait(wait_for_replicaset)
 except amulet.helpers.TimeoutError:
     message = 'The environment did not setup in %d seconds.', seconds
     amulet.raise_status(amulet.SKIP, msg=message)
 except:
     raise
 
+ordered_units = sorted(d.sentry['mongodb'], key=lambda u: u.info['unit'])
 sentry_dict = {
-    'mongodb0-sentry': d.sentry.unit['mongodb/0'],
-    'mongodb1-sentry': d.sentry.unit['mongodb/1'],
+    'mongodb0-sentry': ordered_units[0],
+    'mongodb1-sentry': ordered_units[1]
 }
 
 
@@ -57,12 +58,15 @@
 #############################################################
 def validate_replicaset_setup():
 
-    time.sleep(wait_for_replicaset)
+    d.sentry.wait(seconds)
 
     unit_status = []
 
     for service in sentry_dict:
-        client = MongoClient(sentry_dict[service].info['public-address'])
+        addy = sentry_dict[service].info['public-address']
+        if ":" in addy:
+            addy = "[{}]".format(addy)
+        client = MongoClient(addy)
         r = client.admin.command('replSetGetStatus')
         unit_status.append(r['myState'])
         client.close()
@@ -81,7 +85,10 @@
 validate_status()
 validate_replicaset_setup()
 print("Adding storage relation, and sleeping for 2 min.")
-d.relate('mongodb:data', 'storage:data')
-time.sleep(wait_for_relation)
+try:
+    d.relate('mongodb:data', 'storage:data')
+except OSError as e:
+    print("ignoring error:{}", e)
+d.sentry.wait(wait_for_relation)
 validate_status()
 validate_replicaset_setup()

=== modified file 'tests/50_relate_ceilometer_test.py'
--- tests/50_relate_ceilometer_test.py	2014-12-09 23:36:06 +0000
+++ tests/50_relate_ceilometer_test.py	2015-12-07 22:34:32 +0000
@@ -32,10 +32,13 @@
                 getattr(self, test)()
 
     def test_mongo_relation(self):
-        unit = self.deploy.sentry.unit['ceilometer/0']
-        mongo = self.deploy.sentry.unit['mongodb/0'].info['public-address']
+        unit = self.deploy.sentry['ceilometer'][0]
+        mongo = self.deploy.sentry['mongodb'][0].info['public-address']
+        mongo_reladdr = self.deploy.sentry['mongodb'][0].relation(
+            'database', 'ceilometer:shared-db')
         cont = unit.file_contents('/etc/ceilometer/ceilometer.conf')
-        if mongo not in cont:
+        if (mongo not in cont and mongo_reladdr.get(
+                'hostname', 'I SURE HOPE NOT') not in cont):
             amulet.raise_status(amulet.FAIL, "Unable to verify ceilometer cfg")
 
 if __name__ == '__main__':