Merge lp:~stub/charms/precise/postgresql/enable-integration-tests into lp:charms/postgresql

[Stuart Bishop]

Switch to Amulet infrastructure.

Reenable integration tests by default.

[Review Queue (review-queue)]

This items has failed automated testing! Results available here http://reports.vapour.ws/charm-tests/charm-bundle-test-11095-results

review: Needs Fixing (automated testing)

[Adam Israel (aisrael)]

Hi Stuart,

I had the opportunity to review this merge proposal today.

The Makefile target `testdep` isn't being called by `make test`, so my initial bundletester run failed due to dependencies. I manually ran that step, and was able to get further in the integration tests. Unfortunately, the test eventually hung (after 3 hours of running).

I've posted the full bundletester logs here:
http://pastebin.ubuntu.com/10581906/

There were a couple of integration tests that failed (test_basic_admin, test_roles_granted), but test_upgrade_charm eventually timed out. I suspect that final error, and the length of time the tests took to run, was due to the number of machines spun up and never destroyed during the test -- 29 started, 3 pending.

Bundletester should tear down machines allocated between tests, but that didn't happen in this case. I wonder if it's related at all to the monkeypatched version of Amulet you're using.

How does this line up with your own testing? Anything I should do differently to test? Feel free to ping me in irc or elsewhere if there's more I can do to facilitate getting these tests landed.

I have to NAK this for now, pending further instructions on running tests and/or the resolution of the above issues.

review: Needs Fixing

[Review Queue (review-queue)]

This items has failed automated testing! Results available here http://reports.vapour.ws/charm-tests/charm-bundle-test-11097-results

review: Needs Fixing (automated testing)

[Review Queue (review-queue)]

This items has failed automated testing! Results available here http://reports.vapour.ws/charm-tests/charm-bundle-test-11098-results

review: Needs Fixing (automated testing)

[Stuart Bishop (stub)]

I was relying on the newer Juju behavior (or local provider behavior?), where machines are destroyed by default when the service is destroyed. I'll put back the machine harvesting code I thought might not be needed.

test_basic_admin, test_roles_granted and test_upgrade_charm usually pass, and I suspect their failures was due to the provisioning issues. The more problematic tests are the failover ones, as they sometimes trigger a race condition I need to work out how to fix (if you failover too often too quickly, you might promote a unit to master that has not reached a recoverable state and things die).

bundletester is irrelevant here, as it is just the launcher. As far as bundletester is concerned there is just one huge test.

I've upgraded the Makefile rules to install the test dependencies the first time they are needed.

[Review Queue (review-queue)]

This items has failed automated testing! Results available here http://reports.vapour.ws/charm-tests/charm-bundle-test-11102-results

review: Needs Fixing (automated testing)

[Review Queue (review-queue)]

This items has failed automated testing! Results available here http://reports.vapour.ws/charm-tests/charm-bundle-test-11104-results

review: Needs Fixing (automated testing)

[Stuart Bishop (stub)]

Tests are failing as the mocker library cannot be found, despite it being installed using 'apt-get install python-mocker'. This is likely as the python interpreter in play (the charmguardian vm?) isn't searching the system modules.

[Adam Israel (aisrael)]

Hey stub,

Thanks for pointing that out. I've been working on a review of these integration tests and am tracking a bug in juju core/juju deployer (lp:1421315) that's been causing some of the tests to fail.

Out of curiosity, what version of juju are you running?

[Stuart Bishop (stub)]

I stick to ppa:juju/stable for writing charms.

[Adam Israel (aisrael)]

Excellent, thanks for confirming.

I have some initial notes for you.

I started testing using bundletester but ran into some intermittent test failures due to a bug in bundletester (now fixed).

I've continued testing manually, i.e., running nosetests -sv test:PG91Tests

There's a bug in juju core (lp:1421315) and juju-deployer. I've patched my local version of juju-deployer to work around the issue (a merge for that is pending).

With the above issues resolved, I'm running into another error that maybe you have some insight about.

test.PG91Tests.test_failover ... ERROR cannot read info: lock timeout exceeded
['juju', 'run', '--timeout=21600s', '--unit', 'postgresql/0', 'sudo tail -1 /var/log/juju/unit-postgresql-0.log'] failed: 1

ERROR subprocess encountered error code 1

I'm investigating why that error is being thrown. I wonder if juju run is executing before the unit is stood up. Any thoughts?

[Stuart Bishop (stub)]

This particular error is bubbling up from the juju-wait plugin (lp:juju-wait). This will only call juju run against units that are alive (do not have life: set to dying or dead in the status) and are ready (report an agent-state: of started). The only thing slightly unusual that could be happening here is that it will execute the juju run's in parallel. The quickest way of checking if this is the culprit is to hack /usr/bin/juju-wait and temporarily change max_workers=6 to max_workers=1.

[Adam Israel (aisrael)]

Ack, thanks, I'll check juju-wait.

[Adam Israel (aisrael)]

Ok, I've finished a comprehensive review of the integration tests.

There were several tests failing that I've determined to be a failure in the test infrastructure. I've landed a patch to bundletester that was running tests more than once. I'll be sending patches to amulet, juju-deployer, and juju-wait (the change from max_workers=6 to max_workers=1 fixed the lock timeout error).

With the above patches implemented locally, I've been able to run the 9.1 integration test with only one failure, that looks like it's ultimately erroneous:

test_syslog initially failed on the deploy command. This line:

    self.deployment.add('rsyslog', 'cs:rsyslog', 2)

should read:

    self.deployment.add('rsyslog', 'rsyslog', 2)

"cs:rsyslog" returns a 404 - see https://manage.jujucharms.com/api/3/charm/trusty/cs:rsyslog

With that fixed, the test runs but fails it's assert checks:
_StringException: Traceback (most recent call last):
  File "/charms/trusty/postgresql/test.py", line 697, in test_syslog
    self.failUnless('master {}'.format(token) in out)
  File "/usr/lib/python2.7/unittest/case.py", line 614, in deprecated_func
    return original_func(*args, **kwargs)
  File "/usr/lib/python2.7/unittest/case.py", line 424, in assertTrue
    raise self.failureException(msg)
AssertionError: False is not true

I think the failure is wrong, though. I ssh'd into each rsyslog unit after the failure and `tail -n 100 /var/log/syslog | grep`ped for each string and found it.

While running test_explicit_database, I ran into a case where the replication-relation-broken failed on tearDown, and reset_environment wasn't cycling through to resolve failures. The test passed when I flipped the force flag in amuletfixture.py, but that might not be what you intended.

The stack trace of the failure:
2015-03-26 15:17:01 INFO unit.postgresql/0.replication-relation-broken logger.go:40 Traceback (most recent call last):
2015-03-26 15:17:01 INFO unit.postgresql/0.replication-relation-broken logger.go:40 File "/var/lib/juju/agents/unit-postgresql-0/charm/hooks/replication-relation-broken", line 2701, in <module>
2015-03-26 15:17:01 INFO unit.postgresql/0.replication-relation-broken logger.go:40 hooks.execute(sys.argv)
2015-03-26 15:17:01 INFO unit.postgresql/0.replication-relation-broken logger.go:40 File "/var/lib/juju/agents/unit-postgresql-0/charm/hooks/charmhelpers/core/hookenv.py", line 490, in execute
2015-03-26 15:17:01 INFO unit.postgresql/0.replication-relation-broken logger.go:40 self._hooks[hook_name]()
2015-03-26 15:17:01 INFO unit.postgresql/0.replication-relation-broken logger.go:40 File "/var/lib/juju/agents/unit-postgresql-0/charm/hooks/replication-relation-broken", line 2263, in replication_relation_broken
2015-03-26 15:17:01 INFO unit.postgresql/0.replication-relation-broken logger.go:40 config_changed()
2015-03-26 15:17:01 INFO unit.postgresql/0.replication-relation-broken logger.go:40 File "/var/lib/juju/agents/unit-postgresql-0/charm/hooks/replication-relation-broken", line 1140, in config_changed
2015-03-26 15:17:01 INFO unit.postgresql/0.replication-relation-broken logger.go:40 generate_postgresql_hba(postgresql_hba)
2015-03-26 15:1...

[Stuart Bishop (stub)]

On 7 March 2015 at 06:36, Adam Israel <email address hidden> wrote:

> There were several tests failing that I've determined to be a failure in the test infrastructure. I've landed a patch to bundletester that was running tests more than once. I'll be sending patches to amulet, juju-deployer, and juju-wait (the change from max_workers=6 to max_workers=1 fixed the lock timeout error).

Ok. The multiple workers is of course to speed things up, but
correctness is better. I'll sort juju-wait shortly and kick off the
package build.

> With the above patches implemented locally, I've been able to run the 9.1 integration test with only one failure, that looks like it's ultimately erroneous:
>
> test_syslog initially failed on the deploy command. This line:
>
> self.deployment.add('rsyslog', 'cs:rsyslog', 2)
>
> should read:
>
> self.deployment.add('rsyslog', 'rsyslog', 2)
>
> "cs:rsyslog" returns a 404 - see https://manage.jujucharms.com/api/3/charm/trusty/cs:rsyslog

Aha... when to use prefixes or not has always confused me. I think
this has worked in the past.

> With that fixed, the test runs but fails it's assert checks:
> _StringException: Traceback (most recent call last):
> File "/charms/trusty/postgresql/test.py", line 697, in test_syslog
> self.failUnless('master {}'.format(token) in out)
> File "/usr/lib/python2.7/unittest/case.py", line 614, in deprecated_func
> return original_func(*args, **kwargs)
> File "/usr/lib/python2.7/unittest/case.py", line 424, in assertTrue
> raise self.failureException(msg)
> AssertionError: False is not true
>
> I think the failure is wrong, though. I ssh'd into each rsyslog unit after the failure and `tail -n 100 /var/log/syslog | grep`ped for each string and found it.
>
> While running test_explicit_database, I ran into a case where the replication-relation-broken failed on tearDown, and reset_environment wasn't cycling through to resolve failures. The test passed when I flipped the force flag in amuletfixture.py, but that might not be what you intended.

I consider this a bug, which is why the force flag isn't set.

> The stack trace of the failure:
> 2015-03-26 15:17:01 INFO unit.postgresql/0.replication-relation-broken logger.go:40 Traceback (most recent call last):
> 2015-03-26 15:17:01 INFO unit.postgresql/0.replication-relation-broken logger.go:40 File "/var/lib/juju/agents/unit-postgresql-0/charm/hooks/replication-relation-broken", line 2701, in <module>
> 2015-03-26 15:17:01 INFO unit.postgresql/0.replication-relation-broken logger.go:40 hooks.execute(sys.argv)
> 2015-03-26 15:17:01 INFO unit.postgresql/0.replication-relation-broken logger.go:40 File "/var/lib/juju/agents/unit-postgresql-0/charm/hooks/charmhelpers/core/hookenv.py", line 490, in execute
> 2015-03-26 15:17:01 INFO unit.postgresql/0.replication-relation-broken logger.go:40 self._hooks[hook_name]()
> 2015-03-26 15:17:01 INFO unit.postgresql/0.replication-relation-broken logger.go:40 File "/var/lib/juju/agents/unit-postgresql-0/charm/hooks/replication-relation-broken", line 2263, in replication_relation_broken
> 2015-03-26 15:17:01 INFO unit.postgresql/0.replication-relation-broken logger.go:4...

[Adam Israel (aisrael)]

Sorry, it was the complete PG91Tests test suite that took an hour, not an individual test!

[Stuart Bishop (stub)]

This branch is ready for another test run.

[Tim Van Steenburgh (tvansteenburgh)]

Started tests.

Unmerged revisions

123. By Stuart Bishop on 2015-03-12

Missing dependency

122. By Stuart Bishop on 2015-03-12

test updates

121. By Stuart Bishop on 2015-03-12

Install test dependencies if required

120. By Stuart Bishop on 2015-03-11

Remove dead code

119. By Stuart Bishop on 2015-03-11

silence upgrade-charm test

118. By Stuart Bishop on 2015-03-10

Newer wait() implementation

117. By Stuart Bishop on 2015-03-10

missing config

116. By Stuart Bishop on 2015-03-10

tidy

115. By Stuart Bishop on 2015-03-10

delint

114. By Stuart Bishop on 2015-03-10

quiet teardown

=== modified file '.bzrignore'
--- .bzrignore	2014-10-14 10:12:50 +0000
+++ .bzrignore	2015-03-12 09:42:09 +0000
@@ -2,3 +2,4 @@
 hooks/_trial_temp
 hooks/local_state.pickle
 lib/test-client-charm/hooks/charmhelpers
+.stamp-*

=== modified file 'Makefile'
--- Makefile	2014-10-24 06:31:15 +0000
+++ Makefile	2015-03-12 09:42:09 +0000
@@ -3,58 +3,75 @@
 SERIES := $(juju get-environment default-series)
 
 default:
-	@echo "One of:"
+	@echo "Install test dependencies:"
 	@echo "    make testdep"
+	@echo
+	@echo "Indivitual tests, one of:"
 	@echo "    make lint"
-	@echo "    make unit_test"
-	@echo "    make integration_test"
+	@echo "    make unittest"
 	@echo "    make integration_test_91"
 	@echo "    make integration_test_92"
 	@echo "    make integration_test_93"
 	@echo "    make integration_test_94"
 	@echo
-	@echo "There is no 'make test'"
-
-test_bot_tests:
-	@echo "Installing dependencies and running automatic-testrunner tests"
-	tests/00-setup.sh
-	tests/01-lint.sh
-	tests/02-unit-tests.sh
-	tests/03-basic-amulet.py
-
-testdep:
-	tests/00-setup.sh
-
-unit_test:
+	@echo "For full tests:"
+	@echo "    make test"
+
+clean:
+	rm -f .stamp-*
+
+test: testdep lint unittest integration_test
+	@echo "Done"
+
+testdep: .stamp-testdep
+.stamp-testdep:
+	sudo add-apt-repository -y ppa:juju/stable
+	sudo add-apt-repository -y ppa:stub/juju
+	sudo apt-get update
+	sudo apt-get install -y \
+	    amulet \
+	    python-flake8 \
+	    python-fixtures \
+	    python-jinja2 \
+	    python-mocker \
+	    python-psycopg2 \
+	    python-nose \
+	    python-testtools \
+	    python-mocker \
+	    python-yaml \
+	    juju-wait \
+	    pgtune
+	touch .stamp-testdep
+
+unittest: testdep
 	@echo "Unit tests of hooks"
-	cd hooks && trial test_hooks.py
+	cd hooks && nosetests -sv test_hooks.py
 
-integration_test:
+integration_test: testdep
 	@echo "PostgreSQL integration tests, all non-beta versions, ${SERIES}"
-	trial test.PG91Tests
-	trial test.PG92Tests
-	trial test.PG93Tests
+	nosetests -sv \
+	    test:PG91Tests test:PG92Tests test:PG93Tests test:PG94Tests
 
-integration_test_91:
+integration_test_91: testdep
 	@echo "PostgreSQL 9.1 integration tests, ${SERIES}"
-	trial test.PG91Tests
+	nosetests -sv test:PG91Tests
 
-integration_test_92:
+integration_test_92: testdep
 	@echo "PostgreSQL 9.2 integration tests, ${SERIES}"
-	trial test.PG92Tests
+	nosetests -sv test:PG92Tests
 
-integration_test_93:
+integration_test_93: testdep
 	@echo "PostgreSQL 9.3 integration tests, ${SERIES}"
-	trial test.PG93Tests
-
-integration_test_94:
-	@echo "PostgreSQL 9.4 (beta) integration tests, ${SERIES}"
-	trial test.PG94Tests
-
-lint:
+	nosetests -sv test:PG93Tests
+
+integration_test_94: testdep
+	@echo "PostgreSQL 9.4 integration tests, ${SERIES}"
+	nosetests -sv test:PG94Tests
+
+lint: testdep
 	@echo "Lint check (flake8)"
 	@flake8 -v \
-	    --exclude hooks/charmhelpers,hooks/_trial_temp \
+	    --exclude hooks/charmhelpers \
 	    hooks testing tests test.py
 
 sync:

=== modified file 'config.yaml'
--- config.yaml	2015-02-11 21:05:42 +0000
+++ config.yaml	2015-03-12 09:42:09 +0000
@@ -2,7 +2,7 @@
   admin_addresses:
     default: ""
     type: string
-    description: |
+    description: >
       A comma-separated list of IP Addresses (or single IP) admin tools like 
       pgAdmin3 will connect from, this is most useful for developers running 
       juju in local mode who need to connect tools like pgAdmin to a postgres. 
@@ -12,14 +12,14 @@
   locale:
     default: "C"
     type: string
-    description: |
+    description: >
       Locale of service, defining language, default collation order,
       and default formatting of numbers, currency, dates & times. Can only be
       set when deploying the first unit of a service.
   encoding:
     default: "UTF-8"
     type: string
-    description: |
+    description: >
       Default encoding used to store text in this service. Can only be
       set when deploying the first unit of a service.
   extra-packages:
@@ -29,23 +29,12 @@
   dumpfile_location:
     default: "None"
     type: string
-    description: |
+    description: >
         Path to a dumpfile to load into DB when service is initiated.
-  config_change_command:
-    default: "reload"
-    type: string
-    description: |
-        The command to run whenever config has changed. Accepted values
-        are "reload" or "restart" - any other value will mean neither is
-        executed after a config change (which may be desired, if you're
-        running a production server and would rather handle these out of
-        band). Note that postgresql will still need to be reloaded
-        whenever authentication and access details are updated, so
-        disabling either doesn't mean PostgreSQL will never be reloaded.
   version:
     default: null
     type: string
-    description: |
+    description: >
         Version of PostgreSQL that we want to install. Supported versions
         are "9.1", "9.2", "9.3". The default version for the deployed Ubuntu
         release is used when the version is not specified.
@@ -65,6 +54,13 @@
     default: 100
     type: int
     description: Maximum number of connections to allow to the PG database
+  max_prepared_transactions:
+    default: 0
+    type: int
+    description: >
+      Maximum number of prepared two phase commit transactions, waiting
+      to be committed. Defaults to 0. as using two phase commit without
+      a process to monitor and resolve lost transactions is dangerous.
   ssl:
     default: "True"
     type: string
@@ -72,7 +68,7 @@
   log_min_duration_statement:
     default: -1
     type: int
-    description: |
+    description: >
         -1 is disabled, 0 logs all statements
         and their durations, > 0 logs only
         statements running at least this number
@@ -92,7 +88,7 @@
   log_temp_files:
     default: "-1"
     type: string
-    description: |
+    description: >
         Log creation of temporary files larger than the threshold.
         -1 disables the feature, 0 logs all temporary files, or specify
         the threshold size with an optional unit (eg. "512KB", default
@@ -131,13 +127,13 @@
   autovacuum:
     default: True
     type: boolean
-    description: |
+    description: >
         Autovacuum should almost always be running. If you want to turn this
         off, you are probably following out of date documentation.
   log_autovacuum_min_duration:
     default: -1
     type: int
-    description: |
+    description: >
         -1 disables, 0 logs all actions and their durations, > 0 logs only
         actions running at least this number of milliseconds.
   autovacuum_analyze_threshold:
@@ -155,13 +151,13 @@
   autovacuum_vacuum_cost_delay:
     default: "20ms"
     type: string
-    description: |
+    description: >
         Default vacuum cost delay for autovacuum, in milliseconds;
         -1 means use vacuum_cost_delay
   search_path:
     default: "\"$user\",public"
     type: string
-    description: |
+    description: >
         Comma separated list of schema names for
         the default SQL search path.
   standard_conforming_strings:
@@ -171,22 +167,21 @@
   hot_standby:
     default: False
     type: boolean
-    description: |
-        DEPRECATED.
+    description: >
         Hot standby or warm standby. When True, queries can be run against
         the database when in recovery or standby mode (ie. replicated).
         Overridden when service contains multiple units.
   hot_standby_feedback:
     default: False
     type: boolean
-    description: |
+    description: >
         Hot standby feedback, informing a master about in progress
         transactions on a streaming hot standby and allowing the master to
         defer cleanup and avoid query cancelations on the hot standby.
   wal_level:
     default: minimal
     type: string
-    description: |
+    description: >
         'minimal', 'archive' or 'hot_standby'. Defines how much information
         is written to the WAL. Set to 'minimal' for stand alone databases
         and 'hot_standby' for replicated setups. Overridden by juju when
@@ -194,7 +189,7 @@
   max_wal_senders:
     default: 0
     type: int
-    description: |
+    description: >
         Maximum number of hot standbys that can connect using
         streaming replication. Set this to the expected maximum number of
         hot standby units to avoid unnecessary blocking and database restarts.
@@ -202,7 +197,7 @@
   wal_keep_segments:
     default: 0
     type: int
-    description: |
+    description: >
         Number of old WAL files to keep, providing a larger buffer for
         streaming hot standbys to catch up from when lagged. Each WAL file
         is 16MB in size. The WAL files are the buffer of how far a
@@ -212,7 +207,7 @@
   replicated_wal_keep_segments:
     default: 5000
     type: int
-    description: |
+    description: >
         Value of wal_keep_segments used when this service is replicated.
         This setting only exists to provide a sane default when replication
         is requested (so it doesn't fail) and nobody bothered to change the
@@ -220,7 +215,7 @@
   archive_mode:
     default: False
     type: boolean
-    description: |
+    description: >
         Enable archiving of WAL files using the command specified by
         archive_command. If archive_mode is enabled and archive_command not
         set, then archiving is deferred until archive_command is set and the
@@ -228,25 +223,25 @@
   archive_command:
     default: ""
     type: string
-    description: |
+    description: >
         Command used to archive WAL files when archive_mode is set and
         wal_level > minimal.
   work_mem:
     default: "1MB"
     type: string
-    description: |
+    description: >
         Working Memory.
         Ignored unless 'performance_tuning' is set to 'manual'.
   maintenance_work_mem:
     default: "1MB"
     type: string
-    description: |
+    description: >
         Maintenance working memory.
         Ignored unless 'performance_tuning' is set to 'manual'.
   performance_tuning:
     default: "Mixed"
     type: string
-    description: |
+    description: >
         Possible values here are "manual", "DW" (data warehouse),
         "OLTP" (online transaction processing), "Web" (web application),
         "Desktop" or "Mixed". When this is set to a value other than
@@ -267,14 +262,14 @@
   shared_buffers:
     default: ""
     type: string
-    description: |
+    description: >
         The amount of memory the database server uses for shared memory
         buffers. This string should be of the format '###MB'.
         Ignored unless 'performance_tuning' is set to 'manual'.
   effective_cache_size:
     default: ""
     type: string
-    description: |
+    description: >
         Effective cache size is an estimate of how much memory is available for
         disk caching within the database. (50% to 75% of system memory). This
         string should be of the format '###MB'. Ignored unless
@@ -282,52 +277,59 @@
   default_statistics_target:
     default: -1
     type: int
-    description: |
+    description: >
         Sets the default statistics target for table columns without a
         column-specific target set via ALTER TABLE SET STATISTICS.
         Leave unchanged to use the server default, which in recent
         releases is 100. Ignored unless 'performance_tuning' is 'manual'.
         Larger values increase the time needed to do ANALYZE, but
         might improve the quality of the planner's estimates.
+  collapse_limit:
+    default: -1 
+    type: int
+    description: >
+        Sets the from_collapse_limit and join_collapse_limit query planner
+        options, controlling the maximum number of tables that can be joined
+        before the turns off the table collapse query optimization.
   temp_buffers:
     default: "1MB"
     type: string
-    description: |
+    description: >
         The maximum number of temporary buffers used by each database session.
   wal_buffers:
     default: "-1"
     type: string
-    description: |
+    description: >
         min 32kB, -1 sets based on shared_buffers (change requires restart).
         Ignored unless 'performance_tuning' is set to 'manual'.
   checkpoint_segments:
     default: 3
     type: int
-    description: |
+    description: >
         in logfile segments, min 1, 16MB each.
         Ignored unless 'performance_tuning' is set to 'manual'.
   checkpoint_timeout:
     default: ""
     type: string
-    description: |
+    description: >
         Maximum time between automatic WAL checkpoints. range '30s-1h'.
         If left empty, the default postgresql value will be used.
   fsync:
     type: boolean
     default: True
-    description: |
+    description: >
         Turns forced synchronization on/off. If fsync is turned off, database
         failures are likely to involve database corruption and require
         recreating the unit
   synchronous_commit:
     type: boolean
     default: True
-    description: |
+    description: >
         Immediate fsync after commit.
   full_page_writes:
     type: boolean
     default: True
-    description: |
+    description: >
         Recover from partial page writes.
   random_page_cost:
     default: 4.0
@@ -336,7 +338,7 @@
   extra_pg_auth:
     type: string
     default: ""
-    description: |
+    description: >
         A comma separated extra pg_hba.conf auth rules.
         This will be written to the pg_hba.conf file, one line per rule.
         Note that this should not be needed as db relations already create
@@ -348,7 +350,7 @@
   manual_replication:
     type: boolean
     default: False
-    description: |
+    description: >
         Enable or disable charm managed replication. When manual_replication
         is True, the operator is responsible for maintaining recovery.conf
         and performing any necessary database mirroring. The charm will
@@ -372,7 +374,7 @@
   nagios_context:
     default: "juju"
     type: string
-    description: |
+    description: >
         Used by the nrpe-external-master subordinate charm.
         A string that will be prepended to instance name to set the host name
         in nagios. So for instance the hostname would be something like:
@@ -382,14 +384,14 @@
   nagios_additional_servicegroups:
     default: ""
     type: string
-    description: |
+    description: >
         Used by the nrpe-external-master subordinate charm.
         A comma-separated list of servicegroups to include along with
         nagios_context when generating nagios service check configs.
         This is useful for nagios installations where servicegroups
         are used to apply special treatment to particular checks.
   pgdg:
-    description: |
+    description: >
         Enable the PostgreSQL Global Development Group APT repository
         (https://wiki.postgresql.org/wiki/Apt). This package source provides
         official PostgreSQL packages for Ubuntu LTS releases beyond those
@@ -397,13 +399,13 @@
     type: boolean
     default: false
   install_sources:
-    description: |
+    description: >
         List of extra package sources, per charm-helpers standard.
         YAML format.
     type: string
     default: null
   install_keys:
-    description: |
+    description: >
         List of signing keys for install_sources package sources, per
         charmhelpers standard. YAML format.
     type: string
@@ -411,12 +413,12 @@
   extra_archives:
     default: ""
     type: string
-    description: |
+    description: >
         DEPRECATED & IGNORED. Use install_sources and install_keys. 
   advisory_lock_restart_key:
     default: 765
     type: int
-    description: |
+    description: >
         An advisory lock key used internally by the charm. You do not need
         to change it unless it happens to conflict with an advisory lock key
         being used by your applications.
@@ -424,7 +426,7 @@
   swiftwal_container_prefix:
     type: string
     default: null
-    description: |
+    description: >
       EXPERIMENTAL.
       Swift container prefix for SwiftWAL to use. Must be set if any
       SwiftWAL features are enabled. This will become a simple
@@ -433,13 +435,13 @@
   swiftwal_backup_schedule:
     type: string
     default: null
-    description: |
+    description: >
       EXPERIMENTAL.
       Cron-formatted schedule for SwiftWAL database backups.
   swiftwal_backup_retention:
     type: int
     default: 2
-    description: |
+    description: >
       EXPERIMENTAL.
       Number of recent base backups to retain. You need enough space in
       Swift for this many backups plus one more, as an old backup will only
@@ -447,7 +449,7 @@
   swiftwal_log_shipping:
     type: boolean
     default: false
-    description: |
+    description: >
       EXPERIMENTAL.
       Archive WAL files into Swift. If swiftwal_backup_schedule is set,
       allows point-in-time recovery and WAL files are removed
@@ -474,7 +476,7 @@
   wal_e_backup_schedule:
     type: string
     default: "13 0 * * *"
-    description: |
+    description: >
       EXPERIMENTAL.
       Cron-formatted schedule for WAL-E database backups. If
       wal_e_backup_schedule is unset, WAL files will never be removed from
@@ -482,7 +484,7 @@
   wal_e_backup_retention:
     type: int
     default: 2
-    description: |
+    description: >
       EXPERIMENTAL.
       Number of recent base backups and WAL files to retain.
       You need enough space for this many backups plus one more, as
@@ -491,7 +493,7 @@
   streaming_replication:
     type: boolean
     default: true
-    description: |
+    description: >
       Enable streaming replication. Normally, streaming replication is
       always used, and any log shipping configured is used as a fallback.
       Turning this off without configuring log shipping is an error.
@@ -530,7 +532,7 @@
   package_status:
     default: "install"
     type: string
-    description: |
+    description: >
         The status of service-affecting packages will be set to this
         value in the dpkg database. Useful valid values are "install"
         and "hold".
@@ -538,13 +540,13 @@
   metrics_target:
     default: ""
     type: string
-    description: |
+    description: >
         Destination for statsd-format metrics, format "host:port". If
         not present and valid, metrics disabled.
   metrics_prefix:
     default: "dev.$UNIT.postgresql"
     type: string
-    description: |
+    description: >
         Prefix for metrics. Special value $UNIT can be used to include the
         name of the unit in the prefix.
   metrics_sample_interval:

=== removed file 'hooks/helpers.py'
--- hooks/helpers.py	2012-10-01 14:10:14 +0000
+++ hooks/helpers.py	1970-01-01 00:00:00 +0000
@@ -1,199 +0,0 @@
-# Copyright 2012 Canonical Ltd.  This software is licensed under the
-# GNU Affero General Public License version 3 (see the file LICENSE).
-
-"""Helper functions for writing hooks in python."""
-
-__metaclass__ = type
-__all__ = [
-    'get_config',
-    'juju_status',
-    'log',
-    'log_entry',
-    'log_exit',
-    'make_charm_config_file',
-    'relation_get',
-    'relation_set',
-    'unit_info',
-    'wait_for_machine',
-    'wait_for_page_contents',
-    'wait_for_relation',
-    'wait_for_unit',
-    ]
-
-from contextlib import contextmanager
-import json
-import operator
-from shelltoolbox import (
-    command,
-    run,
-    script_name,
-    )
-import os
-import tempfile
-import time
-import urllib2
-import yaml
-
-
-log = command('juju-log')
-
-
-def log_entry():
-    log("--> Entering {}".format(script_name()))
-
-
-def log_exit():
-    log("<-- Exiting {}".format(script_name()))
-
-
-def get_config():
-    config_get = command('config-get', '--format=json')
-    return json.loads(config_get())
-
-
-def relation_get(*args):
-    cmd = command('relation-get')
-    return cmd(*args).strip()
-
-
-def relation_set(**kwargs):
-    cmd = command('relation-set')
-    args = ['{}={}'.format(k, v) for k, v in kwargs.items()]
-    return cmd(*args)
-
-
-def make_charm_config_file(charm_config):
-    charm_config_file = tempfile.NamedTemporaryFile()
-    charm_config_file.write(yaml.dump(charm_config))
-    charm_config_file.flush()
-    # The NamedTemporaryFile instance is returned instead of just the name
-    # because we want to take advantage of garbage collection-triggered
-    # deletion of the temp file when it goes out of scope in the caller.
-    return charm_config_file
-
-
-def juju_status(key):
-    return yaml.safe_load(run('juju', 'status'))[key]
-
-
-def get_charm_revision(service_name):
-    service = juju_status('services')[service_name]
-    return int(service['charm'].split('-')[-1])
-
-
-def unit_info(service_name, item_name, data=None):
-    services = juju_status('services') if data is None else data['services']
-    service = services.get(service_name)
-    if service is None:
-        # XXX 2012-02-08 gmb:
-        #     This allows us to cope with the race condition that we
-        #     have between deploying a service and having it come up in
-        #     `juju status`. We could probably do with cleaning it up so
-        #     that it fails a bit more noisily after a while.
-        return ''
-    units = service['units']
-    item = units.items()[0][1][item_name]
-    return item
-
-
-@contextmanager
-def maintain_charm_revision(path=None):
-    if path is None:
-        path = os.path.join(os.path.dirname(__file__), '..', 'revision')
-    revision = open(path).read()
-    try:
-        yield revision
-    finally:
-        with open(path, 'w') as f:
-            f.write(revision)
-
-
-def upgrade_charm(service_name, timeout=120):
-    next_revision = get_charm_revision(service_name) + 1
-    start_time = time.time()
-    run('juju', 'upgrade-charm', service_name)
-    while get_charm_revision(service_name) != next_revision:
-        if time.time() - start_time >= timeout:
-            raise RuntimeError('timeout waiting for charm to be upgraded')
-        time.sleep(0.1)
-    return next_revision
-
-
-def wait_for_machine(num_machines=1, timeout=300):
-    """Wait `timeout` seconds for `num_machines` machines to come up.
-
-    This wait_for... function can be called by other wait_for functions
-    whose timeouts might be too short in situations where only a bare
-    Juju setup has been bootstrapped.
-    """
-    # You may think this is a hack, and you'd be right. The easiest way
-    # to tell what environment we're working in (LXC vs EC2) is to check
-    # the dns-name of the first machine. If it's localhost we're in LXC
-    # and we can just return here.
-    if juju_status('machines')[0]['dns-name'] == 'localhost':
-        return
-    start_time = time.time()
-    while True:
-        # Drop the first machine, since it's the Zookeeper and that's
-        # not a machine that we need to wait for. This will only work
-        # for EC2 environments, which is why we return early above if
-        # we're in LXC.
-        machine_data = juju_status('machines')
-        non_zookeeper_machines = [
-            machine_data[key] for key in machine_data.keys()[1:]]
-        if len(non_zookeeper_machines) >= num_machines:
-            all_machines_running = True
-            for machine in non_zookeeper_machines:
-                if machine['instance-state'] != 'running':
-                    all_machines_running = False
-                    break
-            if all_machines_running:
-                break
-        if time.time() - start_time >= timeout:
-            raise RuntimeError('timeout waiting for service to start')
-        time.sleep(0.1)
-
-
-def wait_for_unit(service_name, timeout=480):
-    """Wait `timeout` seconds for a given service name to come up."""
-    wait_for_machine(num_machines=1)
-    start_time = time.time()
-    while True:
-        state = unit_info(service_name, 'state')
-        if 'error' in state or state == 'started':
-            break
-        if time.time() - start_time >= timeout:
-            raise RuntimeError('timeout waiting for service to start')
-        time.sleep(0.1)
-    if state != 'started':
-        raise RuntimeError('unit did not start, state: ' + state)
-
-
-def wait_for_relation(service_name, relation_name, timeout=120):
-    """Wait `timeout` seconds for a given relation to come up."""
-    start_time = time.time()
-    while True:
-        relation = unit_info(service_name, 'relations').get(relation_name)
-        if relation is not None and relation['state'] == 'up':
-            break
-        if time.time() - start_time >= timeout:
-            raise RuntimeError('timeout waiting for relation to be up')
-        time.sleep(0.1)
-
-
-def wait_for_page_contents(url, contents, timeout=120, validate=None):
-    if validate is None:
-        validate = operator.contains
-    start_time = time.time()
-    while True:
-        try:
-            stream = urllib2.urlopen(url)
-        except (urllib2.HTTPError, urllib2.URLError):
-            pass
-        else:
-            page = stream.read()
-            if validate(page, contents):
-                return page
-        if time.time() - start_time >= timeout:
-            raise RuntimeError('timeout waiting for contents of ' + url)
-        time.sleep(0.1)

=== modified file 'hooks/hooks.py'
--- hooks/hooks.py	2015-03-08 23:51:59 +0000
+++ hooks/hooks.py	2015-03-12 09:42:09 +0000
@@ -1769,15 +1769,16 @@
             run("apt-key add lib/{}.asc".format(pgdg_key))
             open(pgdg_list, 'w').write('deb {} {}-pgdg main'.format(
                 'http://apt.postgresql.org/pub/repos/apt/', distro_codename()))
-        if version == '9.4':
-            pgdg_94_list = '/etc/apt/sources.list.d/pgdg_94_{}.list'.format(
-                sanitize(hookenv.local_unit()))
-            if not os.path.exists(pgdg_94_list):
-                need_upgrade = True
-                open(pgdg_94_list, 'w').write(
-                    'deb {} {}-pgdg main 9.4'.format(
-                        'http://apt.postgresql.org/pub/repos/apt/',
-                        distro_codename()))
+        # if version == '9.5':
+        #     pgdg_beta_list = (
+        #         '/etc/apt/sources.list.d/pgdg_beta_{}.list'.format(
+        #             sanitize(hookenv.local_unit())))
+        #     if not os.path.exists(pgdg_beta_list):
+        #         need_upgrade = True
+        #         open(pgdg_beta_list, 'w').write(
+        #             'deb {} {}-pgdg main 9.5'.format(
+        #                 'http://apt.postgresql.org/pub/repos/apt/',
+        #                 distro_codename()))
 
     elif os.path.exists(pgdg_list):
         log(
@@ -2513,16 +2514,15 @@
     if len(relations) == 1 and 'nagios_hostname' in relations[0]:
         nagios_hostname = relations[0]['nagios_hostname']
         log("update_nrpe_checks: Obtained nagios_hostname ({}) "
-            "from nrpe-external-master relation.".format(
-            nagios_hostname))
+            "from nrpe-external-master relation.".format(nagios_hostname))
     else:
         unit = hookenv.local_unit()
         unit_name = unit.replace('/', '-')
         nagios_hostname = "%s-%s" % (config_data['nagios_context'], unit_name)
-        log("update_nrpe_checks: Deduced nagios_hostname ({}) from charm config "
-            "(nagios_hostname not found in nrpe-external-master relation, or "
-            "wrong number of relations found)".format(
-            nagios_hostname))
+        log("update_nrpe_checks: Deduced nagios_hostname ({}) from charm "
+            "config (nagios_hostname not found in nrpe-external-master "
+            "relation, or wrong number of relations "
+            "found)".format(nagios_hostname))
 
     nrpe_service_file = \
         '/var/lib/nagios/export/service__{}_check_pgsql.cfg'.format(

=== modified file 'hooks/test_hooks.py'
--- hooks/test_hooks.py	2015-03-09 10:30:33 +0000
+++ hooks/test_hooks.py	2015-03-12 09:42:09 +0000
@@ -68,7 +68,6 @@
             "encoding": "UTF-8",
             "extra_packages": "",
             "dumpfile_location": "None",
-            "config_change_command": "reload",
             "version": "9.1",
             "cluster_name": "main",
             "listen_ip": "*",

=== added file 'lib/juju-deployer-wrapper.py'
--- lib/juju-deployer-wrapper.py	1970-01-01 00:00:00 +0000
+++ lib/juju-deployer-wrapper.py	2015-03-12 09:42:09 +0000
@@ -0,0 +1,15 @@
+#!/usr/bin/python
+
+import subprocess
+import sys
+
+# Strip the -W option, as its noise messes with test output.
+args = list(sys.argv[1:])
+if '-W' in args:
+    args.remove('-W')
+cmd = ['juju-deployer'] + args
+try:
+    subprocess.check_output(cmd, stderr=subprocess.STDOUT)
+except subprocess.CalledProcessError as x:
+    sys.stderr.write(x.output)
+    sys.exit(x.returncode)

=== modified file 'templates/postgresql.conf.tmpl'
--- templates/postgresql.conf.tmpl	2015-02-11 21:05:42 +0000
+++ templates/postgresql.conf.tmpl	2015-03-12 09:42:09 +0000
@@ -41,6 +41,9 @@
 max_connections = {{max_connections}}
 {% endif -%}
 
+# Broken under all versions of Ubuntu, per Bug #1018307
+ssl_renegotiation_limit=0
+
 
 #------------------------------------------------------------------------------
 # RESOURCE USAGE (except WAL)
@@ -58,6 +61,9 @@
 {% if maintenance_work_mem != "" -%}
 maintenance_work_mem = {{maintenance_work_mem}}
 {% endif -%}
+{% if max_prepared_transactions != "" -%}
+max_prepared_transactions = {{max_prepared_transactions}}
+{% endif -%}
 
 
 #------------------------------------------------------------------------------
@@ -97,6 +103,10 @@
 {% if default_statistics_target != -1 -%}
 default_statistics_target = {{default_statistics_target}}
 {% endif -%}
+{% if collapse_limit != -1 -%}
+from_collapse_limit = {{collapse_limit}}
+join_collapse_limit = {{collapse_limit}}
+{% endif -%}
 
 
 #------------------------------------------------------------------------------

=== modified file 'test.py'
--- test.py	2014-12-04 17:49:22 +0000
+++ test.py	2015-03-12 09:42:09 +0000
@@ -23,7 +23,7 @@
 import psycopg2
 import testtools
 
-from testing.jujufixture import JujuFixture, run
+from testing.amuletfixture import AmuletFixture
 
 
 SERIES = os.environ.get('SERIES', 'trusty').strip()
@@ -92,189 +92,14 @@
             shutil.rmtree(psql_charmhelpers)
         shutil.copytree(main_charmhelpers, psql_charmhelpers)
 
-        self.juju = self.useFixture(JujuFixture(
-            series=SERIES, reuse_machines=True,
-            do_teardown='TEST_DONT_TEARDOWN_JUJU' not in os.environ))
+        self.deployment = AmuletFixture(series=SERIES)
+        self.deployment.setUp()
+        self.addCleanup(self.deployment.tearDown)
 
         # If the charms fail, we don't want tests to hang indefinitely.
-        timeout = int(os.environ.get('TEST_TIMEOUT', 1200))
-        if timeout > 0:
-            self.useFixture(fixtures.Timeout(timeout, gentle=True))
-
-    def wait_until_ready(self, pg_units, relation=True):
-
-        # Per Bug #1200267, it is impossible to know when a juju
-        # environment is actually ready for testing. Instead, we do the
-        # best we can by inspecting all the relation state, and, if it
-        # is at this particular instant in the expected state, hoping
-        # that the system is stable enough to continue testing.
-
-        timeout = time.time() + 600
-        pg_units = frozenset(pg_units)
-
-        # The list of PG units we expect to be related to the psql unit.
-        if relation:
-            rel_pg_units = frozenset(pg_units)
-        else:
-            rel_pg_units = frozenset()
-
-        while True:
-            try:
-                self.juju.wait_until_ready(0)  # Also refreshes status
-
-                status_pg_units = set(self.juju.status[
-                    'services']['postgresql']['units'].keys())
-
-                if pg_units != status_pg_units:
-                    # Confirm the PG units reported by 'juju status'
-                    # match the list we expect.
-                    raise NotReady('units not yet added/removed')
-
-                for psql_unit in self.juju.status['services']['psql']['units']:
-                    self.confirm_psql_unit_ready(psql_unit, rel_pg_units)
-
-                for pg_unit in pg_units:
-                    peers = [u for u in pg_units if u != pg_unit]
-                    self.confirm_postgresql_unit_ready(pg_unit, peers)
-
-                return
-            except NotReady:
-                if time.time() > timeout:
-                    raise
-                time.sleep(10)
-
-    def confirm_psql_unit_ready(self, psql_unit, pg_units):
-        # Confirm the db and db-admin relations are all in a useful
-        # state.
-        psql_rel_info = self.juju.relation_info(psql_unit)
-        if pg_units and not psql_rel_info:
-            raise NotReady('{} waiting for relations'.format(psql_unit))
-        elif not pg_units and psql_rel_info:
-            raise NotReady('{} waiting to drop relations'.format(psql_unit))
-        elif not pg_units and not psql_rel_info:
-            return
-
-        psql_service = psql_unit.split('/', 1)[0]
-
-        # The set of PostgreSQL units related to the psql unit. They
-        # might be related via several db or db-admin relations.
-        all_rel_pg_units = set()
-
-        for rel_name in psql_rel_info:
-            for rel_id, rel_info in psql_rel_info[rel_name].items():
-
-                # The database this relation has requested to use, if any.
-                requested_db = rel_info[psql_unit].get('database', None)
-
-                rel_pg_units = (
-                    [u for u in rel_info if not u.startswith(psql_service)])
-                all_rel_pg_units = all_rel_pg_units.union(rel_pg_units)
-
-                num_masters = 0
-
-                for unit in rel_pg_units:
-                    unit_rel_info = rel_info[unit]
-
-                    # PG unit must be presenting the correct database.
-                    if 'database' not in unit_rel_info:
-                        raise NotReady(
-                            '{} has no database'.format(unit))
-                    if requested_db and (
-                            unit_rel_info['database'] != requested_db):
-                        raise NotReady(
-                            '{} not using requested db {}'.format(
-                                unit, requested_db))
-
-                    # PG unit must be in a valid state.
-                    state = unit_rel_info.get('state', None)
-                    if not state:
-                        raise NotReady(
-                            '{} has no state'.format(unit))
-                    elif state == 'standalone':
-                        if len(rel_pg_units) > 1:
-                            raise NotReady(
-                                '{} is standalone'.format(unit))
-                    elif state == 'master':
-                        num_masters += 1
-                    elif state != 'hot standby':
-                        # Failover state or totally broken.
-                        raise NotReady(
-                            '{} in {} state'.format(unit, state))
-
-                    # PG unit must have authorized this psql client.
-                    allowed_units = unit_rel_info.get(
-                        'allowed-units', '').split()
-                    if psql_unit not in allowed_units:
-                        raise NotReady(
-                            '{} not yet authorized by {} ({})'.format(
-                                psql_unit, unit, allowed_units))
-
-                # Alas, this is no longer true with manual replication.
-                # We must not have multiple masters in this relation.
-                # if len(rel_pg_units) > 1 and num_masters != 1:
-                #     raise NotReady(
-                #         '{} masters'.format(num_masters))
-
-        if pg_units != all_rel_pg_units:
-            raise NotReady(
-                'Expected PG units {} != related units {}'.format(
-                    pg_units, all_rel_pg_units))
-
-    def confirm_postgresql_unit_ready(self, pg_unit, peers=()):
-        pg_rel_info = self.juju.relation_info(pg_unit)
-        if not pg_rel_info:
-            raise NotReady('{} has no relations'.format(pg_unit))
-
-        try:
-            rep_rel_id = pg_rel_info['replication'].keys()[0]
-            actual_peers = set([
-                u for u in pg_rel_info['replication'][rep_rel_id].keys()
-                if u != pg_unit])
-        except (IndexError, KeyError):
-            if peers:
-                raise NotReady('Peer relation does not exist')
-            rep_rel_id = None
-            actual_peers = set()
-
-        if actual_peers != set(peers):
-            raise NotReady('Expecting {} peers, found {}'.format(
-                peers, actual_peers))
-
-        if not peers:
-            return
-
-        pg_rep_rel_info = pg_rel_info['replication'][rep_rel_id].get(
-            pg_unit, None)
-        if not pg_rep_rel_info:
-            raise NotReady('{} has not yet joined the peer relation'.format(
-                pg_unit))
-
-        state = pg_rep_rel_info.get('state', None)
-
-        if not state:
-            raise NotReady('{} has no state'.format(pg_unit))
-
-        if state == 'standalone' and peers:
-            raise NotReady('{} is standalone but has peers'.format(pg_unit))
-
-        if state not in ('standalone', 'master', 'hot standby'):
-            raise NotReady('{} reports failover in progress'.format(pg_unit))
-
-        num_masters = 1 if state in ('master', 'standalone') else 0
-
-        for peer in peers:
-            peer_rel_info = pg_rel_info['replication'][rep_rel_id][peer]
-            peer_state = peer_rel_info.get('state', None)
-            if not peer_state:
-                raise NotReady('{} has no peer state'.format(peer))
-            if peer_state == 'master':
-                num_masters += 1
-            elif peer_state != 'hot standby':
-                raise NotReady('Peer {} in state {}'.format(peer, peer_state))
-
-        # No longer true with manual replication.
-        # if num_masters < 1:
-        #     raise NotReady('No masters seen from {}'.format(pg_unit))
+        self.timeout = int(os.environ.get('TEST_TIMEOUT', 1200))
+        if self.timeout > 0:
+            self.useFixture(fixtures.Timeout(self.timeout, gentle=True))
 
     def sql(self, sql, postgres_unit=None, psql_unit=None, dbname=None):
         '''Run some SQL on postgres_unit from psql_unit.
@@ -289,38 +114,34 @@
         '''
         # Which psql unit we are going to query from.
         if psql_unit is None:
-            psql_unit = (
-                self.juju.status['services']['psql']['units'].keys()[0])
-
-        full_rel_info = self.juju.relation_info(psql_unit)
+            psql_sentry = self.deployment.sentry['psql'][0]
+            psql_unit = psql_sentry.info['unit_name']
+        else:
+            psql_sentry = self.deployment.sentry[psql_unit]
 
         # 'db' or 'db-admin' relation?
         rel_name = 'db-admin' if dbname else 'db'
+        to_rel = 'psql:{}'.format(rel_name)
 
         # Which PostgreSQL unit we want to talk to.
         if postgres_unit is None:
-            postgres_unit = (
-                self.juju.status['services']['postgresql']['units'].keys()[0])
+            postgres_sentry = self.deployment.sentry['postgresql'][0]
+            relinfo = postgres_sentry.relation(rel_name, to_rel)
         elif postgres_unit in ('master', 'hot standby'):
-            for rel_id, rel_info in full_rel_info[rel_name].items():
-                for rel_unit, rel_unit_info in rel_info.items():
-                    if rel_unit_info.get('state') == postgres_unit:
-                        postgres_unit = rel_unit
-        assert postgres_unit not in (None, 'master', 'hot standby'), (
-            'Unable to determine postgresql unit to use')
-
-        # PostgreSQL unit relation info
-        rel_info = None
-        for rel_id in full_rel_info[rel_name]:
-            if postgres_unit in full_rel_info[rel_name][rel_id]:
-                rel_info = full_rel_info[rel_name][rel_id][postgres_unit]
-                break
-        assert rel_info is not None, (
-            'Unable to find pg rel info {} {!r}'.format(
-                postgres_unit, full_rel_info[rel_name]))
+            postgres_sentry = None
+            for postgres_sentry in self.deployment.sentry['postgresql']:
+                relinfo = postgres_sentry.relation(rel_name, to_rel)
+                if relinfo.get('state') == postgres_unit:
+                    break
+            assert postgres_sentry is not None, (
+                'Unable to determine postgresql unit to use')
+            relinfo = postgres_sentry.relation(rel_name, to_rel)
+        else:
+            postgres_sentry = self.deployment.sentry[postgres_unit]
+            relinfo = postgres_sentry.relation(rel_name, to_rel)
 
         if dbname is None:
-            dbname = rel_info['database']
+            dbname = relinfo['database']
 
         # Choose a local port for our tunnel.
         s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
@@ -333,8 +154,8 @@
         # tunnels, as simply killing the 'juju ssh' process doesn't seem
         # to be enough.
         tunnel_cmd = [
-            'juju', 'ssh', psql_unit, '-N', '-L',
-            '{}:{}:{}'.format(local_port, rel_info['host'], rel_info['port'])]
+            'juju', 'ssh', psql_unit, '-q', '-N', '-L',
+            '{}:{}:{}'.format(local_port, relinfo['host'], relinfo['port'])]
         # Don't disable stdout, so we can see when there are SSH
         # failures like bad host keys.
         tunnel_proc = subprocess.Popen(
@@ -360,7 +181,7 @@
             # Execute the query
             con = psycopg2.connect(
                 database=dbname, port=local_port, host='localhost',
-                user=rel_info['user'], password=rel_info['password'])
+                user=relinfo['user'], password=relinfo['password'])
             cur = con.cursor()
             cur.execute(sql)
             if cur.description is None:
@@ -379,28 +200,25 @@
         cmd = [
             'juju', 'run', '--unit', unit,
             'sudo pg_ctlcluster 9.1 main -force {}'.format(command)]
-        run(self, cmd)
+        subprocess.check_output(cmd)
 
     def test_basic(self):
         '''Connect to a a single unit service via the db relationship.'''
-        self.juju.deploy(TEST_CHARM, 'postgresql', config=self.pg_config)
-        self.juju.deploy(PSQL_CHARM, 'psql')
-        self.juju.do(['add-relation', 'postgresql:db', 'psql:db'])
-        self.wait_until_ready(['postgresql/0'])
+        self.deployment.add('psql', PSQL_CHARM)
+        self.deployment.add('postgresql', TEST_CHARM)
+        self.deployment.configure('postgresql', self.pg_config)
+        self.deployment.relate('postgresql:db', 'psql:db')
+        self.deployment.deploy(timeout=self.timeout)
 
         result = self.sql('SELECT TRUE')
         self.assertEqual(result, [(True,)])
 
-        # Confirm that the relation tears down without errors.
-        self.juju.do(['destroy-relation', 'postgresql:db', 'psql:db'])
-        self.wait_until_ready(['postgresql/0'], relation=False)
-
     def test_streaming_replication(self):
-        self.juju.deploy(
-            TEST_CHARM, 'postgresql', num_units=2, config=self.pg_config)
-        self.juju.deploy(PSQL_CHARM, 'psql')
-        self.juju.do(['add-relation', 'postgresql:db', 'psql:db'])
-        self.wait_until_ready(['postgresql/0', 'postgresql/1'])
+        self.deployment.add('psql', PSQL_CHARM)
+        self.deployment.add('postgresql', TEST_CHARM, 2)
+        self.deployment.configure('postgresql', self.pg_config)
+        self.deployment.relate('postgresql:db', 'psql:db')
+        self.deployment.deploy(timeout=self.timeout)
 
         # Confirm that the slave has successfully opened a streaming
         # replication connection.
@@ -431,11 +249,11 @@
                                 stderr=subprocess.STDOUT)
         self.addCleanup(swift_cleanup)
 
-        self.juju.deploy(
-            TEST_CHARM, 'postgresql', num_units=2, config=self.pg_config)
-        self.juju.deploy(PSQL_CHARM, 'psql')
-        self.juju.do(['add-relation', 'postgresql:db-admin', 'psql:db-admin'])
-        self.wait_until_ready(['postgresql/0', 'postgresql/1'])
+        self.deployment.add('psql', PSQL_CHARM)
+        self.deployment.add('postgresql', TEST_CHARM, 2)
+        self.deployment.configure('postgresql', self.pg_config)
+        self.deployment.relate('postgresql:db-admin', 'psql:db-admin')
+        self.deployment.deploy(timeout=self.timeout)
 
         # Confirm that the slave has not opened a streaming
         # replication connection.
@@ -476,11 +294,11 @@
                             stderr=subprocess.STDOUT)
         self.addCleanup(swift_cleanup)
 
-        self.juju.deploy(
-            TEST_CHARM, 'postgresql', num_units=2, config=self.pg_config)
-        self.juju.deploy(PSQL_CHARM, 'psql')
-        self.juju.do(['add-relation', 'postgresql:db-admin', 'psql:db-admin'])
-        self.wait_until_ready(['postgresql/0', 'postgresql/1'])
+        self.deployment.add('psql', PSQL_CHARM)
+        self.deployment.add('postgresql', TEST_CHARM, 2)
+        self.deployment.configure('postgresql', self.pg_config)
+        self.deployment.relate('postgresql:db-admin', 'psql:db-admin')
+        self.deployment.deploy(timeout=self.timeout)
 
         # Confirm that the slave has not opened a streaming
         # replication connection.
@@ -505,20 +323,16 @@
 
     def test_basic_admin(self):
         '''Connect to a single unit service via the db-admin relationship.'''
-        self.juju.deploy(TEST_CHARM, 'postgresql', config=self.pg_config)
-        self.juju.deploy(PSQL_CHARM, 'psql')
-        self.juju.do(['add-relation', 'postgresql:db-admin', 'psql:db-admin'])
-        self.juju.do(['expose', 'postgresql'])
-        self.wait_until_ready(['postgresql/0'])
+        self.deployment.add('psql', PSQL_CHARM)
+        self.deployment.add('postgresql', TEST_CHARM, 2)
+        self.deployment.configure('postgresql', self.pg_config)
+        self.deployment.relate('postgresql:db-admin', 'psql:db-admin')
+        self.deployment.expose('postgresql')
+        self.deployment.deploy(timeout=self.timeout)
 
         result = self.sql('SELECT TRUE', dbname='postgres')
         self.assertEqual(result, [(True,)])
 
-        # Confirm that the relation tears down without errors.
-        self.juju.do([
-            'destroy-relation', 'postgresql:db-admin', 'psql:db-admin'])
-        self.wait_until_ready(['postgresql/0'], relation=False)
-
     def is_master(self, postgres_unit, dbname=None):
         is_master = self.sql(
             'SELECT NOT pg_is_in_recovery()',
@@ -530,20 +344,20 @@
         # Per Bug #1258485, creating a 3 unit service will often fail.
         # Instead, create a 2 unit service, wait for it to be ready,
         # then add a third unit.
-        self.juju.deploy(PSQL_CHARM, 'psql')
-        self.juju.deploy(
-            TEST_CHARM, 'postgresql', num_units=2, config=self.pg_config)
-        self.juju.do(['add-relation', 'postgresql:db', 'psql:db'])
-        self.wait_until_ready(['postgresql/0', 'postgresql/1'])
-        self.juju.add_unit('postgresql')
-        self.wait_until_ready(['postgresql/0', 'postgresql/1', 'postgresql/2'])
+        self.deployment.add('psql', PSQL_CHARM)
+        self.deployment.add('postgresql', TEST_CHARM, 2)
+        self.deployment.configure('postgresql', self.pg_config)
+        self.deployment.relate('postgresql:db', 'psql:db')
+        self.deployment.deploy(timeout=self.timeout)
+        self.deployment.add_unit('postgresql')
+        self.deployment.wait()
 
         # Even on a freshly setup service, we have no idea which unit
         # will become the master as we have no control over which two
         # units join the peer relation first.
-        units = sorted(
-            (self.is_master(unit), unit) for unit in
-            self.juju.status['services']['postgresql']['units'].keys())
+        units = sorted((self.is_master(unit), unit) for unit in
+                       [sentry.info['unit_name']
+                        for sentry in self.deployment.sentry['postgresql']])
         self.assertFalse(units[0][0])
         self.assertFalse(units[1][0])
         self.assertTrue(units[2][0])
@@ -580,8 +394,8 @@
         self.assertIs(True, token_received(standby_unit_2))
 
         # Remove the master unit.
-        self.juju.do(['remove-unit', master_unit])
-        self.wait_until_ready([standby_unit_1, standby_unit_2])
+        self.deployment.remove_unit(master_unit)
+        self.deployment.wait()
 
         # When we failover, the unit that has received the most WAL
         # information from the old master (most in sync) is elected the
@@ -603,8 +417,8 @@
         self.assertIs(True, token_received(standby_unit))
 
         # Remove the master again, leaving a single unit.
-        self.juju.do(['remove-unit', master_unit])
-        self.wait_until_ready([standby_unit])
+        self.deployment.remove_unit(master_unit)
+        self.deployment.wait()
 
         # Last unit is a working, standalone database.
         self.is_master(standby_unit)
@@ -612,35 +426,29 @@
 
         # Confirm that it is actually reporting as 'standalone' rather
         # than 'master'
-        full_relation_info = self.juju.relation_info('psql/0')
-        for rel_info in full_relation_info['db'].values():
-            for unit, unit_rel_info in rel_info.items():
-                if unit == 'psql/0':
-                    pass
-                elif unit == standby_unit:
-                    self.assertEqual(unit_rel_info['state'], 'standalone')
-                else:
-                    raise RuntimeError('Unknown unit {}'.format(unit))
+        relinfo = self.deployment.sentry[standby_unit].relation('db',
+                                                                'psql:db')
+        self.assertEqual(relinfo.get('state'), 'standalone')
 
     def test_failover_election(self):
         """Ensure master elected in a failover is the best choice"""
         # Per Bug #1258485, creating a 3 unit service will often fail.
         # Instead, create a 2 unit service, wait for it to be ready,
         # then add a third unit.
-        self.juju.deploy(
-            TEST_CHARM, 'postgresql', num_units=2, config=self.pg_config)
-        self.juju.deploy(PSQL_CHARM, 'psql')
-        self.juju.do(['add-relation', 'postgresql:db-admin', 'psql:db-admin'])
-        self.wait_until_ready(['postgresql/0', 'postgresql/1'])
-        self.juju.add_unit('postgresql')
-        self.wait_until_ready(['postgresql/0', 'postgresql/1', 'postgresql/2'])
+        self.deployment.add('psql', PSQL_CHARM)
+        self.deployment.add('postgresql', TEST_CHARM, 2)
+        self.deployment.configure('postgresql', self.pg_config)
+        self.deployment.relate('postgresql:db-admin', 'psql:db-admin')
+        self.deployment.deploy(timeout=self.timeout)
+        self.deployment.add_unit('postgresql')
+        self.deployment.wait()
 
         # Even on a freshly setup service, we have no idea which unit
         # will become the master as we have no control over which two
         # units join the peer relation first.
-        units = sorted(
-            (self.is_master(unit, 'postgres'), unit) for unit in
-            self.juju.status['services']['postgresql']['units'].keys())
+        units = sorted((self.is_master(unit, 'postgres'), unit) for unit in
+                       [sentry.info['unit_name']
+                        for sentry in self.deployment.sentry['postgresql']])
         self.assertFalse(units[0][0])
         self.assertFalse(units[1][0])
         self.assertTrue(units[2][0])
@@ -665,8 +473,8 @@
         self.pg_ctlcluster(standby_unit_1, 'start')
 
         # Failover.
-        self.juju.do(['remove-unit', master_unit])
-        self.wait_until_ready([standby_unit_1, standby_unit_2])
+        self.deployment.remove_unit(master_unit)
+        self.deployment.wait()
 
         # Fix replication.
         self.sql(
@@ -685,15 +493,18 @@
         port = 7400 + int((self.VERSION or '66').replace('.', ''))
         self.pg_config['listen_port'] = port
 
-        self.juju.deploy(TEST_CHARM, 'postgresql', config=self.pg_config)
-        self.juju.deploy(PSQL_CHARM, 'psql')
-        self.juju.do(['add-relation', 'postgresql:db-admin', 'psql:db-admin'])
-        self.wait_until_ready(['postgresql/0'])
+        self.deployment.add('psql', PSQL_CHARM)
+        self.deployment.add('postgresql', TEST_CHARM)
+        self.deployment.configure('postgresql', self.pg_config)
+        self.deployment.expose('postgresql')  # This test requires exposure.
+        self.deployment.relate('postgresql:db-admin', 'psql:db-admin')
+        self.deployment.deploy(timeout=self.timeout)
 
         # Determine the IP address that the unit will see.
-        unit = self.juju.status['services']['postgresql']['units'].keys()[0]
-        unit_ip = self.juju.status['services']['postgresql']['units'][
-            unit]['public-address']
+        status = self.deployment.get_status()
+        unit = status['services']['postgresql']['units'].keys()[0]
+        unit_ip = (status['services']['postgresql']['units'][unit]
+                   ['public-address'])
         s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
         s.connect((unit_ip, port))
         my_ip = s.getsockname()[0]
@@ -713,8 +524,8 @@
             psycopg2.OperationalError, psycopg2.connect, conn_str)
 
         # Connections should work after setting the admin-addresses.
-        self.juju.do([
-            'set', 'postgresql', 'admin_addresses={}'.format(my_ip)])
+        self.deployment.configure('postgresql', dict(admin_addresses=my_ip))
+        self.deployment.wait()
         timeout = time.time() + 30
         while True:
             try:
@@ -731,14 +542,14 @@
     def test_explicit_database(self):
         # Two units to ensure both masters and hot standbys
         # present the correct credentials.
-        self.juju.deploy(
-            TEST_CHARM, 'postgresql', num_units=2, config=self.pg_config)
-        self.juju.deploy(PSQL_CHARM, 'psql')
-        self.juju.do(['set', 'psql', 'database=explicit'])
-        self.juju.do(['add-relation', 'postgresql:db', 'psql:db'])
+        self.deployment.add('psql', PSQL_CHARM)
+        self.deployment.configure('psql', dict(database='explicit'))
+        self.deployment.add('postgresql', TEST_CHARM, 2)
+        self.deployment.configure('postgresql', self.pg_config)
+        self.deployment.relate('postgresql:db', 'psql:db')
+        self.deployment.deploy(timeout=self.timeout)
 
         pg_units = ['postgresql/0', 'postgresql/1']
-        self.wait_until_ready(pg_units)
 
         for unit in pg_units:
             result = self.sql('SELECT current_database()', unit)[0][0]
@@ -749,20 +560,20 @@
     def test_roles_granted(self):
         # We use two units to confirm that there is no attempt to
         # grant roles on the hot standby.
-        self.juju.deploy(
-            TEST_CHARM, 'postgresql', num_units=2, config=self.pg_config)
-        self.juju.deploy(PSQL_CHARM, 'psql', config={'roles': 'role_a'})
-        self.juju.do(['add-relation', 'postgresql:db', 'psql:db'])
-        pg_units = ['postgresql/0', 'postgresql/1']
-        self.wait_until_ready(pg_units)
+        self.deployment.add('psql', PSQL_CHARM)
+        self.deployment.configure('psql', dict(roles='role_a'))
+        self.deployment.add('postgresql', TEST_CHARM, 2)
+        self.deployment.configure('postgresql', self.pg_config)
+        self.deployment.relate('postgresql:db', 'psql:db')
+        self.deployment.deploy(timeout=self.timeout)
 
         has_role_a = self.sql('''
             SELECT pg_has_role(current_user, 'role_a', 'MEMBER')
             ''')[0][0]
         self.assertTrue(has_role_a)
 
-        self.juju.do(['set', 'psql', 'roles=role_a,role_b'])
-        self.wait_until_ready(pg_units)
+        self.deployment.configure('psql', dict(roles='role_a,role_b'))
+        self.deployment.wait()
 
         # Retry this for a while. Per Bug #1200267, we can't tell when
         # the hooks have finished running and the role has been granted.
@@ -788,12 +599,12 @@
     def test_roles_revoked(self):
         # We use two units to confirm that there is no attempts to
         # grant roles on the hot standby.
-        self.juju.deploy(
-            TEST_CHARM, 'postgresql', num_units=2, config=self.pg_config)
-        self.juju.deploy(PSQL_CHARM, 'psql', config={'roles': 'role_a,role_b'})
-        self.juju.do(['add-relation', 'postgresql:db', 'psql:db'])
-        pg_units = ['postgresql/0', 'postgresql/1']
-        self.wait_until_ready(pg_units)
+        self.deployment.add('psql', PSQL_CHARM)
+        self.deployment.configure('psql', dict(roles='role_a,role_b'))
+        self.deployment.add('postgresql', TEST_CHARM, 2)
+        self.deployment.configure('postgresql', self.pg_config)
+        self.deployment.relate('postgresql:db', 'psql:db')
+        self.deployment.deploy(timeout=self.timeout)
 
         has_role_a, has_role_b = self.sql('''
             SELECT
@@ -803,8 +614,8 @@
         self.assertTrue(has_role_a)
         self.assertTrue(has_role_b)
 
-        self.juju.do(['set', 'psql', 'roles=role_c'])
-        self.wait_until_ready(pg_units)
+        self.deployment.configure('psql', dict(roles='role_c'))
+        self.deployment.wait()
 
         # Per Bug #1200267, we have to retry a while here and hope.
         # We have of knowing when the pending role changes have
@@ -823,8 +634,8 @@
         self.assertFalse(has_role_b)
         self.assertTrue(has_role_c)
 
-        self.juju.do(['unset', 'psql', 'roles'])
-        self.wait_until_ready(pg_units)
+        self.deployment.configure('psql', dict(roles=None))
+        self.deployment.wait()
 
         timeout = time.time() + 60
         while True:
@@ -844,15 +655,14 @@
         # Deploy 2 PostgreSQL units and 2 rsyslog units to ensure that
         # log messages from every source reach every sink.
         self.pg_config['log_min_duration_statement'] = 0  # Log all statements
-        self.juju.deploy(
-            TEST_CHARM, 'postgresql', num_units=2, config=self.pg_config)
-        self.juju.deploy(PSQL_CHARM, 'psql')
-        self.juju.do(['add-relation', 'postgresql:db', 'psql:db'])
-        self.juju.deploy('cs:rsyslog', 'rsyslog', num_units=2)
-        self.juju.do([
-            'add-relation', 'postgresql:syslog', 'rsyslog:aggregator'])
-        pg_units = ['postgresql/0', 'postgresql/1']
-        self.wait_until_ready(pg_units)
+
+        self.deployment.add('psql', PSQL_CHARM)
+        self.deployment.add('postgresql', TEST_CHARM, 2)
+        self.deployment.configure('postgresql', self.pg_config)
+        self.deployment.relate('postgresql:db', 'psql:db')
+        self.deployment.add('rsyslog', 'cs:rsyslog', 2)
+        self.deployment.relate('postgresql:syslog', 'rsyslog:aggregator')
+        self.deployment.deploy(timeout=self.timeout)
 
         token = str(uuid.uuid1())
 
@@ -861,36 +671,24 @@
         time.sleep(2)
 
         for runit in ['rsyslog/0', 'rsyslog/1']:
-            cmd = ['juju', 'run', '--unit', runit, 'tail -100 /var/log/syslog']
-            out = run(self, cmd)
+            sentry = self.deployment.sentry[runit]
+            out = sentry.run('tail -100 /var/log/syslog')
             self.failUnless('master {}'.format(token) in out)
             self.failUnless('hot standby {}'.format(token) in out)
 
-        # Confirm that the relation tears down correctly.
-        self.juju.do(['destroy-service', 'rsyslog'])
-        timeout = time.time() + 120
-        while time.time() < timeout:
-            status = self.juju.refresh_status()
-            if 'rsyslog' not in status['services']:
-                break
-        self.assert_(
-            'rsyslog' not in status['services'], 'rsyslog failed to die')
-        self.wait_until_ready(pg_units)
-
     def test_upgrade_charm(self):
-        self.juju.deploy(
-            TEST_CHARM, 'postgresql', num_units=2, config=self.pg_config)
-        self.juju.deploy(PSQL_CHARM, 'psql')
-        self.juju.do(['add-relation', 'postgresql:db', 'psql:db'])
-        pg_units = ['postgresql/0', 'postgresql/1']
-        self.wait_until_ready(pg_units)
+        self.deployment.add('psql', PSQL_CHARM)
+        self.deployment.add('postgresql', TEST_CHARM, 2)
+        self.deployment.configure('postgresql', self.pg_config)
+        self.deployment.relate('postgresql:db', 'psql:db')
+        self.deployment.deploy(timeout=self.timeout)
 
         # Create something
         self.sql("CREATE TABLE Foo AS SELECT TRUE", 'master')
 
         # Kick off the upgrade-charm hook
-        self.juju.do(['upgrade-charm', 'postgresql'])
-        self.wait_until_ready(pg_units)
+        subprocess.check_output(['juju', 'upgrade-charm', 'postgresql'])
+        self.deployment.wait()
 
         # Ensure that our data has perservered.
         master_data = self.sql('SELECT * FROM Foo', 'master')[0][0]
@@ -900,12 +698,12 @@
 
     def test_manual_replication(self):
         self.pg_config['manual_replication'] = True
-        self.juju.deploy(
-            TEST_CHARM, 'postgresql', num_units=2, config=self.pg_config)
-        self.juju.deploy(PSQL_CHARM, 'psql')
-        self.juju.do(['add-relation', 'postgresql:db', 'psql:db'])
-        pg_units = ['postgresql/0', 'postgresql/1']
-        self.wait_until_ready(pg_units)
+
+        self.deployment.add('psql', PSQL_CHARM)
+        self.deployment.add('postgresql', TEST_CHARM, 2)
+        self.deployment.configure('postgresql', self.pg_config)
+        self.deployment.relate('postgresql:db', 'psql:db')
+        self.deployment.deploy(timeout=self.timeout)
 
         # Because we have enabled manual_replication mode, and not
         # actually manually configured replication, we will have
@@ -914,12 +712,10 @@
         self.assertTrue(self.is_master('postgresql/1'))
 
         # Both units are advertised as master on the relation.
-        rel_info = self.juju.relation_info('psql/0')
-        for relname in rel_info:
-            for relid in rel_info[relname]:
-                for unit in pg_units:
-                    self.assertEqual(rel_info[relname][relid][unit]['state'],
-                                     'master')
+        for unit in ['postgresql/0', 'postgresql/1']:
+            sentry = self.deployment.sentry[unit]
+            relinfo = sentry.relation('db', 'psql:db')
+            self.assertEqual(relinfo.get('state'), 'master')
 
 
 class PG91Tests(
@@ -948,8 +744,7 @@
 class PG94Tests(
         PostgreSQLCharmBaseTestCase,
         testtools.TestCase, fixtures.TestWithFixtures):
-    # 9.4 is still in beta, with packages only available in the PGDG
-    # archive.
+    # 9.4 is released, but packages are only available in the PGDG archive.
     VERSION = '9.4'
     PGDG = True
 

=== removed file 'testing/README'
--- testing/README	2013-10-10 17:41:02 +0000
+++ testing/README	1970-01-01 00:00:00 +0000
@@ -1,36 +0,0 @@
-This package provides a Juju test fixture to help you write tests for
-your charms as a Python test suite. It requires you to already have a
-bootstrapped environment.
-
-Below is a simple example using the fixture. You can find a more extensive
-example in the PostgreSQL charm.
-
-#!/usr/bin/python
-import os
-import fixtures
-import testtools
-import unittest
-from charmhelpers.testing.jujufixture import JujuFixture
-
-
-class CharmTestCase(testtools.TestCase, fixtures.TestWithFixtures):
-
-    def setUp(self):
-        super(CharmTestCase, self).setUp()
-
-        self.juju = self.useFixture(JujuFixture(reuse_machines=True))
-
-        # If the charms fail, we don't want tests to hang indefinitely.
-        timeout = int(os.environ.get('TEST_TIMEOUT', 900))
-        self.useFixture(fixtures.Timeout(timeout, gentle=True))
-
-    def test_basic(self):
-        self.juju.deploy('local:mycharm', 'test-mycharm', num_units=1)
-        self.juju.add_unit('local:mycharm', 'test-mycharm', num_units=2)
-        self.juju.deploy('local:othercharm')
-        self.juju.do(['add-relation', 'test-mycharm:rel', 'othercharm:rel'])
-        self.juju.wait_until_ready()
-        [ ... test the deployment ... ]
-
-if __name__ == '__main__':
-    unittest.main()

=== added file 'testing/amuletfixture.py'
--- testing/amuletfixture.py	1970-01-01 00:00:00 +0000
+++ testing/amuletfixture.py	2015-03-12 09:42:09 +0000
@@ -0,0 +1,200 @@
+# Copyright 2015 Canonical Ltd.
+#
+# This file is part of the PostgreSQL Charm for Juju.
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 3, as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranties of
+# MERCHANTABILITY, SATISFACTORY QUALITY, or FITNESS FOR A PARTICULAR
+# PURPOSE.  See the GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+from functools import wraps
+import json
+import os
+import shutil
+import subprocess
+import tempfile
+import time
+
+import amulet
+import yaml
+
+
+class AmuletFixture(amulet.Deployment):
+    def __init__(self, series='precise'):
+        # We use a wrapper around juju-deployer so we can fix how it is
+        # invoked. In particular, turn off all the noise so we can
+        # actually read our test output.
+        juju_deployer = os.path.abspath(os.path.join(
+            os.path.dirname(__file__), os.pardir, 'lib',
+            'juju-deployer-wrapper.py'))
+        super(AmuletFixture, self).__init__(series=series,
+                                            juju_deployer=juju_deployer)
+
+    def setUp(self):
+        self._temp_dirs = []
+
+        self.reset_environment(force=True)
+
+        # Repackage our charm to a temporary directory, allowing us
+        # to strip our virtualenv symlinks that would otherwise cause
+        # juju to abort. We also strip the .bzr directory, working
+        # around Bug #1394078.
+        self.repackage_charm()
+
+        # Fix amulet.Deployment so it doesn't depend on environment
+        # variables or the current working directory, but rather the
+        # environment we have introspected.
+        with open(os.path.join(self.charm_dir, 'metadata.yaml'), 'r') as s:
+            self.charm_name = yaml.safe_load(s)['name']
+        self.charm_cache.test_charm = None
+        self.charm_cache.fetch(self.charm_name, self.charm_dir, self.series)
+
+        # Explicitly reset $JUJU_REPOSITORY to ensure amulet and
+        # juju-deployer does not mess with the real one, per Bug #1393792
+        self.org_repo = os.environ.get('JUJU_REPOSITORY', None)
+        temp_repo = tempfile.mkdtemp(suffix='.repo')
+        self._temp_dirs.append(temp_repo)
+        os.environ['JUJU_REPOSITORY'] = temp_repo
+        os.makedirs(os.path.join(temp_repo, self.series), mode=0o700)
+
+    def tearDown(self, reset_environment=True):
+        if reset_environment:
+            self.reset_environment()
+        if self.org_repo is None:
+            del os.environ['JUJU_REPOSITORY']
+        else:
+            os.environ['JUJU_REPOSITORY'] = self.org_repo
+
+    def deploy(self, timeout=None):
+        '''Deploying or updating the configured system.
+
+        Invokes amulet.Deployer.setup with a nicer name and standard
+        timeout handling.
+        '''
+        if timeout is None:
+            timeout = int(os.environ.get('AMULET_TIMEOUT', 900))
+
+        # If setUp fails, tearDown is never called leaving the
+        # environment setup. This is useful for debugging.
+        self.setup(timeout=timeout)
+        self.wait(timeout=timeout)
+
+    def __del__(self):
+        for temp_dir in self._temp_dirs:
+            if os.path.exists(temp_dir):
+                shutil.rmtree(temp_dir, ignore_errors=True)
+
+    def get_status(self):
+        raw = subprocess.check_output(['juju', 'status', '--format=json'],
+                                      universal_newlines=True)
+        if raw:
+            return json.loads(raw)
+        return None
+
+    def wait(self, timeout=None):
+        '''Wait until the environment has reached a stable state.'''
+        cmd = ['juju', 'wait']
+        if timeout:
+            cmd = ['timeout', str(timeout)] + cmd
+        subprocess.check_output(cmd)
+
+    def reset_environment(self, force=False):
+        fails = dict()
+        while True:
+            status = self.get_status()
+            service_items = status.get('services', {}).items()
+            if not service_items:
+                break
+            for service_name, service in service_items:
+                if service.get('life', '') not in ('dying', 'dead'):
+                    subprocess.check_output(['juju', 'destroy-service',
+                                             service_name],
+                                            stderr=subprocess.STDOUT)
+                for unit_name, unit in service.get('units', {}).items():
+                    if unit.get('agent-state', None) == 'error':
+                        if force:
+                            # If any units have failed hooks, unstick them.
+                            try:
+                                subprocess.check_output(
+                                    ['juju', 'resolved', unit_name],
+                                    stderr=subprocess.STDOUT)
+                            except subprocess.CalledProcessError:
+                                # A previous 'resolved' call make cause a
+                                # subsequent one to fail if it is still
+                                # being processed. However, we need to keep
+                                # retrying because after a successful
+                                # resolution a subsequent hook may cause an
+                                # error state.
+                                pass
+                        else:
+                            fails[unit_name] = unit
+            time.sleep(1)
+
+        harvest_machines = []
+        for machine, state in status.get('machines', {}).items():
+            if machine != "0" and state.get('life') not in ('dying', 'dead'):
+                harvest_machines.append(machine)
+
+        if harvest_machines:
+            cmd = ['juju', 'remove-machine', '--force'] + harvest_machines
+            subprocess.check_output(cmd, stderr=subprocess.STDOUT)
+
+        if fails:
+            raise Exception("Teardown failed", fails)
+
+    def repackage_charm(self):
+        """Mirror the charm into a staging area.
+
+        We do this to work around issues with Amulet, juju-deployer
+        and juju. In particular:
+            - symlinks in the Python virtual env pointing outside of the
+            charm directory.
+            - odd bzr interactions, such as tests being run on the committed
+            version of the charm, rather than the working tree.
+
+        Returns the test charm directory.
+        """
+        # Find the charm_dir we are testing
+        src_charm_dir = os.path.dirname(__file__)
+        while True:
+            if os.path.exists(os.path.join(src_charm_dir,
+                                           'metadata.yaml')):
+                break
+            assert src_charm_dir != os.sep, 'metadata.yaml not found'
+            src_charm_dir = os.path.abspath(os.path.join(src_charm_dir,
+                                                         os.pardir))
+
+        with open(os.path.join(src_charm_dir, 'metadata.yaml'), 'r') as s:
+            self.charm_name = yaml.safe_load(s)['name']
+
+        repack_root = tempfile.mkdtemp(suffix='.charm')
+        self._temp_dirs.append(repack_root)
+
+        self.charm_dir = os.path.join(repack_root, self.charm_name)
+
+        # Ignore .bzr to work around weird bzr interactions with
+        # juju-deployer, per Bug #1394078, and ignore .venv
+        # due to a) it containing symlinks juju will reject and b) to avoid
+        # infinite recursion.
+        shutil.copytree(src_charm_dir, self.charm_dir, symlinks=True,
+                        ignore=shutil.ignore_patterns('.venv?', '.bzr'))
+
+
+# Bug #1417097 means we need to monkey patch Amulet for now.
+real_juju = amulet.helpers.juju
+
+
+@wraps(real_juju)
+def patched_juju(args, env=None):
+    args = [str(a) for a in args]
+    return real_juju(args, env)
+
+amulet.helpers.juju = patched_juju
+amulet.deployer.juju = patched_juju

=== removed file 'testing/jujufixture.py'
--- testing/jujufixture.py	2014-10-14 10:12:02 +0000
+++ testing/jujufixture.py	1970-01-01 00:00:00 +0000
@@ -1,297 +0,0 @@
-import json
-import os.path
-import subprocess
-import time
-
-import fixtures
-from testtools.content import text_content
-import yaml
-
-
-__all__ = ['JujuFixture', 'run']
-
-
-class JujuFixture(fixtures.Fixture):
-    """Interact with juju.
-
-    Assumes juju environment is bootstrapped.
-    """
-
-    def __init__(self, series, reuse_machines=False, do_teardown=True):
-        super(JujuFixture, self).__init__()
-
-        self.series = series
-        assert series, 'series not set'
-
-        self.reuse_machines = reuse_machines
-
-        # Optionally, don't teardown services and machines after running
-        # a test. If a subsequent test is run, they will be torn down at
-        # that point. This option is only useful when running a single
-        # test, or when the test harness is set to abort after the first
-        # failed test.
-        self.do_teardown = do_teardown
-
-        self._deployed_services = set()
-
-    def do(self, cmd):
-        cmd = ['juju'] + cmd
-        run(self, cmd)
-
-    def get_result(self, cmd):
-        cmd = ['juju'] + cmd + ['--format=json']
-        out = run(self, cmd)
-        if out:
-            return json.loads(out)
-        return None
-
-    def deploy(self, charm, name=None, num_units=1, config=None):
-        cmd = ['deploy']
-
-        if not charm.startswith('cs:'):
-            charm = self.charm_uri(charm)
-
-        if config:
-            config_path = os.path.join(
-                self.useFixture(fixtures.TempDir()).path, 'config.yaml')
-            cmd.append('--config={}'.format(config_path))
-            config = yaml.safe_dump({name: config}, default_flow_style=False)
-            open(config_path, 'w').write(config)
-            self.addDetail('pgconfig', text_content(config))
-
-        cmd.append(charm)
-
-        if name is None:
-            name = charm.split(':', 1)[-1]
-
-        cmd.append(name)
-        self._deployed_services.add(name)
-
-        if self.reuse_machines and self._free_machines:
-            cmd.extend(['--to', str(self._free_machines.pop())])
-            self.do(cmd)
-            if num_units > 1:
-                self.add_unit(name, num_units - 1)
-        else:
-            cmd.extend(['-n', str(num_units)])
-            self.do(cmd)
-
-    def add_unit(self, name, num_units=1):
-        num_units_spawned = 0
-        while self.reuse_machines and self._free_machines:
-            cmd = ['add-unit', '--to', str(self._free_machines.pop()), name]
-            self.do(cmd)
-            num_units_spawned += 1
-            if num_units_spawned == num_units:
-                return
-
-        cmd = ['add-unit', '-n', str(num_units - num_units_spawned), name]
-        self.do(cmd)
-
-    # The most recent environment status, updated by refresh_status()
-    status = None
-
-    def refresh_status(self):
-        self.status = self.get_result(['status'])
-
-        self._free_machines = set(
-            int(k) for k, m in self.status['machines'].items()
-            if k != '0'
-            and m.get('life', None) not in ('dead', 'dying')
-            and m.get('series', None) == self.series
-            and m.get('agent-state', 'pending') in ('started', 'ready'))
-        for service in self.status.get('services', {}).values():
-            for unit in service.get('units', {}).values():
-                if 'machine' in unit:
-                    self._free_machines.discard(int(unit['machine']))
-
-        return self.status
-
-    def relation_info(self, unit):
-        '''Return all the relation information accessible from a unit.
-
-        relation_info('foo/0')[relation_name][relation_id][unit][key]
-        '''
-        # Get the possible relation names heuristically, per Bug #1298819
-        relation_names = []
-        for service_name, service_info in self.status['services'].items():
-            if service_name == unit.split('/')[0]:
-                relation_names = service_info.get('relations', {}).keys()
-                break
-
-        res = {}
-        juju_run_cmd = ['juju', 'run', '--unit', unit]
-        for rel_name in relation_names:
-            try:
-                relation_ids = run(
-                    self, juju_run_cmd + [
-                        'relation-ids {}'.format(rel_name)]).split()
-            except subprocess.CalledProcessError:
-                # Per Bug #1298819, we can't ask the unit which relation
-                # names are active so we need to use the relation names
-                # reported by 'juju status'. This may cause us to
-                # request relation information that the unit is not yet
-                # aware of.
-                continue
-            res[rel_name] = {}
-            for rel_id in relation_ids:
-                res[rel_name][rel_id] = {}
-                relation_units = [unit] + run(
-                    self, juju_run_cmd + [
-                        'relation-list -r {}'.format(rel_id)]).split()
-                for rel_unit in relation_units:
-                    try:
-                        json_rel_info = run(
-                            self, juju_run_cmd + [
-                                'relation-get --format=json -r {} - {}'.format(
-                                    rel_id, rel_unit)])
-                        res[rel_name][rel_id][rel_unit] = json.loads(
-                            json_rel_info)
-                    except subprocess.CalledProcessError:
-                        res[rel_name][rel_id][rel_unit] = None
-        return res
-
-    def wait_until_ready(self, extra=60):
-        ready = False
-        while not ready:
-            self.refresh_status()
-            ready = True
-            for service in self.status['services']:
-                if self.status['services'][service].get('life', '') == 'dying':
-                    ready = False
-                units = self.status['services'][service].get('units', {})
-                for unit in units.keys():
-                    agent_state = units[unit].get('agent-state', '')
-                    if agent_state == 'error':
-                        raise RuntimeError('{} error: {}'.format(
-                            unit, units[unit].get('agent-state-info', '')))
-                    if agent_state != 'started':
-                        ready = False
-            time.sleep(1)
-        # Unfortunately, there is no way to tell when a system is
-        # actually ready for us to test. Juju only tells us that a
-        # relation has started being setup, and that no errors have been
-        # encountered yet. It utterly fails to inform us when the
-        # cascade of hooks this triggers has finished and the
-        # environment is in a stable and actually testable state.
-        # So as a work around for Bug #1200267, we need to sleep long
-        # enough that our system is probably stable. This means we have
-        # extremely slow and flaky tests, but that is possibly better
-        # than no tests.
-        time.sleep(extra)
-
-    def setUp(self):
-        super(JujuFixture, self).setUp()
-        self.reset()
-        if self.do_teardown:
-            self.addCleanup(self.reset)
-
-        # Setup a temporary repository with the magic to find our charms.
-        self.repo_dir = self.useFixture(fixtures.TempDir()).path
-        self.useFixture(fixtures.EnvironmentVariable('JUJU_REPOSITORY',
-                                                     self.repo_dir))
-        self.repo_series_dir = os.path.join(self.repo_dir, self.series)
-        os.mkdir(self.repo_series_dir)
-
-    def charm_uri(self, charm):
-        meta = yaml.safe_load(open(os.path.join(charm, 'metadata.yaml'), 'rb'))
-        name = meta.get('name')
-        assert name, 'charm {} has no name in metadata.yaml'.format(charm)
-        charm_link = os.path.join(self.repo_series_dir, name)
-
-        # Recreate the charm link, which might have changed from the
-        # last deploy.
-        if os.path.exists(charm_link):
-            os.remove(charm_link)
-        os.symlink(charm, os.path.join(self.repo_series_dir, name))
-
-        return 'local:{}/{}'.format(self.series, name)
-
-    def reset(self):
-        # Tear down any services left running that we know we spawned.
-        while True:
-            found_services = False
-            self.refresh_status()
-
-            # Kill any services started by the deploy() method.
-            for service_name, service in self.status.get(
-                    'services', {}).items():
-                if service_name in self._deployed_services:
-                    found_services = True
-                    if service.get('life', '') not in ('dying', 'dead'):
-                        self.do(['destroy-service', service_name])
-                    # If any units have failed hooks, unstick them.
-                    for unit_name, unit in service.get('units', {}).items():
-                        if unit.get('agent-state', None) == 'error':
-                            try:
-                                self.do(['resolved', unit_name])
-                            except subprocess.CalledProcessError:
-                                # More race conditions in juju. A
-                                # previous 'resolved' call make cause a
-                                # subsequent one to fail if it is still
-                                # being processed. However, we need to
-                                # keep retrying because after a
-                                # successful resolution a subsequent hook
-                                # may cause an error state.
-                                pass
-            if not found_services:
-                break
-            time.sleep(1)
-
-        self._deployed_services = set()
-
-        # We shouldn't reuse machines, as we have no guarantee they are
-        # still in a usable state, so tear them down too. Per
-        # Bug #1190492 (INVALID), in the future this will be much nicer
-        # when we can use containers for isolation and can happily reuse
-        # machines.
-        if self.reuse_machines:
-            # If we are reusing machines, wait until pending machines
-            # are ready and dying machines are dead.
-            while True:
-                for k, machine in self.status['machines'].items():
-                    if (k != 0 and machine.get('agent-state', 'pending')
-                            not in ('ready', 'started')):
-                        time.sleep(1)
-                        self.refresh_status()
-                        continue
-                break
-        else:
-            self.do(['terminate-machine'] + list(self._free_machines))
-
-
-_run_seq = 0
-
-
-def run(detail_collector, cmd, input=''):
-    global _run_seq
-    _run_seq = _run_seq + 1
-
-    # This helper primarily exists to capture the subprocess details,
-    # but this is failing. Details are being captured, but those added
-    # inside the actual test (not setup) are not being reported.
-
-    out, err, returncode = None, None, None
-    try:
-        proc = subprocess.Popen(
-            cmd, stdin=subprocess.PIPE, stdout=subprocess.PIPE,
-            stderr=subprocess.PIPE)
-        (out, err) = proc.communicate(input)
-        returncode = proc.returncode
-        if returncode != 0:
-            raise subprocess.CalledProcessError(returncode, cmd, err)
-        return out
-    except subprocess.CalledProcessError as x:
-        returncode = x.returncode
-        raise
-    finally:
-        if detail_collector is not None:
-            m = {
-                'cmd': ' '.join(cmd),
-                'rc': returncode,
-                'stdout': out,
-                'stderr': err,
-            }
-            detail_collector.addDetail(
-                'run_{}'.format(_run_seq),
-                text_content(yaml.safe_dump(m, default_flow_style=False)))

=== removed file 'tests/00-setup.sh'
--- tests/00-setup.sh	2014-10-14 11:25:24 +0000
+++ tests/00-setup.sh	1970-01-01 00:00:00 +0000
@@ -1,15 +0,0 @@
-#!/bin/sh
-
-sudo add-apt-repository -y ppa:juju/stable
-sudo apt-get update
-sudo apt-get install -y \
-    amulet \
-    python-flake8 \
-    python-fixtures \
-    python-jinja2 \
-    python-mocker \
-    python-psycopg2 \
-    python-testtools \
-    python-twisted-core \
-    python-yaml \
-    pgtune

=== removed file 'tests/01-lint.sh'
--- tests/01-lint.sh	2014-10-14 11:25:24 +0000
+++ tests/01-lint.sh	1970-01-01 00:00:00 +0000
@@ -1,3 +0,0 @@
-#!/bin/sh
-
-make -C $(dirname $0)/.. lint

=== removed file 'tests/02-unit-tests.sh'
--- tests/02-unit-tests.sh	2014-10-14 11:25:24 +0000
+++ tests/02-unit-tests.sh	1970-01-01 00:00:00 +0000
@@ -1,3 +0,0 @@
-#!/bin/bash
-
-make -C $(dirname $0)/.. unit_test

=== removed file 'tests/03-basic-amulet.py'
--- tests/03-basic-amulet.py	2014-10-14 11:25:24 +0000
+++ tests/03-basic-amulet.py	1970-01-01 00:00:00 +0000
@@ -1,19 +0,0 @@
-#!/usr/bin/python3
-
-import amulet
-import os.path
-
-d = amulet.Deployment()
-
-d.add('postgresql', os.path.join(os.path.dirname(__file__), os.pardir))
-d.expose('postgresql')
-
-try:
-    d.setup(timeout=900)
-    d.sentry.wait()
-except amulet.helpers.TimeoutError:
-    amulet.raise_status(amulet.SKIP, msg="Environment wasn't stood up in time")
-except:
-    raise
-
-amulet.raise_status(amulet.PASS)

=== added file 'tests/tests.yaml'
--- tests/tests.yaml	1970-01-01 00:00:00 +0000
+++ tests/tests.yaml	2015-03-12 09:42:09 +0000
@@ -0,0 +1,3 @@
+virtualenv: false
+makefile:
+    - test