Merge lp:~stub/charms/trusty/postgresql/rewrite into lp:charms/trusty/postgresql

[Stuart Bishop]

The PostgreSQL charm is one of the oldest around, crufty and unmaintainable. Its time for a rewrite to make use of modern Juju and improve its reliability.

[Tim Van Steenburgh (tvansteenburgh)]

Hey stub, I haven't seen the tests for this branch pass yet: http://reports.vapour.ws/all-bundle-and-charm-results/lp%3A~stub%252Fcharms%252Ftrusty%252Fpostgresql%252Frewrite

[Stuart Bishop (stub)]

They were passing on the new system being setup. I haven't been
watching the old runner.

--
Stuart Bishop <email address hidden>

[Stuart Bishop (stub)]

On 17 September 2015 at 14:49, Stuart Bishop
<email address hidden> wrote:
> They were passing on the new system being setup. I haven't been
> watching the old runner.

Looks like the green runs are lost in the mists of time. I'm not sure
what has changed in the month this has been waiting for review.

[Stuart Bishop (stub)]

Looking at http://data.vapour.ws/charm-test/charm-bundle-test-azure-651-all-machines-log,
it seems debugging output has been turned up to 11, and I think this
breaks 'juju wait'. Without 'juju wait', it is impossible to get the
tests running reliably

Its hard to tell through all the noise, but I think here is where the
initial deploy actually terminates:

unit-postgresql-0[21563]: 2015-09-17 01:01:47 INFO
unit.postgresql/0.juju-log server.go:254 db-admin:2: *** End
'db-admin-relation-changed' hook

After this, we keep seeing noise about leadership leases, and other
stuff including the juju run commands that are checking that the logs
are quiet (thus shooting itself in the foot, because the only way to
check if the logs are quiet adds noise to the logs...).

This never got picked up locally, as I don't even know how to turn on
this output and don't get it here (using juju stable).

Also see Bug #1496130, where we are fixing things so the OpenStack
mojo tests can use the wait algorithm.

What is confusing me though is that the Cassandra tests last passed
Sep 4th, and I would have expected them to have started failing much
earlier (it too relies on juju wait). What is even more confusing, the
last successful run at
http://reports.vapour.ws/charm-test-details/charm-bundle-test-parent-703
leads me to the AWS log at
http://data.vapour.ws/charm-test/charm-bundle-test-aws-591-all-machines-log
which contains a whole heap of logging information from other tests,
in addition to the Cassandra logs, which means test isolation was
busted and the results bogus.

[Stuart Bishop (stub)]

I see. Prior to ~Sep 4th, tests were often passing but would also
often fail with various provisioning errors. eg. This one is common,
indicating units took longer than 5 minutes to provision:

amulet.helpers.TimeoutError: public-address not set forall units after 300s

After to ~Sep 4th, all integration tests started failing consistently
at 'juju wait', for both Cassandra and the PostgreSQL rewrite.

--
Stuart Bishop <email address hidden>

[Stuart Bishop (stub)]

I added Juju 1.24 specific behavior to the juju wait plugin, which avoids needing to sniff the logs. I've updated the package in ppa:stub/juju and I think I've queued a retest.

[Tim Van Steenburgh (tvansteenburgh)]

> They were passing on the new system being setup. I haven't been
> watching the old runner.
>
Sorry, I'm not familiar, where is the new system (or output from it)?

[Tim Van Steenburgh (tvansteenburgh)]

> Looking at http://data.vapour.ws/charm-test/charm-bundle-test-azure-651-all-
> machines-log,
> it seems debugging output has been turned up to 11, and I think this
> breaks 'juju wait'. Without 'juju wait', it is impossible to get the
> tests running reliably

I'm not sure why this would be, unless the default log level in juju itself got turned up. CI is currently running 1.24.5.1

> What is confusing me though is that the Cassandra tests last passed
> Sep 4th, and I would have expected them to have started failing much
> earlier (it too relies on juju wait). What is even more confusing, the
> last successful run at
> http://reports.vapour.ws/charm-test-details/charm-bundle-test-parent-703
> leads me to the AWS log at
> http://data.vapour.ws/charm-test/charm-bundle-test-aws-591-all-machines-log
> which contains a whole heap of logging information from other tests,
> in addition to the Cassandra logs, which means test isolation was
> busted and the results bogus.

Wow, that is bizarre. Thanks for pointing this out, I had not seen this before. I'm not convinced that test isolation is busted, but log isolation may be. I'll look into this.

[Stuart Bishop (stub)]

stub@aargh:~/charms/postgresql/rewrite$ make test
sudo add-apt-repository -y ppa:juju/stable
gpg: keyring `/tmp/tmppv9t1ria/secring.gpg' created
gpg: keyring `/tmp/tmppv9t1ria/pubring.gpg' created
gpg: requesting key C8068B11 from hkp server keyserver.ubuntu.com
gpg: /tmp/tmppv9t1ria/trustdb.gpg: trustdb created
gpg: key C8068B11: public key "Launchpad Ensemble PPA" imported
gpg: Total number processed: 1
gpg: imported: 1 (RSA: 1)
OK
sudo add-apt-repository -y ppa:stub/juju
gpg: keyring `/tmp/tmpfkk5hwu_/secring.gpg' created
gpg: keyring `/tmp/tmpfkk5hwu_/pubring.gpg' created
gpg: requesting key E4FD7A7A from hkp server keyserver.ubuntu.com
gpg: /tmp/tmpfkk5hwu_/trustdb.gpg: trustdb created
gpg: key E4FD7A7A: public key "Launchpad Stub's Launchpad PPA" imported
gpg: Total number processed: 1
gpg: imported: 1 (RSA: 1)
OK
[...]
Reading package lists... Done
sudo apt-get install -y \
    python3-psycopg2 python3-nose python3-flake8 amulet \
    python3-jinja2 python3-yaml juju-wait bzr \
    python3-nose-cov python3-nose-timer python-swiftclient
Reading package lists... Done
Building dependency tree
Reading state information... Done
bzr is already the newest version.
python-swiftclient is already the newest version.
python3-flake8 is already the newest version.
python3-jinja2 is already the newest version.
python3-nose is already the newest version.
python3-psycopg2 is already the newest version.
python3-yaml is already the newest version.
python3-nose-cov is already the newest version.
python3-nose-timer is already the newest version.
amulet is already the newest version.
juju-wait is already the newest version.
0 to upgrade, 0 to newly install, 0 to remove and 0 not to upgrade.
Charm Proof
I: metadata name (postgresql) must match directory name (rewrite) exactly for local deployment.
Lint check (flake8)
user configuration: /home/stub/.config/flake8
directory hooks
checking hooks/bootstrap.py
checking hooks/client.py
checking hooks/coordinator.py
checking hooks/decorators.py
checking hooks/definitions.py
checking hooks/helpers.py
checking hooks/metrics.py
checking hooks/nagios.py
checking hooks/postgresql.py
checking hooks/replication.py
checking hooks/service.py
checking hooks/storage.py
checking hooks/syslogrel.py
checking hooks/upgrade.py
checking hooks/wal_e.py
directory actions
checking actions/actions.py
directory testing
checking testing/__init__.py
checking testing/amuletfixture.py
directory tests
checking tests/obsolete.py
checking tests/test_integration.py
checking tests/test_postgresql.py
nosetests3 -sv tests/test_postgresql.py --cover-package=bootstrap,nagios,wal_e,syslogrel,replication,definitions,storage,decorators,metrics,upgrade,postgresql,helpers,coordinator,client,service \
    --with-coverage --cover-branches
test_addr_to_range (test_postgresql.TestPostgresql) ... ok
test_connect (test_postgresql.TestPostgresql) ... ok
test_convert_unit (test_postgresql.TestPostgresql) ... ok
test_create_cluster (test_postgresql.TestPostgresql) ... ok
test_drop_cluster (test_postgresql.TestPostgresql) ... ok
test_ensure_database (test_postgresql.TestPostgresql) ... ok
test_ensure_extensions (test_...

[Tim Van Steenburgh (tvansteenburgh)]

@stub, which cloud was that on?

I still have no idea why it literally never passes on CI. Usually takes forever to run too, and often hangs and needs to be killed. I haven't had time to dig deeper yet but I'm hoping to soon.

[Stuart Bishop (stub)]

@tim It was just a different report, so these clouds. I think I was getting confused with the Cassandra runs which I was doing around the same time.

Previously I had been getting tests to pass, but not all the tests. There were always a few that would fail due to provisioning type errors, the amulet error failing to list the contents of a remote directory, or a juju-deployer race.

Then around Sept 4th, something changed on your clouds and the juju logging changed. This broke 'juju wait', and thus all the PostgreSQL tests, and the Cassandra tests to boot. I've since been reworking the t plugin to be less fragile with Juju 1.23 and earlier, and use 'agent-status' with Juju 1.24 and later. This has got the Cassandra tests green again (and the lp:~stub/charms/trusty/cassandra/spike MP green too).

[Stuart Bishop (stub)]

This round is looking good. Tests at http://juju-ci.vapour.ws:8080/job/charm-bundle-test-joyent/729/console running smooth until the env.lock bug kicked in.

[Tim Van Steenburgh (tvansteenburgh)]

I just emailed the juju list about the env.lock problem. If my multiple $JUJU_HOME idea works, it'll be a quick fix. That particular problem is ruining a lot of otherwise green test runs across many charms and bundles.

[Stuart Bishop (stub)]

The remaining failure seems genuine (PG93MultiTests.test_failover, which victimizes test_replication). Given 3 nodes, dropping the master and adding a new node at the same time can cause a situation where there is no master. I have not been able to reproduce this locally. I need to trawl the logs from the CI system and try to reproduce manually on a cloud.

[lp:~stub/charms/trusty/postgresql/rewrite]

248. By Stuart Bishop on 2015-10-26

Work around Bug #1510000

[Stuart Bishop (stub)]

FWIW, this is still waiting for review so I can land it. Despite the failure of one of the tests on the CI system, this branch is still preferable to the existing known broken charm store charm which has its tests disabled.

[lp:~stub/charms/trusty/postgresql/rewrite]

249. By Stuart Bishop on 2015-11-02

Add timings

250. By Stuart Bishop on 2015-11-02

Dependency for timestamps

251. By Stuart Bishop on 2015-11-02

Skip failover tests until Bug #1511659 can be dealt with

252. By Stuart Bishop on 2015-11-02

Update tests for Juju 1.25 unit naming changes

[Matt Bruzek (mbruzek)]

Hello Stuart,

Thanks for the enormous amount of work on this charm! This merge is a huge refactor and a lot of files changed. From what I saw of the code and the extensive tests it looks great. I see the automated tests are passing on Joyent, and Power 8! I also tested this on the KVM local provider and the results were PASS: 13 Total: 13 (7392.816871 sec).

Charm proof now returns no errors or warnings which was not the case with the previous branch. Thanks for working with Tim and the team to get the automated tests passing. The tests are complex and very thorough. The automated tests passing are very important and will help this charm be of the highest quality in the future.

+1 LGTM

review: Approve

=== modified file '.bzrignore'
--- .bzrignore	2014-10-14 10:12:50 +0000
+++ .bzrignore	2015-11-02 12:15:35 +0000
@@ -1,4 +1,6 @@
 _trial_temp
 hooks/_trial_temp
 hooks/local_state.pickle
-lib/test-client-charm/hooks/charmhelpers
+lib/pgclient/hooks/charmhelpers/
+coverage
+.coverage

=== modified file 'Makefile'
--- Makefile	2015-06-25 08:13:25 +0000
+++ Makefile	2015-11-02 12:15:35 +0000
@@ -1,63 +1,72 @@
 CHARM_DIR := $(shell pwd)
 TEST_TIMEOUT := 900
-SERIES := $(juju get-environment default-series)
+SERIES := $(shell juju get-environment default-series)
+HOST_SERIES := $(shell lsb_release -sc)
 
 default:
 	@echo "One of:"
-	@echo "    make testdep"
+	@echo "    make testdeps"
 	@echo "    make lint"
-	@echo "    make unit_test"
-	@echo "    make integration_test"
-	@echo "    make integration_test_91"
-	@echo "    make integration_test_92"
-	@echo "    make integration_test_93"
-	@echo "    make integration_test_94"
-	@echo
-	@echo "There is no 'make test'"
-
-test_bot_tests:
-	@echo "Installing dependencies and running automatic-testrunner tests"
-	tests/00-setup.sh
-	tests/01-lint.sh
-	tests/02-unit-tests.sh
-	tests/03-basic-amulet.py
-
-testdep:
-	tests/00-setup.sh
-
-unit_test:
-	@echo "Unit tests of hooks"
-	cd hooks && trial test_hooks.py
-
-integration_test:
-	@echo "PostgreSQL integration tests, all non-beta versions, ${SERIES}"
-	trial test.PG91Tests
-	trial test.PG92Tests
-	trial test.PG93Tests
-	trial test.PG94Tests
-
-integration_test_91:
-	@echo "PostgreSQL 9.1 integration tests, ${SERIES}"
-	trial test.PG91Tests
-
-integration_test_92:
-	@echo "PostgreSQL 9.2 integration tests, ${SERIES}"
-	trial test.PG92Tests
-
-integration_test_93:
-	@echo "PostgreSQL 9.3 integration tests, ${SERIES}"
-	trial test.PG93Tests
-
-integration_test_94:
-	@echo "PostgreSQL 9.4 integration tests, ${SERIES}"
-	trial test.PG94Tests
+	@echo "    make unittest"
+	@echo "    make integration"
+	@echo "    make coverage (opens browser)"
+
+test: testdeps lint unittest integration
+
+testdeps:
+	sudo add-apt-repository -y ppa:juju/stable
+	sudo add-apt-repository -y ppa:stub/juju
+	sudo apt-get update
+ifeq ($(HOST_SERIES),trusty)
+	sudo apt-get install -y \
+	    python3-psycopg2 python3-nose python3-flake8 amulet \
+	    python3-jinja2 python3-yaml juju-wait bzr python3-amulet \
+	    python-swiftclient moreutils
+else
+	sudo apt-get install -y \
+	    python3-psycopg2 python3-nose python3-flake8 amulet \
+	    python3-jinja2 python3-yaml juju-wait bzr python3-amulet \
+	    python3-nose-cov python3-nose-timer python-swiftclient moreutils
+endif
 
 lint:
+	@echo "Charm Proof"
+	@charm proof
 	@echo "Lint check (flake8)"
 	@flake8 -v \
-	    --exclude hooks/charmhelpers,hooks/_trial_temp \
             --ignore=E402 \
-	    hooks actions testing tests test.py
+	    --exclude=hooks/charmhelpers,__pycache__ \
+	    hooks actions testing tests
+
+_co=,
+_empty=
+_sp=$(_empty) $(_empty)
+
+TESTFILES=$(filter-out %/test_integration.py,$(wildcard tests/test_*.py))
+PACKAGES=$(subst $(_sp),$(_co),$(notdir $(basename $(wildcard hooks/*.py))))
+
+NOSE := nosetests3 -sv
+ifeq ($(HOST_SERIES),trusty)
+TIMING_NOSE := nosetests3 -sv
+else
+TIMING_NOSE := nosetests3 -sv --with-timer
+endif
+
+unittest:
+	${NOSE} ${TESTFILES} --cover-package=${PACKAGES} \
+	    --with-coverage --cover-branches
+	@echo OK: Unit tests pass `date`
+
+coverage:
+	${NOSE} ${TESTFILES} --cover-package=${PACKAGES} \
+	    --with-coverage --cover-branches \
+	    --cover-erase --cover-html --cover-html-dir=coverage \
+	    --cover-min-percentage=100 || \
+		(gnome-open coverage/index.html; false)
+
+integration:
+	${TIMING_NOSE} tests/test_integration.py 2>&1 | ts
+	@echo OK: Integration tests pass `date`
 
 sync:
 	@bzr cat \
@@ -65,3 +74,9 @@
 		> .charm_helpers_sync.py
 	@python .charm_helpers_sync.py -c charm-helpers.yaml
 	@rm .charm_helpers_sync.py
+
+
+# These targets are to separate the test output in the Charm CI system
+test_integration.py%:
+	${TIMING_NOSE} tests/$@ 2>&1 | ts
+	@echo OK: $@ tests pass `date`

=== modified file 'README.md'
--- README.md	2015-06-22 18:40:20 +0000
+++ README.md	2015-11-02 12:15:35 +0000
@@ -26,14 +26,8 @@
 
 # Usage
 
-This charm supports several deployment models:
-
- - A single service containing one unit. This provides a 'standalone'
-   environment.
-
- - A service containing multiple units. One unit will be a 'master', and every
-   other unit is a 'hot standby'. The charm sets up and maintains replication
-for you, using standard PostgreSQL streaming replication.
+This charm can deploy a single standalone PostgreSQL unit, or a service
+containing a single master unit and one or more replicas.
 
 To setup a single 'standalone' service::
 
@@ -42,21 +36,18 @@
 
 ## Scale Out Usage
 
-To replicate this 'standalone' database to a 'hot standby', turning the
-existing unit into a 'master'::
+To add a replica to an existing service::
 
     juju add-unit pg-a
 
-To deploy a new service containing a 'master' and two 'hot standbys'::
+To deploy a new service containing a master and two hot standby replicas::
 
-    juju deploy -n 2 postgresql pg-b
-    [ ... wait until units are stable ... ]
-    juju add-unit pg-b
+    juju deploy -n 3 postgresql pg-b
 
 You can remove units as normal. If the master unit is removed, failover occurs
-and the most up to date 'hot standby' is promoted to 'master'.  The
-'db-relation-changed' and 'db-admin-relation-changed' hooks are fired, letting
-clients adjust::
+and the most up to date hot standby is promoted to the master.  The
+'db-relation-changed' and 'db-admin-relation-changed' hooks are fired,
+letting clients adjust::
 
     juju remove-unit pg-b/0
 
@@ -74,12 +65,6 @@
 
 ## Known Limitations and Issues
 
-⚠ Due to current [limitations][1] with juju, you cannot reliably
-create a service initially containing more than 2 units (eg. juju deploy
--n 3 postgresql). Instead, you must first create a service with 2 units.
-Once the environment is stable and all the hooks have finished running,
-you may add more units.
-
 ⚠ Do not attempt to relate client charms to a PostgreSQL service containing
   multiple units unless you know the charm supports a replicated service.
 
@@ -106,20 +91,23 @@
   practice with your database permissions will make your life difficult
   when you need to recover after failure.
 
-_Always_ set the 'roles' relationship setting when joining a
-relationship. _Always_ grant permissions to database roles for _all_
-database objects your charm creates. _Never_ rely on access permissions
-given directly to a user, either explicitly or implicitly (such as being
-the user who created a table). Consider the users you are provided by
-the PostgreSQL charm as ephemeral. Any rights granted directly to them
-will be lost if relations are recreated, as the generated usernames will
-be different. _If you don't follow this advice, you will need to
-manually repair permissions on all your database objects after any of
-the available recovery mechanisms._
+PostgreSQL has comprehensive database security, including ownership
+and permissions on database objects. By default, any objects a client
+service creates will be owned by a user with the same name as the
+client service and inaccessible to other users. To share data, it
+is best to create new roles, grant the relevant permissions and object
+ownership to the new roles and finally grant these roles to the users
+your services can connect as. This also makes disaster recovery easier.
+If you restore a database into an indentical Juju environment, then
+the service names and usernames will be the same and database permissions
+will match. However, if you restore a database into an environment
+with different client service names then the usernames will not match
+and the new users not have access to your data.
 
 Learn about the SQL `GRANT` statement in the excellect [PostgreSQL
 reference guide][3].
 
+
 ### block-storage-broker
 
 If you are using external storage provided by the block storage broker,
@@ -163,10 +151,9 @@
   will create it if necessary. If your charm sets this, then it must wait
   until a matching `database` value is presented on the PostgreSQL side of
   the relation (ie. `relation-get database` returns the value you set).
-- `roles`: Optional. A comma separated list of database roles to grant the
+- `roles`: Deprecated. A comma separated list of database roles to grant the
   database user. Typically these roles will have been granted permissions to
-  access the tables and other database objects.  Do not grant permissions
-  directly to juju generated database users, as the charm may revoke them.
+  access the tables and other database objects.
 - `extensions`: Optional. A comma separated list of required postgresql
   extensions.
 
@@ -210,28 +197,26 @@
 A PostgreSQL service may contain multiple units (a single master, and
 optionally one or more hot standbys). The client charm can tell which
 unit in a relation is the master and which are hot standbys by
-inspecting the 'state' property on the relation, and it needs to be
-aware of how many units are in the relation by using the 'relation-list'
-hook tool.
-
-If there is a single PostgreSQL unit related, the state will be
-'standalone'. All database connections of course go to this unit.
-
-If there is more than one PostgreSQL unit related, the client charm
-must only use units with state set to 'master' or 'hot standby'.
-The unit with 'master' state can accept read and write connections. The
-units with 'hot standby' state can accept read-only connections, and
-any attempted writes will fail. Units with any other state must not be
-used and should be ignored ('standalone' units are new units joining the
-service that are not yet setup, and 'failover' state will occur when the
-master unit is being shutdown and a new master is being elected).
+inspecting the 'state' property on the relation.
+
+The 'standalone' state is deprecated, and when a unit advertises itself
+as 'standalone' you should treat it like a 'master'. The state only exists
+for backwards compatibility and will be removed soon.
+
+Writes must go to the unit identifying itself as 'master' or 'standalone'.
+If you sent writes to a 'hot standby', they will fail.
+
+Reads may go to any unit. Ideally they should be load balanced over
+the 'hot standby' units. If you need to ensure consistency, you may
+need to read from the 'master'.
+
+Units in any other state, including no state, should not be used and
+connections to them will likely fail. These units may still be setting
+up, or performing a maintenance operation such as a failover.
 
 The client charm needs to watch for state changes in its
-relation-changed hook. New units may be added to a single unit service,
-and the client charm must stop using existing 'standalone' unit and wait
-for 'master' and 'hot standby' units to appear. Units may be removed,
-possibly causing a 'hot standby' unit to be promoted to a master, or
-even having the service revert to a single 'standalone' unit.
+relation-changed hook. During failover, one of the existing 'hot standby' 
+units will change into a 'master'.
 
 
 ## Example client hooks
@@ -249,7 +234,6 @@
     @hook
     def db_relation_joined():
         relation_set('database', config('database'))  # Explicit database name
-        relation_set('roles', 'reporting,standard')  # DB roles required
 
     @hook('db-relation-changed', 'db-relation-departed')
     def db_relation_changed():
@@ -271,9 +255,7 @@
             conn_str = conn_str_tmpl.format(**relation_get(unit=db_unit)
             remote_state = relation_get('state', db_unit)
 
-            if remote_state == 'standalone' and len(active_db_units) == 1:
-                master_conn_str = conn_str
-            elif relation_state == 'master':
+            if remote_state in ('master', 'standalone'):
                 master_conn_str = conn_str
             elif relation_state == 'hot standby':
                 slave_conn_strs.append(conn_str)
@@ -284,46 +266,17 @@
         hooks.execute(sys.argv)
 
 
-## Upgrade-charm hook notes
-
-The PostgreSQL charm has deprecated volume-map and volume-ephemeral-storage
-configuration options in favor of using the storage subordinate charm for
-general external storage management. If the installation being upgraded is
-using these deprecated options, there are a couple of manual steps necessary 
-to finish migration and continue using the current external volumes.
-Even though all data will remain intact, and PostgreSQL service will continue
-running, the upgrade-charm hook will intentionally fail and exit 1 as well to
-raise awareness of the manual procedure which will also be documented in the
-juju logs on the PostgreSQL units.
-
-The following steps must be additionally performed to continue using external
-volume maps for the PostgreSQL units once juju upgrade-charm is run from the
-command line:
-  1. cat > storage.cfg <<EOF
-     storage:
-       provider:block-storage-broker
-       root: /srv/data
-       volume_map: "{postgresql/0: your-vol-id, postgresql/1: your-2nd-vol-id}"
-     EOF
-  2. juju deploy --config storage.cfg storage
-  3. juju deploy block-storage-broker
-  4. juju add-relation block-storage-broker storage
-  5. juju resolved --retry postgresql/0   # for each postgresql unit running
-  6. juju add-relation postgresql storage
-
-
 # Point In Time Recovery
 
-The PostgreSQL charm has experimental support for log shipping and point
-in time recovery. This feature uses the wal-e[2] tool, and requires access
-to Amazon S3, Microsoft Azure Block Storage or Swift. This feature is
-flagged as experimental because it has only been tested with Swift, and
-not yet been tested under load. It also may require some API changes,
-particularly on how authentication credentials are accessed when a standard
-emerges. The charm can be configured to perform regular filesystem backups
-and ship WAL files to the object store. Hot standbys will make use of the
-archived WAL files, allowing them to resync after extended netsplits or
-even let you turn off streaming replication entirely.
+The PostgreSQL charm has support for log shipping and point in time
+recovery. This feature uses the wal-e[2] tool, which will be
+installed from the Launchpad PPA ppa:stub/pgcharm. This feature
+requires access to either Amazon S3, Microsoft Azure Block Storage or
+Swift. This feature is experimental because it has only been tested with
+Swift. The charm can be configured to perform regular filesystem backups
+and ship WAL files to the object store. Hot standbys will make use of
+the archived WAL files, allowing them to resync after extended netsplits
+or even let you turn off streaming replication entirely.
 
 With a base backup and the WAL archive you can perform point in time
 recovery, but this is still a manual process and the charm does not
@@ -336,8 +289,7 @@
 service.
 
 To enable the experimental wal-e support with Swift, you will need to
-use Ubuntu 14.04 (Trusty), and set the service configuration settings
-similar to the following::
+and set the service configuration settings similar to the following::
 
     postgresql:
         wal_e_storage_uri: swift://mycontainer

=== removed file 'TODO'
--- TODO	2013-02-20 13:42:08 +0000
+++ TODO	1970-01-01 00:00:00 +0000
@@ -1,27 +0,0 @@
-- Fix "run" function in hooks.py to log error rather than exiting silently
-  (exceptions emitted at INFO level)
-
-- Pick a better way to get machine specs than hacky bash functions - facter?
-
-- Specify usernames when adding a relation, rather than generating them
-  automagically. This will interact better with connection poolers.
-
-- If we have three services deployed, related via master/slave, then the
-  repmgr command line tools only work on the service containing the master
-  unit, as the slave-only services have neither ssh nor postgresql access
-  to each other. This may only be a problem with master/slave relationships
-  between services and failover, and it is unclear how that would work
-  anyway (get failover working fist for units within a service).
-
-- Drop config_change_command from config.yaml, or make it work.
-  "restart" should be the default, as there may be required config changes
-  to get replication working. "reload" would mean blocking a hook requesting
-  config changes requireing a restart, emitting warnings so the admin can
-  do the restart manually. Is this good juju? An admin can already control
-  when restarts happen by when they change the juju configuration, and can
-  avoid unnecessary restarts when adding replicas by ensuring required
-  configuration changes are made before hand.
-
-- No hook is invoked when removing a unit, leaving the host standby PostgreSQL
-  cluster running and attempting to replicate from a master that is refusing
-  its connections. Bug #872264.

=== modified file 'actions.yaml'
--- actions.yaml	2015-06-25 11:35:33 +0000
+++ actions.yaml	2015-11-02 12:15:35 +0000
@@ -2,3 +2,14 @@
     description: Pause replication replay on a hot standby unit.
 replication-resume:
     description: Resume replication replay on a hot standby unit.
+        
+# Revisit this when actions are more mature. Per Bug #1483525, it seems
+# impossible to return filenames in our results.
+# backup:
+#     description: Run backups
+#     params:
+#         type:
+#             type: string
+#             enum: [dump]
+#             description: Type of backup. Currently only 'dump' supported.
+#             default: dump

=== modified file 'actions/actions.py'
--- actions/actions.py	2015-06-25 11:35:33 +0000
+++ actions/actions.py	2015-11-02 12:15:35 +0000
@@ -25,17 +25,21 @@
     sys.path.append(hooks_dir)
 
 from charmhelpers.core import hookenv
-import hooks
+
+import postgresql
 
 
 def replication_pause(params):
-    offset = hooks.postgresql_wal_received_offset()
-    if offset is None:
+    if not postgresql.is_secondary():
         hookenv.action_fail('Not a hot standby')
         return
+
+    offset = postgresql.wal_received_offset()
     hookenv.action_set(dict(offset=offset))
 
-    cur = hooks.db_cursor(autocommit=True)
+    con = postgresql.connect()
+    con.autocommit = True
+    cur = con.cursor()
     cur.execute('SELECT pg_is_xlog_replay_paused()')
     if cur.fetchone()[0] is True:
         hookenv.action_fail('Already paused')
@@ -45,13 +49,16 @@
 
 
 def replication_resume(params):
-    offset = hooks.postgresql_wal_received_offset()
-    if offset is None:
+    if not postgresql.is_secondary():
         hookenv.action_fail('Not a hot standby')
         return
+
+    offset = postgresql.wal_received_offset()
     hookenv.action_set(dict(offset=offset))
 
-    cur = hooks.db_cursor(autocommit=True)
+    con = postgresql.connect()
+    con.autocommit = True
+    cur = con.cursor()
     cur.execute('SELECT pg_is_xlog_replay_paused()')
     if cur.fetchone()[0] is False:
         hookenv.action_fail('Already resumed')
@@ -60,6 +67,34 @@
     hookenv.action_set(dict(result='Resumed'))
 
 
+# Revisit this when actions are more mature. Per Bug #1483525, it seems
+# impossible to return filenames in our results.
+#
+# def backup(params):
+#     assert params['type'] == 'dump'
+#     script = os.path.join(helpers.scripts_dir(), 'pg_backup_job')
+#     cmd = ['sudo', '-u', 'postgres', '-H', script, str(params['retention'])]
+#     hookenv.action_set(dict(command=' '.join(cmd)))
+#
+#     try:
+#         subprocess.check_call(cmd)
+#     except subprocess.CalledProcessError as x:
+#         hookenv.action_fail(str(x))
+#         return
+#
+#     backups = {}
+#     for filename in os.listdir(backups_dir):
+#         path = os.path.join(backups_dir, filename)
+#         if not is.path.isfile(path):
+#             continue
+#         backups['{}.{}'.format(filename
+#         backups[filename] = dict(name=filename,
+#                                  size=os.path.getsize(path),
+#                                  path=path,
+#                                  scp_path='{}:{}'.format(unit, path))
+#     hookenv.action_set(dict(backups=backups))
+
+
 def main(argv):
     action = os.path.basename(argv[0])
     params = hookenv.action_get()

=== modified file 'config.yaml'
--- config.yaml	2015-07-10 17:05:44 +0000
+++ config.yaml	2015-11-02 12:15:35 +0000
@@ -3,12 +3,12 @@
     default: ""
     type: string
     description: >
-      A comma-separated list of IP Addresses (or single IP) admin tools like 
-      pgAdmin3 will connect from, this is most useful for developers running 
-      juju in local mode who need to connect tools like pgAdmin to a postgres. 
-      The IP addresses added here will be included in the pg_hba.conf file 
-      allowing ip connections to all databases on the server from the given 
-      using md5 password encryption.
+      A comma-separated list of IP Addresses (or single IP) admin tools
+      like pgAdmin3 will connect from. The IP addresses added here will
+      be included in the pg_hba.conf file allowing ip connections to all
+      databases on the server from the given IP addresses using md5
+      password encryption. IP address ranges are also supported, using
+      the standard format described in the PostgreSQL reference guide.
   locale:
     default: "C"
     type: string
@@ -22,53 +22,294 @@
     description: >
       Default encoding used to store text in this service. Can only be
       set when deploying the first unit of a service.
-  extra-packages:
+  relation_database_privileges:
+    default: "ALL"
+    type: string
+    description: >
+      A comma-separated list of database privileges to grant to relation
+      users on their databases. The defaults allow to connect to the
+      database (CONNECT), create objects such as tables (CREATE), and
+      create temporary tables (TEMPORARY). Client charms that create
+      objects in the database are responsible to granting suitable
+      access on those objects to other roles and users (or PUBLIC) using
+      standard GRANT statements.
+  extra_packages:
     default: ""
     type: string
-    description: Extra packages to install on the postgresql service units.
+    description: >
+        Space separated list of extra packages to install.
   dumpfile_location:
     default: "None"
     type: string
     description: >
         Path to a dumpfile to load into DB when service is initiated.
   version:
-    default: null
+    default: ""
     type: string
     description: >
         Version of PostgreSQL that we want to install. Supported versions
-        are "9.1", "9.2", "9.3". The default version for the deployed Ubuntu
-        release is used when the version is not specified.
-  cluster_name:
-    default: "main"
-    type: string
-    description: Name of the cluster we want to install the DBs into
-  listen_ip:
-    default: "*"
-    type: string
-    description: IP to listen on
+        are "9.1", "9.2", "9.3" & "9.4". The default version for the
+        deployed Ubuntu release is used when the version is not specified.
+  extra_pg_conf:
+      # The defaults here match the defaults chosen by the charm,
+      # so removing them will not change them. They are listed
+      # as documentation. The charm actually loads the non-calculated
+      # defaults from this config.yaml file to make it unlikely it will
+      # get out of sync with reality.
+      default: |
+        # Additional service specific postgresql.conf settings.
+        listen_addresses='*'
+        max_connections=100
+        ssl=true
+        log_timezone=UTC
+        log_checkpoints=true
+        log_connections=true
+        log_disconnections=true
+        log_autovacuum_min_duration=-1
+        log_line_prefix='%t [%p]: [%l-1] db=%d,user=%u '
+        archive_mode=true
+        archive_command='/bin/true'
+        hot_standby=true
+        max_wal_senders=80
+        # max_wal_senders=num_units * 2 + 5
+        # wal_level=hot_standby (<9.4) or logical (>=9.4)
+        # shared_buffers=total_ram*0.25
+        # effective_cache_size=total_ram*0.75
+        default_statistics_target=250
+        from_collapse_limit=16
+        join_collapse_limit=16
+        wal_buffers=-1
+        checkpoint_completion_target=0.9
+        password_encryption=true
+        max_connections=100
+      type: string
+      description: >
+          postgresql.conf settings, one per line in standard key=value
+          PostgreSQL format. These settings will generally override
+          any values selected by the charm. The charm however will
+          attempt to ensure minimum requirements for the charm's
+          operation are met.
+  extra_pg_auth:
+    type: string
+    default: ""
+    description: >
+        A comma separated extra pg_hba.conf auth rules.
+        This will be written to the pg_hba.conf file, one line per rule.
+        Note that this should not be needed as db relations already create
+        those rules the right way. Only use this if you really need too
+        (e.g. on a development environment), or are connecting juju managed
+        databases to external managed systems, or configuring replication
+        between unrelated PostgreSQL services using the manual_replication
+        option.
+  performance_tuning:
+    default: "Mixed"
+    type: string
+    description: >
+        DEPRECATED AND IGNORED. The pgtune project has been abandoned
+        and the packages dropped from Debian and Ubuntu. The charm
+        still performs some basic tuning, which users can tweak using
+        extra_pg_config.
+  manual_replication:
+    type: boolean
+    default: False
+    description: >
+        Enable or disable charm managed replication. When manual_replication
+        is True, the operator is responsible for maintaining recovery.conf
+        and performing any necessary database mirroring. The charm will
+        still advertise the unit as standalone, master or hot standby to
+        relations based on whether the system is in recovery mode or not.
+        Note that this option makes it possible to create a PostgreSQL
+        service with multiple master units, which is a very silly thing
+        to do unless you are also using multi-master software like BDR.
+  backup_schedule:
+    default: "13 4 * * *"
+    type: string
+    description: Cron-formatted schedule for database backups.
+  backup_retention_count:
+    default: 7
+    type: int
+    description: Number of recent backups to retain.
+  nagios_context:
+    default: "juju"
+    type: string
+    description: >
+        Used by the nrpe subordinate charms.
+        A string that will be prepended to instance name to set the host name
+        in nagios. So for instance the hostname would be something like:
+            juju-postgresql-0
+        If you're running multiple environments with the same services in them
+        this allows you to differentiate between them.
+  nagios_servicegroups:
+    default: ""
+    type: string
+    description: >
+        A comma-separated list of nagios servicegroups.
+        If left empty, the nagios_context will be used as the servicegroup
+  pgdg:
+    description: >
+        Enable the PostgreSQL Global Development Group APT repository
+        (https://wiki.postgresql.org/wiki/Apt). This package source provides
+        official PostgreSQL packages for Ubuntu LTS releases beyond those
+        provided by the main Ubuntu archive.
+    type: boolean
+    default: false
+  install_sources:
+    description: >
+        List of extra package sources, per charm-helpers standard.
+        YAML format.
+    type: string
+    default: ""
+  install_keys:
+    description: >
+        List of signing keys for install_sources package sources, per
+        charmhelpers standard. YAML format.
+    type: string
+    default: ""
+  wal_e_storage_uri:
+    type: string
+    default: ""
+    description: |
+      EXPERIMENTAL.
+      Specify storage to be used by WAL-E. Every PostgreSQL service must use
+      a unique URI. Backups will be unrecoverable if it is not unique. The
+      URI's scheme must be one of 'swift' (OpenStack Swift), 's3' (Amazon AWS)
+      or 'wabs' (Windows Azure). For example:
+        'swift://some-container/directory/or/whatever'
+        's3://some-bucket/directory/or/whatever'
+        'wabs://some-bucket/directory/or/whatever'
+      Setting the wal_e_storage_uri enables regular WAL-E filesystem level
+      backups (per wal_e_backup_schedule), and log shipping to the configured
+      storage. Point-in-time recovery becomes possible, as is disabling the
+      streaming_replication configuration item and relying solely on
+      log shipping for replication.
+  wal_e_backup_schedule:
+    type: string
+    default: "13 0 * * *"
+    description: >
+      EXPERIMENTAL.
+      Cron-formatted schedule for WAL-E database backups. If
+      wal_e_backup_schedule is unset, WAL files will never be removed from
+      WAL-E storage.
+  wal_e_backup_retention:
+    type: int
+    default: 2
+    description: >
+      EXPERIMENTAL.
+      Number of recent base backups and WAL files to retain.
+      You need enough space for this many backups plus one more, as
+      an old backup will only be removed after a new one has been
+      successfully made to replace it.
+  streaming_replication:
+    type: boolean
+    default: true
+    description: >
+      Enable streaming replication. Normally, streaming replication is
+      always used, and any log shipping configured is used as a fallback.
+      Turning this off without configuring log shipping is an error.
+  os_username:
+    type: string
+    default: ""
+    description: EXPERIMENTAL. OpenStack Swift username.
+  os_password:
+    type: string
+    default: ""
+    description: EXPERIMENTAL. OpenStack Swift password.
+  os_auth_url:
+    type: string
+    default: ""
+    description: EXPERIMENTAL. OpenStack Swift authentication URL.
+  os_tenant_name:
+    type: string
+    default: ""
+    description: EXPERIMENTAL. OpenStack Swift tenant name.
+  aws_access_key_id:
+    type: string
+    default: ""
+    description: EXPERIMENTAL. Amazon AWS access key id.
+  aws_secret_access_key:
+    type: string
+    default: ""
+    description: EXPERIMENTAL. Amazon AWS secret access key.
+  wabs_account_name:
+    type: string
+    default: ""
+    description: EXPERIMENTAL. Windows Azure account name.
+  wabs_access_key:
+    type: string
+    default: ""
+    description: EXPERIMENTAL. Windows Azure access key.
+  package_status:
+    default: "install"
+    type: string
+    description: >
+        The status of service-affecting packages will be set to this
+        value in the dpkg database. Useful valid values are "install"
+        and "hold".
+  # statsd-compatible metrics
+  metrics_target:
+    default: ""
+    type: string
+    description: >
+        Destination for statsd-format metrics, format "host:port". If
+        not present and valid, metrics disabled.
+  metrics_prefix:
+    default: "dev.$UNIT.postgresql"
+    type: string
+    description: >
+        Prefix for metrics. Special value $UNIT can be used to include the
+        name of the unit in the prefix.
+  metrics_sample_interval:
+    default: 5
+    type: int
+    description: Period for metrics cron job to run in minutes
+
+
+  # DEPRECATED SETTINGS.
+  # Remove them one day. They remain here to avoid making existing
+  # configurations fail.
+  advisory_lock_restart_key:
+    default: 765
+    type: int
+    description: >
+        DEPRECATED AND IGNORED.
+        An advisory lock key used internally by the charm. You do not need
+        to change it unless it happens to conflict with an advisory lock key
+        being used by your applications.
+  extra-packages:
+    default: ""
+    type: string
+    description: DEPRECATED. Use extra_packages.
   listen_port:
-    default: null
+    default: -1
     type: int
-    description: Port to listen on. Default is automatically assigned.
+    description: >
+        DEPRECATED. Use extra_pg_conf.
+        Port to listen on. Default is automatically assigned.
   max_connections:
     default: 100
     type: int
-    description: Maximum number of connections to allow to the PG database
+    description: >
+        DEPRECATED. Use extra_pg_conf.
+        Maximum number of connections to allow to the PG database
   max_prepared_transactions:
     default: 0
     type: int
     description: >
-      Maximum number of prepared two phase commit transactions, waiting
-      to be committed. Defaults to 0. as using two phase commit without
-      a process to monitor and resolve lost transactions is dangerous.
+        DEPRECATED. Use extra_pg_conf.
+        Maximum number of prepared two phase commit transactions, waiting
+        to be committed. Defaults to 0. as using two phase commit without
+        a process to monitor and resolve lost transactions is dangerous.
   ssl:
     default: "True"
     type: string
-    description: Whether PostgreSQL should talk SSL
+    description: >
+        DEPRECATED. Use extra_pg_conf.
+        Whether PostgreSQL should talk SSL
   log_min_duration_statement:
     default: -1
     type: int
     description: >
+        DEPRECATED. Use extra_pg_conf.
         -1 is disabled, 0 logs all statements
         and their durations, > 0 logs only
         statements running at least this number
@@ -76,19 +317,21 @@
   log_checkpoints:
     default: False
     type: boolean
-    description: Log checkpoints
+    description: >
+        DEPRECATED. Use extra_pg_conf.
   log_connections:
     default: False
     type: boolean
-    description: Log connections
+    description: DEPRECATED. Use extra_pg_conf.
   log_disconnections:
     default: False
     type: boolean
-    description: Log disconnections
+    description: DEPRECATED. Use extra_pg_conf.
   log_temp_files:
     default: "-1"
     type: string
     description: >
+        DEPRECATED. Use extra_pg_conf.
         Log creation of temporary files larger than the threshold.
         -1 disables the feature, 0 logs all temporary files, or specify
         the threshold size with an optional unit (eg. "512KB", default
@@ -97,6 +340,7 @@
     default: "%t [%p]: [%l-1] db=%d,user=%u "
     type: string
     description: |
+        DEPRECATED. Use extra_pg_conf.
         special values:
           %a = application name
           %u = user name
@@ -119,55 +363,68 @@
   log_lock_waits:
     default: False
     type: boolean
-    description: log lock waits >= deadlock_timeout
+    description: DEPRECATED. Use extra_pg_conf.
   log_timezone:
     default: "UTC"
     type: string
-    description: Log timezone
+    description: DEPRECATED. Use extra_pg_conf.
   autovacuum:
     default: True
     type: boolean
     description: >
+        DEPRECATED. Use extra_pg_conf.
         Autovacuum should almost always be running. If you want to turn this
         off, you are probably following out of date documentation.
   log_autovacuum_min_duration:
     default: -1
     type: int
     description: >
+        DEPRECATED. Use extra_pg_conf.
         -1 disables, 0 logs all actions and their durations, > 0 logs only
         actions running at least this number of milliseconds.
   autovacuum_analyze_threshold:
     default: 50
     type: int
-    description: min number of row updates before analyze
+    description: >
+        DEPRECATED. Use extra_pg_conf.
+        min number of row updates before analyze
   autovacuum_vacuum_scale_factor:
     default: 0.2
     type: float
-    description: Fraction of table size before vacuum
+    description: >
+        DEPRECATED. Use extra_pg_conf.
+        Fraction of table size before vacuum
   autovacuum_analyze_scale_factor:
     default: 0.1  
     type: float
-    description: Fraction of table size before analyze
+    description: >
+        DEPRECATED. Use extra_pg_conf.
+        Fraction of table size before analyze
   autovacuum_vacuum_cost_delay:
     default: "20ms"
     type: string
     description: >
+        DEPRECATED. Use extra_pg_conf.
         Default vacuum cost delay for autovacuum, in milliseconds;
         -1 means use vacuum_cost_delay
   search_path:
     default: "\"$user\",public"
     type: string
     description: >
+        DEPRECATED. Use extra_pg_conf.
         Comma separated list of schema names for
         the default SQL search path.
   standard_conforming_strings:
     default: True
     type: boolean
-    description: Standard conforming strings
+    description: >
+        DEPRECATED. Use extra_pg_conf.
+        Standard conforming strings
   hot_standby:
     default: False
     type: boolean
     description: >
+        DEPRECATED. Use extra_pg_conf.
         Hot standby or warm standby. When True, queries can be run against
         the database when in recovery or standby mode (ie. replicated).
         Overridden when service contains multiple units.
@@ -175,6 +432,7 @@
     default: False
     type: boolean
     description: >
+        DEPRECATED. Use extra_pg_conf.
         Hot standby feedback, informing a master about in progress
         transactions on a streaming hot standby and allowing the master to
         defer cleanup and avoid query cancelations on the hot standby.
@@ -182,6 +440,7 @@
     default: minimal
     type: string
     description: >
+        DEPRECATED. Use extra_pg_conf.
         'minimal', 'archive', 'hot_standby' or 'logical'. Defines how much
         information is written to the WAL. Set to 'minimal' for stand alone
         databases and 'hot_standby' for replicated setups. Overridden by
@@ -190,6 +449,7 @@
     default: 0
     type: int
     description: >
+        DEPRECATED. Use extra_pg_conf.
         Maximum number of hot standbys that can connect using
         streaming replication. Set this to the expected maximum number of
         hot standby units to avoid unnecessary blocking and database restarts.
@@ -198,6 +458,7 @@
     default: 0
     type: int
     description: >
+        DEPRECATED. Use extra_pg_conf.
         Number of old WAL files to keep, providing a larger buffer for
         streaming hot standbys to catch up from when lagged. Each WAL file
         is 16MB in size. The WAL files are the buffer of how far a
@@ -208,6 +469,7 @@
     default: 5000
     type: int
     description: >
+        DEPRECATED. Use extra_pg_conf.
         Value of wal_keep_segments used when this service is replicated.
         This setting only exists to provide a sane default when replication
         is requested (so it doesn't fail) and nobody bothered to change the
@@ -216,6 +478,7 @@
     default: False
     type: boolean
     description: >
+        DEPRECATED. Use extra_pg_conf.
         Enable archiving of WAL files using the command specified by
         archive_command. If archive_mode is enabled and archive_command not
         set, then archiving is deferred until archive_command is set and the
@@ -224,45 +487,38 @@
     default: ""
     type: string
     description: >
+        DEPRECATED. Use extra_pg_conf.
         Command used to archive WAL files when archive_mode is set and
         wal_level > minimal.
   work_mem:
     default: "1MB"
     type: string
     description: >
-        Working Memory.
+        DEPRECATED. Use extra_pg_conf. Working Memory.
         Ignored unless 'performance_tuning' is set to 'manual'.
   maintenance_work_mem:
     default: "1MB"
     type: string
     description: >
-        Maintenance working memory.
+        DEPRECATED. Use extra_pg_conf. Maintenance working memory.
         Ignored unless 'performance_tuning' is set to 'manual'.
-  performance_tuning:
-    default: "Mixed"
-    type: string
-    description: >
-        Possible values here are "manual", "DW" (data warehouse),
-        "OLTP" (online transaction processing), "Web" (web application),
-        "Desktop" or "Mixed". When this is set to a value other than
-        "manual", the charm invokes the pgtune tool to tune a number
-        of performance parameters based on the specified load type.
-        pgtune gathers information about the node on which you are deployed and
-        tries to make intelligent guesses about what tuning parameters to set
-        based on available RAM and CPU under the assumption that it's the only
-        significant service running on this node.
   kernel_shmall:
     default: 0
     type: int
-    description: Total amount of shared memory available, in bytes.
+    description: >
+        DEPRECATED and ignored. 
+        Total amount of shared memory available, in bytes.
   kernel_shmmax:
     default: 0
     type: int
-    description: The maximum size, in bytes, of a shared memory segment.
+    description: >
+        DEPRECATED and ignored.
+        The maximum size, in bytes, of a shared memory segment.
   shared_buffers:
     default: ""
     type: string
     description: >
+        DEPRECATED. Use extra_pg_conf.
         The amount of memory the database server uses for shared memory
         buffers. This string should be of the format '###MB'.
         Ignored unless 'performance_tuning' is set to 'manual'.
@@ -270,6 +526,7 @@
     default: ""
     type: string
     description: >
+        DEPRECATED. Use extra_pg_conf.
         Effective cache size is an estimate of how much memory is available for
         disk caching within the database. (50% to 75% of system memory). This
         string should be of the format '###MB'. Ignored unless
@@ -278,6 +535,7 @@
     default: -1
     type: int
     description: >
+        DEPRECATED. Use extra_pg_conf.
         Sets the default statistics target for table columns without a
         column-specific target set via ALTER TABLE SET STATISTICS.
         Leave unchanged to use the server default, which in recent
@@ -288,6 +546,7 @@
     default: -1 
     type: int
     description: >
+        DEPRECATED. Use extra_pg_conf.
         Sets the from_collapse_limit and join_collapse_limit query planner
         options, controlling the maximum number of tables that can be joined
         before the turns off the table collapse query optimization.
@@ -295,35 +554,41 @@
     default: "1MB"
     type: string
     description: >
+        DEPRECATED. Use extra_pg_conf.
         The maximum number of temporary buffers used by each database session.
   wal_buffers:
     default: "-1"
     type: string
     description: >
+        DEPRECATED. Use extra_pg_conf.
         min 32kB, -1 sets based on shared_buffers (change requires restart).
         Ignored unless 'performance_tuning' is set to 'manual'.
   checkpoint_segments:
     default: 10 
     type: int
     description: >
+        DEPRECATED. Use extra_pg_conf.
         in logfile segments, min 1, 16MB each.
         Ignored unless 'performance_tuning' is set to 'manual'.
   checkpoint_completion_target:
     default: 0.9
     type: float
     description: >
-      checkpoint target duration time, as a fraction of checkpoint_timeout.
-      Range [0.0, 1.0].
+        DEPRECATED. Use extra_pg_conf.
+        checkpoint target duration time, as a fraction of checkpoint_timeout.
+        Range [0.0, 1.0].
   checkpoint_timeout:
     default: ""
     type: string
     description: >
+        DEPRECATED. Use extra_pg_conf.
         Maximum time between automatic WAL checkpoints. range '30s-1h'.
         If left empty, the default postgresql value will be used.
   fsync:
     type: boolean
     default: True
     description: >
+        DEPRECATED. Use extra_pg_conf.
         Turns forced synchronization on/off. If fsync is turned off, database
         failures are likely to involve database corruption and require
         recreating the unit
@@ -331,231 +596,23 @@
     type: boolean
     default: True
     description: >
+        DEPRECATED. Use extra_pg_conf.
         Immediate fsync after commit.
   full_page_writes:
     type: boolean
     default: True
     description: >
+        DEPRECATED. Use extra_pg_conf.
         Recover from partial page writes.
   random_page_cost:
     default: 4.0
     type: float
-    description: Random page cost
-  extra_pg_auth:
-    type: string
-    default: ""
-    description: >
-        A comma separated extra pg_hba.conf auth rules.
-        This will be written to the pg_hba.conf file, one line per rule.
-        Note that this should not be needed as db relations already create
-        those rules the right way. Only use this if you really need too
-        (e.g. on a development environment), or are connecting juju managed
-        databases to external managed systems, or configuring replication
-        between unrelated PostgreSQL services using the manual_replication
-        option.
-  manual_replication:
-    type: boolean
-    default: False
-    description: >
-        Enable or disable charm managed replication. When manual_replication
-        is True, the operator is responsible for maintaining recovery.conf
-        and performing any necessary database mirroring. The charm will
-        still advertise the unit as standalone, master or hot standby to
-        relations based on whether the system is in recovery mode or not.
-        Note that this option makes it possible to create a PostgreSQL
-        service with multiple master units, which is probably a very silly
-        thing to do.
+    description: >
+        DEPRECATED. Use extra_pg_conf. Random page cost
   backup_dir:
     default: "/var/lib/postgresql/backups"
     type: string
-    description: Directory to place backups in
-  backup_schedule:
-    default: "13 4 * * *"
-    type: string
-    description: Cron-formatted schedule for database backups.
-  backup_retention_count:
-    default: 7
-    type: int
-    description: Number of recent backups to retain.
-  nagios_context:
-    default: "juju"
-    type: string
-    description: >
-        Used by the nrpe-external-master subordinate charm.
-        A string that will be prepended to instance name to set the host name
-        in nagios. So for instance the hostname would be something like:
-            juju-postgresql-0
-        If you're running multiple environments with the same services in them
-        this allows you to differentiate between them.
-  nagios_additional_servicegroups:
-    default: ""
-    type: string
-    description: >
-        Used by the nrpe-external-master subordinate charm.
-        A comma-separated list of servicegroups to include along with
-        nagios_context when generating nagios service check configs.
-        This is useful for nagios installations where servicegroups
-        are used to apply special treatment to particular checks.
-  pgdg:
-    description: >
-        Enable the PostgreSQL Global Development Group APT repository
-        (https://wiki.postgresql.org/wiki/Apt). This package source provides
-        official PostgreSQL packages for Ubuntu LTS releases beyond those
-        provided by the main Ubuntu archive.
-    type: boolean
-    default: false
-  install_sources:
-    description: >
-        List of extra package sources, per charm-helpers standard.
-        YAML format.
-    type: string
-    default: null
-  install_keys:
-    description: >
-        List of signing keys for install_sources package sources, per
-        charmhelpers standard. YAML format.
-    type: string
-    default: null
-  extra_archives:
-    default: ""
-    type: string
-    description: >
-        DEPRECATED & IGNORED. Use install_sources and install_keys. 
-  advisory_lock_restart_key:
-    default: 765
-    type: int
-    description: >
-        An advisory lock key used internally by the charm. You do not need
-        to change it unless it happens to conflict with an advisory lock key
-        being used by your applications.
-  # Swift backups and PITR via SwiftWAL
-  swiftwal_container_prefix:
-    type: string
-    default: null
-    description: >
-      EXPERIMENTAL.
-      Swift container prefix for SwiftWAL to use. Must be set if any
-      SwiftWAL features are enabled. This will become a simple
-      swiftwal_container config item when proper leader election is
-      implemented in juju.
-  swiftwal_backup_schedule:
-    type: string
-    default: null
-    description: >
-      EXPERIMENTAL.
-      Cron-formatted schedule for SwiftWAL database backups.
-  swiftwal_backup_retention:
-    type: int
-    default: 2
-    description: >
-      EXPERIMENTAL.
-      Number of recent base backups to retain. You need enough space in
-      Swift for this many backups plus one more, as an old backup will only
-      be removed after a new one has been successfully made to replace it.
-  swiftwal_log_shipping:
-    type: boolean
-    default: false
-    description: >
-      EXPERIMENTAL.
-      Archive WAL files into Swift. If swiftwal_backup_schedule is set,
-      allows point-in-time recovery and WAL files are removed
-      automatically with old backups. If swiftwal_backup_schedule is not set
-      then WAL files are never removed. Enabling this option will override
-      the archive_mode and archive_command settings.
-  wal_e_storage_uri:
-    type: string
-    default: null
-    description: |
-      EXPERIMENTAL.
-      Specify storage to be used by WAL-E. Every PostgreSQL service must use
-      a unique URI. Backups will be unrecoverable if it is not unique. The
-      URI's scheme must be one of 'swift' (OpenStack Swift), 's3' (Amazon AWS)
-      or 'wabs' (Windows Azure). For example:
-        'swift://some-container/directory/or/whatever'
-        's3://some-bucket/directory/or/whatever'
-        'wabs://some-bucket/directory/or/whatever'
-      Setting the wal_e_storage_uri enables regular WAL-E filesystem level
-      backups (per wal_e_backup_schedule), and log shipping to the configured
-      storage. Point-in-time recovery becomes possible, as is disabling the
-      streaming_replication configuration item and relying solely on
-      log shipping for replication.
-  wal_e_backup_schedule:
-    type: string
-    default: "13 0 * * *"
-    description: >
-      EXPERIMENTAL.
-      Cron-formatted schedule for WAL-E database backups. If
-      wal_e_backup_schedule is unset, WAL files will never be removed from
-      WAL-E storage.
-  wal_e_backup_retention:
-    type: int
-    default: 2
-    description: >
-      EXPERIMENTAL.
-      Number of recent base backups and WAL files to retain.
-      You need enough space for this many backups plus one more, as
-      an old backup will only be removed after a new one has been
-      successfully made to replace it.
-  streaming_replication:
-    type: boolean
-    default: true
-    description: >
-      Enable streaming replication. Normally, streaming replication is
-      always used, and any log shipping configured is used as a fallback.
-      Turning this off without configuring log shipping is an error.
-  os_username:
-    type: string
-    default: null
-    description: EXPERIMENTAL. OpenStack Swift username.
-  os_password:
-    type: string
-    default: null
-    description: EXPERIMENTAL. OpenStack Swift password.
-  os_auth_url:
-    type: string
-    default: null
-    description: EXPERIMENTAL. OpenStack Swift authentication URL.
-  os_tenant_name:
-    type: string
-    default: null
-    description: EXPERIMENTAL. OpenStack Swift tenant name.
-  aws_access_key_id:
-    type: string
-    default: null
-    description: EXPERIMENTAL. Amazon AWS access key id.
-  aws_secret_access_key:
-    type: string
-    default: null
-    description: EXPERIMENTAL. Amazon AWS secret access key.
-  wabs_account_name:
-    type: string
-    default: null
-    description: EXPERIMENTAL. Windows Azure account name.
-  wabs_access_key:
-    type: string
-    default: null
-    description: EXPERIMENTAL. Windows Azure access key.
-  package_status:
-    default: "install"
-    type: string
-    description: >
-        The status of service-affecting packages will be set to this
-        value in the dpkg database. Useful valid values are "install"
-        and "hold".
-  # statsd-compatible metrics
-  metrics_target:
-    default: ""
-    type: string
-    description: >
-        Destination for statsd-format metrics, format "host:port". If
-        not present and valid, metrics disabled.
-  metrics_prefix:
-    default: "dev.$UNIT.postgresql"
-    type: string
-    description: >
-        Prefix for metrics. Special value $UNIT can be used to include the
-        name of the unit in the prefix.
-  metrics_sample_interval:
-    default: 5
-    type: int
-    description: Period for metrics cron job to run in minutes
+    description: >
+        DEPRECATED. Directory to place backups in. If you change this,
+        your backups will go to this path and not the 'backups' Juju
+        storage mount.

=== modified file 'copyright'
--- copyright	2011-07-10 09:53:13 +0000
+++ copyright	2015-11-02 12:15:35 +0000
@@ -1,17 +1,16 @@
 Format: http://dep.debian.net/deps/dep5/
 
 Files: *
-Copyright: Copyright 2011, Canonical Ltd., All Rights Reserved.
+Copyright: Copyright 2011-2015, Canonical Ltd.
 License: GPL-3
  This program is free software: you can redistribute it and/or modify
- it under the terms of the GNU General Public License as published by
- the Free Software Foundation, either version 3 of the License, or
- (at your option) any later version.
+ it under the terms of the GNU General Public License version 3, as
+ published by the Free Software Foundation.
  .
  This program is distributed in the hope that it will be useful,
- but WITHOUT ANY WARRANTY; without even the implied warranty of
- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
- GNU General Public License for more details.
+ but WITHOUT ANY WARRANTY; without even the implied warranties of
+ MERCHANTABILITY, SATISFACTORY QUALITY, or FITNESS FOR A PARTICULAR
+ PURPOSE.  See the GNU General Public License for more details.
  .
  You should have received a copy of the GNU General Public License
  along with this program.  If not, see <http://www.gnu.org/licenses/>.

=== added file 'hooks/bootstrap.py'
--- hooks/bootstrap.py	1970-01-01 00:00:00 +0000
+++ hooks/bootstrap.py	2015-11-02 12:15:35 +0000
@@ -0,0 +1,57 @@
+#!/usr/bin/python3
+
+# Copyright 2015 Canonical Ltd.
+#
+# This file is part of the PostgreSQL Charm for Juju.
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 3, as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranties of
+# MERCHANTABILITY, SATISFACTORY QUALITY, or FITNESS FOR A PARTICULAR
+# PURPOSE.  See the GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+from charmhelpers import fetch
+from charmhelpers.core import hookenv
+
+
+def bootstrap():
+    try:
+        import psycopg2  # NOQA: flake8
+        import jinja2  # NOQA: flake8
+    except ImportError:
+        packages = ['python3-psycopg2', 'python3-jinja2']
+        fetch.apt_install(packages, fatal=True)
+        import psycopg2  # NOQA: flake8
+
+
+def block_on_bad_juju():
+    if not hookenv.has_juju_version('1.24'):
+        hookenv.status_set('blocked', 'Requires Juju 1.24 or higher')
+        # Error state, since we don't have 1.24 to give a nice blocked state.
+        raise SystemExit(1)
+
+
+def upgrade_charm():
+    block_on_bad_juju()
+    # This needs to be imported after bootstrap() or required Python
+    # packages may not have been installed.
+    import upgrade
+    upgrade.upgrade_charm()
+
+
+def default_hook():
+    block_on_bad_juju()
+    # This needs to be imported after bootstrap() or required Python
+    # packages may not have been installed.
+    import definitions
+
+    hookenv.log('*** Start {!r} hook'.format(hookenv.hook_name()))
+    sm = definitions.get_service_manager()
+    sm.manage()
+    hookenv.log('*** End {!r} hook'.format(hookenv.hook_name()))

=== added file 'hooks/client.py'
--- hooks/client.py	1970-01-01 00:00:00 +0000
+++ hooks/client.py	2015-11-02 12:15:35 +0000
@@ -0,0 +1,182 @@
+# Copyright 2015 Canonical Ltd.
+#
+# This file is part of the PostgreSQL Charm for Juju.
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 3, as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranties of
+# MERCHANTABILITY, SATISFACTORY QUALITY, or FITNESS FOR A PARTICULAR
+# PURPOSE.  See the GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+from charmhelpers.core import hookenv, host
+
+from decorators import relation_handler, master_only
+import helpers
+import postgresql
+
+
+@relation_handler('db', 'db-admin', 'master')
+def publish_db_relations(rel):
+    if postgresql.is_master():
+        db_relation_master(rel)
+    else:
+        db_relation_mirror(rel)
+    db_relation_common(rel)
+
+
+def _credential_types(rel):
+    superuser = (rel.relname in ('db-admin', 'master'))
+    replication = (rel.relname == 'master')
+    return (superuser, replication)
+
+
+def db_relation_master(rel):
+    '''The master generates credentials and negotiates resources.'''
+    master = rel.local
+    # Pick one remote unit as representative. They should all converge.
+    for remote in rel.values():
+        break
+
+    # The requested database name, the existing database name, or use
+    # the remote service name as a default. We no longer use the
+    # relation id for the database name or usernames, as when a
+    # database dump is restored into a new Juju environment we
+    # are more likely to have matching service names than relation ids
+    # and less likely to have to perform manual permission and ownership
+    # cleanups.
+    if 'database' in remote:
+        master['database'] = remote['database']
+    elif 'database' not in master:
+        master['database'] = remote.service
+
+    superuser, replication = _credential_types(rel)
+
+    if 'user' not in master:
+        user = postgresql.username(remote.service, superuser=superuser,
+                                   replication=replication)
+        password = host.pwgen()
+        master['user'] = user
+        master['password'] = password
+
+        # schema_user has never been documented and is deprecated.
+        if not superuser:
+            master['schema_user'] = user
+            master['schema_password'] = password
+
+    hookenv.log('** Master providing {} ({}/{})'.format(rel,
+                                                        master['database'],
+                                                        master['user']))
+
+    # Reflect these settings back so the client knows when they have
+    # taken effect.
+    if not replication:
+        master['roles'] = remote.get('roles')
+        master['extensions'] = remote.get('extensions')
+
+
+def db_relation_mirror(rel):
+    '''Non-masters mirror relation information from the master.'''
+    master = postgresql.master()
+    master_keys = ['database', 'user', 'password', 'roles',
+                   'schema_user', 'schema_password', 'extensions']
+    master_info = rel.peers.get(master)
+    if master_info is None:
+        hookenv.log('Waiting for {} to join {}'.format(master, rel))
+        return
+    hookenv.log('Mirroring {} database credentials from {}'.format(rel,
+                                                                   master))
+    rel.local.update({k: master_info.get(k) for k in master_keys})
+
+
+def db_relation_common(rel):
+    '''Publish unit specific relation details.'''
+    local = rel.local
+    if 'database' not in local:
+        return  # Not yet ready.
+
+    # Version number, allowing clients to adjust or block if their
+    # expectations are not met.
+    local['version'] = postgresql.version()
+
+    # Calculate the state of this unit. 'standalone' will disappear
+    # in a future version of this interface, as this state was
+    # only needed to deal with race conditions now solved by
+    # Juju leadership.
+    if postgresql.is_primary():
+        if hookenv.is_leader() and len(helpers.peers()) == 0:
+            local['state'] = 'standalone'
+        else:
+            local['state'] = 'master'
+    else:
+        local['state'] = 'hot standby'
+
+    # Host is the private ip address, but this might change and
+    # become the address of an attached proxy or alternative peer
+    # if this unit is in maintenance.
+    local['host'] = hookenv.unit_private_ip()
+
+    # Port will be 5432, unless the user has overridden it or
+    # something very weird happened when the packages where installed.
+    local['port'] = str(postgresql.port())
+
+    # The list of remote units on this relation granted access.
+    # This is to avoid the race condition where a new client unit
+    # joins an existing client relation and sees valid credentials,
+    # before we have had a chance to grant it access.
+    local['allowed-units'] = ' '.join(rel.keys())
+
+
+@master_only
+@relation_handler('db', 'db-admin', 'master')
+def ensure_db_relation_resources(rel):
+    '''Create the database resources needed for the relation.'''
+
+    master = rel.local
+
+    hookenv.log('Ensuring database {!r} and user {!r} exist for {}'
+                ''.format(master['database'], master['user'], rel))
+
+    # First create the database, if it isn't already.
+    postgresql.ensure_database(master['database'])
+
+    # Next, connect to the database to create the rest in a transaction.
+    con = postgresql.connect(database=master['database'])
+
+    superuser, replication = _credential_types(rel)
+    postgresql.ensure_user(con, master['user'], master['password'],
+                           superuser=superuser, replication=replication)
+    if not superuser:
+        postgresql.ensure_user(con,
+                               master['schema_user'],
+                               master['schema_password'])
+
+    # Grant specified privileges on the database to the user. This comes
+    # from the PostgreSQL service configuration, as allowing the
+    # relation to specify how much access it gets is insecure.
+    config = hookenv.config()
+    privs = set(filter(None,
+                       config['relation_database_privileges'].split(',')))
+    postgresql.grant_database_privileges(con, master['user'],
+                                         master['database'], privs)
+    if not superuser:
+        postgresql.grant_database_privileges(con, master['schema_user'],
+                                             master['database'], privs)
+
+    # Reset the roles granted to the user as requested.
+    if 'roles' in master:
+        roles = filter(None, master.get('roles', '').split(','))
+        postgresql.grant_user_roles(con, master['user'], roles)
+
+    # Create requested extensions. We never drop extensions, as there
+    # may be dependent objects.
+    if 'extensions' in master:
+        extensions = filter(None, master.get('extensions', '').split(','))
+        postgresql.ensure_extensions(con, extensions)
+
+    con.commit()  # Don't throw away our changes.

=== removed symlink 'hooks/config-changed'
=== target was u'hooks.py'
=== added file 'hooks/coordinator.py'
--- hooks/coordinator.py	1970-01-01 00:00:00 +0000
+++ hooks/coordinator.py	2015-11-02 12:15:35 +0000
@@ -0,0 +1,19 @@
+# Copyright 2015 Canonical Ltd.
+#
+# This file is part of the PostgreSQL Charm for Juju.
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 3, as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranties of
+# MERCHANTABILITY, SATISFACTORY QUALITY, or FITNESS FOR A PARTICULAR
+# PURPOSE.  See the GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+from charmhelpers.coordinator import Serial
+
+coordinator = Serial()

=== added file 'hooks/data-relation-changed'
--- hooks/data-relation-changed	1970-01-01 00:00:00 +0000
+++ hooks/data-relation-changed	2015-11-02 12:15:35 +0000
@@ -0,0 +1,23 @@
+#!/usr/bin/python3
+
+# Copyright 2015 Canonical Ltd.
+#
+# This file is part of the PostgreSQL Charm for Juju.
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 3, as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranties of
+# MERCHANTABILITY, SATISFACTORY QUALITY, or FITNESS FOR A PARTICULAR
+# PURPOSE.  See the GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+import bootstrap
+
+if __name__ == '__main__':
+    bootstrap.bootstrap()
+    bootstrap.default_hook()

=== removed symlink 'hooks/data-relation-changed'
=== target was u'hooks.py'
=== added file 'hooks/data-relation-departed'
--- hooks/data-relation-departed	1970-01-01 00:00:00 +0000
+++ hooks/data-relation-departed	2015-11-02 12:15:35 +0000
@@ -0,0 +1,23 @@
+#!/usr/bin/python3
+
+# Copyright 2015 Canonical Ltd.
+#
+# This file is part of the PostgreSQL Charm for Juju.
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 3, as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranties of
+# MERCHANTABILITY, SATISFACTORY QUALITY, or FITNESS FOR A PARTICULAR
+# PURPOSE.  See the GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+import bootstrap
+
+if __name__ == '__main__':
+    bootstrap.bootstrap()
+    bootstrap.default_hook()

=== removed symlink 'hooks/data-relation-departed'
=== target was u'hooks.py'
=== removed symlink 'hooks/data-relation-joined'
=== target was u'hooks.py'
=== removed symlink 'hooks/db-admin-relation-broken'
=== target was u'hooks.py'
=== removed symlink 'hooks/db-admin-relation-changed'
=== target was u'hooks.py'
=== added file 'hooks/db-admin-relation-departed'
--- hooks/db-admin-relation-departed	1970-01-01 00:00:00 +0000
+++ hooks/db-admin-relation-departed	2015-11-02 12:15:35 +0000
@@ -0,0 +1,23 @@
+#!/usr/bin/python3
+
+# Copyright 2015 Canonical Ltd.
+#
+# This file is part of the PostgreSQL Charm for Juju.
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 3, as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranties of
+# MERCHANTABILITY, SATISFACTORY QUALITY, or FITNESS FOR A PARTICULAR
+# PURPOSE.  See the GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+import bootstrap
+
+if __name__ == '__main__':
+    bootstrap.bootstrap()
+    bootstrap.default_hook()

=== removed symlink 'hooks/db-admin-relation-joined'
=== target was u'hooks.py'
=== removed symlink 'hooks/db-relation-broken'
=== target was u'hooks.py'
=== removed symlink 'hooks/db-relation-changed'
=== target was u'hooks.py'
=== added file 'hooks/db-relation-departed'
--- hooks/db-relation-departed	1970-01-01 00:00:00 +0000
+++ hooks/db-relation-departed	2015-11-02 12:15:35 +0000
@@ -0,0 +1,23 @@
+#!/usr/bin/python3
+
+# Copyright 2015 Canonical Ltd.
+#
+# This file is part of the PostgreSQL Charm for Juju.
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 3, as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranties of
+# MERCHANTABILITY, SATISFACTORY QUALITY, or FITNESS FOR A PARTICULAR
+# PURPOSE.  See the GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+import bootstrap
+
+if __name__ == '__main__':
+    bootstrap.bootstrap()
+    bootstrap.default_hook()

=== removed symlink 'hooks/db-relation-joined'
=== target was u'hooks.py'
=== added file 'hooks/decorators.py'
--- hooks/decorators.py	1970-01-01 00:00:00 +0000
+++ hooks/decorators.py	2015-11-02 12:15:35 +0000
@@ -0,0 +1,124 @@
+# Copyright 2015 Canonical Ltd.
+#
+# This file is part of the PostgreSQL Charm for Juju.
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 3, as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranties of
+# MERCHANTABILITY, SATISFACTORY QUALITY, or FITNESS FOR A PARTICULAR
+# PURPOSE.  See the GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+from functools import wraps
+
+from charmhelpers import context
+from charmhelpers.core import hookenv
+from charmhelpers.core.hookenv import DEBUG
+
+import helpers
+
+
+def data_ready_action(func):
+    '''Decorate func to be used as a data_ready item.'''
+    @wraps(func)
+    def wrapper(service_name=None):
+        if hookenv.remote_unit():
+            hookenv.log("** Action {}/{} ({})".format(hookenv.hook_name(),
+                                                      func.__name__,
+                                                      hookenv.remote_unit()))
+        else:
+            hookenv.log("** Action {}/{}".format(hookenv.hook_name(),
+                                                 func.__name__))
+        return func()
+    return wrapper
+
+
+class requirement:
+    '''Decorate a function so it can be used as a required_data item.
+
+    Function must True if requirements are met. Sets the unit state
+    to blocked if requirements are not met and the unit not already blocked.
+    '''
+    def __init__(self, func):
+        self._func = func
+
+    def __bool__(self):
+        name = self._func.__name__
+        if self._func():
+            hookenv.log('** Requirement {} passed'.format(name))
+            return True
+        else:
+            if hookenv.status_get() != 'blocked':
+                helpers.status_set('blocked',
+                                   'Requirement {} failed'.format(name))
+            return False
+
+
+def relation_handler(*relnames):
+    '''Invoke the decorated function once per matching relation.
+
+    The decorated function should accept the Relation() instance
+    as its single parameter.
+    '''
+    assert relnames, 'relation names required'
+
+    def decorator(func):
+        @wraps(func)
+        def wrapper(service_name=None):
+            rels = context.Relations()
+            for relname in relnames:
+                for rel in rels[relname].values():
+                    if rel:
+                        func(rel)
+        return wrapper
+    return decorator
+
+
+def leader_only(func):
+    '''Only run on the service leader.'''
+    @wraps(func)
+    def wrapper(*args, **kw):
+        if hookenv.is_leader():
+            return func(*args, **kw)
+        else:
+            hookenv.log('Not the leader', DEBUG)
+    return wrapper
+
+
+def not_leader(func):
+    '''Only run on the service leader.'''
+    @wraps(func)
+    def wrapper(*args, **kw):
+        if not hookenv.is_leader():
+            return func(*args, **kw)
+        else:
+            hookenv.log("I'm the leader", DEBUG)
+    return wrapper
+
+
+def master_only(func):
+    '''Only run on the appointed master.'''
+    @wraps(func)
+    def wrapper(*args, **kw):
+        import postgresql
+        if postgresql.is_master():
+            return func(*args, **kw)
+        else:
+            hookenv.log('Not the master', DEBUG)
+    return wrapper
+
+
+def not_master(func):
+    '''Don't run on the appointed master.'''
+    @wraps(func)
+    def wrapper(*args, **kw):
+        import postgresql
+        if postgresql.is_master():
+            hookenv.log("I'm the master", DEBUG)
+        else:
+            return func(*args, **kw)
+    return wrapper

=== added file 'hooks/definitions.py'
--- hooks/definitions.py	1970-01-01 00:00:00 +0000
+++ hooks/definitions.py	2015-11-02 12:15:35 +0000
@@ -0,0 +1,86 @@
+# Copyright 2011-2015 Canonical Ltd.
+#
+# This file is part of the PostgreSQL Charm for Juju.
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 3, as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranties of
+# MERCHANTABILITY, SATISFACTORY QUALITY, or FITNESS FOR A PARTICULAR
+# PURPOSE.  See the GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+from charmhelpers.core import services
+
+import client
+import nagios
+import replication
+import service
+import storage
+import syslogrel
+import wal_e
+
+
+SERVICE_DEFINITION = [
+    dict(service='postgresql',
+         required_data=[service.valid_config],
+         data_ready=[service.preinstall,
+                     service.configure_sources,
+                     service.install_packages,
+                     service.ensure_package_status,
+                     service.update_kernel_settings,
+                     service.appoint_master,
+
+                     service.wait_for_peers,  # Exit if there are no peers.
+
+                     nagios.ensure_nagios_credentials,
+                     replication.ensure_replication_credentials,
+                     replication.publish_replication_details,
+
+                     # Exit if required leader settings are not set.
+                     service.wait_for_leader,
+
+                     service.ensure_cluster,
+                     service.update_pgpass,
+                     service.update_pg_hba_conf,
+                     service.update_pg_ident_conf,
+                     service.update_postgresql_conf,
+                     syslogrel.handle_syslog_relations,
+                     storage.handle_storage_relation,
+                     wal_e.update_wal_e_env_dir,
+                     service.request_restart,
+
+                     service.wait_for_restart,  # Exit if cannot restart yet.
+
+                     replication.promote_master,
+                     storage.remount,
+                     replication.clone_master,  # Exit if cannot clone yet.
+                     replication.update_recovery_conf,
+                     service.restart_or_reload,
+
+                     replication.ensure_replication_user,
+                     nagios.ensure_nagios_user,
+                     service.install_administrative_scripts,
+                     service.update_postgresql_crontab,
+
+                     client.publish_db_relations,
+                     client.ensure_db_relation_resources,
+
+                     service.update_pg_hba_conf,  # Again, after client setup.
+                     service.reload_config,
+
+                     service.set_active,
+
+                     # At the end, as people check the end of logs
+                     # most frequently.
+                     service.emit_deprecated_option_warnings],
+         start=[service.open_ports],
+         stop=[service.stop_postgresql, service.close_ports])]
+
+
+def get_service_manager():
+    return services.ServiceManager(SERVICE_DEFINITION)

=== added file 'hooks/helpers.py'
--- hooks/helpers.py	1970-01-01 00:00:00 +0000
+++ hooks/helpers.py	2015-11-02 12:15:35 +0000
@@ -0,0 +1,151 @@
+# Copyright 2015 Canonical Ltd.
+#
+# This file is part of the PostgreSQL Charm for Juju.
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 3, as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranties of
+# MERCHANTABILITY, SATISFACTORY QUALITY, or FITNESS FOR A PARTICULAR
+# PURPOSE.  See the GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+from contextlib import contextmanager
+import os
+import shutil
+import stat
+import tempfile
+
+import yaml
+
+from charmhelpers import context
+from charmhelpers.core import hookenv, host
+from charmhelpers.core.hookenv import INFO, CRITICAL
+
+
+def status_set(status_or_msg, msg=None):
+    '''Set the unit status message, and log the change too.'''
+    if msg is None:
+        msg = status_or_msg
+        status = hookenv.status_get()
+    else:
+        status = status_or_msg
+
+    if status == 'blocked':
+        lvl = CRITICAL
+    else:
+        lvl = INFO
+    hookenv.log('{}: {}'.format(status, msg), lvl)
+    hookenv.status_set(status, msg)
+
+
+def distro_codename():
+    """Return the distro release code name, eg. 'precise' or 'trusty'."""
+    return host.lsb_release()['DISTRIB_CODENAME']
+
+
+def extra_packages():
+    config = hookenv.config()
+    packages = set()
+
+    packages.update(set(config['extra_packages'].split()))
+    packages.update(set(config['extra-packages'].split()))  # Deprecated.
+
+    if config['wal_e_storage_uri']:
+        packages.add('daemontools')
+        packages.add('wal-e')
+
+    return packages
+
+
+def peers():
+    '''Return the set of peers, not including the local unit.'''
+    rel = context.Relations().peer
+    return frozenset(rel.keys()) if rel else frozenset()
+
+
+def rewrite(path, content):
+    '''Rewrite a file atomically, preserving ownership and permissions.'''
+    attr = os.lstat(path)
+    write(path, content,
+          mode=stat.S_IMODE(attr.st_mode),
+          user=attr[stat.ST_UID],
+          group=attr[stat.ST_GID])
+
+
+def write(path, content, mode=0o640, user='root', group='root'):
+    '''Write a file atomically.'''
+    open_mode = 'wb' if isinstance(content, bytes) else 'w'
+    with tempfile.NamedTemporaryFile(mode=open_mode, delete=False) as f:
+        try:
+            f.write(content)
+            f.flush()
+            shutil.chown(f.name, user, group)
+            os.chmod(f.name, mode)
+            os.replace(f.name, path)
+        finally:
+            if os.path.exists(f.name):
+                os.unlink(f.name)
+
+
+def makedirs(path, mode=0o750, user='root', group='root'):
+    if os.path.exists(path):
+        assert os.path.isdir(path), '{} is not a directory'
+    else:
+        os.makedirs(path, mode=mode)
+    shutil.chown(path, user, group)
+    os.chmod(path, mode)
+
+
+@contextmanager
+def switch_cwd(new_working_directory='/tmp'):
+    'Switch working directory.'
+    org_dir = os.getcwd()
+    os.chdir(new_working_directory)
+    try:
+        yield new_working_directory
+    finally:
+        os.chdir(org_dir)
+
+
+def config_yaml():
+    config_yaml_path = os.path.join(hookenv.charm_dir(), 'config.yaml')
+    with open(config_yaml_path, 'r') as f:
+        return yaml.load(f)
+
+
+def deprecated_config_in_use():
+    options = config_yaml()['options']
+    config = hookenv.config()
+    deprecated = [key for key in options
+                  if ('DEPRECATED' in options[key]['description'] and
+                      config[key] != options[key]['default'])]
+    return set(deprecated)
+
+
+def cron_dir():
+    '''Where we put crontab files.'''
+    return '/etc/cron.d'
+
+
+def scripts_dir():
+    '''Where the charm puts adminstrative scripts.'''
+    return '/var/lib/postgresql/scripts'
+
+
+def logs_dir():
+    '''Where the charm administrative scripts log their output.'''
+    return '/var/lib/postgresql/logs'
+
+
+def backups_dir():
+    '''Where pg_dump backups are stored.'''
+    return hookenv.config()['backup_dir']
+
+
+def backups_log_path():
+    return os.path.join(logs_dir(), 'backups.log')

=== removed file 'hooks/helpers.py'
--- hooks/helpers.py	2015-02-24 16:57:31 +0000
+++ hooks/helpers.py	1970-01-01 00:00:00 +0000
@@ -1,197 +0,0 @@
-# Copyright 2012 Canonical Ltd.  This software is licensed under the
-# GNU Affero General Public License version 3 (see the file LICENSE).
-
-"""Helper functions for writing hooks in python."""
-
-__metaclass__ = type
-__all__ = [
-    'get_config',
-    'juju_status',
-    'log',
-    'log_entry',
-    'log_exit',
-    'make_charm_config_file',
-    'relation_get',
-    'relation_set',
-    'unit_info',
-    'wait_for_machine',
-    'wait_for_page_contents',
-    'wait_for_relation',
-    'wait_for_unit']
-
-from contextlib import contextmanager
-import json
-import operator
-from shelltoolbox import (
-    command,
-    run,
-    script_name)
-import os
-import tempfile
-import time
-import urllib2
-import yaml
-
-
-log = command('juju-log')
-
-
-def log_entry():
-    log("--> Entering {}".format(script_name()))
-
-
-def log_exit():
-    log("<-- Exiting {}".format(script_name()))
-
-
-def get_config():
-    config_get = command('config-get', '--format=json')
-    return json.loads(config_get())
-
-
-def relation_get(*args):
-    cmd = command('relation-get')
-    return cmd(*args).strip()
-
-
-def relation_set(**kwargs):
-    cmd = command('relation-set')
-    args = ['{}={}'.format(k, v) for k, v in kwargs.items()]
-    return cmd(*args)
-
-
-def make_charm_config_file(charm_config):
-    charm_config_file = tempfile.NamedTemporaryFile()
-    charm_config_file.write(yaml.dump(charm_config))
-    charm_config_file.flush()
-    # The NamedTemporaryFile instance is returned instead of just the name
-    # because we want to take advantage of garbage collection-triggered
-    # deletion of the temp file when it goes out of scope in the caller.
-    return charm_config_file
-
-
-def juju_status(key):
-    return yaml.safe_load(run('juju', 'status'))[key]
-
-
-def get_charm_revision(service_name):
-    service = juju_status('services')[service_name]
-    return int(service['charm'].split('-')[-1])
-
-
-def unit_info(service_name, item_name, data=None):
-    services = juju_status('services') if data is None else data['services']
-    service = services.get(service_name)
-    if service is None:
-        # XXX 2012-02-08 gmb:
-        #     This allows us to cope with the race condition that we
-        #     have between deploying a service and having it come up in
-        #     `juju status`. We could probably do with cleaning it up so
-        #     that it fails a bit more noisily after a while.
-        return ''
-    units = service['units']
-    item = units.items()[0][1][item_name]
-    return item
-
-
-@contextmanager
-def maintain_charm_revision(path=None):
-    if path is None:
-        path = os.path.join(os.path.dirname(__file__), '..', 'revision')
-    revision = open(path).read()
-    try:
-        yield revision
-    finally:
-        with open(path, 'w') as f:
-            f.write(revision)
-
-
-def upgrade_charm(service_name, timeout=120):
-    next_revision = get_charm_revision(service_name) + 1
-    start_time = time.time()
-    run('juju', 'upgrade-charm', service_name)
-    while get_charm_revision(service_name) != next_revision:
-        if time.time() - start_time >= timeout:
-            raise RuntimeError('timeout waiting for charm to be upgraded')
-        time.sleep(0.1)
-    return next_revision
-
-
-def wait_for_machine(num_machines=1, timeout=300):
-    """Wait `timeout` seconds for `num_machines` machines to come up.
-
-    This wait_for... function can be called by other wait_for functions
-    whose timeouts might be too short in situations where only a bare
-    Juju setup has been bootstrapped.
-    """
-    # You may think this is a hack, and you'd be right. The easiest way
-    # to tell what environment we're working in (LXC vs EC2) is to check
-    # the dns-name of the first machine. If it's localhost we're in LXC
-    # and we can just return here.
-    if juju_status('machines')[0]['dns-name'] == 'localhost':
-        return
-    start_time = time.time()
-    while True:
-        # Drop the first machine, since it's the Zookeeper and that's
-        # not a machine that we need to wait for. This will only work
-        # for EC2 environments, which is why we return early above if
-        # we're in LXC.
-        machine_data = juju_status('machines')
-        non_zookeeper_machines = [
-            machine_data[key] for key in machine_data.keys()[1:]]
-        if len(non_zookeeper_machines) >= num_machines:
-            all_machines_running = True
-            for machine in non_zookeeper_machines:
-                if machine['instance-state'] != 'running':
-                    all_machines_running = False
-                    break
-            if all_machines_running:
-                break
-        if time.time() - start_time >= timeout:
-            raise RuntimeError('timeout waiting for service to start')
-        time.sleep(0.1)
-
-
-def wait_for_unit(service_name, timeout=480):
-    """Wait `timeout` seconds for a given service name to come up."""
-    wait_for_machine(num_machines=1)
-    start_time = time.time()
-    while True:
-        state = unit_info(service_name, 'state')
-        if 'error' in state or state == 'started':
-            break
-        if time.time() - start_time >= timeout:
-            raise RuntimeError('timeout waiting for service to start')
-        time.sleep(0.1)
-    if state != 'started':
-        raise RuntimeError('unit did not start, state: ' + state)
-
-
-def wait_for_relation(service_name, relation_name, timeout=120):
-    """Wait `timeout` seconds for a given relation to come up."""
-    start_time = time.time()
-    while True:
-        relation = unit_info(service_name, 'relations').get(relation_name)
-        if relation is not None and relation['state'] == 'up':
-            break
-        if time.time() - start_time >= timeout:
-            raise RuntimeError('timeout waiting for relation to be up')
-        time.sleep(0.1)
-
-
-def wait_for_page_contents(url, contents, timeout=120, validate=None):
-    if validate is None:
-        validate = operator.contains
-    start_time = time.time()
-    while True:
-        try:
-            stream = urllib2.urlopen(url)
-        except (urllib2.HTTPError, urllib2.URLError):
-            pass
-        else:
-            page = stream.read()
-            if validate(page, contents):
-                return page
-        if time.time() - start_time >= timeout:
-            raise RuntimeError('timeout waiting for contents of ' + url)
-        time.sleep(0.1)

=== removed file 'hooks/hooks.py'
--- hooks/hooks.py	2015-08-11 11:15:27 +0000
+++ hooks/hooks.py	1970-01-01 00:00:00 +0000
@@ -1,2820 +0,0 @@
-#!/usr/bin/env python
-# vim: et ai ts=4 sw=4:
-
-from contextlib import contextmanager
-import commands
-import cPickle as pickle
-from distutils.version import StrictVersion
-import glob
-from grp import getgrnam
-import os
-from pwd import getpwnam
-import re
-import shutil
-import socket
-import subprocess
-import sys
-from tempfile import NamedTemporaryFile
-from textwrap import dedent
-import time
-import urlparse
-
-from charmhelpers import fetch
-from charmhelpers.core import hookenv, host
-from charmhelpers.core.hookenv import (
-    CRITICAL, ERROR, WARNING, INFO, DEBUG)
-
-try:
-    import psycopg2
-    from jinja2 import Template
-except ImportError:
-    fetch.apt_update(fatal=True)
-    fetch.apt_install(['python-psycopg2', 'python-jinja2'], fatal=True)
-    import psycopg2
-    from jinja2 import Template
-
-from psycopg2.extensions import AsIs
-from jinja2 import Environment, FileSystemLoader
-
-
-hooks = hookenv.Hooks()
-
-
-def log(msg, lvl=INFO):
-    '''Log a message.
-
-    Per Bug #1208787, log messages sent via juju-log are being lost.
-    Spit messages out to a log file to work around the problem.
-    It is also rather nice to have the log messages we explicitly emit
-    in a separate log file, rather than just mashed up with all the
-    juju noise.
-    '''
-    myname = hookenv.local_unit().replace('/', '-')
-    ts = time.strftime('%Y-%m-%d %H:%M:%S', time.gmtime())
-    with open('{}/{}-debug.log'.format(juju_log_dir, myname), 'a') as f:
-        f.write('{} {}: {}\n'.format(ts, lvl, msg))
-    hookenv.log(msg, lvl)
-
-
-def pg_version():
-    '''Return pg_version to use.
-
-    Return "version" config item if set, else use version from "postgresql"
-    package candidate, saving it in local_state for later.
-    '''
-    config_data = hookenv.config()
-    if 'pg_version' in local_state:
-        version = local_state['pg_version']
-    elif 'version' in config_data:
-        version = config_data['version']
-    else:
-        log("map version from distro release ...")
-        version_map = {'precise': '9.1',
-                       'trusty': '9.3'}
-        version = version_map.get(distro_codename())
-        if not version:
-            log("No PG version map for distro_codename={}, "
-                "you'll need to explicitly set it".format(distro_codename()),
-                CRITICAL)
-            sys.exit(1)
-        log("version={} from distro_codename='{}'".format(
-            version, distro_codename()))
-        # save it for later
-        local_state.setdefault('pg_version', version)
-        local_state.save()
-
-    assert version, "pg_version couldn't find a version to use"
-    return version
-
-
-def distro_codename():
-    """Return the distro release code name, eg. 'precise' or 'trusty'."""
-    return host.lsb_release()['DISTRIB_CODENAME']
-
-
-def render_template(template_name, vars):
-    # deferred import so install hook can install jinja2
-    templates_dir = os.path.join(os.environ['CHARM_DIR'], 'templates')
-    template_env = Environment(loader=FileSystemLoader(templates_dir))
-    template = template_env.get_template(template_name)
-    return template.render(vars)
-
-
-class State(dict):
-    """Encapsulate state common to the unit for republishing to relations."""
-    def __init__(self, state_file):
-        super(State, self).__init__()
-        self._state_file = state_file
-        self.load()
-
-    def load(self):
-        '''Load stored state from local disk.'''
-        if os.path.exists(self._state_file):
-            state = pickle.load(open(self._state_file, 'rb'))
-        else:
-            state = {}
-        self.clear()
-
-        self.update(state)
-
-    def save(self):
-        '''Store state to local disk.'''
-        state = {}
-        state.update(self)
-        old_mask = os.umask(0o077)  # This file contains database passwords!
-        try:
-            pickle.dump(state, open(self._state_file, 'wb'))
-        finally:
-            os.umask(old_mask)
-
-    def publish(self):
-        """Publish relevant unit state to relations"""
-
-        def add(state_dict, key):
-            if key in self:
-                state_dict[key] = self[key]
-
-        client_state = {}
-        add(client_state, 'state')
-
-        for relid in hookenv.relation_ids('db'):
-            hookenv.relation_set(relid, client_state)
-
-        for relid in hookenv.relation_ids('db-admin'):
-            hookenv.relation_set(relid, client_state)
-
-        replication_state = dict(client_state)
-
-        add(replication_state, 'replication_password')
-        add(replication_state, 'port')
-        add(replication_state, 'wal_received_offset')
-        add(replication_state, 'following')
-        add(replication_state, 'client_relations')
-
-        authorized = self.get('authorized', None)
-        if authorized:
-            replication_state['authorized'] = ' '.join(sorted(authorized))
-
-        for relid in hookenv.relation_ids('replication'):
-            hookenv.relation_set(relid, replication_state)
-
-        for relid in hookenv.relation_ids('master'):
-            hookenv.relation_set(relid, state=self.get('state'))
-
-        log('saving local state', DEBUG)
-        self.save()
-
-
-def volume_get_all_mounted():
-    command = ("mount |egrep %s" % external_volume_mount)
-    status, output = commands.getstatusoutput(command)
-    if status != 0:
-        return None
-    return output
-
-
-def postgresql_autostart(enabled):
-    postgresql_config_dir = _get_postgresql_config_dir()
-    startup_file = os.path.join(postgresql_config_dir, 'start.conf')
-    if enabled:
-        log("Enabling PostgreSQL startup in {}".format(startup_file))
-        mode = 'auto'
-    else:
-        log("Disabling PostgreSQL startup in {}".format(startup_file))
-        mode = 'manual'
-    template_file = "{}/templates/start_conf.tmpl".format(hookenv.charm_dir())
-    contents = Template(open(template_file).read()).render({'mode': mode})
-    host.write_file(
-        startup_file, contents, 'postgres', 'postgres', perms=0o644)
-
-
-def run(command, exit_on_error=True, quiet=False):
-    '''Run a command and return the output.'''
-    if not quiet:
-        log("Running {!r}".format(command), DEBUG)
-    p = subprocess.Popen(
-        command, stdin=subprocess.PIPE, stdout=subprocess.PIPE,
-        shell=isinstance(command, basestring))
-    p.stdin.close()
-    lines = []
-    for line in p.stdout:
-        if line:
-            # LP:1274460 & LP:1259490 mean juju-log is no where near as
-            # useful as we would like, so just shove a copy of the
-            # output to stdout for logging.
-            # log("> {}".format(line), DEBUG)
-            if not quiet:
-                print line
-            lines.append(line)
-        elif p.poll() is not None:
-            break
-
-    p.wait()
-
-    if p.returncode == 0:
-        return '\n'.join(lines)
-
-    if p.returncode != 0 and exit_on_error:
-        log("ERROR: {}".format(p.returncode), ERROR)
-        sys.exit(p.returncode)
-
-    raise subprocess.CalledProcessError(p.returncode, command,
-                                        '\n'.join(lines))
-
-
-def postgresql_is_running():
-    '''Return true if PostgreSQL is running.'''
-    for version, name, _, status in lsclusters(slice(4)):
-        if (version, name) == (pg_version(), hookenv.config('cluster_name')):
-            if 'online' in status.split(','):
-                log('PostgreSQL is running', DEBUG)
-                return True
-            else:
-                log('PostgreSQL is not running', DEBUG)
-                return False
-    assert False, 'Cluster {} {} not found'.format(
-        pg_version(), hookenv.config('cluster_name'))
-
-
-def postgresql_stop():
-    '''Shutdown PostgreSQL.'''
-    if postgresql_is_running():
-        run([
-            'pg_ctlcluster', '--force',
-            pg_version(), hookenv.config('cluster_name'), 'stop'])
-        log('PostgreSQL shut down')
-
-
-def postgresql_start():
-    '''Start PostgreSQL if it is not already running.'''
-    if not postgresql_is_running():
-        run([
-            'pg_ctlcluster', pg_version(),
-            hookenv.config('cluster_name'), 'start'])
-        log('PostgreSQL started')
-
-
-def postgresql_restart():
-    '''Restart PostgreSQL, or start it if it is not already running.'''
-    if postgresql_is_running():
-        with restart_lock(hookenv.local_unit(), True):
-            run([
-                'pg_ctlcluster', '--force',
-                pg_version(), hookenv.config('cluster_name'), 'restart'])
-            log('PostgreSQL restarted')
-    else:
-        postgresql_start()
-
-    assert postgresql_is_running()
-
-    # Store a copy of our known live configuration so
-    # postgresql_reload_or_restart() can make good choices.
-    if 'saved_config' in local_state:
-        local_state['live_config'] = local_state['saved_config']
-        local_state.save()
-
-
-def postgresql_reload():
-    '''Make PostgreSQL reload its configuration.'''
-    # reload returns a reliable exit status
-    if postgresql_is_running():
-        # I'm using the PostgreSQL function to avoid as much indirection
-        # as possible.
-        success = run_select_as_postgres('SELECT pg_reload_conf()')[1][0][0]
-        assert success, 'Failed to reload PostgreSQL configuration'
-        log('PostgreSQL configuration reloaded')
-    return postgresql_start()
-
-
-def requires_restart():
-    '''Check for configuration changes requiring a restart to take effect.'''
-    if not postgresql_is_running():
-        return True
-
-    saved_config = local_state.get('saved_config', None)
-    if not saved_config:
-        log("No record of postgresql.conf state. Better restart.")
-        return True
-
-    live_config = local_state.setdefault('live_config', {})
-
-    # Pull in a list of PostgreSQL settings.
-    cur = db_cursor()
-    cur.execute("SELECT name, context FROM pg_settings")
-    restart = False
-    for name, context in cur.fetchall():
-        live_value = live_config.get(name, None)
-        new_value = saved_config.get(name, None)
-
-        if new_value != live_value:
-            if live_config:
-                log("Changed {} from {!r} to {!r}".format(
-                    name, live_value, new_value), DEBUG)
-            if context == 'postmaster':
-                # A setting has changed that requires PostgreSQL to be
-                # restarted before it will take effect.
-                restart = True
-                log('{} changed from {} to {}. Restart required.'.format(
-                    name, live_value, new_value), DEBUG)
-    return restart
-
-
-def postgresql_reload_or_restart():
-    """Reload PostgreSQL configuration, restarting if necessary."""
-    if requires_restart():
-        log("Configuration change requires PostgreSQL restart", WARNING)
-        postgresql_restart()
-        assert not requires_restart(), "Configuration changes failed to apply"
-    else:
-        postgresql_reload()
-
-    local_state['saved_config'] = local_state['live_config']
-    local_state.save()
-
-
-def get_service_port():
-    '''Return the port PostgreSQL is listening on.'''
-    for version, name, port in lsclusters(slice(3)):
-        if (version, name) == (pg_version(), hookenv.config('cluster_name')):
-            return int(port)
-
-    assert False, 'No port found for {!r} {!r}'.format(
-        pg_version(), hookenv.config['cluster_name'])
-
-
-def lsclusters(s=slice(0, -1)):
-    for line in run('pg_lsclusters', quiet=True).splitlines()[1:]:
-        if line:
-            yield line.split()[s]
-
-
-def createcluster():
-    with switch_cwd('/tmp'):  # Ensure cwd is readable as the postgres user
-        create_cmd = [
-            "pg_createcluster",
-            "--locale", hookenv.config('locale'),
-            "-e", hookenv.config('encoding')]
-        if hookenv.config('listen_port'):
-            create_cmd.extend(["-p", str(hookenv.config('listen_port'))])
-        version = pg_version()
-        create_cmd.append(version)
-        create_cmd.append(hookenv.config('cluster_name'))
-
-        # With 9.3+, we make an opinionated decision to always enable
-        # data checksums. This seems to be best practice. We could
-        # turn this into a configuration item if there is need. There
-        # is no way to enable this option on existing clusters.
-        if StrictVersion(version) >= StrictVersion('9.3'):
-            create_cmd.extend(['--', '--data-checksums'])
-
-        run(create_cmd)
-        # Ensure SSL certificates exist, as we enable SSL by default.
-        create_ssl_cert(os.path.join(
-            postgresql_data_dir, pg_version(), hookenv.config('cluster_name')))
-
-
-def _get_system_ram():
-    """ Return the system ram in Megabytes """
-    import psutil
-    return psutil.phymem_usage()[0] / (1024 ** 2)
-
-
-def _get_page_size():
-    """ Return the operating system's configured PAGE_SIZE """
-    return int(run("getconf PAGE_SIZE"))   # frequently 4096
-
-
-def _run_sysctl(postgresql_sysctl):
-    """sysctl -p postgresql_sysctl, helper for easy test mocking."""
-    # Do not error out when this fails. It is not likely to work under LXC.
-    return run("sysctl -p {}".format(postgresql_sysctl), exit_on_error=False)
-
-
-def create_postgresql_config(config_file):
-    '''Create the postgresql.conf file'''
-    config_data = hookenv.config()
-    if not config_data.get('listen_port', None):
-        config_data['listen_port'] = get_service_port()
-    if config_data["performance_tuning"].lower() != "manual":
-        total_ram = _get_system_ram()
-        config_data["kernel_shmmax"] = (int(total_ram) * 1024 * 1024) + 1024
-        config_data["kernel_shmall"] = config_data["kernel_shmmax"]
-
-    # XXX: This is very messy - should probably be a subordinate charm
-    lines = ["kernel.sem = 250 32000 100 1024\n"]
-    if config_data["kernel_shmall"] > 0:
-        # Convert config kernel_shmall (bytes) to pages
-        page_size = _get_page_size()
-        num_pages = config_data["kernel_shmall"] / page_size
-        if (config_data["kernel_shmall"] % page_size) > 0:
-            num_pages += 1
-        lines.append("kernel.shmall = %s\n" % num_pages)
-    if config_data["kernel_shmmax"] > 0:
-        lines.append("kernel.shmmax = %s\n" % config_data["kernel_shmmax"])
-    host.write_file(postgresql_sysctl, ''.join(lines), perms=0600)
-    _run_sysctl(postgresql_sysctl)
-
-    # Our config file specifies a default wal_level that only works
-    # with PostgreSQL 9.4. Downgrade this for earlier versions of
-    # PostgreSQL. We have this default so more things Just Work.
-    if pg_version() < '9.4' and config_data['wal_level'] == 'logical':
-        config_data['wal_level'] = 'hot_standby'
-
-    # If we are replicating, some settings may need to be overridden to
-    # certain minimum levels.
-    num_slaves = slave_count()
-    if num_slaves > 0:
-        log('{} hot standbys in peer relation.'.format(num_slaves))
-        log('Ensuring minimal replication settings')
-        config_data['hot_standby'] = True
-        if config_data['wal_level'] != 'logical':
-            config_data['wal_level'] = 'hot_standby'
-        config_data['wal_keep_segments'] = max(
-            config_data['wal_keep_segments'],
-            config_data['replicated_wal_keep_segments'])
-        # We need this set even if config_data['streaming_replication']
-        # is False, because the replication connection is still needed
-        # by pg_basebackup to build a hot standby.
-        config_data['max_wal_senders'] = max(
-            num_slaves, config_data['max_wal_senders'])
-
-    # Log shipping to Swift using SwiftWAL. This could be for
-    # non-streaming replication, or for PITR.
-    if config_data.get('swiftwal_log_shipping', None):
-        config_data['archive_mode'] = True
-        if config_data['wal_level'] != 'logical':
-            config_data['wal_level'] = 'hot_standby'
-        config_data['archive_command'] = swiftwal_archive_command()
-
-    if config_data.get('wal_e_storage_uri', None):
-        config_data['archive_mode'] = True
-        if config_data['wal_level'] != 'logical':
-            config_data['wal_level'] = 'hot_standby'
-        config_data['archive_command'] = wal_e_archive_command()
-
-    # Send config data to the template
-    # Return it as pg_config
-    charm_dir = hookenv.charm_dir()
-    template_file = "{}/templates/postgresql.conf.tmpl".format(charm_dir)
-    if not config_data.get('version', None):
-        config_data['version'] = pg_version()
-    pg_config = Template(
-        open(template_file).read()).render(config_data)
-    host.write_file(
-        config_file, pg_config,
-        owner="postgres", group="postgres", perms=0600)
-
-    # Create or update files included from postgresql.conf.
-    configure_log_destination(os.path.dirname(config_file))
-
-    tune_postgresql_config(config_file)
-
-    local_state['saved_config'] = dict(config_data)
-    local_state.save()
-
-
-def tune_postgresql_config(config_file):
-    tune_workload = hookenv.config('performance_tuning').lower()
-    if tune_workload == "manual":
-        return  # Requested no autotuning.
-
-    if tune_workload == "auto":
-        tune_workload = "mixed"  # Pre-pgtune backwards compatibility.
-
-    with NamedTemporaryFile() as tmp_config:
-        run(['pgtune', '-i', config_file, '-o', tmp_config.name,
-             '-T', tune_workload,
-             '-c', str(hookenv.config('max_connections'))])
-        host.write_file(
-            config_file, open(tmp_config.name, 'r').read(),
-            owner='postgres', group='postgres', perms=0o600)
-
-
-def create_postgresql_ident(output_file):
-    '''Create the pg_ident.conf file.'''
-    ident_data = {}
-    charm_dir = hookenv.charm_dir()
-    template_file = "{}/templates/pg_ident.conf.tmpl".format(charm_dir)
-    pg_ident_template = Template(open(template_file).read())
-    host.write_file(
-        output_file, pg_ident_template.render(ident_data),
-        owner="postgres", group="postgres", perms=0600)
-
-
-def generate_postgresql_hba(
-        output_file, user=None, schema_user=None, database=None):
-    '''Create the pg_hba.conf file.'''
-
-    # Per Bug #1117542, when generating the postgresql_hba file we
-    # need to cope with private-address being either an IP address
-    # or a hostname.
-    def munge_address(addr):
-        # http://stackoverflow.com/q/319279/196832
-        try:
-            socket.inet_aton(addr)
-            return "%s/32" % addr
-        except socket.error:
-            # It's not an IP address.
-            # XXX workaround for MAAS bug
-            # https://bugs.launchpad.net/maas/+bug/1250435
-            # If it's a CNAME, use the A record it points to.
-            # If it fails for some reason, return the original address
-            try:
-                output = run("dig +short -t CNAME %s" % addr, True).strip()
-            except:
-                return addr
-            if len(output) != 0:
-                return output.rstrip(".")  # trailing dot
-            return addr
-
-    config_data = hookenv.config()
-    allowed_units = set()
-    relation_data = []
-    relids = hookenv.relation_ids('db') + hookenv.relation_ids('db-admin')
-    for relid in relids:
-        local_relation = hookenv.relation_get(
-            unit=hookenv.local_unit(), rid=relid)
-
-        # We might see relations that have not yet been setup enough.
-        # At a minimum, the relation-joined hook needs to have been run
-        # on the server so we have information about the usernames and
-        # databases to allow in.
-        if 'user' not in local_relation:
-            continue
-
-        for unit in hookenv.related_units(relid):
-            relation = hookenv.relation_get(unit=unit, rid=relid)
-
-            relation['relation-id'] = relid
-            relation['unit'] = unit
-
-            if relid.startswith('db-admin:'):
-                relation['user'] = 'all'
-                relation['database'] = 'all'
-            elif relid.startswith('db:'):
-                relation['user'] = local_relation.get('user', user)
-                relation['schema_user'] = local_relation.get('schema_user',
-                                                             schema_user)
-                relation['database'] = local_relation.get('database', database)
-
-                if ((relation['user'] is None
-                     or relation['schema_user'] is None
-                     or relation['database'] is None)):
-                    # Missing info in relation for this unit, so skip it.
-                    continue
-            else:
-                raise RuntimeError(
-                    'Unknown relation type {}'.format(repr(relid)))
-
-            allowed_units.add(unit)
-            relation['private-address'] = munge_address(
-                relation['private-address'])
-            relation_data.append(relation)
-
-    log(str(relation_data), INFO)
-
-    # Replication connections. Each unit needs to be able to connect to
-    # every other unit's postgres database and the magic replication
-    # database. It also needs to be able to connect to its own postgres
-    # database.
-    relids = hookenv.relation_ids('replication')
-    for relid in relids:
-        for unit in hookenv.related_units(relid):
-            relation = hookenv.relation_get(unit=unit, rid=relid)
-            remote_addr = munge_address(relation['private-address'])
-            remote_replication = {'database': 'replication',
-                                  'user': 'juju_replication',
-                                  'private-address': remote_addr,
-                                  'relation-id': relid,
-                                  'unit': unit,
-                                  }
-            relation_data.append(remote_replication)
-            remote_pgdb = {'database': 'postgres',
-                           'user': 'juju_replication',
-                           'private-address': remote_addr,
-                           'relation-id': relid,
-                           'unit': unit,
-                           }
-            relation_data.append(remote_pgdb)
-
-    # More replication connections, this time from external services.
-    # Somewhat different than before, as we do not share credentials
-    # and services using 9.4's logical replication feature will want
-    # to specify the database name.
-    relids = hookenv.relation_ids('master')
-    for relid in relids:
-        for unit in hookenv.related_units(relid):
-            remote_rel = hookenv.relation_get(unit=unit, rid=relid)
-            local_rel = hookenv.relation_get(unit=hookenv.local_unit(),
-                                             rid=relid)
-            remote_addr = munge_address(remote_rel['private-address'])
-            remote_replication = {'database': 'replication',
-                                  'user': local_rel['user'],
-                                  'private-address': remote_addr,
-                                  'relation-id': relid,
-                                  'unit': unit,
-                                  }
-            relation_data.append(remote_replication)
-            if 'database' in local_rel:
-                remote_pgdb = {'database': local_rel['database'],
-                            'user': local_rel['user'],
-                            'private-address': remote_addr,
-                            'relation-id': relid,
-                            'unit': unit,
-                            }
-                relation_data.append(remote_pgdb)
-
-    # Local hooks also need permissions to setup replication.
-    for relid in hookenv.relation_ids('replication'):
-        local_replication = {'database': 'postgres',
-                             'user': 'juju_replication',
-                             'private-address': munge_address(
-                                 hookenv.unit_private_ip()),
-                             'relation-id': relid,
-                             'unit': hookenv.local_unit(),
-                             }
-        relation_data.append(local_replication)
-
-    # Admin IP addresses for people using tools like pgAdminIII in a local JuJu
-    # We accept a single IP or a comma separated list of IPs, these are added
-    # to the list of relations that end up in pg_hba.conf thus granting
-    # the IP addresses socket access to the postgres server.
-    if config_data["admin_addresses"] != '':
-        if "," in config_data["admin_addresses"]:
-            admin_ip_list = config_data["admin_addresses"].split(",")
-        else:
-            admin_ip_list = [config_data["admin_addresses"]]
-
-        for admin_ip in admin_ip_list:
-            admin_host = {
-                'database': 'all',
-                'user': 'all',
-                'private-address': munge_address(admin_ip)}
-            relation_data.append(admin_host)
-
-    extra_pg_auth = [pg_auth.strip() for pg_auth in
-                     config_data["extra_pg_auth"].split(',') if pg_auth]
-
-    template_file = "{}/templates/pg_hba.conf.tmpl".format(hookenv.charm_dir())
-    pg_hba_template = Template(open(template_file).read())
-    pg_hba_rendered = pg_hba_template.render(extra_pg_auth=extra_pg_auth,
-                                             access_list=relation_data)
-    host.write_file(
-        output_file, pg_hba_rendered,
-        owner="postgres", group="postgres", perms=0600)
-    postgresql_reload()
-
-    # Loop through all db relations, making sure each knows what are the list
-    # of allowed hosts that were just added. lp:#1187508
-    # We sort the list to ensure stability, probably unnecessarily.
-    for relid in hookenv.relation_ids('db') + hookenv.relation_ids('db-admin'):
-        hookenv.relation_set(
-            relid, {"allowed-units": " ".join(unit_sorted(allowed_units))})
-
-
-def install_postgresql_crontab(output_file):
-    '''Create the postgres user's crontab'''
-    config_data = hookenv.config()
-    config_data['scripts_dir'] = postgresql_scripts_dir
-    config_data['swiftwal_backup_command'] = swiftwal_backup_command()
-    config_data['swiftwal_prune_command'] = swiftwal_prune_command()
-    config_data['wal_e_backup_command'] = wal_e_backup_command()
-    config_data['wal_e_prune_command'] = wal_e_prune_command()
-
-    charm_dir = hookenv.charm_dir()
-    template_file = "{}/templates/postgres.cron.tmpl".format(charm_dir)
-    crontab_template = Template(
-        open(template_file).read()).render(config_data)
-    host.write_file(output_file, crontab_template, perms=0600)
-
-
-def create_recovery_conf(master_host, master_port, restart_on_change=False):
-    if hookenv.config('manual_replication'):
-        log('manual_replication; should not be here', CRITICAL)
-        raise RuntimeError('manual_replication; should not be here')
-
-    version = pg_version()
-    cluster_name = hookenv.config('cluster_name')
-    postgresql_cluster_dir = os.path.join(
-        postgresql_data_dir, version, cluster_name)
-
-    recovery_conf_path = os.path.join(postgresql_cluster_dir, 'recovery.conf')
-    if os.path.exists(recovery_conf_path):
-        old_recovery_conf = open(recovery_conf_path, 'r').read()
-    else:
-        old_recovery_conf = None
-
-    charm_dir = hookenv.charm_dir()
-    streaming_replication = hookenv.config('streaming_replication')
-    template_file = "{}/templates/recovery.conf.tmpl".format(charm_dir)
-    params = dict(
-        host=master_host, port=master_port,
-        password=local_state['replication_password'],
-        streaming_replication=streaming_replication)
-    if hookenv.config('wal_e_storage_uri'):
-        params['restore_command'] = wal_e_restore_command()
-    elif hookenv.config('swiftwal_log_shipping'):
-        params['restore_command'] = swiftwal_restore_command()
-    recovery_conf = Template(open(template_file).read()).render(params)
-    log(recovery_conf, DEBUG)
-    host.write_file(
-        os.path.join(postgresql_cluster_dir, 'recovery.conf'),
-        recovery_conf, owner="postgres", group="postgres", perms=0o600)
-
-    if restart_on_change and old_recovery_conf != recovery_conf:
-        log("recovery.conf updated. Restarting to take effect.")
-        postgresql_restart()
-
-
-def ensure_swift_container(container):
-    from swiftclient import client as swiftclient
-    config = hookenv.config()
-    con = swiftclient.Connection(
-        authurl=config.get('os_auth_url', ''),
-        user=config.get('os_username', ''),
-        key=config.get('os_password', ''),
-        tenant_name=config.get('os_tenant_name', ''),
-        auth_version='2.0',
-        retries=0)
-    try:
-        con.head_container(container)
-    except swiftclient.ClientException:
-        con.put_container(container)
-
-
-def wal_e_envdir():
-    '''The envdir(1) environment location used to drive WAL-E.'''
-    return os.path.join(_get_postgresql_config_dir(), 'wal-e.env')
-
-
-def create_wal_e_envdir():
-    '''Regenerate the envdir(1) environment used to drive WAL-E.'''
-    config = hookenv.config()
-    env = dict(
-        SWIFT_AUTHURL=config.get('os_auth_url', ''),
-        SWIFT_TENANT=config.get('os_tenant_name', ''),
-        SWIFT_USER=config.get('os_username', ''),
-        SWIFT_PASSWORD=config.get('os_password', ''),
-        AWS_ACCESS_KEY_ID=config.get('aws_access_key_id', ''),
-        AWS_SECRET_ACCESS_KEY=config.get('aws_secret_access_key', ''),
-        WABS_ACCOUNT_NAME=config.get('wabs_account_name', ''),
-        WABS_ACCESS_KEY=config.get('wabs_access_key', ''),
-        WALE_SWIFT_PREFIX='',
-        WALE_S3_PREFIX='',
-        WALE_WABS_PREFIX='')
-
-    uri = config.get('wal_e_storage_uri', None)
-
-    if uri:
-        # Until juju provides us with proper leader election, we have a
-        # state where units do not know if they are alone or part of a
-        # cluster. To avoid units stomping on each others WAL and backups,
-        # we use a unique container for each unit when they are not
-        # part of the peer relation. Once they are part of the peer
-        # relation, they share a container.
-        if local_state.get('state', 'standalone') == 'standalone':
-            if not uri.endswith('/'):
-                uri += '/'
-            uri += hookenv.local_unit().split('/')[-1]
-
-        parsed_uri = urlparse.urlparse(uri)
-
-        required_env = []
-        if parsed_uri.scheme == 'swift':
-            env['WALE_SWIFT_PREFIX'] = uri
-            required_env = ['SWIFT_AUTHURL', 'SWIFT_TENANT',
-                            'SWIFT_USER', 'SWIFT_PASSWORD']
-            ensure_swift_container(parsed_uri.netloc)
-        elif parsed_uri.scheme == 's3':
-            env['WALE_S3_PREFIX'] = uri
-            required_env = ['AWS_ACCESS_KEY_ID', 'AWS_SECRET_ACCESS_KEY']
-        elif parsed_uri.scheme == 'wabs':
-            env['WALE_WABS_PREFIX'] = uri
-            required_env = ['WABS_ACCOUNT_NAME', 'WABS_ACCESS_KEY']
-        else:
-            log('Invalid wal_e_storage_uri {}'.format(uri), ERROR)
-
-        for env_key in required_env:
-            if not env[env_key].strip():
-                log('Missing {}'.format(env_key), ERROR)
-
-    # Regenerate the envdir(1) environment recommended by WAL-E.
-    # All possible keys are rewritten to ensure we remove old secrets.
-    host.mkdir(wal_e_envdir(), 'postgres', 'postgres', 0o750)
-    for k, v in env.items():
-        host.write_file(
-            os.path.join(wal_e_envdir(), k), v.strip(),
-            'postgres', 'postgres', 0o640)
-
-
-def wal_e_archive_command():
-    '''Return the archive_command needed in postgresql.conf.'''
-    return 'envdir {} wal-e wal-push %p'.format(wal_e_envdir())
-
-
-def wal_e_restore_command():
-    return 'envdir {} wal-e wal-fetch "%f" "%p"'.format(wal_e_envdir())
-
-
-def wal_e_backup_command():
-    postgresql_cluster_dir = os.path.join(
-        postgresql_data_dir, pg_version(), hookenv.config('cluster_name'))
-    return 'envdir {} wal-e backup-push {}'.format(
-        wal_e_envdir(), postgresql_cluster_dir)
-
-
-def wal_e_prune_command():
-    return 'envdir {} wal-e delete --confirm retain {}'.format(
-        wal_e_envdir(), hookenv.config('wal_e_backup_retention'))
-
-
-def swiftwal_config():
-    postgresql_config_dir = _get_postgresql_config_dir()
-    return os.path.join(postgresql_config_dir, "swiftwal.conf")
-
-
-def create_swiftwal_config():
-    if not hookenv.config('swiftwal_container_prefix'):
-        return
-
-    # Until juju provides us with proper leader election, we have a
-    # state where units do not know if they are alone or part of a
-    # cluster. To avoid units stomping on each others WAL and backups,
-    # we use a unique Swift container for each unit when they are not
-    # part of the peer relation. Once they are part of the peer
-    # relation, they share a container.
-    if local_state.get('state', 'standalone') == 'standalone':
-        container = '{}_{}'.format(hookenv.config('swiftwal_container_prefix'),
-                                   hookenv.local_unit().split('/')[-1])
-    else:
-        container = hookenv.config('swiftwal_container_prefix')
-
-    template_file = os.path.join(hookenv.charm_dir(),
-                                 'templates', 'swiftwal.conf.tmpl')
-    params = dict(hookenv.config())
-    params['swiftwal_container'] = container
-    content = Template(open(template_file).read()).render(params)
-    host.write_file(swiftwal_config(), content, "postgres", "postgres", 0o600)
-
-
-def swiftwal_archive_command():
-    '''Return the archive_command needed in postgresql.conf'''
-    return 'swiftwal --config={} archive-wal %p'.format(swiftwal_config())
-
-
-def swiftwal_restore_command():
-    '''Return the restore_command needed in recovery.conf'''
-    return 'swiftwal --config={} restore-wal %f %p'.format(swiftwal_config())
-
-
-def swiftwal_backup_command():
-    '''Return the backup command needed in postgres' crontab'''
-    cmd = 'swiftwal --config={} backup --port={}'.format(swiftwal_config(),
-                                                         get_service_port())
-    if not hookenv.config('swiftwal_log_shipping'):
-        cmd += ' --xlog'
-    return cmd
-
-
-def swiftwal_prune_command():
-    '''Return the backup & wal pruning command needed in postgres' crontab'''
-    config = hookenv.config()
-    args = '--keep-backups={} --keep-wals={}'.format(
-        config.get('swiftwal_backup_retention', 0),
-        max(config['wal_keep_segments'],
-            config['replicated_wal_keep_segments']))
-    return 'swiftwal --config={} prune {}'.format(swiftwal_config(), args)
-
-
-def update_service_port():
-    old_port = local_state.get('listen_port', None)
-    new_port = get_service_port()
-    if old_port != new_port:
-        if new_port:
-            hookenv.open_port(new_port)
-        if old_port:
-            hookenv.close_port(old_port)
-        local_state['listen_port'] = new_port
-        local_state.save()
-
-
-def create_ssl_cert(cluster_dir):
-    # PostgreSQL expects SSL certificates in the datadir.
-    server_crt = os.path.join(cluster_dir, 'server.crt')
-    server_key = os.path.join(cluster_dir, 'server.key')
-    if not os.path.exists(server_crt):
-        os.symlink('/etc/ssl/certs/ssl-cert-snakeoil.pem',
-                   server_crt)
-    if not os.path.exists(server_key):
-        os.symlink('/etc/ssl/private/ssl-cert-snakeoil.key',
-                   server_key)
-
-
-def set_password(user, password):
-    if not os.path.isdir("passwords"):
-        os.makedirs("passwords")
-    old_umask = os.umask(0o077)
-    try:
-        with open("passwords/%s" % user, "w") as pwfile:
-            pwfile.write(password)
-    finally:
-        os.umask(old_umask)
-
-
-def get_password(user):
-    try:
-        with open("passwords/%s" % user) as pwfile:
-            return pwfile.read()
-    except IOError:
-        return None
-
-
-def db_cursor(autocommit=False, db='postgres', user='postgres',
-              host=None, port=None, timeout=30):
-    if port is None:
-        port = get_service_port()
-    if host:
-        conn_str = "dbname={} host={} port={} user={}".format(
-            db, host, port, user)
-    else:
-        conn_str = "dbname={} port={} user={}".format(db, port, user)
-    # There are often race conditions in opening database connections,
-    # such as a reload having just happened to change pg_hba.conf
-    # settings or a hot standby being restarted and needing to catch up
-    # with its master. To protect our automation against these sorts of
-    # race conditions, by default we always retry failed connections
-    # until a timeout is reached.
-    start = time.time()
-    while True:
-        try:
-            with pgpass():
-                conn = psycopg2.connect(conn_str)
-            break
-        except psycopg2.Error, x:
-            if time.time() > start + timeout:
-                log("Database connection {!r} failed".format(
-                    conn_str), CRITICAL)
-                raise
-            log("Unable to open connection ({}), retrying.".format(x))
-            time.sleep(1)
-    conn.autocommit = autocommit
-    return conn.cursor()
-
-
-def run_sql_as_postgres(sql, *parameters):
-    cur = db_cursor(autocommit=True)
-    try:
-        cur.execute(sql, parameters)
-        return cur.statusmessage
-    except psycopg2.ProgrammingError:
-        log(sql, CRITICAL)
-        raise
-
-
-def run_select_as_postgres(sql, *parameters):
-    cur = db_cursor()
-    cur.execute(sql, parameters)
-    # NB. Need to suck in the results before the rowcount is valid.
-    results = cur.fetchall()
-    return (cur.rowcount, results)
-
-
-def validate_config():
-    """
-    Sanity check charm configuration, aborting the script if
-    we have bogus config values or config changes the charm does not yet
-    (or cannot) support.
-    """
-    valid = True
-    config_data = hookenv.config()
-
-    version = config_data.get('version', None)
-    if version:
-        if version not in ('9.1', '9.2', '9.3', '9.4'):
-            valid = False
-            log("Invalid or unsupported version {!r} requested".format(
-                version), CRITICAL)
-
-    if config_data['cluster_name'] != 'main':
-        valid = False
-        log("Cluster names other than 'main' do not work per LP:1271835",
-            CRITICAL)
-
-    if config_data['listen_ip'] != '*':
-        valid = False
-        log("listen_ip values other than '*' do not work per LP:1271837",
-            CRITICAL)
-
-    valid_workloads = [
-        'dw', 'oltp', 'web', 'mixed', 'desktop', 'manual', 'auto']
-    requested_workload = config_data['performance_tuning'].lower()
-    if requested_workload not in valid_workloads:
-        valid = False
-        log('Invalid performance_tuning setting {}'.format(requested_workload),
-            CRITICAL)
-    if requested_workload == 'auto':
-        log("'auto' performance_tuning deprecated. Using 'mixed' tuning",
-            WARNING)
-
-    unchangeable_config = [
-        'locale', 'encoding', 'version', 'cluster_name', 'pgdg']
-
-    for name in unchangeable_config:
-        if config_data._prev_dict is not None and config_data.changed(name):
-            valid = False
-            log("Cannot change {!r} setting after install.".format(name))
-        local_state[name] = config_data.get(name, None)
-    local_state.save()
-
-    package_status = config_data['package_status']
-    if package_status not in ['install', 'hold']:
-        valid = False
-        log("package_status must be 'install' or 'hold' not '{}'"
-            "".format(package_status), CRITICAL)
-
-    if not valid:
-        sys.exit(99)
-
-
-def ensure_package_status(package, status):
-    selections = ''.join(['{} {}\n'.format(package, status)])
-    dpkg = subprocess.Popen(
-        ['dpkg', '--set-selections'], stdin=subprocess.PIPE)
-    dpkg.communicate(input=selections)
-
-
-# -----------------------------------------------------------------------------
-# Core logic for permanent storage changes:
-# NOTE the only 2 "True" return points:
-#   1) symlink already pointing to existing storage (no-op)
-#   2) new storage properly initialized:
-#     - if fresh new storage dir: rsync existing data
-#     - manipulate /var/lib/postgresql/VERSION/CLUSTER symlink
-# -----------------------------------------------------------------------------
-def config_changed_volume_apply(mount_point):
-    version = pg_version()
-    cluster_name = hookenv.config('cluster_name')
-    data_directory_path = os.path.join(
-        postgresql_data_dir, version, cluster_name)
-
-    assert(data_directory_path)
-
-    if not os.path.exists(data_directory_path):
-        log(
-            "postgresql data dir {} not found, "
-            "not applying changes.".format(data_directory_path),
-            CRITICAL)
-        return False
-
-    new_pg_dir = os.path.join(mount_point, "postgresql")
-    new_pg_version_cluster_dir = os.path.join(
-        new_pg_dir, version, cluster_name)
-    if not mount_point:
-        log(
-            "invalid mount point = {}, "
-            "not applying changes.".format(mount_point), ERROR)
-        return False
-
-    if ((os.path.islink(data_directory_path) and
-         os.readlink(data_directory_path) == new_pg_version_cluster_dir and
-         os.path.isdir(new_pg_version_cluster_dir))):
-        log(
-            "postgresql data dir '{}' already points "
-            "to {}, skipping storage changes.".format(
-                data_directory_path, new_pg_version_cluster_dir))
-        log(
-            "existing-symlink: to fix/avoid UID changes from "
-            "previous units, doing: "
-            "chown -R postgres:postgres {}".format(new_pg_dir))
-        run("chown -R postgres:postgres %s" % new_pg_dir)
-        return True
-
-    # Create a directory structure below "new" mount_point as
-    #   external_volume_mount/postgresql/9.1/main
-    for new_dir in [new_pg_dir,
-                    os.path.join(new_pg_dir, version),
-                    new_pg_version_cluster_dir]:
-        if not os.path.isdir(new_dir):
-            log("mkdir %s".format(new_dir))
-            host.mkdir(new_dir, owner="postgres", perms=0o700)
-    # Carefully build this symlink, e.g.:
-    # /var/lib/postgresql/9.1/main ->
-    # external_volume_mount/postgresql/9.1/main
-    # but keep previous "main/"  directory, by renaming it to
-    # main-$TIMESTAMP
-    if not postgresql_stop() and postgresql_is_running():
-        log("postgresql_stop() failed - can't migrate data.", ERROR)
-        return False
-    if not os.path.exists(os.path.join(
-            new_pg_version_cluster_dir, "PG_VERSION")):
-        log("migrating PG data {}/ -> {}/".format(
-            data_directory_path, new_pg_version_cluster_dir), WARNING)
-        # void copying PID file to perm storage (shouldn't be any...)
-        command = "rsync -a --exclude postmaster.pid {}/ {}/".format(
-            data_directory_path, new_pg_version_cluster_dir)
-        log("run: {}".format(command))
-        run(command)
-    try:
-        os.rename(data_directory_path, "{}-{}".format(
-            data_directory_path, int(time.time())))
-        log("NOTICE: symlinking {} -> {}".format(
-            new_pg_version_cluster_dir, data_directory_path))
-        os.symlink(new_pg_version_cluster_dir, data_directory_path)
-        run("chown -h postgres:postgres {}".format(data_directory_path))
-        log(
-            "after-symlink: to fix/avoid UID changes from "
-            "previous units, doing: "
-            "chown -R postgres:postgres {}".format(new_pg_dir))
-        run("chown -R postgres:postgres {}".format(new_pg_dir))
-        return True
-    except OSError:
-        log("failed to symlink {} -> {}".format(
-            data_directory_path, mount_point), CRITICAL)
-        return False
-
-
-def reset_manual_replication_state():
-    '''In manual replication mode, the state of the local database cluster
-    is outside of Juju's control. We need to detect and update the charm
-    state to match reality.
-    '''
-    if hookenv.config('manual_replication'):
-        if os.path.exists('recovery.conf'):
-            local_state['state'] = 'hot standby'
-        elif slave_count():
-            local_state['state'] = 'master'
-        else:
-            local_state['state'] = 'standalone'
-        local_state.publish()
-
-
[e-mail]()
-def config_changed(force_restart=False, mount_point=None):
-    validate_config()
-    config_data = hookenv.config()
-    update_repos_and_packages()
-
-    if mount_point is not None:
-        # config_changed_volume_apply will stop the service if it finds
-        # it necessary, ie: new volume setup
-        if config_changed_volume_apply(mount_point=mount_point):
-            postgresql_autostart(True)
-        else:
-            postgresql_autostart(False)
-            postgresql_stop()
-            mounts = volume_get_all_mounted()
-            if mounts:
-                log("current mounted volumes: {}".format(mounts))
-            log(
-                "Disabled and stopped postgresql service "
-                "(config_changed_volume_apply failure)", ERROR)
-            sys.exit(1)
-
-    reset_manual_replication_state()
-
-    postgresql_config_dir = _get_postgresql_config_dir(config_data)
-    postgresql_config = os.path.join(postgresql_config_dir, "postgresql.conf")
-    postgresql_hba = os.path.join(postgresql_config_dir, "pg_hba.conf")
-    postgresql_ident = os.path.join(postgresql_config_dir, "pg_ident.conf")
-
-    create_postgresql_config(postgresql_config)
-    create_postgresql_ident(postgresql_ident)  # Do this before pg_hba.conf.
-    generate_postgresql_hba(postgresql_hba)
-    create_ssl_cert(os.path.join(
-        postgresql_data_dir, pg_version(), config_data['cluster_name']))
-    create_swiftwal_config()
-    create_wal_e_envdir()
-    update_service_port()
-    update_nrpe_checks()
-    write_metrics_cronjob('/usr/local/bin/postgres_to_statsd.py',
-                          '/etc/cron.d/postgres_metrics')
-
-    # If an external mountpoint has caused an old, existing DB to be
-    # mounted, we need to ensure that all the users, databases, roles
-    # etc. exist with known passwords.
-    if local_state['state'] in ('standalone', 'master'):
-        client_relids = (
-            hookenv.relation_ids('db') + hookenv.relation_ids('db-admin'))
-        for relid in client_relids:
-            rel = hookenv.relation_get(rid=relid, unit=hookenv.local_unit())
-            client_rel = None
-            for unit in hookenv.related_units(relid):
-                client_rel = hookenv.relation_get(unit=unit, rid=relid)
-            if not client_rel:
-                continue  # No client units - in between departed and broken?
-
-            database = rel.get('database')
-            if database is None:
-                continue  # The relation exists, but we haven't joined it yet.
-
-            roles = filter(None, (client_rel.get('roles') or '').split(","))
-            user = rel.get('user')
-            if user:
-                admin = relid.startswith('db-admin')
-                password = create_user(user, admin=admin)
-                reset_user_roles(user, roles)
-                hookenv.relation_set(relid, password=password)
-
-            schema_user = rel.get('schema_user')
-            if schema_user:
-                schema_password = create_user(schema_user)
-                hookenv.relation_set(relid, schema_password=schema_password)
-
-            if user and schema_user and not (
-                    database is None or database == 'all'):
-                ensure_database(user, schema_user, database)
-
-    if force_restart:
-        postgresql_restart()
-    postgresql_reload_or_restart()
-
-    # In case the log_line_prefix has changed, inform syslog consumers.
-    for relid in hookenv.relation_ids('syslog'):
-        hookenv.relation_set(
-            relid, log_line_prefix=hookenv.config('log_line_prefix'))
-
-
[e-mail]()
-def install(run_pre=True, force_restart=True):
-    if run_pre:
-        for f in glob.glob('exec.d/*/charm-pre-install'):
-            if os.path.isfile(f) and os.access(f, os.X_OK):
-                subprocess.check_call(['sh', '-c', f])
-
-    validate_config()
-
-    config_data = hookenv.config()
-    update_repos_and_packages()
-    if 'state' not in local_state:
-        log('state not in {}'.format(local_state.keys()), DEBUG)
-        # Fresh installation. Because this function is invoked by both
-        # the install hook and the upgrade-charm hook, we need to guard
-        # any non-idempotent setup. We should probably fix this; it
-        # seems rather fragile.
-        local_state.setdefault('state', 'standalone')
-        log(repr(local_state.keys()), DEBUG)
-
-        # Drop the cluster created when the postgresql package was
-        # installed, and rebuild it with the requested locale and encoding.
-        version = pg_version()
-        for ver, name in lsclusters(slice(2)):
-            if version == ver and name == 'main':
-                run("pg_dropcluster --stop {} main".format(version))
-        listen_port = config_data.get('listen_port', None)
-        if listen_port:
-            port_opt = "--port={}".format(config_data['listen_port'])
-        else:
-            port_opt = ''
-        createcluster()
-        assert (
-            not port_opt
-            or get_service_port() == config_data['listen_port']), (
-            'allocated port {!r} != {!r}'.format(
-                get_service_port(), config_data['listen_port']))
-        local_state['port'] = get_service_port()
-        log('publishing state', DEBUG)
-        local_state.publish()
-
-    postgresql_backups_dir = (
-        config_data['backup_dir'].strip() or
-        os.path.join(postgresql_data_dir, 'backups'))
-
-    host.mkdir(postgresql_backups_dir, owner="postgres", perms=0o755)
-    host.mkdir(postgresql_scripts_dir, owner="postgres", perms=0o755)
-    host.mkdir(postgresql_logs_dir, owner="postgres", perms=0o755)
-    paths = {
-        'base_dir': postgresql_data_dir,
-        'backup_dir': postgresql_backups_dir,
-        'scripts_dir': postgresql_scripts_dir,
-        'logs_dir': postgresql_logs_dir,
-    }
-    charm_dir = hookenv.charm_dir()
-    template_file = "{}/templates/pg_backup_job.tmpl".format(charm_dir)
-    backup_job = Template(open(template_file).read()).render(paths)
-    host.write_file(
-        os.path.join(postgresql_scripts_dir, 'dump-pg-db'),
-        open('scripts/pgbackup.py', 'r').read(), perms=0o755)
-    host.write_file(
-        os.path.join(postgresql_scripts_dir, 'pg_backup_job'),
-        backup_job, perms=0755)
-    install_postgresql_crontab(postgresql_crontab)
-
-    # Create this empty log file on installation to avoid triggering
-    # spurious monitoring system alerts, per Bug #1329816.
-    if not os.path.exists(backup_log):
-        host.write_file(backup_log, '', 'postgres', 'postgres', 0664)
-
-    hookenv.open_port(get_service_port())
-
-    # Ensure at least minimal access granted for hooks to run.
-    # Reload because we are using the default cluster setup and started
-    # when we installed the PostgreSQL packages.
-    config_changed(force_restart=force_restart)
-
-    snapshot_relations()
-
-
[e-mail]()
-def upgrade_charm():
-    """Handle saving state during an upgrade-charm hook.
-
-    When upgrading from an installation using volume-map, we migrate
-    that installation to use the storage subordinate charm by remounting
-    a mountpath that the storage subordinate maintains. We exit(1) only to
-    raise visibility to manual procedure that we log in juju logs below for the
-    juju admin to finish the migration by relating postgresql to the storage
-    and block-storage-broker services. These steps are generalised in the
-    README as well.
-    """
-    install(run_pre=False, force_restart=False)
-    snapshot_relations()
-    version = pg_version()
-    cluster_name = hookenv.config('cluster_name')
-    data_directory_path = os.path.join(
-        postgresql_data_dir, version, cluster_name)
-    if (os.path.islink(data_directory_path)):
-        link_target = os.readlink(data_directory_path)
-        if "/srv/juju" in link_target:
-            # Then we just upgraded from an installation that was using
-            # charm config volume_map definitions. We need to stop postgresql
-            # and remount the device where the storage subordinate expects to
-            # control the mount in the future if relations/units change
-            volume_id = link_target.split("/")[3]
-            unit_name = hookenv.local_unit()
-            new_mount_root = external_volume_mount
-            new_pg_version_cluster_dir = os.path.join(
-                new_mount_root, "postgresql", version, cluster_name)
-            if not os.exists(new_mount_root):
-                os.mkdir(new_mount_root)
-            log("\n"
-                "WARNING: %s unit has external volume id %s mounted via the\n"
-                "deprecated volume-map and volume-ephemeral-storage\n"
-                "configuration parameters.\n"
-                "These parameters are no longer available in the postgresql\n"
-                "charm in favor of using the volume_map parameter in the\n"
-                "storage subordinate charm.\n"
-                "We are migrating the attached volume to a mount path which\n"
-                "can be managed by the storage subordinate charm. To\n"
-                "continue using this volume_id with the storage subordinate\n"
-                "follow this procedure.\n-----------------------------------\n"
-                "1. cat > storage.cfg <<EOF\nstorage:\n"
-                "  provider: block-storage-broker\n"
-                "  root: %s\n"
-                "  volume_map: \"{%s: %s}\"\nEOF\n2. juju deploy "
-                "--config storage.cfg storage\n"
-                "3. juju deploy block-storage-broker\n4. juju add-relation "
-                "block-storage-broker storage\n5. juju resolved --retry "
-                "%s\n6. juju add-relation postgresql storage\n"
-                "-----------------------------------\n" %
-                (unit_name, volume_id, new_mount_root, unit_name, volume_id,
-                 unit_name), WARNING)
-            postgresql_stop()
-            os.unlink(data_directory_path)
-            log("Unmounting external storage due to charm upgrade: %s" %
-                link_target)
-            try:
-                subprocess.check_output(
-                    "umount /srv/juju/%s" % volume_id, shell=True)
-                # Since e2label truncates labels to 16 characters use only the
-                # first 16 characters of the volume_id as that's what was
-                # set by old versions of postgresql charm
-                subprocess.check_call(
-                    "mount -t ext4 LABEL=%s %s" %
-                    (volume_id[:16], new_mount_root), shell=True)
-            except subprocess.CalledProcessError, e:
-                log("upgrade-charm mount migration failed. %s" % str(e), ERROR)
-                sys.exit(1)
-
-            log("NOTICE: symlinking {} -> {}".format(
-                new_pg_version_cluster_dir, data_directory_path))
-            os.symlink(new_pg_version_cluster_dir, data_directory_path)
-            run("chown -h postgres:postgres {}".format(data_directory_path))
-            postgresql_start()  # Will exit(1) if issues
-            log("Remount and restart success for this external volume.\n"
-                "This current running installation will break upon\n"
-                "add/remove postgresql units or relations if you do not\n"
-                "follow the above procedure to ensure your external\n"
-                "volumes are preserved by the storage subordinate charm.",
-                WARNING)
-            # So juju admins can see the hook fail and note the steps to fix
-            # per our WARNINGs above
-            sys.exit(1)
-
-
[e-mail]()
-def start():
-    postgresql_reload_or_restart()
-
-
[e-mail]()
-def stop():
-    if postgresql_is_running():
-        with restart_lock(hookenv.local_unit(), True):
-            postgresql_stop()
-
-
-def quote_identifier(identifier):
-    r'''Quote an identifier, such as a table or role name.
-
-    In SQL, identifiers are quoted using " rather than ' (which is reserved
-    for strings).
-
-    >>> print(quote_identifier('hello'))
-    "hello"
-
-    Quotes and Unicode are handled if you make use of them in your
-    identifiers.
-
-    >>> print(quote_identifier("'"))
-    "'"
-    >>> print(quote_identifier('"'))
-    """"
-    >>> print(quote_identifier("\\"))
-    "\"
-    >>> print(quote_identifier('\\"'))
-    "\"""
-    >>> print(quote_identifier('\\ aargh \u0441\u043b\u043e\u043d'))
-    U&"\\ aargh \0441\043b\043e\043d"
-    '''
-    try:
-        return '"%s"' % identifier.encode('US-ASCII').replace('"', '""')
-    except UnicodeEncodeError:
-        escaped = []
-        for c in identifier:
-            if c == '\\':
-                escaped.append('\\\\')
-            elif c == '"':
-                escaped.append('""')
-            else:
-                c = c.encode('US-ASCII', 'backslashreplace')
-                # Note Python only supports 32 bit unicode, so we use
-                # the 4 hexdigit PostgreSQL syntax (\1234) rather than
-                # the 6 hexdigit format (\+123456).
-                if c.startswith('\\u'):
-                    c = '\\' + c[2:]
-                escaped.append(c)
-        return 'U&"%s"' % ''.join(escaped)
-
-
-def sanitize(s):
-    s = s.replace(':', '_')
-    s = s.replace('-', '_')
-    s = s.replace('/', '_')
-    s = s.replace('"', '_')
-    s = s.replace("'", '_')
-    return s
-
-
-def user_name(relid, remote_unit, admin=False, schema=False):
-    # Per Bug #1160530, don't append the remote unit number to the user name.
-    components = [sanitize(relid), sanitize(re.split("/", remote_unit)[0])]
-    if admin:
-        components.append("admin")
-    elif schema:
-        components.append("schema")
-    return "_".join(components)
-
-
-def user_exists(user):
-    sql = "SELECT rolname FROM pg_roles WHERE rolname = %s"
-    if run_select_as_postgres(sql, user)[0] > 0:
-        return True
-    else:
-        return False
-
-
-def create_user(user, admin=False, replication=False):
-    password = get_password(user)
-    if password is None:
-        password = host.pwgen()
-        set_password(user, password)
-    if user_exists(user):
-        log("Updating {} user".format(user))
-        action = ["ALTER ROLE"]
-    else:
-        log("Creating {} user".format(user))
-        action = ["CREATE ROLE"]
-    action.append('%s WITH LOGIN')
-    if admin:
-        action.append('SUPERUSER')
-    else:
-        action.append('NOSUPERUSER')
-    if replication:
-        action.append('REPLICATION')
-    else:
-        action.append('NOREPLICATION')
-    action.append('PASSWORD %s')
-    sql = ' '.join(action)
-    run_sql_as_postgres(sql, AsIs(quote_identifier(user)), password)
-    return password
-
-
-def reset_user_roles(user, roles):
-    wanted_roles = set(roles)
-
-    sql = """
-        SELECT role.rolname
-        FROM
-            pg_roles AS role,
-            pg_roles AS member,
-            pg_auth_members
-        WHERE
-            member.oid = pg_auth_members.member
-            AND role.oid = pg_auth_members.roleid
-            AND member.rolname = %s
-        """
-    existing_roles = set(r[0] for r in run_select_as_postgres(sql, user)[1])
-
-    roles_to_grant = wanted_roles.difference(existing_roles)
-
-    for role in roles_to_grant:
-        ensure_role(role)
-
-    if roles_to_grant:
-        log("Granting {} to {}".format(",".join(roles_to_grant), user), INFO)
-
-    for role in roles_to_grant:
-        run_sql_as_postgres(
-            "GRANT %s TO %s",
-            AsIs(quote_identifier(role)), AsIs(quote_identifier(user)))
-
-    roles_to_revoke = existing_roles.difference(wanted_roles)
-
-    if roles_to_revoke:
-        log("Revoking {} from {}".format(",".join(roles_to_grant), user), INFO)
-
-    for role in roles_to_revoke:
-        run_sql_as_postgres(
-            "REVOKE %s FROM %s",
-            AsIs(quote_identifier(role)), AsIs(quote_identifier(user)))
-
-
-def ensure_role(role):
-    sql = "SELECT oid FROM pg_roles WHERE rolname = %s"
-    if run_select_as_postgres(sql, role)[0] == 0:
-        sql = "CREATE ROLE %s INHERIT NOLOGIN"
-        run_sql_as_postgres(sql, AsIs(quote_identifier(role)))
-
-
-def ensure_database(user, schema_user, database):
-    sql = "SELECT datname FROM pg_database WHERE datname = %s"
-    if run_select_as_postgres(sql, database)[0] != 0:
-        # DB already exists
-        pass
-    else:
-        sql = "CREATE DATABASE %s"
-        run_sql_as_postgres(sql, AsIs(quote_identifier(database)))
-    sql = "GRANT ALL PRIVILEGES ON DATABASE %s TO %s"
-    run_sql_as_postgres(sql, AsIs(quote_identifier(database)),
-                        AsIs(quote_identifier(schema_user)))
-    sql = "GRANT CONNECT ON DATABASE %s TO %s"
-    run_sql_as_postgres(sql, AsIs(quote_identifier(database)),
-                        AsIs(quote_identifier(user)))
-
-
-def ensure_extensions(extensions, database):
-    if extensions:
-        cur = db_cursor(db=database, autocommit=True)
-        try:
-            cur.execute('SELECT extname FROM pg_extension')
-            installed_extensions = frozenset(x[0] for x in cur.fetchall())
-            log("ensure_extensions({}), have {}"
-                .format(extensions, installed_extensions),
-                DEBUG)
-            extensions_set = frozenset(extensions)
-            extensions_to_create = \
-                extensions_set.difference(installed_extensions)
-            for ext in extensions_to_create:
-                log("creating extension {}".format(ext), DEBUG)
-                cur.execute('CREATE EXTENSION %s',
-                            (AsIs(quote_identifier(ext)),))
-        finally:
-            cur.close()
-
-
-def snapshot_relations():
-    '''Snapshot our relation information into local state.
-
-    We need this information to be available in -broken
-    hooks letting us actually clean up properly. Bug #1190996.
-    '''
-    log("Snapshotting relations", DEBUG)
-    local_state['relations'] = hookenv.relations()
-    local_state.save()
-
-
-# Each database unit needs to publish connection details to the
-# client. This is problematic, because 1) the user and database are
-# only created on the master unit and this is replicated to the
-# slave units outside of juju and 2) we have no control over the
-# order that units join the relation.
-#
-# The simplest approach of generating usernames and passwords in
-# the master units db-relation-joined hook fails because slave
-# units may well have already run their hooks and found no
-# connection details to republish. When the master unit publishes
-# the connection details it only triggers relation-changed hooks
-# on the client units, not the relation-changed hook on other peer
-# units.
-#
-# A more complex approach is for the first database unit that joins
-# the relation to generate the usernames and passwords and publish
-# this to the relation. Subsequent units can retrieve this
-# information and republish it. Of course, the master unit also
-# creates the database and users when it joins the relation.
-# This approach should work reliably on the server side. However,
-# there is a window from when a slave unit joins a client relation
-# until the master unit has joined that relation when the
-# credentials published by the slave unit are invalid. These
-# credentials will only become valid after the master unit has
-# actually created the user and database.
-#
-# The implemented approach is for the master unit's
-# db-relation-joined hook to create the user and database and
-# publish the connection details, and in addition update a list
-# of active relations to the service's peer 'replication' relation.
-# After the master unit has updated the peer relationship, the
-# slave unit's peer replication-relation-changed hook will
-# be triggered and it will have an opportunity to republish the
-# connection details. Of course, it may not be able to do so if the
-# slave unit's db-relation-joined hook has yet been run, so we must
-# also attempt to to republish the connection settings there.
-# This way we are guaranteed at least one chance to republish the
-# connection details after the database and user have actually been
-# created and both the master and slave units have joined the
-# relation.
-#
-# The order of relevant hooks firing may be:
-#
-# master db-relation-joined (publish)
-# slave db-relation-joined (republish)
-# slave replication-relation-changed (noop)
-#
-# slave db-relation-joined (noop)
-# master db-relation-joined (publish)
-# slave replication-relation-changed (republish)
-#
-# master db-relation-joined (publish)
-# slave replication-relation-changed (noop; slave not yet joined db rel)
-# slave db-relation-joined (republish)
-
-
[e-mail]('db-relation-joined', 'db-relation-changed')
-def db_relation_joined_changed():
-    reset_manual_replication_state()
-    if local_state['state'] == 'hot standby':
-        publish_hot_standby_credentials()
-        return
-
-    # By default, we create a database named after the remote
-    # servicename. The remote service can override this by setting
-    # the database property on the relation.
-    database = hookenv.relation_get('database')
-    if not database:
-        database = hookenv.remote_unit().split('/')[0]
-
-    # Generate a unique username for this relation to use.
-    user = user_name(hookenv.relation_id(), hookenv.remote_unit())
-
-    roles = filter(None, (hookenv.relation_get('roles') or '').split(","))
-
-    extensions = filter(None,
-                        (hookenv.relation_get('extensions') or '').split(","))
-
-    log('{} unit publishing credentials'.format(local_state['state']))
-
-    password = create_user(user)
-    reset_user_roles(user, roles)
-    schema_user = "{}_schema".format(user)
-    schema_password = create_user(schema_user)
-    ensure_database(user, schema_user, database)
-    ensure_extensions(extensions, database)
-    host = hookenv.unit_private_ip()
-    port = get_service_port()
-    state = local_state['state']  # master, hot standby, standalone
-
-    # Publish connection details.
-    connection_settings = dict(
-        user=user, password=password,
-        schema_user=schema_user, schema_password=schema_password,
-        host=host, database=database, port=port, state=state)
-    log("Connection settings {!r}".format(connection_settings), DEBUG)
-    hookenv.relation_set(relation_settings=connection_settings)
-
-    # Update the peer relation, notifying any hot standby units
-    # to republish connection details to the client relation.
-    local_state['client_relations'] = ' '.join(sorted(
-        hookenv.relation_ids('db') + hookenv.relation_ids('db-admin')))
-    log("Client relations {}".format(local_state['client_relations']))
-    local_state.publish()
-
-    postgresql_hba = os.path.join(_get_postgresql_config_dir(), "pg_hba.conf")
-    generate_postgresql_hba(postgresql_hba, user=user,
-                            schema_user=schema_user,
-                            database=database)
-
-    snapshot_relations()
-
-
[e-mail]('db-admin-relation-joined', 'db-admin-relation-changed')
-def db_admin_relation_joined_changed():
-    reset_manual_replication_state()
-    if local_state['state'] == 'hot standby':
-        publish_hot_standby_credentials()
-        return
-
-    user = user_name(
-        hookenv.relation_id(), hookenv.remote_unit(), admin=True)
-
-    log('{} unit publishing credentials'.format(local_state['state']))
-
-    password = create_user(user, admin=True)
-    host = hookenv.unit_private_ip()
-    port = get_service_port()
-    state = local_state['state']  # master, hot standby, standalone
-
-    # Publish connection details.
-    connection_settings = dict(
-        user=user, password=password,
-        host=host, database='all', port=port, state=state)
-    log("Connection settings {!r}".format(connection_settings), DEBUG)
-    hookenv.relation_set(relation_settings=connection_settings)
-
-    # Update the peer relation, notifying any hot standby units
-    # to republish connection details to the client relation.
-    local_state['client_relations'] = ' '.join(
-        hookenv.relation_ids('db') + hookenv.relation_ids('db-admin'))
-    log("Client relations {}".format(local_state['client_relations']))
-    local_state.publish()
-
-    postgresql_hba = os.path.join(_get_postgresql_config_dir(), "pg_hba.conf")
-    generate_postgresql_hba(postgresql_hba)
-
-    snapshot_relations()
-
-
[e-mail]()
-def db_relation_broken():
-    relid = hookenv.relation_id()
-    if relid not in local_state['relations']['db']:
-        # This was to be a hot standby, but it had not yet got as far as
-        # receiving and handling credentials from the master.
-        log("db-relation-broken called before relation finished setup", DEBUG)
-        return
-
-    # The relation no longer exists, so we can't pull the database name
-    # we used from there. Instead, we have to persist this information
-    # ourselves.
-    relation = local_state['relations']['db'][relid]
-    unit_relation_data = relation[hookenv.local_unit()]
-
-    if local_state['state'] in ('master', 'standalone'):
-        user = unit_relation_data.get('user', None)
-        database = unit_relation_data['database']
-
-        # We need to check that the database still exists before
-        # attempting to revoke privileges because the local PostgreSQL
-        # cluster may have been rebuilt by another hook.
-        sql = "SELECT datname FROM pg_database WHERE datname = %s"
-        if run_select_as_postgres(sql, database)[0] != 0:
-            sql = "REVOKE ALL PRIVILEGES ON DATABASE %s FROM %s"
-            run_sql_as_postgres(sql, AsIs(quote_identifier(database)),
-                                AsIs(quote_identifier(user)))
-            run_sql_as_postgres(sql, AsIs(quote_identifier(database)),
-                                AsIs(quote_identifier(user + "_schema")))
-
-    postgresql_hba = os.path.join(_get_postgresql_config_dir(), "pg_hba.conf")
-    generate_postgresql_hba(postgresql_hba)
-
-    # Cleanup our local state.
-    snapshot_relations()
-
-
[e-mail]()
-def db_admin_relation_broken():
-    if local_state['state'] in ('master', 'standalone'):
-        user = hookenv.relation_get('user', unit=hookenv.local_unit())
-        if user:
-            # We need to check that the user still exists before
-            # attempting to revoke privileges because the local PostgreSQL
-            # cluster may have been rebuilt by another hook.
-            sql = "SELECT usename FROM pg_user WHERE usename = %s"
-            if run_select_as_postgres(sql, user)[0] != 0:
-                sql = "ALTER USER %s NOSUPERUSER"
-                run_sql_as_postgres(sql, AsIs(quote_identifier(user)))
-
-    postgresql_hba = os.path.join(_get_postgresql_config_dir(), "pg_hba.conf")
-    generate_postgresql_hba(postgresql_hba)
-
-    # Cleanup our local state.
-    snapshot_relations()
-
-
-def update_repos_and_packages():
-    need_upgrade = False
-
-    version = pg_version()
-
-    # Add the PGDG APT repository if it is enabled. Setting this boolean
-    # is simpler than requiring the magic URL and key be added to
-    # install_sources and install_keys. In addition, per Bug #1271148,
-    # install_keys is likely a security hole for this sort of remote
-    # archive. Instead, we keep a copy of the signing key in the charm
-    # and can add it securely.
-    pgdg_list = '/etc/apt/sources.list.d/pgdg_{}.list'.format(
-        sanitize(hookenv.local_unit()))
-    pgdg_key = 'ACCC4CF8'
-
-    if hookenv.config('pgdg'):
-        if not os.path.exists(pgdg_list):
-            # We need to upgrade, as if we have Ubuntu main packages
-            # installed they may be incompatible with the PGDG ones.
-            # This is unlikely to ever happen outside of the test suite,
-            # and never if you don't reuse machines.
-            need_upgrade = True
-            run("apt-key add lib/{}.asc".format(pgdg_key))
-            open(pgdg_list, 'w').write('deb {} {}-pgdg main'.format(
-                'http://apt.postgresql.org/pub/repos/apt/', distro_codename()))
-        if version == '9.4':
-            pgdg_94_list = '/etc/apt/sources.list.d/pgdg_94_{}.list'.format(
-                sanitize(hookenv.local_unit()))
-            if not os.path.exists(pgdg_94_list):
-                need_upgrade = True
-                open(pgdg_94_list, 'w').write(
-                    'deb {} {}-pgdg main 9.4'.format(
-                        'http://apt.postgresql.org/pub/repos/apt/',
-                        distro_codename()))
-
-    elif os.path.exists(pgdg_list):
-        log(
-            "PGDG apt source not requested, but already in place in this "
-            "container", WARNING)
-        # We can't just remove a source, as we may have packages
-        # installed that conflict with ones from the other configured
-        # sources. In particular, if we have postgresql-common installed
-        # from the PGDG Apt source, PostgreSQL packages from Ubuntu main
-        # will fail to install.
-        # os.unlink(pgdg_list)
-
-    # Try to optimize our calls to fetch.configure_sources(), as it
-    # cannot do this itself due to lack of state.
-    if (need_upgrade
-        or local_state.get('install_sources', None)
-            != hookenv.config('install_sources')
-        or local_state.get('install_keys', None)
-            != hookenv.config('install_keys')):
-        # Support the standard mechanism implemented by charm-helpers. Pulls
-        # from the default 'install_sources' and 'install_keys' config
-        # options. This also does 'apt-get update', pulling in the PGDG data
-        # if we just configured it.
-        fetch.configure_sources(True)
-        local_state['install_sources'] = hookenv.config('install_sources')
-        local_state['install_keys'] = hookenv.config('install_keys')
-        local_state.save()
-
-    # Ensure that the desired database locale is possible.
-    if hookenv.config('locale') != 'C':
-        run(["locale-gen", "{}.{}".format(
-            hookenv.config('locale'), hookenv.config('encoding'))])
-
-    if need_upgrade:
-        run("apt-get -y upgrade")
-
-    # It might have been better for debversion and plpython to only get
-    # installed if they were listed in the extra-packages config item,
-    # but they predate this feature.
-    packages = ["python-psutil",  # to obtain system RAM from python
-                "libc-bin",       # for getconf
-                "postgresql-{}".format(version),
-                "postgresql-contrib-{}".format(version),
-                "postgresql-plpython-{}".format(version),
-                "python-jinja2", "python-psycopg2"]
-
-    # PGDG currently doesn't have debversion for 9.3 & 9.4. Put this back
-    # when it does.
-    if not (hookenv.config('pgdg') and version in ('9.3', '9.4')):
-        packages.append("postgresql-{}-debversion".format(version))
-
-    if hookenv.config('performance_tuning').lower() != 'manual':
-        packages.append('pgtune')
-
-    if hookenv.config('swiftwal_container_prefix'):
-        packages.append('swiftwal')
-
-    if hookenv.config('wal_e_storage_uri'):
-        packages.extend(['wal-e', 'daemontools'])
-
-    packages.extend((hookenv.config('extra-packages') or '').split())
-    packages = fetch.filter_installed_packages(packages)
-    # Set package state for main postgresql package if installed
-    if 'postgresql-{}'.format(version) not in packages:
-        ensure_package_status('postgresql-{}'.format(version),
-                              hookenv.config('package_status'))
-    fetch.apt_update(fatal=True)
-    fetch.apt_install(packages, fatal=True)
-
-
-@contextmanager
-def pgpass():
-    passwords = {}
-
-    # Replication.
-    # pg_basebackup only works with the password in .pgpass, or entered
-    # at the command prompt.
-    if 'replication_password' in local_state:
-        passwords['juju_replication'] = local_state['replication_password']
-
-    pgpass_contents = '\n'.join(
-        "*:*:*:{}:{}".format(username, password)
-        for username, password in passwords.items())
-    pgpass_file = NamedTemporaryFile()
-    pgpass_file.write(pgpass_contents)
-    pgpass_file.flush()
-    os.chown(pgpass_file.name, getpwnam('postgres').pw_uid, -1)
-    os.chmod(pgpass_file.name, 0o400)
-    org_pgpassfile = os.environ.get('PGPASSFILE', None)
-    os.environ['PGPASSFILE'] = pgpass_file.name
-    try:
-        yield pgpass_file.name
-    finally:
-        if org_pgpassfile is None:
-            del os.environ['PGPASSFILE']
-        else:
-            os.environ['PGPASSFILE'] = org_pgpassfile
-
-
-def authorized_by(unit):
-    '''Return True if the peer has authorized our database connections.'''
-    for relid in hookenv.relation_ids('replication'):
-        relation = hookenv.relation_get(unit=unit, rid=relid)
-        authorized = relation.get('authorized', '').split()
-        return hookenv.local_unit() in authorized
-
-
-def promote_database():
-    '''Take the database out of recovery mode.'''
-    config_data = hookenv.config()
-    version = pg_version()
-    cluster_name = config_data['cluster_name']
-    postgresql_cluster_dir = os.path.join(
-        postgresql_data_dir, version, cluster_name)
-    recovery_conf = os.path.join(postgresql_cluster_dir, 'recovery.conf')
-    if os.path.exists(recovery_conf):
-        # Rather than using 'pg_ctl promote', we do the promotion
-        # this way to avoid creating a timeline change. Switch this
-        # to using 'pg_ctl promote' once PostgreSQL propagates
-        # timeline changes via streaming replication.
-        os.unlink(recovery_conf)
-        postgresql_restart()
-
-
-def follow_database(master):
-    '''Connect the database as a streaming replica of the master.'''
-    master_relation = hookenv.relation_get(unit=master)
-    create_recovery_conf(
-        master_relation['private-address'],
-        master_relation['port'], restart_on_change=True)
-
-
-def elected_master():
-    """Return the unit that should be master, or None if we don't yet know."""
-    if local_state['state'] == 'master':
-        log("I am already the master", DEBUG)
-        return hookenv.local_unit()
-
-    if local_state['state'] == 'hot standby':
-        log("I am already following {}".format(
-            local_state['following']), DEBUG)
-        return local_state['following']
-
-    replication_relid = hookenv.relation_ids('replication')[0]
-    replication_units = hookenv.related_units(replication_relid)
-
-    if local_state['state'] == 'standalone':
-        log("I'm a standalone unit wanting to participate in replication")
-        existing_replication = False
-        for unit in replication_units:
-            # If another peer thinks it is the master, believe it.
-            remote_state = hookenv.relation_get(
-                'state', unit, replication_relid)
-            if remote_state == 'master':
-                log("{} thinks it is the master, believing it".format(
-                    unit), DEBUG)
-                return unit
-
-            # If we find a peer that isn't standalone, we know
-            # replication has already been setup at some point.
-            if remote_state != 'standalone':
-                existing_replication = True
-
-        # If we are joining a peer relation where replication has
-        # already been setup, but there is currently no master, wait
-        # until one of the remaining participating units has been
-        # promoted to master. Only they have the data we need to
-        # preserve.
-        if existing_replication:
-            log("Peers participating in replication need to elect a master",
-                DEBUG)
-            return None
-
-        # There are no peers claiming to be master, and there is no
-        # election in progress, so lowest numbered unit wins.
-        units = replication_units + [hookenv.local_unit()]
-        master = unit_sorted(units)[0]
-        if master == hookenv.local_unit():
-            log("I'm Master - lowest numbered unit in new peer group")
-            return master
-        else:
-            log("Waiting on {} to declare itself Master".format(master), DEBUG)
-            return None
-
-    if local_state['state'] == 'failover':
-        former_master = local_state['following']
-        log("Failover from {}".format(former_master))
-
-        units_not_in_failover = set()
-        candidates = set()
-        for unit in replication_units:
-            if unit == former_master:
-                log("Found dying master {}".format(unit), DEBUG)
-                continue
-
-            relation = hookenv.relation_get(unit=unit, rid=replication_relid)
-
-            if relation['state'] == 'master':
-                log("{} says it already won the election".format(unit),
-                    INFO)
-                return unit
-
-            if relation['state'] == 'failover':
-                candidates.add(unit)
-
-            elif relation['state'] != 'standalone':
-                units_not_in_failover.add(unit)
-
-        if units_not_in_failover:
-            log("{} unaware of impending election. Deferring result.".format(
-                " ".join(unit_sorted(units_not_in_failover))))
-            return None
-
-        log("Election in progress")
-        winner = None
-        winning_offset = -1
-        candidates.add(hookenv.local_unit())
-        # Sort the unit lists so we get consistent results in a tie
-        # and lowest unit number wins.
-        for unit in unit_sorted(candidates):
-            relation = hookenv.relation_get(unit=unit, rid=replication_relid)
-            if int(relation['wal_received_offset']) > winning_offset:
-                winner = unit
-                winning_offset = int(relation['wal_received_offset'])
-
-        # All remaining hot standbys are in failover mode and have
-        # reported their wal_received_offset. We can declare victory.
-        if winner == hookenv.local_unit():
-            log("I won the election, announcing myself winner")
-            return winner
-        else:
-            log("Waiting for {} to announce its victory".format(winner),
-                DEBUG)
-            return None
-
-
[e-mail]('replication-relation-joined', 'replication-relation-changed')
-def replication_relation_joined_changed():
-    config_changed()  # Ensure minimal replication settings.
-
-    # Now that pg_hba.conf has been regenerated and loaded, inform related
-    # units that they have been granted replication access.
-    authorized_units = set()
-    for unit in hookenv.related_units():
-        authorized_units.add(unit)
-    local_state['authorized'] = authorized_units
-
-    if hookenv.config('manual_replication'):
-        log('manual_replication, nothing to do')
-        return
-
-    master = elected_master()
-
-    # Handle state changes:
-    #  - Fresh install becoming the master
-    #  - Fresh install becoming a hot standby
-    #  - Hot standby being promoted to master
-
-    if master is None:
-        log("Master is not yet elected. Deferring.")
-
-    elif master == hookenv.local_unit():
-        if local_state['state'] != 'master':
-            log("I have elected myself master")
-            promote_database()
-            if 'following' in local_state:
-                del local_state['following']
-            if 'wal_received_offset' in local_state:
-                del local_state['wal_received_offset']
-            if 'paused_at_failover' in local_state:
-                del local_state['paused_at_failover']
-            local_state['state'] = 'master'
-
-            # Publish credentials to hot standbys so they can connect.
-            replication_password = create_user(
-                'juju_replication', replication=True)
-            local_state['replication_password'] = replication_password
-            local_state['client_relations'] = ' '.join(
-                hookenv.relation_ids('db') + hookenv.relation_ids('db-admin'))
-            local_state.publish()
-
-        else:
-            log("I am master and remain master")
-
-    elif not authorized_by(master):
-        log("I need to follow {} but am not yet authorized".format(master))
-
-    else:
-        log("Syncing replication_password from {}".format(master), DEBUG)
-        local_state['replication_password'] = hookenv.relation_get(
-            'replication_password', master)
-
-        if 'following' not in local_state:
-            log("Fresh unit. I will clone {} and become a hot standby".format(
-                master))
-
-            master_ip = hookenv.relation_get('private-address', master)
-            master_port = hookenv.relation_get('port', master)
-            assert master_port is not None, 'No master port set'
-
-            clone_database(master, master_ip, master_port)
-
-            local_state['state'] = 'hot standby'
-            local_state['following'] = master
-            if 'wal_received_offset' in local_state:
-                del local_state['wal_received_offset']
-
-        elif local_state['following'] == master:
-            log("I am a hot standby already following {}".format(master))
-
-            # Replication connection details may have changed, so
-            # ensure we are still following.
-            follow_database(master)
-
-        else:
-            log("I am a hot standby following new master {}".format(master))
-            follow_database(master)
-            if not local_state.get("paused_at_failover", None):
-                run_sql_as_postgres("SELECT pg_xlog_replay_resume()")
-            local_state['state'] = 'hot standby'
-            local_state['following'] = master
-            del local_state['wal_received_offset']
-            del local_state['paused_at_failover']
-
-        publish_hot_standby_credentials()
-        postgresql_hba = os.path.join(
-            _get_postgresql_config_dir(), "pg_hba.conf")
-        generate_postgresql_hba(postgresql_hba)
-
-    # Swift container name make have changed, so regenerate the SwiftWAL
-    # config. This can go away when we have real leader election and can
-    # safely share a single container.
-    create_swiftwal_config()
-    create_wal_e_envdir()
-
-    local_state.publish()
-
-
-def publish_hot_standby_credentials():
-    '''
-    If a hot standby joins a client relation before the master
-    unit, it is unable to publish connection details. However,
-    when the master does join it updates the client_relations
-    value in the peer relation causing the replication-relation-changed
-    hook to be invoked. This gives us a second opertunity to publish
-    connection details.
-
-    This function is invoked from both the client and peer
-    relation-changed hook. One of these will work depending on the order
-    the master and hot standby joined the client relation.
-    '''
-    master = local_state['following']
-    if not master:
-        log("I will be a hot standby, but no master yet")
-        return
-
-    if not authorized_by(master):
-        log("Master {} has not yet authorized us".format(master))
-        return
-
-    client_relations = hookenv.relation_get(
-        'client_relations', master, hookenv.relation_ids('replication')[0])
-
-    if client_relations is None:
-        log("Master {} has not yet joined any client relations".format(
-            master), DEBUG)
-        return
-
-    # Build the set of client relations that both the master and this
-    # unit have joined.
-    possible_client_relations = set(hookenv.relation_ids('db') +
-                                    hookenv.relation_ids('db-admin') +
-                                    hookenv.relation_ids('master'))
-    active_client_relations = possible_client_relations.intersection(
-        set(client_relations.split()))
-
-    for client_relation in active_client_relations:
-        # We need to pull the credentials from the master unit's
-        # end of the client relation.
-        log('Hot standby republishing credentials from {} to {}'.format(
-            master, client_relation))
-
-        connection_settings = hookenv.relation_get(
-            unit=master, rid=client_relation)
-
-        # Override unit specific connection details
-        connection_settings['host'] = hookenv.unit_private_ip()
-        connection_settings['port'] = get_service_port()
-        connection_settings['state'] = local_state['state']
-        requested_db = hookenv.relation_get('database')
-        # A hot standby might have seen a database name change before
-        # the master, so override. This is no problem because we block
-        # until this database has been created on the master and
-        # replicated through to this unit.
-        if requested_db:
-            connection_settings['database'] = requested_db
-
-        # Block until users and database has replicated, so we know the
-        # connection details we publish are actually valid. This will
-        # normally be pretty much instantaneous. Do not block if we are
-        # running in manual replication mode, as it is outside of juju's
-        # control when replication is actually setup and running.
-        if not hookenv.config('manual_replication'):
-            timeout = 60
-            start = time.time()
-            while time.time() < start + timeout:
-                cur = db_cursor(autocommit=True)
-                cur.execute('select datname from pg_database')
-                if cur.fetchone() is not None:
-                    break
-                del cur
-                log('Waiting for database {} to be replicated'.format(
-                    connection_settings['database']))
-                time.sleep(10)
-
-        log("Relation {} connection settings {!r}".format(
-            client_relation, connection_settings), DEBUG)
-        hookenv.relation_set(
-            client_relation, relation_settings=connection_settings)
-
-
[e-mail]()
-def replication_relation_departed():
-    '''A unit has left the replication peer group.'''
-    remote_unit = hookenv.remote_unit()
-
-    assert remote_unit is not None
-
-    log("{} has left the peer group".format(remote_unit))
-
-    # If we are the last unit standing, we become standalone
-    remaining_peers = set(hookenv.related_units(hookenv.relation_id()))
-    remaining_peers.discard(remote_unit)  # Bug #1192433
-
-    # True if we were following the departed unit.
-    following_departed = (local_state.get('following', None) == remote_unit)
-
-    if remaining_peers and not following_departed:
-        log("Remaining {}".format(local_state['state']))
-
-    elif remaining_peers and following_departed:
-        # If the unit being removed was our master, prepare for failover.
-        # We need to suspend replication to ensure that the replay point
-        # remains consistent throughout the election, and publish that
-        # replay point. Once all units have entered this steady state,
-        # we can identify the most up to date hot standby and promote it
-        # to be the new master.
-        log("Entering failover state")
-        cur = db_cursor(autocommit=True)
-        cur.execute("SELECT pg_is_xlog_replay_paused()")
-        already_paused = cur.fetchone()[0]
-        local_state["paused_at_failover"] = already_paused
-        if not already_paused:
-            cur.execute("SELECT pg_xlog_replay_pause()")
-        # Switch to failover state. Don't cleanup the 'following'
-        # setting because having access to the former master is still
-        # useful.
-        local_state['state'] = 'failover'
-        local_state['wal_received_offset'] = postgresql_wal_received_offset()
-
-    else:
-        log("Last unit standing. Switching from {} to standalone.".format(
-            local_state['state']))
-        promote_database()
-        local_state['state'] = 'standalone'
-        if 'following' in local_state:
-            del local_state['following']
-        if 'wal_received_offset' in local_state:
-            del local_state['wal_received_offset']
-        if 'paused_at_failover' in local_state:
-            del local_state['paused_at_failover']
-
-    config_changed()
-    local_state.publish()
-
-
[e-mail]()
-def replication_relation_broken():
-    # This unit has been removed from the service.
-    promote_database()
-    config_changed()
-
-
-@contextmanager
-def switch_cwd(new_working_directory):
-    org_dir = os.getcwd()
-    os.chdir(new_working_directory)
-    try:
-        yield new_working_directory
-    finally:
-        os.chdir(org_dir)
-
-
-@contextmanager
-def restart_lock(unit, exclusive):
-    '''Aquire the database restart lock on the given unit.
-
-    A database needing a restart should grab an exclusive lock before
-    doing so. To block a remote database from doing a restart, grab a shared
-    lock.
-    '''
-    key = long(hookenv.config('advisory_lock_restart_key'))
-    if exclusive:
-        lock_function = 'pg_advisory_lock'
-    else:
-        lock_function = 'pg_advisory_lock_shared'
-    q = 'SELECT {}({})'.format(lock_function, key)
-
-    # We will get an exception if the database is rebooted while waiting
-    # for a shared lock. If the connection is killed, we retry a few
-    # times to cope.
-    num_retries = 3
-
-    for count in range(0, num_retries):
-        try:
-            if unit == hookenv.local_unit():
-                cur = db_cursor(autocommit=True)
-            else:
-                host = hookenv.relation_get('private-address', unit)
-                port = hookenv.relation_get('port', unit)
-                cur = db_cursor(
-                    autocommit=True, db='postgres', user='juju_replication',
-                    host=host, port=port)
-            cur.execute(q)
-            break
-        except psycopg2.Error:
-            if count == num_retries - 1:
-                raise
-
-    try:
-        yield
-    finally:
-        # Close our connection, swallowing any exceptions as the database
-        # may be being rebooted now we have released our lock.
-        try:
-            del cur
-        except psycopg2.Error:
-            pass
-
-
-def clone_database(master_unit, master_host, master_port):
-    with restart_lock(master_unit, False):
-        postgresql_stop()
-        log("Cloning master {}".format(master_unit))
-
-        config_data = hookenv.config()
-        version = pg_version()
-        cluster_name = config_data['cluster_name']
-        postgresql_cluster_dir = os.path.join(
-            postgresql_data_dir, version, cluster_name)
-        postgresql_config_dir = _get_postgresql_config_dir(config_data)
-        cmd = [
-            'sudo', '-E',  # -E needed to locate pgpass file.
-            '-u', 'postgres', 'pg_basebackup', '-D', postgresql_cluster_dir,
-            '--xlog', '--checkpoint=fast', '--no-password',
-            '-h', master_host, '-p', master_port,
-            '--username=juju_replication']
-        log(' '.join(cmd), DEBUG)
-
-        if os.path.isdir(postgresql_cluster_dir):
-            shutil.rmtree(postgresql_cluster_dir)
-
-        try:
-            # Change directory the postgres user can read, and need
-            # .pgpass too.
-            with switch_cwd('/tmp'), pgpass():
-                # Clone the master with pg_basebackup.
-                output = subprocess.check_output(cmd, stderr=subprocess.STDOUT)
-            log(output, DEBUG)
-            # SSL certificates need to exist in the datadir.
-            create_ssl_cert(postgresql_cluster_dir)
-            create_recovery_conf(master_host, master_port)
-        except subprocess.CalledProcessError as x:
-            # We failed, and this cluster is broken. Rebuild a
-            # working cluster so start/stop etc. works and we
-            # can retry hooks again. Even assuming the charm is
-            # functioning correctly, the clone may still fail
-            # due to eg. lack of disk space.
-            log(x.output, ERROR)
-            log("Clone failed, local db destroyed", ERROR)
-            if os.path.exists(postgresql_cluster_dir):
-                shutil.rmtree(postgresql_cluster_dir)
-            if os.path.exists(postgresql_config_dir):
-                shutil.rmtree(postgresql_config_dir)
-            createcluster()
-            config_changed()
-            raise
-        finally:
-            postgresql_start()
-            wait_for_db()
-
-
-def slave_count():
-    num_slaves = 0
-    for relid in hookenv.relation_ids('replication'):
-        num_slaves += len(hookenv.related_units(relid))
-    for relid in hookenv.relation_ids('master'):
-        num_slaves += len(hookenv.related_units(relid))
-    return num_slaves
-
-
-def postgresql_is_in_backup_mode():
-    version = pg_version()
-    cluster_name = hookenv.config('cluster_name')
-    postgresql_cluster_dir = os.path.join(
-        postgresql_data_dir, version, cluster_name)
-
-    return os.path.exists(
-        os.path.join(postgresql_cluster_dir, 'backup_label'))
-
-
-def pg_basebackup_is_running():
-    cur = db_cursor(autocommit=True)
-    cur.execute("""
-        SELECT count(*) FROM pg_stat_activity
-        WHERE usename='juju_replication' AND application_name='pg_basebackup'
-        """)
-    return cur.fetchone()[0] > 0
-
-
-def postgresql_wal_received_offset():
-    """How much WAL we have.
-
-    WAL is replicated asynchronously from the master to hot standbys.
-    The more WAL a hot standby has received, the better a candidate it
-    makes for master during failover.
-
-    Note that this is not quite the same as how in sync the hot standby is.
-    That depends on how much WAL has been replayed. WAL is replayed after
-    it is received.
-    """
-    cur = db_cursor(autocommit=True)
-    cur.execute('SELECT pg_is_in_recovery(), pg_last_xlog_receive_location()')
-    is_in_recovery, xlog_received = cur.fetchone()
-    if is_in_recovery:
-        return wal_location_to_bytes(xlog_received)
-    return None
-
-
-def wal_location_to_bytes(wal_location):
-    """Convert WAL + offset to num bytes, so they can be compared."""
-    logid, offset = wal_location.split('/')
-    return int(logid, 16) * 16 * 1024 * 1024 * 255 + int(offset, 16)
-
-
-def wait_for_db(
-        timeout=120, db='postgres', user='postgres', host=None, port=None):
-    '''Wait until the db is fully up.'''
-    db_cursor(db=db, user=user, host=host, port=port, timeout=timeout)
-
-
-def unit_sorted(units):
-    """Return a sorted list of unit names."""
-    return sorted(
-        units, lambda a, b: cmp(int(a.split('/')[-1]), int(b.split('/')[-1])))
-
-
-def delete_metrics_cronjob(cron_path):
-    try:
-        os.unlink(cron_path)
-    except OSError:
-        pass
-
-
-def write_metrics_cronjob(script_path, cron_path):
-    config_data = hookenv.config()
-
-    # need the following two configs to be valid
-    metrics_target = config_data['metrics_target'].strip()
-    metrics_sample_interval = config_data['metrics_sample_interval']
-    if (not metrics_target
-            or ':' not in metrics_target
-            or not metrics_sample_interval):
-        log("Required config not found or invalid "
-            "(metrics_target, metrics_sample_interval), "
-            "disabling statsd metrics", DEBUG)
-        delete_metrics_cronjob(cron_path)
-        return
-
-    charm_dir = os.environ['CHARM_DIR']
-    statsd_host, statsd_port = metrics_target.split(':', 1)
-    metrics_prefix = config_data['metrics_prefix'].strip()
-    metrics_prefix = metrics_prefix.replace(
-        "$UNIT", hookenv.local_unit().replace('.', '-').replace('/', '-'))
-
-    # ensure script installed
-    charm_script = os.path.join(charm_dir, 'files', 'metrics',
-                                'postgres_to_statsd.py')
-    host.write_file(script_path, open(charm_script, 'rb').read(), perms=0755)
-
-    # write the crontab
-    with open(cron_path, 'w') as cronjob:
-        cronjob.write(render_template("metrics_cronjob.template", {
-            'interval': config_data['metrics_sample_interval'],
-            'script': script_path,
-            'metrics_prefix': metrics_prefix,
-            'metrics_sample_interval': metrics_sample_interval,
-            'statsd_host': statsd_host,
-            'statsd_port': statsd_port,
-        }))
-
-
[e-mail]('nrpe-external-master-relation-changed')
-def update_nrpe_checks():
-    config_data = hookenv.config()
-    try:
-        nagios_uid = getpwnam('nagios').pw_uid
-        nagios_gid = getgrnam('nagios').gr_gid
-    except Exception:
-        hookenv.log("Nagios user not set up.", hookenv.DEBUG)
-        return
-
-    try:
-        nagios_password = create_user('nagios')
-        pg_pass_entry = '*:*:*:nagios:%s' % (nagios_password)
-        with open('/var/lib/nagios/.pgpass', 'w') as target:
-            os.fchown(target.fileno(), nagios_uid, nagios_gid)
-            os.fchmod(target.fileno(), 0400)
-            target.write(pg_pass_entry)
-    except psycopg2.InternalError:
-        if config_data['manual_replication']:
-            log("update_nrpe_checks(): manual_replication: "
-                "ignoring psycopg2.InternalError caught creating 'nagios' "
-                "postgres role; assuming we're already replicating")
-        else:
-            raise
-
-    relids = hookenv.relation_ids('nrpe-external-master')
-    relations = []
-    for relid in relids:
-        for unit in hookenv.related_units(relid):
-            relations.append(hookenv.relation_get(unit=unit, rid=relid))
-
-    if len(relations) == 1 and 'nagios_hostname' in relations[0]:
-        nagios_hostname = relations[0]['nagios_hostname']
-        log("update_nrpe_checks: Obtained nagios_hostname ({}) "
-            "from nrpe-external-master relation.".format(nagios_hostname))
-    else:
-        unit = hookenv.local_unit()
-        unit_name = unit.replace('/', '-')
-        nagios_hostname = "%s-%s" % (config_data['nagios_context'], unit_name)
-        log("update_nrpe_checks: Deduced nagios_hostname ({}) from charm "
-            "config (nagios_hostname not found in nrpe-external-master "
-            "relation, or wrong number of relations "
-            "found)".format(nagios_hostname))
-
-    nrpe_service_file = \
-        '/var/lib/nagios/export/service__{}_check_pgsql.cfg'.format(
-            nagios_hostname)
-    nagios_logdir = '/var/log/nagios'
-    if not os.path.exists(nagios_logdir):
-        os.mkdir(nagios_logdir)
-        os.chown(nagios_logdir, nagios_uid, nagios_gid)
-    for f in os.listdir('/var/lib/nagios/export/'):
-        if re.search('.*check_pgsql.cfg', f):
-            os.remove(os.path.join('/var/lib/nagios/export/', f))