Merge lp:~stub/charms/precise/postgresql/integration into lp:charms/postgresql

[Stuart Bishop]

Various pending bug fixes and feature work.

Integration branch containing all of stub's PostgreSQL charm merge proposals currently in the review queue.

Separate, smaller merge proposals also exist, with each branch in the sequence dependent on the previous. This branch is here in case anyone wants to review everything at once.

[Review Queue (review-queue)]

The results (PASS) are in and available here: http://reports.vapour.ws/charm-tests/charm-bundle-test-960-results

review: Approve (cbt)

[Adam Israel (aisrael)]

Hi Stuart,

Thanks for all your work on the Postgresql charm! I've spent some time reviewing this cumulative MP against postgresql.

I should note that the above comment from review-queue is incorrect. The results did not pass automated testing.

There were a handful of errors running make lint. I fixed those and pushed the change to my personal namespace[1].

Moving on to the integration tests[2], I was able to get 19 tests to pass, with 14 failures.

I notice that a couple of the failed tests is attempting to execute `juju run --unit postgresql/1 relation-list -r replication:168`, however, relation-list only seems to be valid within the context of hook execution.

I had a great deal more failures initially, with deploy failing. I set JUJU_REPOSITORY and those tests were able to fun.

If you think it would be useful in debugging the failed tests, I can also include any juju/machine logs.

[1] lp:~aisrael/charms/precise/postgresql/fix-lint
[2] http://pastebin.ubuntu.com/8371905/

review: Needs Fixing

[Stuart Bishop (stub)]

'make lint' is clean when run on my trusty box. I did merge in your branch, but it introduces lint failures:

stub@belch:~/charms/postgresql/integration$ make lint
Lint check (flake8)
directory hooks
checking hooks/helpers.py
checking hooks/hooks.py
hooks/hooks.py:991:80: E501 line too long (80 > 79 characters)
hooks/hooks.py:998:80: E501 line too long (80 > 79 characters)
checking hooks/test_hooks.py
directory testing
checking testing/__init__.py
checking testing/jujufixture.py
directory tests
checking test.py
make: *** [lint] Error 1

I suspect you are running from Precise, with older versions of the lint checkers? Or a more modern version? Possibly more modern since one of the comments changed predates my maintainership. Apart from the trivially fixed long lines, your other changes look benign so I've merged it in anyway.

relation-list does work with 'juju run', because 'juju run' runs the command in a hook context (and is why it is functionally different to juju ssh). I do notice you are running juju 1.18, and I'm running the test suite with juju 1.20 (latest stable). I am not aware of any 1.20 features or bug fixes that I'm relying on, so 1.18 could be fine.

JUJU_REPOSITORY needs to be set, yes. I think we are stuck with this until I can port the tests to Amulet. Or when juju lets me deploy the cwd as a charm rather than requiring this baroque structure.

The tests are flaky, and will alas remain flaky until Bug #1200267 is addressed. I can increase the various sleeps scattered around the code base to make things better with slower providers, which will of course make the test suite even slower. I have increased what I think is the relevant timeout, and am now ignoring all failures from 'juju run' in this section (rather than just errcode==2, which is what we usually see when juju is caught with its pants down) - I think there was a race here where I might attempt to list units in a relation before the relation has been joined and an error code returned (the juju run failures you see).

The test_failover failure is interesting. I think for you it was failing just due to not waiting long enough. For me, I think that a race condition has been introduced with juju 1.20. I need to investigate this further.

[Adam Israel (aisrael)]

Hey Stuart,

Interesting. I'm not sure why make lint is producing different results. My juju version is 1.20.7-trusty-amd64, on a recently updated Trusty host, but the machine's agent version is 1.18.4.1.

Perhaps it's an issue of running inside a Vagrant VM.

I'll re-run the tests natively and see if there are any differences. It's entirely possible that something in the vagrant cloud image is different.

[Adam Israel (aisrael)]

Confirmed -- my Vagrant VM has a slightly newer version of flake8, which I suspect was installed via pip.

[Stuart Bishop (stub)]

I think I've addressed everything from Adam's review now.

[Cory Johns (johnsca)]

Stuart,

I still had test failures, which are all errors in the replication-relation-joined hook, and which doesn't appear to be directly related to a change from this set.

* Test run output: http://pastebin.ubuntu.com/8473480/
* Hook error from log: http://pastebin.ubuntu.com/8473481/

Also, while definitely not related to this change set, the tests for this charm do have some significant hurdles to passing in the automated test environment. Specifically:

* The `make test` target doesn't force the `make testdep` target, causing the `make test` target to fail if run first (which is done by the automated test runner). Once the runner manually discovers and runs 00_setup.py, the rest of the tests will fail, but `make test` will already have failed by that point. (This might be best fixed by having the runner discover and run the testdep target, though I do think it would be more intuitive to have the test target depend on the testdep target.)

* As you noted in your previous comment, the tests rely on the JUJU_REPOSITORY and SERIES (if being tested against precise) environment variables. The former may be set by the test runner, but it is unlikely that the latter will be.

* The tests for this charm require a local checkout of the postgresql-psql charm, with no fallback to installing from the store (which is what's causing this failure: http://reports.vapour.ws/charm-tests/charm-bundle-test-960-results/charm/charm-testing-aws/6).

Even though these weren't introduced with this changeset, we really need to address them to get the postgresql charm passing in the automated test environment.

review: Needs Fixing

[Stuart Bishop (stub)]

The exception traceback you pastebinned is very interesting. The "user
= rel['user']" line does not exist in this branch, so I think an old
version of the charm is being deployed somehow. The actual exception
looks familiar, and I think it is one that is fixed in this branch.
Hopefully it is something simple like the trusty charm is being run
from JUJU_REPOSITORY, whereas you were hoping for the precise branch
(the test run appears to be using trusty as the series). I beat my
head against a similar wall the other day, never finding out how juju
was managing to deploy a charm that didn't exist, until I totally
rebuild my JUJU_REPOSITORY.

I can add testdep as a requirement to the makefile easily enough. Is
'charm test' no longer the proposed method of running the tests?
(done)

I don't know why SERIES is required. The makefile does "SERIES :=
$(juju get-environment default-series)" at the top, so unless it is
overridden it will be using the default series of the environment.
(unchanged - charm version issue?)

I will fix things to use the charmstore version of postgresql-psql if
the series is Precise. There is not a version in the charm store for
Trusty. Ideally, there will never be as I'd like to just include this
charm inside the PostgreSQL charm somehow (in much the same way that
Amulet launches its sentinal charms). I haven't looked at this yet, as
I've been putting it off until I have time to switch things to Amulet.
Perhaps there is some syntax that will always use the precise branch
of the postgresql-psql charm no matter if I'm deploying to trusty or
precise hosts? (done for precise)

I'd be interested in hearing thoughts on the code changes and style btw.

On 2 October 2014 00:05, Cory Johns <email address hidden> wrote:

> Even though these weren't introduced with this changeset, we really need to address them to get the postgresql charm passing in the automated test environment.

Does this block landing? Let me know (and how to go about checking),
as I've been landing other branches to this charm from other
contributers without such scrutiny.

The current branch will also fail on the automated test environment,
and it doesn't seem helpful to block on this (and it means nobody’s
changes to the charm can land, since I doubt anyone else will be
attempting to fix this). I can simply disable most of the tests if it
is a requirement (and still have more tests than most charms).

--
Stuart Bishop <email address hidden>

[Cory Johns (johnsca)]

> The "user = rel['user']" line does not exist in this branch,
> so I think an old version of the charm is being deployed somehow.

I was testing by merging this branch into trunk, and the line is there. It looks like it was added upstream in this commit: http://bazaar.launchpad.net/~charmers/charms/precise/postgresql/trunk/revision/102

> Is 'charm test' no longer the proposed method of running the tests?

The automated test runner uses bundletester, which does the equivalent of `charm test` but also does some additional discovery, which includes running `make test` before doing `charm test`. Also, it was just less intuitive for me to check for a testdep target before running `make test`; I was all ready to nack you for missing dependencies. :p

Tim Van Steenburgh is working on a blog post with detailed info about the automated test runner and recommended way for running tests locally to ensure they pass for the automated runner, which we hope to have out soon. In the meantime, the bundletester tool can be `pip install`ed from https://pypi.python.org/pypi/bundletester/0.3.7

> I don't know why SERIES is required.

You're absolutely right. I checked the charms out under my local precise namespace, but was running with the default series set to trusty and was confused by why it was trying to deploy the charms as trusty, so that's my bad.

> Perhaps there is some syntax that will always use the precise branch
> of the postgresql-psql charm no matter if I'm deploying to trusty or
> precise hosts?

Just removing the conditional and hard-coding PSQL_CHARM to "cs:precise/postgresql-psql" should work.

> Does this block landing?

No, not really. I certainly won't push to block this MP based on upstream test failures, but that doesn't mean I won't push for you to fix them if you're willing. :)

Although, failing tests from upstream *do* make it more difficult to review the new changes, especially on a complex charm like this one.

Also, our goal is to gate reviews on the automated tests, to improve quality and help us manage the queue of reviews more effectively. One of our milestones on that goal is to get all of the failing charms on http://reports.vapour.ws/charm-tests-by-charm passing, or to back them out of Recommended status (unpromulgate) if we can't fix them or get the maintainer to fix them.

[Cory Johns (johnsca)]

For reference, here is the blog post I mentioned: http://blog.juju.solutions/cloud/juju/2014/10/02/charm-testing.html

Stuart, I will take a look at your updates shortly to see if they resolve the issue I was seeing. Thanks.

[Stuart Bishop (stub)]

This is where I'm at at the moment:

PostgreSQL integration tests, all non-beta versions,
trial test.PG91Tests
test
  PG91Tests
    test_admin_addresses ... [OK]
    test_basic ... [OK]
    test_basic_admin ... [OK]
    test_explicit_database ... [OK]
    test_failover ... [ERROR]
    test_failover_election ... [OK]
    test_roles_granted ... [FAIL]
    test_roles_revoked ... [FAIL]
    test_streaming_replication ... [OK]
    test_swiftwal_logshipping_replication ... [OK]
    test_syslog ... [OK]
    test_upgrade_charm ... [OK]
    test_wal_e_swift_logshipping ... [OK]

[Stuart Bishop (stub)]

Looks sorted now. Bugs were on trunk, so trunk is busted until this branch lands.

[lp:~stub/charms/precise/postgresql/integration]

114. By Stuart Bishop on 2014-10-08

Merge trunk

[Review Queue (review-queue)]

This items has failed automated testing! Results available here http://reports.vapour.ws/charm-tests/charm-bundle-test-1227-results

review: Needs Fixing (automated testing)

[lp:~stub/charms/precise/postgresql/integration]

115. By Stuart Bishop on 2014-10-10

Make hooks.py executable

116. By Stuart Bishop on 2014-10-14

Embed test client charm and remove JUJU_REPOSITORY requirement

117. By Stuart Bishop on 2014-10-14

Allow charm store charms, fixing rsyslog test

118. By Stuart Bishop on 2014-10-14

Ignore magic synced charmhelpers

[Whit Morriss (whitmo)]

Hey Stuart,

I'm running into issues getting bundletester to run the tests. It gets through lint and proof, then hangs, and exits after printing "Alarm clock" to stdout. psql and postgres seem to have deployed ok.

This may be an issue with my environment. still investigating.

-w

[Stuart Bishop (stub)]

Yeah, I've got another MP up disabling the integration tests by default as having them is not being helpful. If you work out what assumptions bundletester is making (or vice versa), we can fix that in a separate branch. My prime concern with this branch is actually landing it, as trunk is broken and fixes have been stuck here for months.

[Charles Butler (lazypower)]

Stub,

I've taken some time to re-review this branch with a mindset of "Focus on the code and verify the charm works" - And indeed. This branch of postgres fixes the upstream brokeneness that I've seen.

I'm going to tentatively approve this branch on the good faith promise that there will be an update correcting the testing story - as the PG91 tests are whats failing consistently when I test - but with the severity and longevity of this response cycle - I see value in whats here with additional work scheduled elsewhere to correct our infra problems re: testing.

Thanks again for the continued effort and work to keep the postgres charm a high quality member of the charm store.

If you have any questions/comments/concerns about the review contact us in #juju on irc.freenode.net or email the mailing list <email address hidden>, or ask a question tagged with "juju" on http://askubuntu.com.

review: Approve (deploy + code)

=== modified file '.bzrignore'
--- .bzrignore	2014-02-25 19:47:03 +0000
+++ .bzrignore	2014-10-14 10:13:22 +0000
@@ -1,3 +1,4 @@
 _trial_temp
 hooks/_trial_temp
 hooks/local_state.pickle
+lib/test-client-charm/hooks/charmhelpers

=== modified file 'Makefile'
--- Makefile	2014-06-27 10:04:40 +0000
+++ Makefile	2014-10-14 10:13:22 +0000
@@ -14,7 +14,7 @@
 	@echo "    make integration_test_93"
 	@echo "    make integration_test_94"
 
-test: lint unit_test integration_test
+test: testdep lint unit_test integration_test
 
 testdep:
 	tests/00_setup.test
@@ -25,7 +25,9 @@
 
 integration_test:
 	@echo "PostgreSQL integration tests, all non-beta versions, ${SERIES}"
-	trial test.PG91Tests test.PG92Tests test.PG93Tests
+	trial test.PG91Tests
+	trial test.PG92Tests
+	trial test.PG93Tests
 
 integration_test_91:
 	@echo "PostgreSQL 9.1 integration tests, ${SERIES}"
@@ -48,3 +50,10 @@
 	@flake8 -v \
 	    --exclude hooks/charmhelpers,hooks/_trial_temp \
 	    hooks testing tests test.py
+
+sync:
+	@bzr cat \
+	    lp:charm-helpers/tools/charm_helpers_sync/charm_helpers_sync.py \
+		> .charm_helpers_sync.py
+	@python .charm_helpers_sync.py -c charm-helpers.yaml
+	@rm .charm_helpers_sync.py

=== modified file 'README.md'
--- README.md	2014-06-27 05:27:12 +0000
+++ README.md	2014-10-14 10:13:22 +0000
@@ -255,6 +255,44 @@
   6. juju add-relation postgresql storage
 
 
+# Point In Time Recovery
+
+The PostgreSQL charm has experimental support for log shipping and point
+in time recovery. This feature uses the wal-e[2] tool, and requires access
+to Amazon S3, Microsoft Azure Block Storage or Swift. This feature is
+flagged as experimental because it has only been tested with Swift, and
+not yet been tested under load. It also may require some API changes,
+particularly on how authentication credentials are accessed when a standard
+emerges. The charm can be configured to perform regular filesystem backups
+and ship WAL files to the object store. Hot standbys will make use of the
+archived WAL files, allowing them to resync after extended netsplits or
+even let you turn off streaming replication entirely.
+
+With a base backup and the WAL archive you can perform point in time
+recovery, but this is still a manual process and the charm does not
+yet help you do it. The simplest approach would be to create a new
+PostgreSQL service containing a single unit, 'juju ssh' in and use
+wal-e to replace the database after shutting it down, create a
+recovery.conf to replay the archived WAL files using wal-e, restart the
+database and wait for it to recover. Once recovered, new hot standby
+units can be added and client services related to the new database
+service.
+
+To enable the experimental wal-e support with Swift, you will need to
+use Ubuntu 14.04 (Trusty), and set the service configuration settings
+similar to the following::
+
+    postgresql:
+        wal_e_storage_uri: swift://mycontainer
+        os_username: my_swift_username
+        os_password: my_swift_password
+        os_auth_url: https://keystone.auth.url.example.com:8080/v2/
+        os_tenant_name: my_tenant_name
+        install_sources: |
+            - ppa:stub/pgcharm
+            - cloud:icehouse
+
+
 # Contact Information
 
 ## PostgreSQL 
@@ -265,3 +303,4 @@
 - [PostgreSQL Mailing List](http://www.postgresql.org/list/)
 
   [1]: https://bugs.launchpad.net/charms/+source/postgresql/+bug/1258485
+  [2]: https://github.com/wal-e/wal-e

=== modified file 'charm-helpers.yaml'
--- charm-helpers.yaml	2014-06-05 11:04:11 +0000
+++ charm-helpers.yaml	2014-10-14 10:13:22 +0000
@@ -1,6 +1,5 @@
 destination: hooks/charmhelpers
-#branch: lp:charm-helpers
-branch: lp:~stub/charm-helpers/fix-configure_sources
+branch: lp:charm-helpers
 include:
     - core
     - fetch

=== modified file 'config.yaml'
--- config.yaml	2014-06-03 09:24:14 +0000
+++ config.yaml	2014-10-14 10:13:22 +0000
@@ -89,6 +89,14 @@
     default: False
     type: boolean
     description: Log disconnections
+  log_temp_files:
+    default: "-1"
+    type: string
+    description: |
+        Log creation of temporary files larger than the threshold.
+        -1 disables the feature, 0 logs all temporary files, or specify
+        the threshold size with an optional unit (eg. "512KB", default
+        unit is kilobytes).
   log_line_prefix:
     default: "%t [%p]: [%l-1] db=%d,user=%u "
     type: string
@@ -164,9 +172,10 @@
     default: False
     type: boolean
     description: |
+        DEPRECATED.
         Hot standby or warm standby. When True, queries can be run against
         the database when in recovery or standby mode (ie. replicated).
-        Overridden by juju when master/slave relations are used.
+        Overridden when service contains multiple units.
   hot_standby_feedback:
     default: False
     type: boolean
@@ -181,7 +190,7 @@
         'minimal', 'archive' or 'hot_standby'. Defines how much information
         is written to the WAL. Set to 'minimal' for stand alone databases
         and 'hot_standby' for replicated setups. Overridden by juju when
-        replication s used.
+        replication is used.
   max_wal_senders:
     default: 0
     type: int
@@ -380,14 +389,16 @@
   # Swift backups and PITR via SwiftWAL
   swiftwal_container_prefix:
     type: string
-    default: ""
+    default: null
     description: |
       EXPERIMENTAL.
       Swift container prefix for SwiftWAL to use. Must be set if any
-      SwiftWAL features are enabled.
+      SwiftWAL features are enabled. This will become a simple
+      swiftwal_container config item when proper leader election is
+      implemented in juju.
   swiftwal_backup_schedule:
     type: string
-    default: ""
+    default: null
     description: |
       EXPERIMENTAL.
       Cron-formatted schedule for SwiftWAL database backups.
@@ -405,34 +416,83 @@
     description: |
       EXPERIMENTAL.
       Archive WAL files into Swift. If swiftwal_backup_schedule is set,
-      this allows point-in-time recovery and WAL files are removed
+      allows point-in-time recovery and WAL files are removed
       automatically with old backups. If swiftwal_backup_schedule is not set
       then WAL files are never removed. Enabling this option will override
       the archive_mode and archive_command settings.
+  wal_e_storage_uri:
+    type: string
+    default: null
+    description: |
+      EXPERIMENTAL.
+      Specify storage to be used by WAL-E. Every PostgreSQL service must use
+      a unique URI. Backups will be unrecoverable if it is not unique. The
+      URI's scheme must be one of 'swift' (OpenStack Swift), 's3' (Amazon AWS)
+      or 'wabs' (Windows Azure). For example:
+        'swift://some-container/directory/or/whatever'
+        's3://some-bucket/directory/or/whatever'
+        'wabs://some-bucket/directory/or/whatever'
+      Setting the wal_e_storage_uri enables regular WAL-E filesystem level
+      backups (per wal_e_backup_schedule), and log shipping to the configured
+      storage. Point-in-time recovery becomes possible, as is disabling the
+      streaming_replication configuration item and relying solely on
+      log shipping for replication.
+  wal_e_backup_schedule:
+    type: string
+    default: "13 0 * * *"
+    description: |
+      EXPERIMENTAL.
+      Cron-formatted schedule for WAL-E database backups. If
+      wal_e_backup_schedule is unset, WAL files will never be removed from
+      WAL-E storage.
+  wal_e_backup_retention:
+    type: int
+    default: 2
+    description: |
+      EXPERIMENTAL.
+      Number of recent base backups and WAL files to retain.
+      You need enough space for this many backups plus one more, as
+      an old backup will only be removed after a new one has been
+      successfully made to replace it.
   streaming_replication:
     type: boolean
     default: true
     description: |
-      EXPERIMENTAL.
       Enable streaming replication. Normally, streaming replication is
       always used, and any log shipping configured is used as a fallback.
       Turning this off without configuring log shipping is an error.
   os_username:
     type: string
-    default: ""
+    default: null
     description: EXPERIMENTAL. OpenStack Swift username.
   os_password:
     type: string
-    default: ""
+    default: null
     description: EXPERIMENTAL. OpenStack Swift password.
   os_auth_url:
     type: string
-    default: ""
+    default: null
     description: EXPERIMENTAL. OpenStack Swift authentication URL.
   os_tenant_name:
     type: string
-    default: ""
+    default: null
     description: EXPERIMENTAL. OpenStack Swift tenant name.
+  aws_access_key_id:
+    type: string
+    default: null
+    description: EXPERIMENTAL. Amazon AWS access key id.
+  aws_secret_access_key:
+    type: string
+    default: null
+    description: EXPERIMENTAL. Amazon AWS secret access key.
+  wabs_account_name:
+    type: string
+    default: null
+    description: EXPERIMENTAL. Windows Azure account name.
+  wabs_access_key:
+    type: string
+    default: null
+    description: EXPERIMENTAL. Windows Azure access key.
   package_status:
     default: "install"
     type: string

=== modified file 'hooks/charmhelpers/core/fstab.py'
--- hooks/charmhelpers/core/fstab.py	2014-06-05 11:21:55 +0000
+++ hooks/charmhelpers/core/fstab.py	2014-10-14 10:13:22 +0000
@@ -48,9 +48,11 @@
         file.__init__(self, self._path, 'r+')
 
     def _hydrate_entry(self, line):
+        # NOTE: use split with no arguments to split on any
+        #       whitespace including tabs
         return Fstab.Entry(*filter(
             lambda x: x not in ('', None),
-            line.strip("\n").split(" ")))
+            line.strip("\n").split()))
 
     @property
     def entries(self):

=== modified file 'hooks/charmhelpers/core/hookenv.py'
--- hooks/charmhelpers/core/hookenv.py	2014-05-26 07:10:14 +0000
+++ hooks/charmhelpers/core/hookenv.py	2014-10-14 10:13:22 +0000
@@ -25,7 +25,7 @@
 def cached(func):
     """Cache return values for multiple executions of func + args
 
-    For example:
+    For example::
 
         @cached
         def unit_get(attribute):
@@ -156,12 +156,15 @@
 
 
 class Config(dict):
-    """A Juju charm config dictionary that can write itself to
-    disk (as json) and track which values have changed since
-    the previous hook invocation.
-
-    Do not instantiate this object directly - instead call
-    ``hookenv.config()``
+    """A dictionary representation of the charm's config.yaml, with some
+    extra features:
+
+    - See which values in the dictionary have changed since the previous hook.
+    - For values that have changed, see what the previous value was.
+    - Store arbitrary data for use in a later hook.
+
+    NOTE: Do not instantiate this object directly - instead call
+    ``hookenv.config()``, which will return an instance of :class:`Config`.
 
     Example usage::
 
@@ -170,8 +173,8 @@
         >>> config = hookenv.config()
         >>> config['foo']
         'bar'
+        >>> # store a new key/value for later use
         >>> config['mykey'] = 'myval'
-        >>> config.save()
 
 
         >>> # user runs `juju set mycharm foo=baz`
@@ -188,22 +191,23 @@
         >>> # keys/values that we add are preserved across hooks
         >>> config['mykey']
         'myval'
-        >>> # don't forget to save at the end of hook!
-        >>> config.save()
 
     """
     CONFIG_FILE_NAME = '.juju-persistent-config'
 
     def __init__(self, *args, **kw):
         super(Config, self).__init__(*args, **kw)
+        self.implicit_save = True
         self._prev_dict = None
         self.path = os.path.join(charm_dir(), Config.CONFIG_FILE_NAME)
         if os.path.exists(self.path):
             self.load_previous()
 
     def load_previous(self, path=None):
-        """Load previous copy of config from disk so that current values
-        can be compared to previous values.
+        """Load previous copy of config from disk.
+
+        In normal usage you don't need to call this method directly - it
+        is called automatically at object initialization.
 
         :param path:
 
@@ -218,8 +222,8 @@
             self._prev_dict = json.load(f)
 
     def changed(self, key):
-        """Return true if the value for this key has changed since
-        the last save.
+        """Return True if the current value for this key is different from
+        the previous value.
 
         """
         if self._prev_dict is None:
@@ -228,7 +232,7 @@
 
     def previous(self, key):
         """Return previous value for this key, or None if there
-        is no "previous" value.
+        is no previous value.
 
         """
         if self._prev_dict:
@@ -238,7 +242,13 @@
     def save(self):
         """Save this config to disk.
 
-        Preserves items in _prev_dict that do not exist in self.
+        If the charm is using the :mod:`Services Framework <services.base>`
+        or :meth:'@hook <Hooks.hook>' decorator, this
+        is called automatically at the end of successful hook execution.
+        Otherwise, it should be called directly by user code.
+
+        To disable automatic saves, set ``implicit_save=False`` on this
+        instance.
 
         """
         if self._prev_dict:
@@ -285,8 +295,9 @@
         raise
 
 
-def relation_set(relation_id=None, relation_settings={}, **kwargs):
+def relation_set(relation_id=None, relation_settings=None, **kwargs):
     """Set relation information for the current unit"""
+    relation_settings = relation_settings if relation_settings else {}
     relation_cmd_line = ['relation-set']
     if relation_id is not None:
         relation_cmd_line.extend(('-r', relation_id))
@@ -445,18 +456,19 @@
 class Hooks(object):
     """A convenient handler for hook functions.
 
-    Example:
+    Example::
+
         hooks = Hooks()
 
         # register a hook, taking its name from the function name
         @hooks.hook()
         def install():
-            ...
+            pass  # your code here
 
         # register a hook, providing a custom hook name
         @hooks.hook("config-changed")
         def config_changed():
-            ...
+            pass  # your code here
 
         if __name__ == "__main__":
             # execute a hook based on the name the program is called by
@@ -476,6 +488,9 @@
         hook_name = os.path.basename(args[0])
         if hook_name in self._hooks:
             self._hooks[hook_name]()
+            cfg = config()
+            if cfg.implicit_save:
+                cfg.save()
         else:
             raise UnregisteredHookError(hook_name)
 

=== modified file 'hooks/charmhelpers/core/host.py'
--- hooks/charmhelpers/core/host.py	2014-06-05 11:04:11 +0000
+++ hooks/charmhelpers/core/host.py	2014-10-14 10:13:22 +0000
@@ -12,7 +12,8 @@
 import string
 import subprocess
 import hashlib
-import apt_pkg
+import shutil
+from contextlib import contextmanager
 
 from collections import OrderedDict
 
@@ -53,7 +54,7 @@
 def service_running(service):
     """Determine whether a system service is running"""
     try:
-        output = subprocess.check_output(['service', service, 'status'])
+        output = subprocess.check_output(['service', service, 'status'], stderr=subprocess.STDOUT)
     except subprocess.CalledProcessError:
         return False
     else:
@@ -63,6 +64,16 @@
             return False
 
 
+def service_available(service_name):
+    """Determine whether a system service is available"""
+    try:
+        subprocess.check_output(['service', service_name, 'status'], stderr=subprocess.STDOUT)
+    except subprocess.CalledProcessError:
+        return False
+    else:
+        return True
+
+
 def adduser(username, password=None, shell='/bin/bash', system_user=False):
     """Add a user to the system"""
     try:
@@ -212,13 +223,13 @@
 def restart_on_change(restart_map, stopstart=False):
     """Restart services based on configuration files changing
 
-    This function is used a decorator, for example
+    This function is used a decorator, for example::
 
         @restart_on_change({
             '/etc/ceph/ceph.conf': [ 'cinder-api', 'cinder-volume' ]
             })
         def ceph_client_changed():
-            ...
+            pass  # your code here
 
     In this example, the cinder-api and cinder-volume services
     would be restarted if /etc/ceph/ceph.conf is changed by the
@@ -314,12 +325,36 @@
 
 def cmp_pkgrevno(package, revno, pkgcache=None):
     '''Compare supplied revno with the revno of the installed package
-       1 => Installed revno is greater than supplied arg
-       0 => Installed revno is the same as supplied arg
-      -1 => Installed revno is less than supplied arg
+
+    *  1 => Installed revno is greater than supplied arg
+    *  0 => Installed revno is the same as supplied arg
+    * -1 => Installed revno is less than supplied arg
+
     '''
+    import apt_pkg
+    from charmhelpers.fetch import apt_cache
     if not pkgcache:
-        apt_pkg.init()
-        pkgcache = apt_pkg.Cache()
+        pkgcache = apt_cache()
     pkg = pkgcache[package]
     return apt_pkg.version_compare(pkg.current_ver.ver_str, revno)
+
+
+@contextmanager
+def chdir(d):
+    cur = os.getcwd()
+    try:
+        yield os.chdir(d)
+    finally:
+        os.chdir(cur)
+
+
+def chownr(path, owner, group):
+    uid = pwd.getpwnam(owner).pw_uid
+    gid = grp.getgrnam(group).gr_gid
+
+    for root, dirs, files in os.walk(path):
+        for name in dirs + files:
+            full = os.path.join(root, name)
+            broken_symlink = os.path.lexists(full) and not os.path.exists(full)
+            if not broken_symlink:
+                os.chown(full, uid, gid)

=== added directory 'hooks/charmhelpers/core/services'
=== added file 'hooks/charmhelpers/core/services/__init__.py'
--- hooks/charmhelpers/core/services/__init__.py	1970-01-01 00:00:00 +0000
+++ hooks/charmhelpers/core/services/__init__.py	2014-10-14 10:13:22 +0000
@@ -0,0 +1,2 @@
+from .base import *
+from .helpers import *

=== added file 'hooks/charmhelpers/core/services/base.py'
--- hooks/charmhelpers/core/services/base.py	1970-01-01 00:00:00 +0000
+++ hooks/charmhelpers/core/services/base.py	2014-10-14 10:13:22 +0000
@@ -0,0 +1,313 @@
+import os
+import re
+import json
+from collections import Iterable
+
+from charmhelpers.core import host
+from charmhelpers.core import hookenv
+
+
+__all__ = ['ServiceManager', 'ManagerCallback',
+           'PortManagerCallback', 'open_ports', 'close_ports', 'manage_ports',
+           'service_restart', 'service_stop']
+
+
+class ServiceManager(object):
+    def __init__(self, services=None):
+        """
+        Register a list of services, given their definitions.
+
+        Service definitions are dicts in the following formats (all keys except
+        'service' are optional)::
+
+            {
+                "service": <service name>,
+                "required_data": <list of required data contexts>,
+                "provided_data": <list of provided data contexts>,
+                "data_ready": <one or more callbacks>,
+                "data_lost": <one or more callbacks>,
+                "start": <one or more callbacks>,
+                "stop": <one or more callbacks>,
+                "ports": <list of ports to manage>,
+            }
+
+        The 'required_data' list should contain dicts of required data (or
+        dependency managers that act like dicts and know how to collect the data).
+        Only when all items in the 'required_data' list are populated are the list
+        of 'data_ready' and 'start' callbacks executed.  See `is_ready()` for more
+        information.
+
+        The 'provided_data' list should contain relation data providers, most likely
+        a subclass of :class:`charmhelpers.core.services.helpers.RelationContext`,
+        that will indicate a set of data to set on a given relation.
+
+        The 'data_ready' value should be either a single callback, or a list of
+        callbacks, to be called when all items in 'required_data' pass `is_ready()`.
+        Each callback will be called with the service name as the only parameter.
+        After all of the 'data_ready' callbacks are called, the 'start' callbacks
+        are fired.
+
+        The 'data_lost' value should be either a single callback, or a list of
+        callbacks, to be called when a 'required_data' item no longer passes
+        `is_ready()`.  Each callback will be called with the service name as the
+        only parameter.  After all of the 'data_lost' callbacks are called,
+        the 'stop' callbacks are fired.
+
+        The 'start' value should be either a single callback, or a list of
+        callbacks, to be called when starting the service, after the 'data_ready'
+        callbacks are complete.  Each callback will be called with the service
+        name as the only parameter.  This defaults to
+        `[host.service_start, services.open_ports]`.
+
+        The 'stop' value should be either a single callback, or a list of
+        callbacks, to be called when stopping the service.  If the service is
+        being stopped because it no longer has all of its 'required_data', this
+        will be called after all of the 'data_lost' callbacks are complete.
+        Each callback will be called with the service name as the only parameter.
+        This defaults to `[services.close_ports, host.service_stop]`.
+
+        The 'ports' value should be a list of ports to manage.  The default
+        'start' handler will open the ports after the service is started,
+        and the default 'stop' handler will close the ports prior to stopping
+        the service.
+
+
+        Examples:
+
+        The following registers an Upstart service called bingod that depends on
+        a mongodb relation and which runs a custom `db_migrate` function prior to
+        restarting the service, and a Runit service called spadesd::
+
+            manager = services.ServiceManager([
+                {
+                    'service': 'bingod',
+                    'ports': [80, 443],
+                    'required_data': [MongoRelation(), config(), {'my': 'data'}],
+                    'data_ready': [
+                        services.template(source='bingod.conf'),
+                        services.template(source='bingod.ini',
+                                          target='/etc/bingod.ini',
+                                          owner='bingo', perms=0400),
+                    ],
+                },
+                {
+                    'service': 'spadesd',
+                    'data_ready': services.template(source='spadesd_run.j2',
+                                                    target='/etc/sv/spadesd/run',
+                                                    perms=0555),
+                    'start': runit_start,
+                    'stop': runit_stop,
+                },
+            ])
+            manager.manage()
+        """
+        self._ready_file = os.path.join(hookenv.charm_dir(), 'READY-SERVICES.json')
+        self._ready = None
+        self.services = {}
+        for service in services or []:
+            service_name = service['service']
+            self.services[service_name] = service
+
+    def manage(self):
+        """
+        Handle the current hook by doing The Right Thing with the registered services.
+        """
+        hook_name = hookenv.hook_name()
+        if hook_name == 'stop':
+            self.stop_services()
+        else:
+            self.provide_data()
+            self.reconfigure_services()
+        cfg = hookenv.config()
+        if cfg.implicit_save:
+            cfg.save()
+
+    def provide_data(self):
+        """
+        Set the relation data for each provider in the ``provided_data`` list.
+
+        A provider must have a `name` attribute, which indicates which relation
+        to set data on, and a `provide_data()` method, which returns a dict of
+        data to set.
+        """
+        hook_name = hookenv.hook_name()
+        for service in self.services.values():
+            for provider in service.get('provided_data', []):
+                if re.match(r'{}-relation-(joined|changed)'.format(provider.name), hook_name):
+                    data = provider.provide_data()
+                    _ready = provider._is_ready(data) if hasattr(provider, '_is_ready') else data
+                    if _ready:
+                        hookenv.relation_set(None, data)
+
+    def reconfigure_services(self, *service_names):
+        """
+        Update all files for one or more registered services, and,
+        if ready, optionally restart them.
+
+        If no service names are given, reconfigures all registered services.
+        """
+        for service_name in service_names or self.services.keys():
+            if self.is_ready(service_name):
+                self.fire_event('data_ready', service_name)
+                self.fire_event('start', service_name, default=[
+                    service_restart,
+                    manage_ports])
+                self.save_ready(service_name)
+            else:
+                if self.was_ready(service_name):
+                    self.fire_event('data_lost', service_name)
+                self.fire_event('stop', service_name, default=[
+                    manage_ports,
+                    service_stop])
+                self.save_lost(service_name)
+
+    def stop_services(self, *service_names):
+        """
+        Stop one or more registered services, by name.
+
+        If no service names are given, stops all registered services.
+        """
+        for service_name in service_names or self.services.keys():
+            self.fire_event('stop', service_name, default=[
+                manage_ports,
+                service_stop])
+
+    def get_service(self, service_name):
+        """
+        Given the name of a registered service, return its service definition.
+        """
+        service = self.services.get(service_name)
+        if not service:
+            raise KeyError('Service not registered: %s' % service_name)
+        return service
+
+    def fire_event(self, event_name, service_name, default=None):
+        """
+        Fire a data_ready, data_lost, start, or stop event on a given service.
+        """
+        service = self.get_service(service_name)
+        callbacks = service.get(event_name, default)
+        if not callbacks:
+            return
+        if not isinstance(callbacks, Iterable):
+            callbacks = [callbacks]
+        for callback in callbacks:
+            if isinstance(callback, ManagerCallback):
+                callback(self, service_name, event_name)
+            else:
+                callback(service_name)
+
+    def is_ready(self, service_name):
+        """
+        Determine if a registered service is ready, by checking its 'required_data'.
+
+        A 'required_data' item can be any mapping type, and is considered ready
+        if `bool(item)` evaluates as True.
+        """
+        service = self.get_service(service_name)
+        reqs = service.get('required_data', [])
+        return all(bool(req) for req in reqs)
+
+    def _load_ready_file(self):
+        if self._ready is not None:
+            return
+        if os.path.exists(self._ready_file):
+            with open(self._ready_file) as fp:
+                self._ready = set(json.load(fp))
+        else:
+            self._ready = set()
+
+    def _save_ready_file(self):
+        if self._ready is None:
+            return
+        with open(self._ready_file, 'w') as fp:
+            json.dump(list(self._ready), fp)
+
+    def save_ready(self, service_name):
+        """
+        Save an indicator that the given service is now data_ready.
+        """
+        self._load_ready_file()
+        self._ready.add(service_name)
+        self._save_ready_file()
+
+    def save_lost(self, service_name):
+        """
+        Save an indicator that the given service is no longer data_ready.
+        """
+        self._load_ready_file()
+        self._ready.discard(service_name)
+        self._save_ready_file()
+
+    def was_ready(self, service_name):
+        """
+        Determine if the given service was previously data_ready.
+        """
+        self._load_ready_file()
+        return service_name in self._ready
+
+
+class ManagerCallback(object):
+    """
+    Special case of a callback that takes the `ServiceManager` instance
+    in addition to the service name.
+
+    Subclasses should implement `__call__` which should accept three parameters:
+
+        * `manager`       The `ServiceManager` instance
+        * `service_name`  The name of the service it's being triggered for
+        * `event_name`    The name of the event that this callback is handling
+    """
+    def __call__(self, manager, service_name, event_name):
+        raise NotImplementedError()
+
+
+class PortManagerCallback(ManagerCallback):
+    """
+    Callback class that will open or close ports, for use as either
+    a start or stop action.
+    """
+    def __call__(self, manager, service_name, event_name):
+        service = manager.get_service(service_name)
+        new_ports = service.get('ports', [])
+        port_file = os.path.join(hookenv.charm_dir(), '.{}.ports'.format(service_name))
+        if os.path.exists(port_file):
+            with open(port_file) as fp:
+                old_ports = fp.read().split(',')
+            for old_port in old_ports:
+                if bool(old_port):
+                    old_port = int(old_port)
+                    if old_port not in new_ports:
+                        hookenv.close_port(old_port)
+        with open(port_file, 'w') as fp:
+            fp.write(','.join(str(port) for port in new_ports))
+        for port in new_ports:
+            if event_name == 'start':
+                hookenv.open_port(port)
+            elif event_name == 'stop':
+                hookenv.close_port(port)
+
+
+def service_stop(service_name):
+    """
+    Wrapper around host.service_stop to prevent spurious "unknown service"
+    messages in the logs.
+    """
+    if host.service_running(service_name):
+        host.service_stop(service_name)
+
+
+def service_restart(service_name):
+    """
+    Wrapper around host.service_restart to prevent spurious "unknown service"
+    messages in the logs.
+    """
+    if host.service_available(service_name):
+        if host.service_running(service_name):
+            host.service_restart(service_name)
+        else:
+            host.service_start(service_name)
+
+
+# Convenience aliases
+open_ports = close_ports = manage_ports = PortManagerCallback()

=== added file 'hooks/charmhelpers/core/services/helpers.py'
--- hooks/charmhelpers/core/services/helpers.py	1970-01-01 00:00:00 +0000
+++ hooks/charmhelpers/core/services/helpers.py	2014-10-14 10:13:22 +0000
@@ -0,0 +1,125 @@
+from charmhelpers.core import hookenv
+from charmhelpers.core import templating
+
+from charmhelpers.core.services.base import ManagerCallback
+
+
+__all__ = ['RelationContext', 'TemplateCallback',
+           'render_template', 'template']
+
+
+class RelationContext(dict):
+    """
+    Base class for a context generator that gets relation data from juju.
+
+    Subclasses must provide the attributes `name`, which is the name of the
+    interface of interest, `interface`, which is the type of the interface of
+    interest, and `required_keys`, which is the set of keys required for the
+    relation to be considered complete.  The data for all interfaces matching
+    the `name` attribute that are complete will used to populate the dictionary
+    values (see `get_data`, below).
+
+    The generated context will be namespaced under the interface type, to prevent
+    potential naming conflicts.
+    """
+    name = None
+    interface = None
+    required_keys = []
+
+    def __init__(self, *args, **kwargs):
+        super(RelationContext, self).__init__(*args, **kwargs)
+        self.get_data()
+
+    def __bool__(self):
+        """
+        Returns True if all of the required_keys are available.
+        """
+        return self.is_ready()
+
+    __nonzero__ = __bool__
+
+    def __repr__(self):
+        return super(RelationContext, self).__repr__()
+
+    def is_ready(self):
+        """
+        Returns True if all of the `required_keys` are available from any units.
+        """
+        ready = len(self.get(self.name, [])) > 0
+        if not ready:
+            hookenv.log('Incomplete relation: {}'.format(self.__class__.__name__), hookenv.DEBUG)
+        return ready
+
+    def _is_ready(self, unit_data):
+        """
+        Helper method that tests a set of relation data and returns True if
+        all of the `required_keys` are present.
+        """
+        return set(unit_data.keys()).issuperset(set(self.required_keys))
+
+    def get_data(self):
+        """
+        Retrieve the relation data for each unit involved in a relation and,
+        if complete, store it in a list under `self[self.name]`.  This
+        is automatically called when the RelationContext is instantiated.
+
+        The units are sorted lexographically first by the service ID, then by
+        the unit ID.  Thus, if an interface has two other services, 'db:1'
+        and 'db:2', with 'db:1' having two units, 'wordpress/0' and 'wordpress/1',
+        and 'db:2' having one unit, 'mediawiki/0', all of which have a complete
+        set of data, the relation data for the units will be stored in the
+        order: 'wordpress/0', 'wordpress/1', 'mediawiki/0'.
+
+        If you only care about a single unit on the relation, you can just
+        access it as `{{ interface[0]['key'] }}`.  However, if you can at all
+        support multiple units on a relation, you should iterate over the list,
+        like::
+
+            {% for unit in interface -%}
+                {{ unit['key'] }}{% if not loop.last %},{% endif %}
+            {%- endfor %}
+
+        Note that since all sets of relation data from all related services and
+        units are in a single list, if you need to know which service or unit a
+        set of data came from, you'll need to extend this class to preserve
+        that information.
+        """
+        if not hookenv.relation_ids(self.name):
+            return
+
+        ns = self.setdefault(self.name, [])
+        for rid in sorted(hookenv.relation_ids(self.name)):
+            for unit in sorted(hookenv.related_units(rid)):
+                reldata = hookenv.relation_get(rid=rid, unit=unit)
+                if self._is_ready(reldata):
+                    ns.append(reldata)
+
+    def provide_data(self):
+        """
+        Return data to be relation_set for this interface.
+        """
+        return {}
+
+
+class TemplateCallback(ManagerCallback):
+    """
+    Callback class that will render a template, for use as a ready action.
+    """
+    def __init__(self, source, target, owner='root', group='root', perms=0444):
+        self.source = source
+        self.target = target
+        self.owner = owner
+        self.group = group
+        self.perms = perms
+
+    def __call__(self, manager, service_name, event_name):
+        service = manager.get_service(service_name)
+        context = {}
+        for ctx in service.get('required_data', []):
+            context.update(ctx)
+        templating.render(self.source, self.target, context,
+                          self.owner, self.group, self.perms)
+
+
+# Convenience aliases for templates
+render_template = template = TemplateCallback

=== added file 'hooks/charmhelpers/core/templating.py'
--- hooks/charmhelpers/core/templating.py	1970-01-01 00:00:00 +0000
+++ hooks/charmhelpers/core/templating.py	2014-10-14 10:13:22 +0000
@@ -0,0 +1,51 @@
+import os
+
+from charmhelpers.core import host
+from charmhelpers.core import hookenv
+
+
+def render(source, target, context, owner='root', group='root', perms=0444, templates_dir=None):
+    """
+    Render a template.
+
+    The `source` path, if not absolute, is relative to the `templates_dir`.
+
+    The `target` path should be absolute.
+
+    The context should be a dict containing the values to be replaced in the
+    template.
+
+    The `owner`, `group`, and `perms` options will be passed to `write_file`.
+
+    If omitted, `templates_dir` defaults to the `templates` folder in the charm.
+
+    Note: Using this requires python-jinja2; if it is not installed, calling
+    this will attempt to use charmhelpers.fetch.apt_install to install it.
+    """
+    try:
+        from jinja2 import FileSystemLoader, Environment, exceptions
+    except ImportError:
+        try:
+            from charmhelpers.fetch import apt_install
+        except ImportError:
+            hookenv.log('Could not import jinja2, and could not import '
+                        'charmhelpers.fetch to install it',
+                        level=hookenv.ERROR)
+            raise
+        apt_install('python-jinja2', fatal=True)
+        from jinja2 import FileSystemLoader, Environment, exceptions
+
+    if templates_dir is None:
+        templates_dir = os.path.join(hookenv.charm_dir(), 'templates')
+    loader = Environment(loader=FileSystemLoader(templates_dir))
+    try:
+        source = source
+        template = loader.get_template(source)
+    except exceptions.TemplateNotFound as e:
+        hookenv.log('Could not load template %s from %s.' %
+                    (source, templates_dir),
+                    level=hookenv.ERROR)
+        raise e
+    content = template.render(context)
+    host.mkdir(os.path.dirname(target))
+    host.write_file(target, content, owner, group, perms)

=== modified file 'hooks/charmhelpers/fetch/__init__.py'
--- hooks/charmhelpers/fetch/__init__.py	2014-06-05 11:04:11 +0000
+++ hooks/charmhelpers/fetch/__init__.py	2014-10-14 10:13:22 +0000
@@ -1,4 +1,5 @@
 import importlib
+from tempfile import NamedTemporaryFile
 import time
 from yaml import safe_load
 from charmhelpers.core.host import (
@@ -13,7 +14,6 @@
     config,
     log,
 )
-import apt_pkg
 import os
 
 
@@ -117,13 +117,7 @@
 
 def filter_installed_packages(packages):
     """Returns a list of packages that require installation"""
-    apt_pkg.init()
-
-    # Tell apt to build an in-memory cache to prevent race conditions (if
-    # another process is already building the cache).
-    apt_pkg.config.set("Dir::Cache::pkgcache", "")
-
-    cache = apt_pkg.Cache()
+    cache = apt_cache()
     _pkgs = []
     for package in packages:
         try:
@@ -136,6 +130,16 @@
     return _pkgs
 
 
+def apt_cache(in_memory=True):
+    """Build and return an apt cache"""
+    import apt_pkg
+    apt_pkg.init()
+    if in_memory:
+        apt_pkg.config.set("Dir::Cache::pkgcache", "")
+        apt_pkg.config.set("Dir::Cache::srcpkgcache", "")
+    return apt_pkg.Cache()
+
+
 def apt_install(packages, options=None, fatal=False):
     """Install one or more packages"""
     if options is None:
@@ -201,6 +205,27 @@
 
 
 def add_source(source, key=None):
+    """Add a package source to this system.
+
+    @param source: a URL or sources.list entry, as supported by
+    add-apt-repository(1). Examples:
+        ppa:charmers/example
+        deb https://stub:[e-mail]/ubuntu trusty main
+
+    In addition:
+        'proposed:' may be used to enable the standard 'proposed'
+        pocket for the release.
+        'cloud:' may be used to activate official cloud archive pockets,
+        such as 'cloud:icehouse'
+
+    @param key: A key to be added to the system's APT keyring and used
+    to verify the signatures on packages. Ideally, this should be an
+    ASCII format GPG public key including the block headers. A GPG key
+    id may also be used, but be aware that only insecure protocols are
+    available to retrieve the actual public key from a public keyserver
+    placing your Juju environment at risk. ppa and cloud archive keys
+    are securely added automtically, so sould not be provided.
+    """
     if source is None:
         log('Source is not present. Skipping')
         return
@@ -225,10 +250,23 @@
         release = lsb_release()['DISTRIB_CODENAME']
         with open('/etc/apt/sources.list.d/proposed.list', 'w') as apt:
             apt.write(PROPOSED_POCKET.format(release))
+    else:
+        raise SourceConfigError("Unknown source: {!r}".format(source))
+
     if key:
-        subprocess.check_call(['apt-key', 'adv', '--keyserver',
-                               'hkp://keyserver.ubuntu.com:80', '--recv',
-                               key])
+        if '-----BEGIN PGP PUBLIC KEY BLOCK-----' in key:
+            with NamedTemporaryFile() as key_file:
+                key_file.write(key)
+                key_file.flush()
+                key_file.seek(0)
+                subprocess.check_call(['apt-key', 'add', '-'], stdin=key_file)
+        else:
+            # Note that hkp: is in no way a secure protocol. Using a
+            # GPG key id is pointless from a security POV unless you
+            # absolutely trust your network and DNS.
+            subprocess.check_call(['apt-key', 'adv', '--keyserver',
+                                   'hkp://keyserver.ubuntu.com:80', '--recv',
+                                   key])
 
 
 def configure_sources(update=False,
@@ -238,7 +276,8 @@
     Configure multiple sources from charm configuration.
 
     The lists are encoded as yaml fragments in the configuration.
-    The frament needs to be included as a string.
+    The frament needs to be included as a string. Sources and their
+    corresponding keys are of the types supported by add_source().
 
     Example config:
         install_sources: |

=== modified file 'hooks/hooks.py'
--- hooks/hooks.py	2014-10-08 07:39:16 +0000
+++ hooks/hooks.py	2014-10-14 10:13:22 +0000
@@ -4,6 +4,7 @@
 from contextlib import contextmanager
 import commands
 import cPickle as pickle
+from distutils.version import StrictVersion
 import glob
 from grp import getgrnam
 import os.path
@@ -16,6 +17,7 @@
 from tempfile import NamedTemporaryFile
 from textwrap import dedent
 import time
+import urlparse
 
 from charmhelpers import fetch
 from charmhelpers.core import hookenv, host
@@ -144,6 +146,7 @@
         for relid in hookenv.relation_ids('replication'):
             hookenv.relation_set(relid, replication_state)
 
+        log('saving local state', DEBUG)
         self.save()
 
 
@@ -339,8 +342,17 @@
             "-e", hookenv.config('encoding')]
         if hookenv.config('listen_port'):
             create_cmd.extend(["-p", str(hookenv.config('listen_port'))])
-        create_cmd.append(pg_version())
+        version = pg_version()
+        create_cmd.append(version)
         create_cmd.append(hookenv.config('cluster_name'))
+
+        # With 9.3+, we make an opinionated decision to always enable
+        # data checksums. This seems to be best practice. We could
+        # turn this into a configuration item if there is need. There
+        # is no way to enable this option on existing clusters.
+        if StrictVersion(version) >= StrictVersion('9.3'):
+            create_cmd.extend(['--', '--data-checksums'])
+
         run(create_cmd)
         # Ensure SSL certificates exist, as we enable SSL by default.
         create_ssl_cert(os.path.join(
@@ -396,11 +408,26 @@
         log('Ensuring minimal replication settings')
         config_data['hot_standby'] = True
         config_data['wal_level'] = 'hot_standby'
-        config_data['max_wal_senders'] = max(
-            num_slaves, config_data['max_wal_senders'])
         config_data['wal_keep_segments'] = max(
             config_data['wal_keep_segments'],
             config_data['replicated_wal_keep_segments'])
+        # We need this set even if config_data['streaming_replication']
+        # is False, because the replication connection is still needed
+        # by pg_basebackup to build a hot standby.
+        config_data['max_wal_senders'] = max(
+            num_slaves, config_data['max_wal_senders'])
+
+    # Log shipping to Swift using SwiftWAL. This could be for
+    # non-streaming replication, or for PITR.
+    if config_data.get('swiftwal_log_shipping', None):
+        config_data['archive_mode'] = True
+        config_data['wal_level'] = 'hot_standby'
+        config_data['archive_command'] = swiftwal_archive_command()
+
+    if config_data.get('wal_e_storage_uri', None):
+        config_data['archive_mode'] = True
+        config_data['wal_level'] = 'hot_standby'
+        config_data['archive_command'] = wal_e_archive_command()
 
     # Send config data to the template
     # Return it as pg_config
@@ -419,7 +446,7 @@
 
     tune_postgresql_config(config_file)
 
-    local_state['saved_config'] = config_data
+    local_state['saved_config'] = dict(config_data)
     local_state.save()
 
 
@@ -597,15 +624,16 @@
 def install_postgresql_crontab(output_file):
     '''Create the postgres user's crontab'''
     config_data = hookenv.config()
-    crontab_data = {
-        'backup_schedule': config_data["backup_schedule"],
-        'scripts_dir': postgresql_scripts_dir,
-        'backup_days': config_data["backup_retention_count"],
-    }
+    config_data['scripts_dir'] = postgresql_scripts_dir
+    config_data['swiftwal_backup_command'] = swiftwal_backup_command()
+    config_data['swiftwal_prune_command'] = swiftwal_prune_command()
+    config_data['wal_e_backup_command'] = wal_e_backup_command()
+    config_data['wal_e_prune_command'] = wal_e_prune_command()
+
     charm_dir = hookenv.charm_dir()
     template_file = "{}/templates/postgres.cron.tmpl".format(charm_dir)
     crontab_template = Template(
-        open(template_file).read()).render(crontab_data)
+        open(template_file).read()).render(config_data)
     host.write_file(output_file, crontab_template, perms=0600)
 
 
@@ -624,11 +652,15 @@
     charm_dir = hookenv.charm_dir()
     streaming_replication = hookenv.config('streaming_replication')
     template_file = "{}/templates/recovery.conf.tmpl".format(charm_dir)
-    recovery_conf = Template(open(template_file).read()).render({
-        'host': master_host,
-        'port': master_port,
-        'password': local_state['replication_password'],
-        'streaming_replication': streaming_replication})
+    params = dict(
+        host=master_host, port=master_port,
+        password=local_state['replication_password'],
+        streaming_replication=streaming_replication)
+    if hookenv.config('wal_e_storage_uri'):
+        params['restore_command'] = wal_e_restore_command()
+    elif hookenv.config('swiftwal_log_shipping'):
+        params['restore_command'] = swiftwal_restore_command()
+    recovery_conf = Template(open(template_file).read()).render(params)
     log(recovery_conf, DEBUG)
     host.write_file(
         os.path.join(postgresql_cluster_dir, 'recovery.conf'),
@@ -639,17 +671,164 @@
         postgresql_restart()
 
 
-# ------------------------------------------------------------------------------
-# load_postgresql_config:  Convenience function that loads (as a string) the
-#                          current postgresql configuration file.
-#                          Returns a string containing the postgresql config or
-#                          None
-# ------------------------------------------------------------------------------
-def load_postgresql_config(config_file):
-    if os.path.isfile(config_file):
-        return(open(config_file).read())
+def ensure_swift_container(container):
+    from swiftclient import client as swiftclient
+    config = hookenv.config()
+    con = swiftclient.Connection(
+        authurl=config.get('os_auth_url', ''),
+        user=config.get('os_username', ''),
+        key=config.get('os_password', ''),
+        tenant_name=config.get('os_tenant_name', ''),
+        auth_version='2.0',
+        retries=0)
+    try:
+        con.head_container(container)
+    except swiftclient.ClientException:
+        con.put_container(container)
+
+
+def wal_e_envdir():
+    '''The envdir(1) environment location used to drive WAL-E.'''
+    return os.path.join(_get_postgresql_config_dir(), 'wal-e.env')
+
+
+def create_wal_e_envdir():
+    '''Regenerate the envdir(1) environment used to drive WAL-E.'''
+    config = hookenv.config()
+    env = dict(
+        SWIFT_AUTHURL=config.get('os_auth_url', ''),
+        SWIFT_TENANT=config.get('os_tenant_name', ''),
+        SWIFT_USER=config.get('os_username', ''),
+        SWIFT_PASSWORD=config.get('os_password', ''),
+        AWS_ACCESS_KEY_ID=config.get('aws_access_key_id', ''),
+        AWS_SECRET_ACCESS_KEY=config.get('aws_secret_access_key', ''),
+        WABS_ACCOUNT_NAME=config.get('wabs_account_name', ''),
+        WABS_ACCESS_KEY=config.get('wabs_access_key', ''),
+        WALE_SWIFT_PREFIX='',
+        WALE_S3_PREFIX='',
+        WALE_WABS_PREFIX='')
+
+    uri = config.get('wal_e_storage_uri', None)
+
+    if uri:
+        # Until juju provides us with proper leader election, we have a
+        # state where units do not know if they are alone or part of a
+        # cluster. To avoid units stomping on each others WAL and backups,
+        # we use a unique container for each unit when they are not
+        # part of the peer relation. Once they are part of the peer
+        # relation, they share a container.
+        if local_state.get('state', 'standalone') == 'standalone':
+            if not uri.endswith('/'):
+                uri += '/'
+            uri += hookenv.local_unit().split('/')[-1]
+
+        parsed_uri = urlparse.urlparse(uri)
+
+        required_env = []
+        if parsed_uri.scheme == 'swift':
+            env['WALE_SWIFT_PREFIX'] = uri
+            required_env = ['SWIFT_AUTHURL', 'SWIFT_TENANT',
+                            'SWIFT_USER', 'SWIFT_PASSWORD']
+            ensure_swift_container(parsed_uri.netloc)
+        elif parsed_uri.scheme == 's3':
+            env['WALE_S3_PREFIX'] = uri
+            required_env = ['AWS_ACCESS_KEY_ID', 'AWS_SECRET_ACCESS_KEY']
+        elif parsed_uri.scheme == 'wabs':
+            env['WALE_WABS_PREFIX'] = uri
+            required_env = ['WABS_ACCOUNT_NAME', 'WABS_ACCESS_KEY']
+        else:
+            log('Invalid wal_e_storage_uri {}'.format(uri), ERROR)
+
+        for env_key in required_env:
+            if not env[env_key].strip():
+                log('Missing {}'.format(env_key), ERROR)
+
+    # Regenerate the envdir(1) environment recommended by WAL-E.
+    # All possible keys are rewritten to ensure we remove old secrets.
+    host.mkdir(wal_e_envdir(), 'postgres', 'postgres', 0o750)
+    for k, v in env.items():
+        host.write_file(
+            os.path.join(wal_e_envdir(), k), v.strip(),
+            'postgres', 'postgres', 0o640)
+
+
+def wal_e_archive_command():
+    '''Return the archive_command needed in postgresql.conf.'''
+    return 'envdir {} wal-e wal-push %p'.format(wal_e_envdir())
+
+
+def wal_e_restore_command():
+    return 'envdir {} wal-e wal-fetch "%f" "%p"'.format(wal_e_envdir())
+
+
+def wal_e_backup_command():
+    postgresql_cluster_dir = os.path.join(
+        postgresql_data_dir, pg_version(), hookenv.config('cluster_name'))
+    return 'envdir {} wal-e backup-push {}'.format(
+        wal_e_envdir(), postgresql_cluster_dir)
+
+
+def wal_e_prune_command():
+    return 'envdir {} wal-e delete --confirm retain {}'.format(
+        wal_e_envdir(), hookenv.config('wal_e_backup_retention'))
+
+
+def swiftwal_config():
+    postgresql_config_dir = _get_postgresql_config_dir()
+    return os.path.join(postgresql_config_dir, "swiftwal.conf")
+
+
+def create_swiftwal_config():
+    if not hookenv.config('swiftwal_container_prefix'):
+        return
+
+    # Until juju provides us with proper leader election, we have a
+    # state where units do not know if they are alone or part of a
+    # cluster. To avoid units stomping on each others WAL and backups,
+    # we use a unique Swift container for each unit when they are not
+    # part of the peer relation. Once they are part of the peer
+    # relation, they share a container.
+    if local_state.get('state', 'standalone') == 'standalone':
+        container = '{}_{}'.format(hookenv.config('swiftwal_container_prefix'),
+                                   hookenv.local_unit().split('/')[-1])
     else:
-        return(None)
+        container = hookenv.config('swiftwal_container_prefix')
+
+    template_file = os.path.join(hookenv.charm_dir(),
+                                 'templates', 'swiftwal.conf.tmpl')
+    params = dict(hookenv.config())
+    params['swiftwal_container'] = container
+    content = Template(open(template_file).read()).render(params)
+    host.write_file(swiftwal_config(), content, "postgres", "postgres", 0o600)
+
+
+def swiftwal_archive_command():
+    '''Return the archive_command needed in postgresql.conf'''
+    return 'swiftwal --config={} archive-wal %p'.format(swiftwal_config())
+
+
+def swiftwal_restore_command():
+    '''Return the restore_command needed in recovery.conf'''
+    return 'swiftwal --config={} restore-wal %f %p'.format(swiftwal_config())
+
+
+def swiftwal_backup_command():
+    '''Return the backup command needed in postgres' crontab'''
+    cmd = 'swiftwal --config={} backup --port={}'.format(swiftwal_config(),
+                                                         get_service_port())
+    if not hookenv.config('swiftwal_log_shipping'):
+        cmd += ' --xlog'
+    return cmd
+
+
+def swiftwal_prune_command():
+    '''Return the backup & wal pruning command needed in postgres' crontab'''
+    config = hookenv.config()
+    args = '--keep-backups={} --keep-wals={}'.format(
+        config.get('swiftwal_backup_retention', 0),
+        max(config['wal_keep_segments'],
+            config['replicated_wal_keep_segments']))
+    return 'swiftwal --config={} prune {}'.format(swiftwal_config(), args)
 
 
 def update_service_port():
@@ -810,14 +989,14 @@
     dpkg.communicate(input=selections)
 
 
-# ------------------------------------------------------------------------------
+# -----------------------------------------------------------------------------
 # Core logic for permanent storage changes:
 # NOTE the only 2 "True" return points:
 #   1) symlink already pointing to existing storage (no-op)
 #   2) new storage properly initialized:
 #     - if fresh new storage dir: rsync existing data
 #     - manipulate /var/lib/postgresql/VERSION/CLUSTER symlink
-# ------------------------------------------------------------------------------
+# -----------------------------------------------------------------------------
 def config_changed_volume_apply(mount_point):
     version = pg_version()
     cluster_name = hookenv.config('cluster_name')
@@ -900,13 +1079,6 @@
         return False
 
 
-def token_sql_safe(value):
-    # Only allow alphanumeric + underscore in database identifiers
-    if re.search('[^A-Za-z0-9_]', value):
-        return False
-    return True
-
-
 @hooks.hook()
 def config_changed(force_restart=False, mount_point=None):
     validate_config()
@@ -939,28 +1111,45 @@
     generate_postgresql_hba(postgresql_hba)
     create_ssl_cert(os.path.join(
         postgresql_data_dir, pg_version(), config_data['cluster_name']))
+    create_swiftwal_config()
+    create_wal_e_envdir()
     update_service_port()
     update_nrpe_checks()
 
-    # Ensure client credentials match in the case that an external mountpoint
-    # has been mounted with an existing DB.
-    client_relids = (hookenv.relation_ids('db')
-                     + hookenv.relation_ids('db-admin'))
-    for relid in client_relids:
-        rel = hookenv.relation_get(rid=relid, unit=hookenv.local_unit())
-
-        database = rel.get('database')
-        roles = filter(None, (rel.get('roles') or '').split(","))
-        user = rel['user']
-        password = create_user(user)
-        reset_user_roles(user, roles)
-        schema_user = rel['schema_user']
-        schema_password = create_user(schema_user)
-        hookenv.relation_set(relid,
-                             password=password,
-                             schema_password=schema_password)
-        if not (database is None or database == 'all'):
-            ensure_database(user, schema_user, database)
+    # If an external mountpoint has caused an old, existing DB to be
+    # mounted, we need to ensure that all the users, databases, roles
+    # etc. exist with known passwords.
+    if local_state['state'] in ('standalone', 'master'):
+        client_relids = (
+            hookenv.relation_ids('db') + hookenv.relation_ids('db-admin'))
+        for relid in client_relids:
+            rel = hookenv.relation_get(rid=relid, unit=hookenv.local_unit())
+            client_rel = None
+            for unit in hookenv.related_units(relid):
+                client_rel = hookenv.relation_get(unit=unit, rid=relid)
+            if not client_rel:
+                continue  # No client units - in between departed and broken?
+
+            database = rel.get('database')
+            if database is None:
+                continue  # The relation exists, but we haven't joined it yet.
+
+            roles = filter(None, (client_rel.get('roles') or '').split(","))
+            user = rel.get('user')
+            if user:
+                admin = relid.startswith('db-admin')
+                password = create_user(user, admin=admin)
+                reset_user_roles(user, roles)
+                hookenv.relation_set(relid, password=password)
+
+            schema_user = rel.get('schema_user')
+            if schema_user:
+                schema_password = create_user(schema_user)
+                hookenv.relation_set(relid, schema_password=schema_password)
+
+            if user and schema_user and not (
+                    database is None or database == 'all'):
+                ensure_database(user, schema_user, database)
 
     if force_restart:
         postgresql_restart()
@@ -984,11 +1173,13 @@
     config_data = hookenv.config()
     update_repos_and_packages()
     if 'state' not in local_state:
+        log('state not in {}'.format(local_state.keys()), DEBUG)
         # Fresh installation. Because this function is invoked by both
         # the install hook and the upgrade-charm hook, we need to guard
         # any non-idempotent setup. We should probably fix this; it
         # seems rather fragile.
         local_state.setdefault('state', 'standalone')
+        log(repr(local_state.keys()), DEBUG)
 
         # Drop the cluster created when the postgresql package was
         # installed, and rebuild it with the requested locale and encoding.
@@ -1008,6 +1199,7 @@
             'allocated port {!r} != {!r}'.format(
                 get_service_port(), config_data['listen_port']))
         local_state['port'] = get_service_port()
+        log('publishing state', DEBUG)
         local_state.publish()
 
     postgresql_backups_dir = (
@@ -1598,12 +1790,21 @@
                 "postgresql-contrib-{}".format(version),
                 "postgresql-plpython-{}".format(version),
                 "python-jinja2", "python-psycopg2"]
+
     # PGDG currently doesn't have debversion for 9.3 & 9.4. Put this back
     # when it does.
     if not (hookenv.config('pgdg') and version in ('9.3', '9.4')):
         packages.append("postgresql-{}-debversion".format(version))
+
     if hookenv.config('performance_tuning').lower() != 'manual':
         packages.append('pgtune')
+
+    if hookenv.config('swiftwal_container_prefix'):
+        packages.append('swiftwal')
+
+    if hookenv.config('wal_e_storage_uri'):
+        packages.extend(['wal-e', 'daemontools'])
+
     packages.extend((hookenv.config('extra-packages') or '').split())
     packages = fetch.filter_installed_packages(packages)
     # Set package state for main postgresql package if installed
@@ -1644,9 +1845,10 @@
 
 def authorized_by(unit):
     '''Return True if the peer has authorized our database connections.'''
-    relation = hookenv.relation_get(unit=unit)
-    authorized = relation.get('authorized', '').split()
-    return hookenv.local_unit() in authorized
+    for relid in hookenv.relation_ids('replication'):
+        relation = hookenv.relation_get(unit=unit, rid=relid)
+        authorized = relation.get('authorized', '').split()
+        return hookenv.local_unit() in authorized
 
 
 def promote_database():
@@ -1867,6 +2069,12 @@
             _get_postgresql_config_dir(), "pg_hba.conf")
         generate_postgresql_hba(postgresql_hba)
 
+    # Swift container name make have changed, so regenerate the SwiftWAL
+    # config. This can go away when we have real leader election and can
+    # safely share a single container.
+    create_swiftwal_config()
+    create_wal_e_envdir()
+
     local_state.publish()
 
 
@@ -1884,6 +2092,13 @@
     the master and hot standby joined the client relation.
     '''
     master = local_state['following']
+    if not master:
+        log("I will be a hot standby, but no master yet")
+        return
+
+    if not authorized_by(master):
+        log("Master {} has not yet authorized us".format(master))
+        return
 
     client_relations = hookenv.relation_get(
         'client_relations', master, hookenv.relation_ids('replication')[0])
@@ -1926,7 +2141,7 @@
         # Block until users and database has replicated, so we know the
         # connection details we publish are actually valid. This will
         # normally be pretty much instantaneous.
-        timeout = 900
+        timeout = 60
         start = time.time()
         while time.time() < start + timeout:
             cur = db_cursor(autocommit=True)
@@ -2352,7 +2567,6 @@
 postgresql_sysctl = "/etc/sysctl.d/50-postgresql.conf"
 postgresql_crontab = "/etc/cron.d/postgresql"
 postgresql_service_config_dir = "/var/run/postgresql"
-replication_relation_types = ['master', 'slave', 'replication']
 local_state = State('local_state.pickle')
 hook_name = os.path.basename(sys.argv[0])
 juju_log_dir = "/var/log/juju"
@@ -2369,3 +2583,4 @@
         log("Relation {} with {}".format(
             hookenv.relation_id(), hookenv.remote_unit()))
     hooks.execute(sys.argv)
+    log("Completed {} hook".format(hook_name))

=== added directory 'lib/test-client-charm'
=== added file 'lib/test-client-charm/config.yaml'
--- lib/test-client-charm/config.yaml	1970-01-01 00:00:00 +0000
+++ lib/test-client-charm/config.yaml	2014-10-14 10:13:22 +0000
@@ -0,0 +1,13 @@
+options:
+  database:
+    default: ""
+    type: string
+    description: |
+        Database to connect 'db' relationships to, overriding the
+        generated default database name.
+  roles:
+    default: ""
+    type: string
+    description: |
+        Comma separated list of roles for PostgreSQL to grant to the database
+        user.

=== added directory 'lib/test-client-charm/hooks'
=== added symlink 'lib/test-client-charm/hooks/config-changed'
=== target is u'hooks.py'
=== added symlink 'lib/test-client-charm/hooks/db-admin-relation-broken'
=== target is u'./hooks.py'
=== added symlink 'lib/test-client-charm/hooks/db-admin-relation-changed'
=== target is u'./hooks.py'
=== added symlink 'lib/test-client-charm/hooks/db-admin-relation-joined'
=== target is u'./hooks.py'
=== added symlink 'lib/test-client-charm/hooks/db-relation-broken'
=== target is u'./hooks.py'
=== added symlink 'lib/test-client-charm/hooks/db-relation-changed'
=== target is u'./hooks.py'
=== added symlink 'lib/test-client-charm/hooks/db-relation-joined'
=== target is u'./hooks.py'
=== added file 'lib/test-client-charm/hooks/hooks.py'
--- lib/test-client-charm/hooks/hooks.py	1970-01-01 00:00:00 +0000
+++ lib/test-client-charm/hooks/hooks.py	2014-10-14 10:13:22 +0000
@@ -0,0 +1,165 @@
+#!/usr/bin/env python
+
+import os.path
+import re
+import shutil
+import sys
+from textwrap import dedent
+
+from charmhelpers.core import hookenv, host
+from charmhelpers.core.hookenv import log, DEBUG, INFO
+from charmhelpers import fetch
+
+
+CLIENT_RELATION_TYPES = frozenset(['db', 'db-admin'])
+
+DATA_DIR = os.path.join(
+    '/var/lib/units', hookenv.local_unit().replace('/', '-'))
+SCRIPT_DIR = os.path.join(DATA_DIR, 'bin')
+PGPASS_DIR = os.path.join(DATA_DIR, 'pgpass')
+
+
+def update_system_path():
+    org_lines = open('/etc/environment', 'rb').readlines()
+    env_lines = []
+
+    for line in org_lines:
+        if line.startswith('PATH=') and SCRIPT_DIR not in line:
+            line = re.sub(
+                """(['"]?)$""",
+                ":{}\\1".format(SCRIPT_DIR),
+                line, 1)
+        env_lines.append(line)
+
+    if org_lines != env_lines:
+        content = '\n'.join(env_lines)
+        host.write_file('/etc/environment', content, perms=0o644)
+
+
+def all_relations(relation_types=CLIENT_RELATION_TYPES):
+    for reltype in relation_types:
+        for relid in hookenv.relation_ids(reltype):
+            for unit in hookenv.related_units(relid):
+                yield reltype, relid, unit, hookenv.relation_get(
+                    unit=unit, rid=relid)
+
+
+def rebuild_all_relations():
+    config = hookenv.config()
+
+    # Clear out old scripts and pgpass files
+    if os.path.exists(SCRIPT_DIR):
+        shutil.rmtree(SCRIPT_DIR)
+    if os.path.exists(PGPASS_DIR):
+        shutil.rmtree(PGPASS_DIR)
+    host.mkdir(DATA_DIR, perms=0o755)
+    host.mkdir(SCRIPT_DIR, perms=0o755)
+    host.mkdir(PGPASS_DIR, group='ubuntu', perms=0o750)
+
+    for _, relid, unit, relation in all_relations(relation_types=['db']):
+        log("{} {} {!r}".format(relid, unit, relation), DEBUG)
+
+        def_str = '<DEFAULT>'
+        if config['database'] != relation.get('database', ''):
+            log("Switching from database {} to {}".format(
+                relation.get('database', '') or def_str,
+                config['database'] or def_str), INFO)
+
+        if config['roles'] != relation.get('roles', ''):
+            log("Updating granted roles from {} to {}".format(
+                relation.get('roles', '') or def_str,
+                config['roles'] or def_str))
+
+        hookenv.relation_set(
+            relid, database=config['database'], roles=config['roles'])
+
+        if 'user' in relation:
+            rebuild_relation(relid, unit, relation)
+
+    for _, relid, unit, relation in all_relations(relation_types=['db-admin']):
+        log("{} {} {!r}".format(relid, unit, relation), DEBUG)
+        if 'user' in relation:
+            rebuild_relation(relid, unit, relation)
+
+
+def rebuild_relation(relid, unit, relation):
+    relname = relid.split(':')[0]
+    unitname = unit.replace('/', '-')
+    this_unit = hookenv.local_unit()
+
+    allowed_units = relation.get('allowed-units', '')
+    if this_unit not in allowed_units.split():
+        log("Not yet authorized on {}".format(relid), INFO)
+        return
+
+    script_name = 'psql-{}-{}'.format(relname, unitname)
+    build_script(script_name, relation)
+    state = relation.get('state', None)
+    if state in ('master', 'hot standby'):
+        script_name = 'psql-{}-{}'.format(relname, state.replace(' ', '-'))
+        build_script(script_name, relation)
+
+
+def build_script(script_name, relation):
+    # Install a wrapper to psql that connects it to the desired database
+    # by default. One wrapper per unit per relation.
+    script_path = os.path.abspath(os.path.join(SCRIPT_DIR, script_name))
+    pgpass_path = os.path.abspath(os.path.join(PGPASS_DIR, script_name))
+    script = dedent("""\
+        #!/bin/sh
+        exec env \\
+            PGHOST={host} PGPORT={port} PGDATABASE={database} \\
+            PGUSER={user} PGPASSFILE={pgpass} \\
+            psql $@
+        """).format(
+            host=relation['host'],
+            port=relation['port'],
+            database=relation.get('database', ''),  # db-admin has no database
+            user=relation['user'],
+            pgpass=pgpass_path)
+    log("Generating wrapper {}".format(script_path), INFO)
+    host.write_file(
+        script_path, script, owner="ubuntu", group="ubuntu", perms=0o700)
+
+    # The wrapper requires access to the password, stored in a .pgpass
+    # file so it isn't exposed in an environment variable or on the
+    # command line.
+    pgpass = "*:*:*:{user}:{password}".format(
+        user=relation['user'], password=relation['password'])
+    host.write_file(
+        pgpass_path, pgpass, owner="ubuntu", group="ubuntu", perms=0o400)
+
+
+hooks = hookenv.Hooks()
+
+
[e-mail]()
+def install():
+    fetch.apt_install(
+        ['language-pack-en', 'postgresql-client', 'python-psycopg2'],
+        fatal=True)
+    update_system_path()
+
+
[e-mail]()
+def upgrade_charm():
+    # Per Bug #1205286, we can't store scripts and passwords in the
+    # charm directory.
+    if os.path.exists('bin'):
+        shutil.rmtree('bin')
+    if os.path.exists('pgpass'):
+        shutil.rmtree('pgpass')
+    update_system_path()
+    return rebuild_all_relations()
+
+
[e-mail](
+    'config-changed', 'db-admin-relation-broken',
+    'db-admin-relation-changed', 'db-admin-relation-joined',
+    'db-relation-broken', 'db-relation-changed', 'db-relation-joined')
+def rebuild_hook():
+    return rebuild_all_relations()
+
+
+if __name__ == '__main__':
+    hooks.execute(sys.argv)

=== added symlink 'lib/test-client-charm/hooks/install'
=== target is u'./hooks.py'
=== added file 'lib/test-client-charm/hooks/start'
--- lib/test-client-charm/hooks/start	1970-01-01 00:00:00 +0000
+++ lib/test-client-charm/hooks/start	2014-10-14 10:13:22 +0000
@@ -0,0 +1,3 @@
+#!/bin/sh
+# Do nothing.
+exit 0

=== added file 'lib/test-client-charm/hooks/stop'
--- lib/test-client-charm/hooks/stop	1970-01-01 00:00:00 +0000
+++ lib/test-client-charm/hooks/stop	2014-10-14 10:13:22 +0000
@@ -0,0 +1,3 @@
+#!/bin/sh
+# Do nothing.
+exit 0

=== added symlink 'lib/test-client-charm/hooks/upgrade-charm'
=== target is u'hooks.py'
=== added file 'lib/test-client-charm/metadata.yaml'
--- lib/test-client-charm/metadata.yaml	1970-01-01 00:00:00 +0000
+++ lib/test-client-charm/metadata.yaml	2014-10-14 10:13:22 +0000
@@ -0,0 +1,13 @@
+name: postgresql-psql
+summary: psql command line access to PostgreSQL services.
+maintainer: [e-mail]
+description: |
+    Access to related PostgreSQL services via the
+    standard psql command line utility.
+categories:
+    - databases
+requires:
+    db:
+        interface: pgsql
+    db-admin:
+        interface: pgsql

=== modified file 'templates/postgres.cron.tmpl'
--- templates/postgres.cron.tmpl	2013-11-19 11:38:45 +0000
+++ templates/postgres.cron.tmpl	2014-10-14 10:13:22 +0000
@@ -1,14 +1,16 @@
-{{backup_schedule}} postgres {{scripts_dir}}/pg_backup_job {{backup_days}}
-{% if swiftwal_container -%}
+# Maintained by juju
+#
+{% if backup_schedule -%}
+{{backup_schedule}} postgres \
+    {{scripts_dir}}/pg_backup_job {{backup_retention_count}}
+{% endif -%}
+
 {% if swiftwal_backup_schedule -%}
-{% if swiftwal_log_shipping -%}
-{{swiftwal_backup_schedule}} postgres \
-  swiftwal --config={{swiftwal_config}} backup && \
-  swiftwal --config={{swiftwal_config}} prune -n {{swiftwal_backup_retention}}
-{% else -%}
-{{swiftwal_backup_schedule}} postgres \
-  swiftwal --config={{swiftwal_config}} backup --xlog && \
-  swiftwal --config={{swiftwal_config}} prune -n {{swiftwal_backup_retention}}
-{% endif -%}
-{% endif -%}
+{{swiftwal_backup_schedule}} postgres \
+    {{swiftwal_backup_command}} && {{swiftwal_prune_command}}
+{% endif -%}
+
+{% if wal_e_backup_schedule -%}
+{{wal_e_backup_schedule}} postgres \
+    {{wal_e_backup_command}} && {{wal_e_prune_command}}
 {% endif -%}

=== modified file 'templates/postgresql.conf.tmpl'
--- templates/postgresql.conf.tmpl	2014-05-13 21:05:40 +0000
+++ templates/postgresql.conf.tmpl	2014-10-14 10:13:22 +0000
@@ -120,6 +120,9 @@
 {% if log_lock_waits  != "" -%}
 log_lock_waits = {{log_lock_waits}}
 {% endif -%}
+{% if log_temp_files != "" -%}
+log_temp_files = {{log_temp_files}}
+{% endif -%}
 
 log_timezone = UTC
 

=== modified file 'templates/swiftwal.conf.tmpl'
--- templates/swiftwal.conf.tmpl	2013-11-22 13:41:01 +0000
+++ templates/swiftwal.conf.tmpl	2014-10-14 10:13:22 +0000
@@ -1,6 +1,6 @@
 # Generated and maintained by juju
-OS_USERNAME: {{config.os_username}}
-OS_TENANT_NAME: {{config.os_tenant_name}}
-OS_PASSWORD: {{config.os_password}}
-OS_AUTH_URL: {{config.os_auth_url}}
-CONTAINER: {{vars.container}}
+OS_USERNAME: {{os_username}}
+OS_TENANT_NAME: {{os_tenant_name}}
+OS_PASSWORD: {{os_password}}
+OS_AUTH_URL: {{os_auth_url}}
+CONTAINER: {{swiftwal_container}}

=== modified file 'test.py' (properties changed: +x to -x)
--- test.py	2014-10-08 07:39:16 +0000
+++ test.py	2014-10-14 10:13:22 +0000
@@ -9,7 +9,9 @@
     juju destroy-environment
 """
 
+from datetime import datetime
 import os.path
+import shutil
 import signal
 import socket
 import subprocess
@@ -24,15 +26,40 @@
 from testing.jujufixture import JujuFixture, run
 
 
-SERIES = os.environ.get('SERIES', 'precise').strip()
-TEST_CHARM = 'local:{}/postgresql'.format(SERIES)
-PSQL_CHARM = 'local:{}/postgresql-psql'.format(SERIES)
+SERIES = os.environ.get('SERIES', 'trusty').strip()
+TEST_CHARM = os.path.dirname(__file__)
+PSQL_CHARM = os.path.join(TEST_CHARM, 'lib', 'test-client-charm')
 
 
 class NotReady(Exception):
     pass
 
 
+def skip_if_swift_is_unavailable():
+    os_keys = set(['OS_TENANT_NAME', 'OS_AUTH_URL',
+                   'OS_USERNAME', 'OS_PASSWORD'])
+    for os_key in os_keys:
+        if os_key not in os.environ:
+            return unittest.skip('Swift is unavailable')
+    return lambda x: x
+
+
+def skip_if_s3_is_unavailable():
+    os_keys = set(['AWS_ACCESS_KEY_ID', 'AWS_SECRET_ACCESS_KEY'])
+    for os_key in os_keys:
+        if os_key not in os.environ:
+            return unittest.skip('S3 is unavailable')
+    return lambda x: x
+
+
+def skip_if_wabs_is_unavailable():
+    os_keys = set(['WABS_ACCOUNT_NAME', 'WABS_ACCESS_KEY'])
+    for os_key in os_keys:
+        if os_key not in os.environ:
+            return unittest.skip('WABS is unavailable')
+    return lambda x: x
+
+
 class PostgreSQLCharmBaseTestCase(object):
 
     # Override these in subclasses to run these tests multiple times
@@ -55,12 +82,22 @@
         # Tests may add or change options.
         self.pg_config = dict(version=self.VERSION, pgdg=self.PGDG)
 
+        # Mirror charmhelpers into our support charms, since charms
+        # can't symlink out of their subtree.
+        here = os.path.abspath(os.path.dirname(__file__))
+        main_charmhelpers = os.path.join(here, 'hooks', 'charmhelpers')
+        psql_charmhelpers = os.path.join(here, 'lib', 'test-client-charm',
+                                         'hooks', 'charmhelpers')
+        if os.path.exists(psql_charmhelpers):
+            shutil.rmtree(psql_charmhelpers)
+        shutil.copytree(main_charmhelpers, psql_charmhelpers)
+
         self.juju = self.useFixture(JujuFixture(
             series=SERIES, reuse_machines=True,
             do_teardown='TEST_DONT_TEARDOWN_JUJU' not in os.environ))
 
         # If the charms fail, we don't want tests to hang indefinitely.
-        timeout = int(os.environ.get('TEST_TIMEOUT', 900))
+        timeout = int(os.environ.get('TEST_TIMEOUT', 1200))
         if timeout > 0:
             self.useFixture(fixtures.Timeout(timeout, gentle=True))
 
@@ -72,7 +109,7 @@
         # is at this particular instant in the expected state, hoping
         # that the system is stable enough to continue testing.
 
-        timeout = time.time() + 180
+        timeout = time.time() + 600
         pg_units = frozenset(pg_units)
 
         # The list of PG units we expect to be related to the psql unit.
@@ -104,7 +141,7 @@
             except NotReady:
                 if time.time() > timeout:
                     raise
-                time.sleep(3)
+                time.sleep(10)
 
     def confirm_psql_unit_ready(self, psql_unit, pg_units):
         # Confirm the db and db-admin relations are all in a useful
@@ -276,8 +313,9 @@
             if postgres_unit in full_rel_info[rel_name][rel_id]:
                 rel_info = full_rel_info[rel_name][rel_id][postgres_unit]
                 break
-        assert rel_info is not None, 'Unable to find pg rel info {!r}'.format(
-            full_rel_info[rel_name])
+        assert rel_info is not None, (
+            'Unable to find pg rel info {} {!r}'.format(
+                postgres_unit, full_rel_info[rel_name]))
 
         if dbname is None:
             dbname = rel_info['database']
@@ -295,11 +333,10 @@
         tunnel_cmd = [
             'juju', 'ssh', psql_unit, '-N', '-L',
             '{}:{}:{}'.format(local_port, rel_info['host'], rel_info['port'])]
+        # Don't disable stdout, so we can see when there are SSH
+        # failures like bad host keys.
         tunnel_proc = subprocess.Popen(
             tunnel_cmd, stdin=subprocess.PIPE, preexec_fn=os.setpgrp)
-        # Don't disable stdout, so we can see when there are SSH
-        # failures like bad host keys.
-        # stdout=open('/dev/null', 'ab'), stderr=subprocess.STDOUT)
         tunnel_proc.stdin.close()
 
         try:
@@ -371,6 +408,99 @@
 
         self.assertEqual(num_slaves, 1, 'Slave not connected')
 
+    @skip_if_swift_is_unavailable()
+    def test_swiftwal_logshipping_replication(self):
+        os_keys = set(['OS_TENANT_NAME', 'OS_AUTH_URL',
+                       'OS_USERNAME', 'OS_PASSWORD'])
+        for os_key in os_keys:
+            self.pg_config[os_key.lower()] = os.environ[os_key]
+        self.pg_config['streaming_replication'] = False
+        self.pg_config['swiftwal_log_shipping'] = True
+        self.pg_config['swiftwal_container_prefix'] = '{}_{}'.format(
+            '_juju_pg_tests', datetime.utcnow().strftime('%Y%m%dT%H%M%SZ'))
+        self.pg_config['install_sources'] = 'ppa:stub/pgcharm'
+
+        def swift_cleanup():
+            prefix = self.pg_config['swiftwal_container_prefix']
+            for container in [prefix, prefix + '_1', prefix + '_2']:
+                # Ignore errors and output
+                subprocess.call(['swift', 'delete', container],
+                                stdout=subprocess.PIPE,
+                                stderr=subprocess.STDOUT)
+        self.addCleanup(swift_cleanup)
+
+        self.juju.deploy(
+            TEST_CHARM, 'postgresql', num_units=2, config=self.pg_config)
+        self.juju.deploy(PSQL_CHARM, 'psql')
+        self.juju.do(['add-relation', 'postgresql:db-admin', 'psql:db-admin'])
+        self.wait_until_ready(['postgresql/0', 'postgresql/1'])
+
+        # Confirm that the slave has not opened a streaming
+        # replication connection.
+        num_slaves = self.sql('SELECT COUNT(*) FROM pg_stat_replication',
+                              'master', dbname='postgres')[0][0]
+        self.assertEqual(num_slaves, 0, 'Streaming connection found')
+
+        # Confirm that replication is actually happening.
+        # Create a table and force a WAL change.
+        self.sql('CREATE TABLE foo AS SELECT generate_series(0,100)',
+                 'master', dbname='postgres')
+        self.sql('SELECT pg_switch_xlog()',
+                 'master', dbname='postgres')
+        timeout = time.time() + 120
+        table_found = False
+        while time.time() < timeout and not table_found:
+            time.sleep(1)
+            if self.sql("SELECT TRUE from pg_class WHERE relname='foo'",
+                        'hot standby', dbname='postgres'):
+                table_found = True
+        self.assertTrue(table_found, "Replication not replicating")
+
+    @skip_if_swift_is_unavailable()
+    def test_wal_e_swift_logshipping(self):
+        os_keys = set(['OS_TENANT_NAME', 'OS_AUTH_URL',
+                       'OS_USERNAME', 'OS_PASSWORD'])
+        container = '_juju_pg_tests'
+        for os_key in os_keys:
+            self.pg_config[os_key.lower()] = os.environ[os_key]
+        self.pg_config['streaming_replication'] = False
+        self.pg_config['wal_e_storage_uri'] = 'swift://{}/{}'.format(
+            container, datetime.utcnow().strftime('%Y%m%dT%H%M%SZ'))
+        self.pg_config['install_sources'] = 'ppa:stub/pgcharm'
+
+        def swift_cleanup():
+            subprocess.call(['swift', 'delete', container],
+                            stdout=open(os.devnull, 'wb'),
+                            stderr=subprocess.STDOUT)
+        self.addCleanup(swift_cleanup)
+
+        self.juju.deploy(
+            TEST_CHARM, 'postgresql', num_units=2, config=self.pg_config)
+        self.juju.deploy(PSQL_CHARM, 'psql')
+        self.juju.do(['add-relation', 'postgresql:db-admin', 'psql:db-admin'])
+        self.wait_until_ready(['postgresql/0', 'postgresql/1'])
+
+        # Confirm that the slave has not opened a streaming
+        # replication connection.
+        num_slaves = self.sql('SELECT COUNT(*) FROM pg_stat_replication',
+                              'master', dbname='postgres')[0][0]
+        self.assertEqual(num_slaves, 0, 'Streaming connection found')
+
+        # Confirm that replication is actually happening.
+        # Create a table and force a WAL change.
+        self.sql('CREATE TABLE foo AS SELECT generate_series(0,100)',
+                 'master', dbname='postgres')
+        self.sql('SELECT pg_switch_xlog()',
+                 'master', dbname='postgres')
+        timeout = time.time() + 120
+        table_found = False
+        while time.time() < timeout and not table_found:
+            time.sleep(1)
+            if self.sql("SELECT TRUE from pg_class WHERE relname='foo'",
+                        'hot standby', dbname='postgres'):
+                table_found = True
+        self.assertTrue(table_found, "Replication not replicating")
+
     def test_basic_admin(self):
         '''Connect to a single unit service via the db-admin relationship.'''
         self.juju.deploy(TEST_CHARM, 'postgresql', config=self.pg_config)
@@ -546,7 +676,6 @@
         self.assertIs(False, self.is_master(standby_unit_1, 'postgres'))
 
     def test_admin_addresses(self):
-
         # This test also tests explicit port assignment. We need
         # a different port for each PostgreSQL version we might be
         # testing, because clusters from previous tests of different
@@ -746,6 +875,27 @@
             'rsyslog' not in status['services'], 'rsyslog failed to die')
         self.wait_until_ready(pg_units)
 
+    def test_upgrade_charm(self):
+        self.juju.deploy(
+            TEST_CHARM, 'postgresql', num_units=2, config=self.pg_config)
+        self.juju.deploy(PSQL_CHARM, 'psql')
+        self.juju.do(['add-relation', 'postgresql:db', 'psql:db'])
+        pg_units = ['postgresql/0', 'postgresql/1']
+        self.wait_until_ready(pg_units)
+
+        # Create something
+        self.sql("CREATE TABLE Foo AS SELECT TRUE", 'master')
+
+        # Kick off the upgrade-charm hook
+        self.juju.do(['upgrade-charm', 'postgresql'])
+        self.wait_until_ready(pg_units)
+
+        # Ensure that our data has perservered.
+        master_data = self.sql('SELECT * FROM Foo', 'master')[0][0]
+        self.assertTrue(master_data)
+        standby_data = self.sql('SELECT * FROM Foo', 'hot standby')[0][0]
+        self.assertTrue(standby_data)
+
 
 class PG91Tests(
         PostgreSQLCharmBaseTestCase,

=== modified file 'testing/jujufixture.py'
--- testing/jujufixture.py	2014-06-10 11:36:17 +0000
+++ testing/jujufixture.py	2014-10-14 10:13:22 +0000
@@ -21,6 +21,7 @@
         super(JujuFixture, self).__init__()
 
         self.series = series
+        assert series, 'series not set'
 
         self.reuse_machines = reuse_machines
 
@@ -47,6 +48,9 @@
     def deploy(self, charm, name=None, num_units=1, config=None):
         cmd = ['deploy']
 
+        if not charm.startswith('cs:'):
+            charm = self.charm_uri(charm)
+
         if config:
             config_path = os.path.join(
                 self.useFixture(fixtures.TempDir()).path, 'config.yaml')
@@ -143,11 +147,8 @@
                                     rel_id, rel_unit)])
                         res[rel_name][rel_id][rel_unit] = json.loads(
                             json_rel_info)
-                    except subprocess.CalledProcessError as x:
-                        if x.returncode == 2:
-                            res[rel_name][rel_id][rel_unit] = None
-                        else:
-                            raise
+                    except subprocess.CalledProcessError:
+                        res[rel_name][rel_id][rel_unit] = None
         return res
 
     def wait_until_ready(self, extra=60):
@@ -185,6 +186,27 @@
         if self.do_teardown:
             self.addCleanup(self.reset)
 
+        # Setup a temporary repository with the magic to find our charms.
+        self.repo_dir = self.useFixture(fixtures.TempDir()).path
+        self.useFixture(fixtures.EnvironmentVariable('JUJU_REPOSITORY',
+                                                     self.repo_dir))
+        self.repo_series_dir = os.path.join(self.repo_dir, self.series)
+        os.mkdir(self.repo_series_dir)
+
+    def charm_uri(self, charm):
+        meta = yaml.safe_load(open(os.path.join(charm, 'metadata.yaml'), 'rb'))
+        name = meta.get('name')
+        assert name, 'charm {} has no name in metadata.yaml'.format(charm)
+        charm_link = os.path.join(self.repo_series_dir, name)
+
+        # Recreate the charm link, which might have changed from the
+        # last deploy.
+        if os.path.exists(charm_link):
+            os.remove(charm_link)
+        os.symlink(charm, os.path.join(self.repo_series_dir, name))
+
+        return 'local:{}/{}'.format(self.series, name)
+
     def reset(self):
         # Tear down any services left running that we know we spawned.
         while True: