Merge lp:~stub/charms/precise/postgresql/use-swiftwal into lp:charms/postgresql

[Stuart Bishop]

Experimental SwiftWAL support, for PostgreSQL log shipping and PITR into Swift.

SwiftWAL is a tool I've written to do PostgreSQL log shipping into Swift. This branch adds experimental support for the tool. Experimental because there are no tests, and we need it in the charm to test it with real staging and production environments.

Unfortunatly, the OpenStack Swift charms do not work with the local provider. I'd like to have a test suite for this functionality before promoting this from experimental to supported.

[lp:~stub/charms/precise/postgresql/use-swiftwal]

118. By Stuart Bishop on 2014-01-08

Merged tests into use-swiftwal.

119. By Stuart Bishop on 2014-01-08

Merge trunk

120. By Stuart Bishop on 2014-01-08

Merged tests into use-swiftwal.

121. By Stuart Bishop on 2014-01-08

Merged tests into use-swiftwal.

[Marco Ceppi (marcoceppi)]

Approved, please make sure when this is no longer experimental that the README gets updated with instructions

review: Approve

[lp:~stub/charms/precise/postgresql/use-swiftwal]

122. By Stuart Bishop on 2014-01-15

Merged tests into use-swiftwal.

123. By Stuart Bishop on 2014-01-22

Merged tests into use-swiftwal.

124. By Stuart Bishop on 2014-01-22

Remove cruft

=== modified file 'config.yaml'
--- config.yaml	2014-01-17 03:18:52 +0000
+++ config.yaml	2014-01-22 07:21:40 +0000
@@ -343,3 +343,60 @@
         An advisory lock key used internally by the charm. You do not need
         to change it unless it happens to conflict with an advisory lock key
         being used by your applications.
+
+  # Swift backups and PITR via SwiftWAL
+  swiftwal_container_prefix:
+    type: string
+    default: ""
+    description: |
+      EXPERIMENTAL.
+      Swift container prefix for SwiftWAL to use. Must be set if any
+      SwiftWAL features are enabled.
+  swiftwal_backup_schedule:
+    type: string
+    default: ""
+    description: |
+      EXPERIMENTAL.
+      Cron-formatted schedule for SwiftWAL database backups.
+  swiftwal_backup_retention:
+    type: int
+    default: 2
+    description: |
+      EXPERIMENTAL.
+      Number of recent base backups to retain. You need enough space in
+      Swift for this many backups plus one more, as an old backup will only
+      be removed after a new one has been successfully made to replace it.
+  swiftwal_log_shipping:
+    type: boolean
+    default: false
+    description: |
+      EXPERIMENTAL.
+      Archive WAL files into Swift. If swiftwal_backup_schedule is set,
+      this allows point-in-time recovery and WAL files are removed
+      automatically with old backups. If swiftwal_backup_schedule is not set
+      then WAL files are never removed. Enabling this option will override
+      the archive_mode and archive_command settings.
+  streaming_replication:
+    type: boolean
+    default: true
+    description: |
+      EXPERIMENTAL.
+      Enable streaming replication. Normally, streaming replication is
+      always used, and any log shipping configured is used as a fallback.
+      Turning this off without configuring log shipping is an error.
+  os_username:
+    type: string
+    default: ""
+    description: EXPERIMENTAL. OpenStack Swift username.
+  os_password:
+    type: string
+    default: ""
+    description: EXPERIMENTAL. OpenStack Swift password.
+  os_auth_url:
+    type: string
+    default: ""
+    description: EXPERIMENTAL. OpenStack Swift authentication URL.
+  os_tenant_name:
+    type: string
+    default: ""
+    description: EXPERIMENTAL. OpenStack Swift tenant name.

=== modified file 'hooks/hooks.py' (properties changed: +x to -x)
=== modified file 'templates/pg_hba.conf.tmpl'
--- templates/pg_hba.conf.tmpl	2013-04-22 20:55:37 +0000
+++ templates/pg_hba.conf.tmpl	2014-01-22 07:21:40 +0000
@@ -3,8 +3,9 @@
 #------------------------------------------------------------------------------
 
 # Database administrative login by UNIX sockets
-local   all        postgres                          ident map=superusers
-local   all        nagios                            md5
+local   all         postgres                          ident map=superusers
+local   replication root,postgres                     ident map=superusers
+local   all         nagios                            md5
 
 {% if access_list is defined -%}
 {% for unit in access_list -%}

=== modified file 'templates/postgres.cron.tmpl'
--- templates/postgres.cron.tmpl	2013-02-05 14:09:18 +0000
+++ templates/postgres.cron.tmpl	2014-01-22 07:21:40 +0000
@@ -1,2 +1,14 @@
 {{backup_schedule}} postgres {{scripts_dir}}/pg_backup_job {{backup_days}}
-
+{% if swiftwal_container -%}
+{% if swiftwal_backup_schedule -%}
+{% if swiftwal_log_shipping -%}
+{{swiftwal_backup_schedule}} postgres \
+  swiftwal --config={{swiftwal_config}} backup && \
+  swiftwal --config={{swiftwal_config}} prune -n {{swiftwal_backup_retention}}
+{% else -%}
+{{swiftwal_backup_schedule}} postgres \
+  swiftwal --config={{swiftwal_config}} backup --xlog && \
+  swiftwal --config={{swiftwal_config}} prune -n {{swiftwal_backup_retention}}
+{% endif -%}
+{% endif -%}
+{% endif -%}

=== modified file 'templates/recovery.conf.tmpl'
--- templates/recovery.conf.tmpl	2013-04-03 14:25:47 +0000
+++ templates/recovery.conf.tmpl	2014-01-22 07:21:40 +0000
@@ -2,4 +2,10 @@
 # This file is managed by Juju
 #------------------------------------------------------------------------------
 standby_mode = on
+recovery_target_timeline = latest
+{% if streaming_replication %}
 primary_conninfo = 'host={{host}} user=juju_replication password={{password}} requirepeer=postgres'
+{% endif %}
+{% if restore_command %}
+restore_command = '{{restore_command}}'
+{% endif %}

=== added file 'templates/swiftwal.conf.tmpl'
--- templates/swiftwal.conf.tmpl	1970-01-01 00:00:00 +0000
+++ templates/swiftwal.conf.tmpl	2014-01-22 07:21:40 +0000
@@ -0,0 +1,6 @@
+# Generated and maintained by juju
+OS_USERNAME: {{config.os_username}}
+OS_TENANT_NAME: {{config.os_tenant_name}}
+OS_PASSWORD: {{config.os_password}}
+OS_AUTH_URL: {{config.os_auth_url}}
+CONTAINER: {{vars.container}}

=== modified file 'test.py'
--- test.py	2013-12-23 10:35:30 +0000
+++ test.py	2014-01-22 07:21:40 +0000
@@ -39,7 +39,8 @@
 
         # If the charms fail, we don't want tests to hang indefinitely.
         timeout = int(os.environ.get('TEST_TIMEOUT', 900))
-        self.useFixture(fixtures.Timeout(timeout, gentle=True))
+        if timeout > 0:
+            self.useFixture(fixtures.Timeout(timeout, gentle=True))
 
     def sql(self, sql, postgres_unit=None, psql_unit=None, dbname=None):
         '''Run some SQL on postgres_unit from psql_unit.

=== modified file 'testing/jujufixture.py'
--- testing/jujufixture.py	2013-12-23 10:35:30 +0000
+++ testing/jujufixture.py	2014-01-22 07:21:40 +0000
@@ -104,7 +104,7 @@
 
         return self.status
 
-    def wait_until_ready(self, extra=45):
+    def wait_until_ready(self, extra=60):
         ready = False
         while not ready:
             self.refresh_status()