Merge lp:~stub/launchpad/replication into lp:launchpad/db-devel

[Stuart Bishop]

[Stuart Bishop (stub)]

A standalone authentication service is being assembled, and needs to be tested on staging (it has to be staging as things are still partially integrated). There are some extra tables and data that is contained in the database dumps, but we need to teach the staging rebuild scripts to not mess with them by trying to stuff them into the Launchpad replication sets.

[Jonathan Lange (jml)]

Hey Stuart,

The branch looks good to me. A couple of minor points though.

 * In database/replication/helpers.py, the new items in the list should probably be newline separated
 * In the same file, it's not clear how you figured out those table names, nor is it clear how I'd find out how you figured them out. Would it be possible to expand the comment explaining how they were determined?

jml

review: Needs Fixing

[Jonathan Lange (jml)]

Thanks. You've spelled "separate" incorrectly. Other than that, looks fine.

review: Approve

=== modified file 'database/replication/Makefile'
--- database/replication/Makefile	2009-12-14 13:00:03 +0000
+++ database/replication/Makefile	2010-01-08 05:18:15 +0000
@@ -100,7 +100,7 @@
 	# up when roles don't exist in this cluster, and we rebuild it later
 	# with security.py anyway.
 	pg_restore --dbname=lpmain_staging_new \
-	    --no-acl --no-owner --exit-on-error ${STAGING_DUMP}
+	    --no-owner --exit-on-error ${STAGING_DUMP}
 	psql -q -d lpmain_staging_new -f authdb_drop.sql
 	psql -q -d lpmain_staging_new -f authdb_create.sql \
 	    2>&1 | grep -v _sl || true
@@ -187,7 +187,9 @@
 	@echo Running fti.py `date`
 	${SHHH} ../schema/fti.py
 	@echo Running security.py `date`
+	./slon_ctl.py stop # security.py can deadlock with slony
 	${SHHH} ../schema/security.py --cluster -U slony
+	./slon_ctl.py --lag="0 seconds" start
 	# Migrate tables to the authdb replication set, creating the set
 	# and subscribing nodes to it as necessary.
 	./populate_auth_replication_set.py -U slony

=== modified file 'database/replication/helpers.py'
--- database/replication/helpers.py	2009-11-30 11:35:04 +0000
+++ database/replication/helpers.py	2010-01-08 05:18:15 +0000
@@ -71,8 +71,27 @@
     'public.lp_person',
     'public.lp_personlocation',
     'public.lp_teamparticipation',
+    # Ubuntu SSO database. These tables where created manually by ISD
+    # and the Launchpad scripts should not mess with them. Eventually
+    # these tables will be in a totally separate database.
+    'public.auth_permission',
+    'public.auth_group',
+    'public.auth_user',
+    'public.auth_message',
+    'public.django_content_type',
+    'public.auth_permission',
+    'public.django_session',
+    'public.django_site',
+    'public.django_admin_log',
+    'public.ssoopenidrpconfig',
+    'public.auth_group_permissions',
+    'public.auth_user_groups',
+    'public.auth_user_user_permissions',
     ])
 
+# Calculate IGNORED_SEQUENCES
+IGNORED_SEQUENCES = set('%s_id_seq' % table for table in IGNORED_TABLES)
+
 
 def slony_installed(con):
     """Return True if the connected database is part of a Launchpad Slony-I
@@ -447,7 +466,7 @@
 
     return (
         all_tables - replicated_tables - IGNORED_TABLES,
-        all_sequences - replicated_sequences)
+        all_sequences - replicated_sequences - IGNORED_SEQUENCES)
 
 
 class ReplicationConfigError(Exception):

=== modified file 'database/replication/initialize.py'
--- database/replication/initialize.py	2009-10-17 14:06:03 +0000
+++ database/replication/initialize.py	2010-01-08 05:18:15 +0000
@@ -224,7 +224,8 @@
             fails += 1
     for sequence in all_sequences_in_schema(cur, 'public'):
         times_seen = 0
-        for sequence_set in [authdb_sequences, lpmain_sequences]:
+        for sequence_set in [
+            authdb_sequences, lpmain_sequences, helpers.IGNORED_SEQUENCES]:
             if sequence in sequence_set:
                 times_seen += 1
         if times_seen == 0:

=== added file 'database/replication/sync.py'
--- database/replication/sync.py	1970-01-01 00:00:00 +0000
+++ database/replication/sync.py	2010-01-08 05:18:15 +0000
@@ -0,0 +1,26 @@
+#!/usr/bin/python2.5
+#
+# Copyright 2010 Canonical Ltd.  This software is licensed under the
+# GNU Affero General Public License version 3 (see the file LICENSE).
+
+"""Block until the replication cluster synchronizes."""
+
+__metaclass__ = type
+__all__ = []
+
+import _pythonpath
+
+from optparse import OptionParser
+
+from canonical.launchpad.scripts import logger_options, db_options
+from replication.helpers import sync
+
+if __name__ == '__main__':
+    parser = OptionParser()
+    parser.add_option(
+        "-t", "--timeout", dest="timeout", metavar="SECS", type="int",
+        help="Abort if no sync after SECS seconds.", default=0)
+    logger_options(parser)
+    db_options(parser)
+    options, args = parser.parse_args()
+    sync(options.timeout)