Merge lp:~stub/launchpad/dbpolicy-syntax into lp:launchpad

[Stuart Bishop]

Turn database policies into Python context managers for use with the 'with' statement

[Stuart Bishop (stub)]

People have been asking how to get scripts making use of the slave databases (yay!). This can already be done, but I'm taking the opportunity to improve the syntax of installing database policies and thus nicer, readable documentation (yay!).

[lp:~stub/launchpad/dbpolicy-syntax]

10357. By Stuart Bishop on 2010-02-22

Interfaces don't have self arguments

10358. By Stuart Bishop on 2010-02-22

typos

[Aaron Bentley (abentley)]

review: Approve

=== added file 'lib/canonical/launchpad/doc/db-policy.txt'
--- lib/canonical/launchpad/doc/db-policy.txt	1970-01-01 00:00:00 +0000
+++ lib/canonical/launchpad/doc/db-policy.txt	2010-02-22 12:18:22 +0000
@@ -0,0 +1,126 @@
+Storm Stores & Database Policies
+================================
+
+Launchpad has multiple master and slave databases. Changes to data are
+made on the master and replicated asynchronously to the slave
+databases. Slave databases will usually lag a few seconds behind their
+master. Under high load they may lag a few minutes behind, during
+maintenance they may lag a few hours behind and if things explode
+while admins are on holiday they may lag days behind.
+
+If know your code needs to change data, or must have the latest posible
+information, you retrieve objects from the master databases that stores
+the data for your database class.
+
+    >>> from canonical.launchpad.interfaces.lpstorm import IMasterStore
+    >>> from lp.registry.model.person import Person
+    >>> import transaction
+
+    >>> writable_janitor = IMasterStore(Person).find(
+    ...     Person, Person.name == 'janitor').one()
+
+    >>> writable_janitor.displayname = 'Jack the Janitor'
+    >>> transaction.commit()
+
+Sometimes though we know we will not make changes and don't care much
+if the information is a little out of date. In these cases you should
+explicitly retrieve objects from a slave.
+
+The more agressively we retrieve objects from slave databases instead
+of the master, the better the overall performance of Launchpad will be.
+We can distribute this load over many slave databases but are limited to
+a single master.
+
+    >>> from canonical.launchpad.interfaces.lpstorm import ISlaveStore
+    >>> ro_janitor = ISlaveStore(Person).find(
+    ...     Person, Person.name == 'janitor').one()
+    >>> ro_janitor is writable_janitor
+    False
+
+    >>> ro_janitor.displayname = 'Janice the Janitor'
+    >>> transaction.commit()
+    Traceback (most recent call last):
+    ...
+    InternalError: transaction is read-only
+
+    >>> transaction.abort()
+
+Much of our code does not know if the objects being retrieved need to be
+updatable to or have to be absolutely up to date. In this case, we
+retrieve objects from the default store. What object being returned
+depends on the currently installed database policy.
+
+    >>> from canonical.launchpad.interfaces.lpstorm import IStore
+    >>> default_janitor = IStore(Person).find(
+    ...     Person, Person.name == 'janitor').one()
+    >>> default_janitor is writable_janitor
+    True
+
+As you can see, the default database policy retrieves objects from
+the master database. This allows our code written before database
+replication was implemented to keep working.
+
+To alter this behavior, you can install a different database policy.
+
+    >>> from canonical.launchpad.webapp.dbpolicy import SlaveDatabasePolicy
+    >>> with SlaveDatabasePolicy():
+    ...     default_janitor = IStore(Person).find(
+    ...         Person, Person.name == 'janitor').one()
+    >>> default_janitor is writable_janitor
+    False
+
+The database policy can also affect what happens when objects are
+explicitly retrieved from a slave or master database. For example,
+if we have code that needs to run during database maintenance or
+code we want to prove only accesses slave database resources, we can
+raise an exception if an attempt is made to access master database
+resources.
+
+    >>> from canonical.launchpad.webapp.dbpolicy import (
+    ...     SlaveOnlyDatabasePolicy)
+    >>> with SlaveOnlyDatabasePolicy():
+    ...     whoops = IMasterStore(Person).find(
+    ...         Person, Person.name == 'janitor').one()
+    Traceback (most recent call last):
+    ...
+    DisallowedStore: master
+
+We can even ensure no database activity occurs at all, for instance
+if we need to guarantee a potentially long running call doesn't access
+the database at all starting a new and potentially long running
+database transaction.
+
+    >>> from canonical.launchpad.webapp.dbpolicy import DatabaseBlockedPolicy
+    >>> with DatabaseBlockedPolicy():
+    ...     whoops = IStore(Person).find(
+    ...         Person, Person.name == 'janitor').one()
+    Traceback (most recent call last):
+    ...
+    DisallowedStore: ('main', 'default')
+
+Database policies can also be installed and uninstalled using the
+IStoreSelector utility for cases where the 'with' syntax cannot
+be used.
+
+    >>> from canonical.launchpad.webapp.interfaces import IStoreSelector
+    >>> getUtility(IStoreSelector).push(SlaveDatabasePolicy())
+    >>> try:
+    ...     default_janitor = IStore(Person).find(
+    ...         Person, Person.name == 'janitor').one()
+    ... finally:
+    ...     db_policy = getUtility(IStoreSelector).pop()
+    >>> default_janitor is ro_janitor
+    True
+
+Casting
+-------
+
+If you need to change an object you have a read only copy of, or are
+unsure if the object is writable or not, you can easily cast it
+to a writable copy. This is a noop if the object is already writable
+so is good defensive programming.
+
+    >>> from canonical.launchpad.interfaces.lpstorm import IMasterObject
+    >>> IMasterObject(ro_janitor) is writable_janitor
+    True
+

=== modified file 'lib/canonical/launchpad/doc/storm.txt'
--- lib/canonical/launchpad/doc/storm.txt	2009-08-21 17:43:28 +0000
+++ lib/canonical/launchpad/doc/storm.txt	2010-02-22 12:18:22 +0000
@@ -1,3 +1,8 @@
+Note: A more readable version of this is in db-policy.txt. Most of this
+doctest will disappear soon when the auth replication set is collapsed
+back into the main replication set as part of login server seperation.
+-- StuartBishop 20100222
+
 In addition to what Storm provides, we also have some Launchpad
 specific Storm tools to cope with our master and slave store arrangement.
 

=== modified file 'lib/canonical/launchpad/ftests/test_system_documentation.py'
--- lib/canonical/launchpad/ftests/test_system_documentation.py	2010-02-10 23:14:56 +0000
+++ lib/canonical/launchpad/ftests/test_system_documentation.py	2010-02-22 12:18:22 +0000
@@ -7,6 +7,8 @@
 """
 # pylint: disable-msg=C0103
 
+from __future__ import with_statement
+
 import logging
 import os
 import unittest
@@ -391,6 +393,8 @@
         one_test = LayeredDocFileSuite(
             path, setUp=setUp, tearDown=tearDown,
             layer=LaunchpadFunctionalLayer,
+            # 'icky way of running doctests with __future__ imports
+            globs={'with_statement': with_statement},
             stdout_logging_level=logging.WARNING
             )
         suite.addTest(one_test)

=== modified file 'lib/canonical/launchpad/webapp/dbpolicy.py'
--- lib/canonical/launchpad/webapp/dbpolicy.py	2010-01-20 22:09:26 +0000
+++ lib/canonical/launchpad/webapp/dbpolicy.py	2010-02-22 12:18:22 +0000
@@ -111,6 +111,17 @@
         """See `IDatabasePolicy`."""
         pass
 
+    def __enter__(self):
+        """See `IDatabasePolicy`."""
+        getUtility(IStoreSelector).push(self)
+
+    def __exit__(self, exc_type, exc_value, traceback):
+        """See `IDatabasePolicy`."""
+        policy = getUtility(IStoreSelector).pop()
+        assert policy is self, (
+            "Unexpected database policy %s returned by store selector"
+            % repr(policy))
+
 
 class DatabaseBlockedPolicy(BaseDatabasePolicy):
     """`IDatabasePolicy` that blocks all access to the database."""

=== modified file 'lib/canonical/launchpad/webapp/interfaces.py'
--- lib/canonical/launchpad/webapp/interfaces.py	2010-02-17 11:13:06 +0000
+++ lib/canonical/launchpad/webapp/interfaces.py	2010-02-22 12:18:22 +0000
@@ -756,6 +756,20 @@
     The publisher adapts the request to `IDatabasePolicy` to
     instantiate the policy for the current request.
     """
+    def __enter__():
+        """Standard Python context manager interface.
+
+        The IDatabasePolicy will install itself using the IStoreSelector
+        utility.
+        """
+
+    def __exit__(exc_type, exc_value, traceback):
+        """Standard Python context manager interface.
+
+        The IDatabasePolicy will uninstall itself using the IStoreSelector
+        utility.
+        """
+
     def getStore(name, flavor):
         """Retrieve a Store.