Merge lp:~mwhudson/launchpad/aiee-everything-is-broken into lp:launchpad

[Michael Hudson-Doyle]

Fix implausible breakage in the test environment by changing the launchpadlib Launchpad object pagetest glob to a function that creates one. Fix a couple of tests that were failing but not detected as such.

http://<email address hidden>/msg02999.html contains the long winded explanation.

More fixes might have to be accumulated into this branch if the current run in ec2 finds more lurking test failures.

[lp:~mwhudson/launchpad/aiee-everything-is-broken]

10598. By Michael Hudson-Doyle on 2010-03-26

fix xx-product-add.txt

10599. By Michael Hudson-Doyle on 2010-03-26

reenable the launchpadlib test using a new getLaunchpadObjectForPerson glob

10600. By Michael Hudson-Doyle on 2010-03-26

docstrings

[Tim Penhey (thumper)]

Looked at http://pastebin.ubuntu.com/401568/.

It's all good.

review: Approve

[lp:~mwhudson/launchpad/aiee-everything-is-broken]

10601. By Michael Hudson-Doyle on 2010-03-28

grr

10602. By Michael Hudson-Doyle on 2010-03-28

merge some trunk

10603. By Michael Hudson-Doyle on 2010-03-28

null merge trunk r10598.1.1

10604. By Michael Hudson-Doyle on 2010-03-28

merge rest of trunk

Unmerged revisions

10604. By Michael Hudson-Doyle on 2010-03-28

merge rest of trunk

10603. By Michael Hudson-Doyle on 2010-03-28

null merge trunk r10598.1.1

10602. By Michael Hudson-Doyle on 2010-03-28

merge some trunk

10601. By Michael Hudson-Doyle on 2010-03-28

grr

10600. By Michael Hudson-Doyle on 2010-03-26

docstrings

10599. By Michael Hudson-Doyle on 2010-03-26

reenable the launchpadlib test using a new getLaunchpadObjectForPerson glob

10598. By Michael Hudson-Doyle on 2010-03-26

fix xx-product-add.txt

10597. By William Grant on 2010-03-25

fix only known failure that has crept in

10596. By Michael Hudson-Doyle on 2010-03-25

remove launchpad from pagetest globs; bowlderize page test

=== modified file 'Makefile'
--- Makefile	2010-03-26 13:23:20 +0000
+++ Makefile	2010-03-28 20:04:28 +0000
@@ -324,11 +324,7 @@
 	$(RM) -r lib/mailman
 	$(RM) -rf lib/canonical/launchpad/icing/build/*
 	$(RM) -r $(CODEHOSTING_ROOT)
-	mv $(APIDOC_DIR)/wadl-testrunner-devel.xml \
-	    $(APIDOC_DIR)/wadl-testrunner-devel.xml.bak
 	$(RM) $(APIDOC_DIR)/wadl*.xml $(APIDOC_DIR)/*.html
-	mv $(APIDOC_DIR)/wadl-testrunner-devel.xml.bak \
-	    $(APIDOC_DIR)/wadl-testrunner-devel.xml
 	$(RM) -rf $(APIDOC_DIR).tmp
 	$(RM) $(BZR_VERSION_INFO)
 	$(RM) _pythonpath.py

=== removed file 'lib/canonical/launchpad/apidoc/wadl-testrunner-devel.xml'
--- lib/canonical/launchpad/apidoc/wadl-testrunner-devel.xml	2010-03-26 16:47:42 +0000
+++ lib/canonical/launchpad/apidoc/wadl-testrunner-devel.xml	1970-01-01 00:00:00 +0000
@@ -1,10 +0,0 @@
-<?xml version="1.0"?>
-<wadl:application xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
-                  xmlns="http://research.sun.com/wadl/2006/10"
-                  xmlns:wadl="http://research.sun.com/wadl/2006/10"
-                  xsi:schemaLocation="http://research.sun.com/wadl/2006/10/wadl.xsd">
-
-  <!--This file is for testing purposes only.  See
-      canonical/launcpad/pagetests/webservice/xx-wadl.txt -->
-
-</wadl:application>

=== added file 'lib/canonical/launchpad/pagetests/webservice/launchpadlib.txt'
--- lib/canonical/launchpad/pagetests/webservice/launchpadlib.txt	1970-01-01 00:00:00 +0000
+++ lib/canonical/launchpad/pagetests/webservice/launchpadlib.txt	2010-03-28 20:04:28 +0000
@@ -0,0 +1,16 @@
+Using launchpadlib in pagetests
+===============================
+
+As an alternative to crafting HTTP requests with the 'webservice'
+object, you can write pagetests using launchpadlib. The
+'getLaunchpadObjectForPerson' function returns a launchpadlib
+Launchpad object that's pre-authenticated as the passed in
+user. Anything you can do with 'webservice', you can do with this
+Lanchpad object.
+
+  >>> login(ANONYMOUS)
+  >>> bob = factory.makePerson(name='bob')
+  >>> logout()
+  >>> launchpad = getLaunchpadObjectForPerson(bob)
+  >>> print launchpad.me.name
+  bob

=== modified file 'lib/canonical/launchpad/pagetests/webservice/xx-wadl.txt'
--- lib/canonical/launchpad/pagetests/webservice/xx-wadl.txt	2010-03-26 13:23:20 +0000
+++ lib/canonical/launchpad/pagetests/webservice/xx-wadl.txt	2010-03-28 20:04:28 +0000
@@ -3,55 +3,83 @@
 Because Launchpad's main WADL files are so big, we cache them
 internally: one WADL file for every version of the web service.
 Because the WADL only changes when the Launchpad software changes,
-these documents are cached to files.  Right now, the Launchpad
-webservice serves a special test file
-(canonical/launchpad/apidoc/wadl-testrunner-devel.xml) when a client
-asks for the big WADL definition for the 'devel' version. The
-'testrunner' part comes from canonical.config.config.instance_name, so
-a development instance will use the file
-canonical/launchpad/apidoc/wadl-development-{version}.xml.
-
-    >>> test_wadl = webservice.get(
-    ...     '/', 'application/vd.sun.wadl+xml', api_version='devel').body
-    >>> print test_wadl
-    <?xml version="1.0"?>
-    <wadl:application xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
-                      xmlns="http://research.sun.com/wadl/2006/10"
-                      xmlns:wadl="http://research.sun.com/wadl/2006/10"
-                      xsi:schemaLocation="http://research.sun.com/wadl/2006/10/wadl.xsd">
-    <BLANKLINE>
-      <!--This file is for testing purposes only.  See
-          canonical/launcpad/pagetests/webservice/xx-wadl.txt -->
-    <BLANKLINE>
-    </wadl:application>
-    <BLANKLINE>
-
-Let's look at the real contents, though.  To do this, we need to
-deactivate the cache. Simply clearing it out will just cause it to be
-filled up again.
+these documents are cached to files.
+
+This test shows how the cache works. We'll start by temporarily
+clearing the cache.
 
     >>> from canonical.launchpad.systemhomes import WebServiceApplication
     >>> old_cached_wadl = WebServiceApplication.cached_wadl
-    >>> WebServiceApplication.cached_wadl = None
+    >>> WebServiceApplication.cached_wadl = {}
+
+If WADL is present in a certain file on disk--the filename depends on
+the Launchpad configuration and the web service version--it will be
+loaded from disk and not generated from scratch. But the testrunner
+does not have any WADL files written to disk.
+
+    >>> import os
+    >>> wadl_filename = WebServiceApplication.cachedWADLPath(
+    ...     'testrunner', 'devel')
+    >>> os.path.exists(wadl_filename)
+    False
+
+Let's write some fake WADL to disk.
+
+    >>> fd = open(wadl_filename, "w")
+    >>> fd.write("Some fake WADL.")
+    >>> fd.close()
+
+When we request the WADL for version "devel", the fake WADL is loaded
+from disk.
+
+    >>> print webservice.get(
+    ...     '/', 'application/vd.sun.wadl+xml', api_version='devel').body
+    Some fake WADL.
+
+The fake WADL is now present in the cache.
+
+    >>> WebServiceApplication.cached_wadl
+    {u'devel': u'Some fake WADL.'}
+
+Change the cached value, and we change the document served.
+
+    >>> WebServiceApplication.cached_wadl['devel'] = "More fake WADL."
+
+    >>> print webservice.get(
+    ...     '/', 'application/vd.sun.wadl+xml', api_version='devel').body
+    More fake WADL.
+
+If there's no value in the cache and no cached file on disk, the WADL
+is generated from scratch.
+
+    >>> WebServiceApplication.cached_wadl = {}
+    >>> os.remove(wadl_filename)
 
     >>> wadl = webservice.get(
     ...     '/', 'application/vd.sun.wadl+xml', api_version='devel').body
     >>> wadl = wadl.decode('UTF-8')
 
-The real file is much bigger than the test file.
-
-    >>> len(wadl) > len(test_wadl)
+Unlike the test strings we used earlier, this is a valid WADL file.
+
+    >>> from lazr.restful import WADL_SCHEMA_FILE
+    >>> from canonical.lazr.xml import XMLValidator
+    >>> wadl_schema = XMLValidator(WADL_SCHEMA_FILE)
+
+    # We need to replace the nbsp entity, because the validator
+    # doesn't support embedded definition.
+    >>> wadl_schema.validate(
+    ...     wadl.replace('&nbsp;', '&#160;').encode('UTF-8'))
     True
 
 The WADL we received is keyed to the 'devel' version of the web
-service. This version will always be present.
+service. The URL to this version's service root will always be
+present.
 
     >>> 'http://api.launchpad.dev/devel/' in wadl
     True
 
 If we retrieve the WADL for the '1.0' version of the web service,
-we'll get a document keyed to the '1.0' version. The '1.0' version
-will be deprecated along with the Lucid release of Ubuntu.
+we'll get a document keyed to the '1.0' version.
 
     >>> wadl_10 = webservice.get(
     ...     '/', 'application/vd.sun.wadl+xml', api_version='1.0').body
@@ -60,31 +88,12 @@
     >>> 'http://api.launchpad.dev/1.0/' in wadl_10
     True
 
-If we retrieve the WADL for the 'beta' version of the web service,
-we'll get a document keyed to the 'beta' version. The '1.0' version
-will be deprecated along with the Karmic release of Ubuntu.
-
-    >>> wadl_beta = webservice.get(
-    ...     '/', 'application/vd.sun.wadl+xml', api_version='beta').body
-    >>> wadl_beta = wadl_beta.decode('UTF-8')
-
-    >>> 'http://api.launchpad.dev/beta/' in wadl_beta
-    True
-
-We don't need the cache anymore, so we'll reinstate the testing
-version. This way, other tests will have a clean slate.
+All of these documents were cached as they were generated:
+
+    >>> sorted(WebServiceApplication.cached_wadl.keys())
+    [u'1.0', u'devel']
+
+Finally, restore the cache so that other tests will have a clean
+slate.
 
     >>> WebServiceApplication.cached_wadl = old_cached_wadl
-
-Like all lazr.restful applications, Launchpad's web service generates
-valid WADL.
-
-    >>> from lazr.restful import WADL_SCHEMA_FILE
-    >>> from canonical.lazr.xml import XMLValidator
-    >>> wadl_schema = XMLValidator(WADL_SCHEMA_FILE)
-
-    # We need to replace the nbsp entity, because the validator
-    # doesn't support embedded definition.
-    >>> wadl_schema.validate(
-    ...     wadl.replace('&nbsp;', '&#160;').encode('UTF-8'))
-    True

=== modified file 'lib/canonical/launchpad/systemhomes.py'
--- lib/canonical/launchpad/systemhomes.py	2010-03-26 13:23:20 +0000
+++ lib/canonical/launchpad/systemhomes.py	2010-03-28 20:04:28 +0000
@@ -354,6 +354,13 @@
 
     cached_wadl = {}
 
+    @classmethod
+    def cachedWADLPath(cls, instance_name, version):
+        """Helper method to calculate the path to a cached WADL file."""
+        return os.path.join(
+            os.path.dirname(os.path.normpath(__file__)),
+            'apidoc', 'wadl-%s-%s.xml' % (instance_name, version))
+
     def toWADL(self):
         """See `IWebServiceApplication`.
 
@@ -370,10 +377,8 @@
             return super(WebServiceApplication, self).toWADL()
         if  version not in self.__class__.cached_wadl:
             # It's not cached. Look for it on disk.
-            _wadl_filename = os.path.join(
-                os.path.dirname(os.path.normpath(__file__)),
-                'apidoc', 'wadl-%s-%s.xml' % (config.instance_name, version))
-
+            _wadl_filename = self.cachedWADLPath(
+                config.instance_name, version)
             _wadl_fd = None
             try:
                 _wadl_fd = codecs.open(_wadl_filename, encoding='UTF-8')

=== modified file 'lib/canonical/launchpad/testing/pages.py'
--- lib/canonical/launchpad/testing/pages.py	2010-03-26 13:23:20 +0000
+++ lib/canonical/launchpad/testing/pages.py	2010-03-28 20:04:28 +0000
@@ -15,13 +15,16 @@
 import re
 import transaction
 import unittest
+from urlparse import urljoin
 
 from BeautifulSoup import (
     BeautifulSoup, CData, Comment, Declaration, NavigableString, PageElement,
     ProcessingInstruction, SoupStrainer, Tag)
+
 from contrib.oauth import (
     OAuthConsumer, OAuthRequest, OAuthSignatureMethod_PLAINTEXT, OAuthToken)
-from urlparse import urljoin
+
+from launchpadlib.launchpad import Launchpad
 
 from zope.app.testing.functional import HTTPCaller, SimpleCookie
 from zope.component import getUtility
@@ -647,14 +650,10 @@
     return str(canonical_url(*args, **kwargs))
 
 
-def webservice_for_person(person, consumer_key='launchpad-library',
-                          permission=OAuthPermission.READ_PUBLIC,
-                          context=None):
-    """Return a valid LaunchpadWebServiceCaller for the person.
-
-    Use this method to create a way to test the webservice that doesn't depend
-    on sample data.
-    """
+def _create_access_token_for_person(person, consumer_key,
+                                    permission=OAuthPermission.READ_PUBLIC,
+                                    context=None):
+    """Create an IOAuthAccessToken for `person`."""
     if person.is_team:
         raise AssertionError('This cannot be used with teams.')
     login(ANONYMOUS)
@@ -666,9 +665,34 @@
     request_token.review(person, permission, context)
     access_token = request_token.createAccessToken()
     logout()
+    return access_token
+
+
+def webservice_for_person(person, consumer_key='launchpad-library',
+                          permission=OAuthPermission.READ_PUBLIC,
+                          context=None):
+    """Return a valid LaunchpadWebServiceCaller for the person.
+
+    Use this method to create a way to test the webservice that doesn't depend
+    on sample data.
+    """
+    access_token = _create_access_token_for_person(
+        person, consumer_key, permission, context)
     return LaunchpadWebServiceCaller(consumer_key, access_token.key)
 
 
+def getLaunchpadObjectForPerson(person, consumer_key='launchpad-library',
+                                permission=OAuthPermission.READ_PUBLIC,
+                                context=None):
+    """Get a launchpadlib Launchpad object authenticated as the given person.
+    """
+    access_token = _create_access_token_for_person(
+        person, consumer_key, permission, context)
+    return Launchpad.login(
+        consumer_key, access_token.key, access_token.secret,
+        service_root='http://api.launchpad.dev/', version="devel")
+
+
 def setupDTCBrowser():
     """Testbrowser configured for Distribution Translations Coordinators.
 
@@ -725,6 +749,7 @@
         'launchpad-library', 'nopriv-read-nonprivate')
     test.globs['anon_webservice'] = LaunchpadWebServiceCaller(
         'launchpad-library', '')
+    test.globs['getLaunchpadObjectForPerson'] = getLaunchpadObjectForPerson
     test.globs['setupBrowser'] = setupBrowser
     test.globs['setupDTCBrowser'] = setupDTCBrowser
     test.globs['setupRosettaExpertBrowser'] = setupRosettaExpertBrowser

=== modified file 'lib/canonical/testing/layers.py'
--- lib/canonical/testing/layers.py	2010-03-26 13:23:20 +0000
+++ lib/canonical/testing/layers.py	2010-03-28 20:04:28 +0000
@@ -73,6 +73,7 @@
 from storm.zope.interfaces import IZStorm
 import transaction
 import wsgi_intercept
+from wsgi_intercept import httplib2_intercept
 
 from lazr.restful.utils import safe_hasattr
 
@@ -938,12 +939,18 @@
         register_launchpad_request_publication_factories()
         wsgi_intercept.add_wsgi_intercept(
             'localhost', 80, lambda: wsgi_application)
+        wsgi_intercept.add_wsgi_intercept(
+            'api.launchpad.dev', 80, lambda: wsgi_application)
+        httplib2_intercept.install()
+
 
     @classmethod
     @profiled
     def tearDown(cls):
         FunctionalLayer.isSetUp = False
         wsgi_intercept.remove_wsgi_intercept('localhost', 80)
+        wsgi_intercept.remove_wsgi_intercept('api.launchpad.dev', 80)
+        httplib2_intercept.uninstall()
         # Signal Layer cannot be torn down fully
         raise NotImplementedError