Merge lp:~cjwatson/launchpad/wsgi-ppa-auth into lp:launchpad

[Colin Watson]

Add a WSGI authenticator for private PPAs.

I got sufficiently annoyed in the process of fixing bug 1722209 to see how hard it would be to fix it properly, since the existing htpasswd scheme has been a thorn in our side for a while now. The answer appears to be "a bit, but not very".

This will let us remove our reliance on htpasswd files, and probably eventually make it easier to do things like issue time-limited tokens (or even macaroons?) to builders.

There are obviously various deployment issues to sort out, and generate-ppa-htaccess still deals with things like deactivation and cancellation emails which we'll need to move elsewhere; it may be possible to just inline the relevant checks into the DB queries in ArchiveAuthTokenSet, since we're always looking at a single subscriber or named token name and so the result set sizes are very small.

The way that mod_wsgi loads the entry point is peculiar. I tried to handle it with the existing buildout-based build system, but that ended up being too fiddly, so I decided to just depend on my virtualenv/pip conversion branch. I considered separating the client code off entirely from Launchpad, but this turns out to be light enough in terms of memory use and startup time that I don't think it's worth the effort at the moment.

[lp:~cjwatson/launchpad/wsgi-ppa-auth]

18475. By Colin Watson on 2017-11-19

Beef up MemcacheFixture a bit to support expiry times and to reject non-str keys.

18476. By Colin Watson on 2017-11-19

Separate out request-timeline handling so that memcache_client_factory can be used to create a basic client with minimal dependencies.

18477. By Colin Watson on 2017-11-19

Use memcached instead of timedcache, storing hashed passwords.

18478. By Colin Watson on 2017-11-19

Merge virtualenv-pip.

[Colin Watson (cjwatson)]

I've rewritten part of this using memcached rather than timedcache, so the cost of using multiple workers here should now be negligible.

[Colin Watson (cjwatson)]

Superseded by https://code.launchpad.net/~cjwatson/launchpad/+git/launchpad/+merge/399617.

Unmerged revisions

18478. By Colin Watson on 2017-11-19

Merge virtualenv-pip.

18477. By Colin Watson on 2017-11-19

Use memcached instead of timedcache, storing hashed passwords.

18476. By Colin Watson on 2017-11-19

Separate out request-timeline handling so that memcache_client_factory can be used to create a basic client with minimal dependencies.

18475. By Colin Watson on 2017-11-19

Beef up MemcacheFixture a bit to support expiry times and to reject non-str keys.

18474. By Colin Watson on 2017-10-11

Add a WSGI authenticator for private PPAs.

=== modified file 'Makefile'
--- Makefile	2017-11-19 12:35:56 +0000
+++ Makefile	2017-11-19 12:35:56 +0000
@@ -463,6 +463,7 @@
 		base=local-launchpad; \
 	fi; \
 	sed -e 's,%BRANCH_REWRITE%,$(shell pwd)/scripts/branch-rewrite.py,' \
+		-e 's,%WSGI_ARCHIVE_AUTH%,$(shell pwd)/scripts/wsgi-archive-auth.py,' \
 		-e 's,%LISTEN_ADDRESS%,$(LISTEN_ADDRESS),' \
 		configs/development/local-launchpad-apache > \
 		/etc/apache2/sites-available/$$base
 
=== modified file 'configs/development/local-launchpad-apache'
--- configs/development/local-launchpad-apache	2017-01-10 17:26:29 +0000
+++ configs/development/local-launchpad-apache	2017-11-19 12:35:56 +0000
@@ -134,7 +134,6 @@
 
 <VirtualHost %LISTEN_ADDRESS%:80>
   ServerName ppa.launchpad.dev
-  ServerAlias private-ppa.launchpad.dev
   LogLevel debug
 
   DocumentRoot /var/tmp/ppa
@@ -147,8 +146,36 @@
       Deny from all
       Allow from 127.0.0.0/255.0.0.0
     </IfVersion>
-    AllowOverride AuthConfig
-    Options Indexes
+    AllowOverride None
+    Options Indexes
+  </Directory>
+</VirtualHost>
+
+<VirtualHost %LISTEN_ADDRESS%:80>
+  ServerName private-ppa.launchpad.dev
+  LogLevel debug
+
+  DocumentRoot /var/tmp/ppa
+  <Directory /var/tmp/ppa/>
+    <IfVersion >= 2.4>
+      <RequireAll>
+        Require ip 127.0.0.0/255.0.0.0
+        Require valid-user
+      </RequireAll>
+    </IfVersion>
+    <IfVersion < 2.4>
+      Order Deny,Allow
+      Deny from all
+      Allow from 127.0.0.0/255.0.0.0
+      Require valid-user
+      Satisfy All
+    </IfVersion>
+    AllowOverride None
+    Options Indexes
+    AuthType Basic
+    AuthName "Token Required"
+    AuthBasicProvider wsgi
+    WSGIAuthUserScript %WSGI_ARCHIVE_AUTH% application-group=lp
   </Directory>
 </VirtualHost>
 
 
=== modified file 'lib/lp/services/config/schema-lazr.conf'
--- lib/lp/services/config/schema-lazr.conf	2017-09-07 13:25:13 +0000
+++ lib/lp/services/config/schema-lazr.conf	2017-11-19 12:35:56 +0000
@@ -1460,6 +1460,11 @@
 # datatype: boolean
 require_signing_keys: false
 
+# The URL to the internal archive API endpoint.  This should implement
+# IArchiveAPI.
+# datatype: string
+archive_api_endpoint: http://xmlrpc-private.launchpad.dev:8087/archive
+
 
 [ppa_apache_log_parser]
 logs_root: /srv/ppa.launchpad.net-logs
 
=== modified file 'lib/lp/services/memcache/client.py'
--- lib/lp/services/memcache/client.py	2017-09-27 10:59:13 +0000
+++ lib/lp/services/memcache/client.py	2017-11-19 12:35:56 +0000
@@ -4,64 +4,26 @@
 """Launchpad Memcache client."""
 
 __metaclass__ = type
-__all__ = []
+__all__ = [
+    'memcache_client_factory',
+    ]
 
-import logging
 import re
 
-from lazr.restful.utils import get_current_browser_request
-import memcache
-
-from lp.services import features
 from lp.services.config import config
-from lp.services.timeline.requesttimeline import get_request_timeline
-
-
-def memcache_client_factory():
+
+
+def memcache_client_factory(timeline=True):
     """Return a memcache.Client for Launchpad."""
     servers = [
         (host, int(weight)) for host, weight in re.findall(
             r'\((.+?),(\d+)\)', config.memcache.servers)]
     assert len(servers) > 0, "Invalid memcached server list %r" % (
         config.memcache.servers,)
-    return TimelineRecordingClient(servers)
-
-
-class TimelineRecordingClient(memcache.Client):
-
-    def __get_timeline_action(self, suffix, key):
-        request = get_current_browser_request()
-        timeline = get_request_timeline(request)
-        return timeline.start("memcache-%s" % suffix, key)
-
-    @property
-    def _enabled(self):
-        configured_value = features.getFeatureFlag('memcache')
-        if configured_value is None:
-            return True
-        else:
-            return configured_value
-
-    def get(self, key):
-        if not self._enabled:
-            return None
-        action = self.__get_timeline_action("get", key)
-        try:
-            return memcache.Client.get(self, key)
-        finally:
-            action.finish()
-
-    def set(self, key, value, time=0, min_compress_len=0):
-        if not self._enabled:
-            return None
-        action = self.__get_timeline_action("set", key)
-        try:
-            success = memcache.Client.set(self, key, value, time=time,
-                min_compress_len=min_compress_len)
-            if success:
-                logging.debug("Memcache set succeeded for %s", key)
-            else:
-                logging.warn("Memcache set failed for %s", key)
-            return success
-        finally:
-            action.finish()
+    if timeline:
+        from lp.services.memcache.timeline import TimelineRecordingClient
+        client_factory = TimelineRecordingClient
+    else:
+        import memcache
+        client_factory = memcache.Client
+    return client_factory(servers)
 
=== modified file 'lib/lp/services/memcache/testing.py'
--- lib/lp/services/memcache/testing.py	2016-09-07 03:43:36 +0000
+++ lib/lp/services/memcache/testing.py	2017-11-19 12:35:56 +0000
@@ -1,4 +1,4 @@
-# Copyright 2016 Canonical Ltd.  This software is licensed under the
+# Copyright 2016-2017 Canonical Ltd.  This software is licensed under the
 # GNU Affero General Public License version 3 (see the file LICENSE).
 
 __metaclass__ = type
@@ -6,6 +6,8 @@
     'MemcacheFixture',
     ]
 
+import time as _time
+
 import fixtures
 
 from lp.services.memcache.interfaces import IMemcacheClient
@@ -22,14 +24,33 @@
         super(MemcacheFixture, self).setUp()
         self.useFixture(ZopeUtilityFixture(self, IMemcacheClient))
 
+    def check_key(self, key):
+        # A subset of the checks performed by the real memcache library;
+        # this one is particularly easy to get wrong in Launchpad code.
+        if not isinstance(key, str):
+            raise TypeError("Key must be str.")
+
     def get(self, key):
-        return self._cache.get(key)
+        self.check_key(key)
+        value, expiry_time = self._cache.get(key, (None, None))
+        if expiry_time and _time.time() >= expiry_time:
+            self.delete(key)
+            return None
+        else:
+            return value
 
-    def set(self, key, val):
-        self._cache[key] = val
+    def set(self, key, val, time=0):
+        self.check_key(key)
+        # memcached accepts either delta-seconds from the current time or
+        # absolute epoch-seconds, and tells them apart using a magic
+        # threshold.  See memcached/memcached.c:realtime.
+        if time and time <= 60 * 60 * 24 * 30:
+            time = _time.time() + time
+        self._cache[key] = (val, time)
         return 1
 
     def delete(self, key):
+        self.check_key(key)
         self._cache.pop(key, None)
         return 1
 
 
=== added file 'lib/lp/services/memcache/timeline.py'
--- lib/lp/services/memcache/timeline.py	1970-01-01 00:00:00 +0000
+++ lib/lp/services/memcache/timeline.py	2017-11-19 12:35:56 +0000
@@ -0,0 +1,57 @@
+# Copyright 2017 Canonical Ltd.  This software is licensed under the
+# GNU Affero General Public License version 3 (see the file LICENSE).
+
+"""Timeline-friendly Launchpad Memcache client."""
+
+__metaclass__ = type
+__all__ = [
+    'TimelineRecordingClient',
+    ]
+
+import logging
+
+from lazr.restful.utils import get_current_browser_request
+import memcache
+
+from lp.services import features
+from lp.services.timeline.requesttimeline import get_request_timeline
+
+
+class TimelineRecordingClient(memcache.Client):
+
+    def __get_timeline_action(self, suffix, key):
+        request = get_current_browser_request()
+        timeline = get_request_timeline(request)
+        return timeline.start("memcache-%s" % suffix, key)
+
+    @property
+    def _enabled(self):
+        configured_value = features.getFeatureFlag('memcache')
+        if configured_value is None:
+            return True
+        else:
+            return configured_value
+
+    def get(self, key):
+        if not self._enabled:
+            return None
+        action = self.__get_timeline_action("get", key)
+        try:
+            return memcache.Client.get(self, key)
+        finally:
+            action.finish()
+
+    def set(self, key, value, time=0, min_compress_len=0):
+        if not self._enabled:
+            return None
+        action = self.__get_timeline_action("set", key)
+        try:
+            success = memcache.Client.set(self, key, value, time=time,
+                min_compress_len=min_compress_len)
+            if success:
+                logging.debug("Memcache set succeeded for %s", key)
+            else:
+                logging.warn("Memcache set failed for %s", key)
+            return success
+        finally:
+            action.finish()
 
=== modified file 'lib/lp/soyuz/doc/archiveauthtoken.txt'
--- lib/lp/soyuz/doc/archiveauthtoken.txt	2012-04-10 14:01:17 +0000
+++ lib/lp/soyuz/doc/archiveauthtoken.txt	2017-11-19 12:35:56 +0000
@@ -139,12 +139,35 @@
     ...     print token.person.name
     bradsmith
 
-Tokens can also be retreived by archive and person:
+Tokens can also be retrieved by archive and person:
 
     >>> print token_set.getActiveTokenForArchiveAndPerson(
     ...     new_token.archive, new_token.person).token
     testtoken
 
+Or by archive and person name:
+
+    >>> print token_set.getActiveTokenForArchiveAndPersonName(
+    ...     new_token.archive, "bradsmith").token
+    testtoken
+
+Tokens are only returned if they match a current subscription:
+
+    >>> from zope.security.proxy import removeSecurityProxy
+    >>> from lp.soyuz.enums import ArchiveSubscriberStatus
+    >>> removeSecurityProxy(subscription_to_joe_private_ppa).status = (
+    ...     ArchiveSubscriberStatus.EXPIRED)
+
+    >>> print token_set.getActiveTokenForArchiveAndPerson(
+    ...     new_token.archive, new_token.person)
+    None
+    >>> print token_set.getActiveTokenForArchiveAndPersonName(
+    ...     new_token.archive, "bradsmith")
+    None
+
+    >>> removeSecurityProxy(subscription_to_joe_private_ppa).status = (
+    ...     ArchiveSubscriberStatus.CURRENT)
+
 
 == Amending Tokens ==
 
 
=== added file 'lib/lp/soyuz/interfaces/archiveapi.py'
--- lib/lp/soyuz/interfaces/archiveapi.py	1970-01-01 00:00:00 +0000
+++ lib/lp/soyuz/interfaces/archiveapi.py	2017-11-19 12:35:56 +0000
@@ -0,0 +1,46 @@
+# Copyright 2017 Canonical Ltd.  This software is licensed under the
+# GNU Affero General Public License version 3 (see the file LICENSE).
+
+"""Interfaces for internal archive APIs."""
+
+from __future__ import absolute_import, print_function, unicode_literals
+
+__metaclass__ = type
+__all__ = [
+    'IArchiveAPI',
+    'IArchiveApplication',
+    ]
+
+from zope.interface import Interface
+
+from lp.services.webapp.interfaces import ILaunchpadApplication
+
+
+class IArchiveApplication(ILaunchpadApplication):
+    """Archive application root."""
+
+
+class IArchiveAPI(Interface):
+    """The Soyuz archive XML-RPC interface to Launchpad.
+
+    Published at "archive" on the private XML-RPC server.
+
+    PPA frontends use this to check archive authorization tokens.
+    """
+
+    def checkArchiveAuthToken(archive_reference, username, password):
+        """Check an archive authorization token.
+
+        :param archive_reference: The reference form of the archive to check.
+        :param username: The username sent using HTTP Basic Authentication;
+            this should either be a `Person.name` or "+" followed by the
+            name of a named authorization token.
+        :param password: The password sent using HTTP Basic Authentication;
+            this should be a corresponding `ArchiveAuthToken.token`.
+
+        :returns: A `NotFound` fault if `archive_reference` does not
+            identify an archive or the username does not identify a valid
+            token for this archive; an `Unauthorized` fault if the password
+            is not equal to the selected token for this archive; otherwise
+            None.
+        """
 
=== modified file 'lib/lp/soyuz/interfaces/archiveauthtoken.py'
--- lib/lp/soyuz/interfaces/archiveauthtoken.py	2016-07-14 16:06:01 +0000
+++ lib/lp/soyuz/interfaces/archiveauthtoken.py	2017-11-19 12:35:56 +0000
@@ -96,23 +96,33 @@
         :return: An object conforming to `IArchiveAuthToken`.
         """
 
-    def getByArchive(archive):
+    def getByArchive(archive, valid=False):
         """Retrieve all the tokens for an archive.
 
         :param archive: The context archive.
+        :param valid: If True, only return valid tokens.
         :return: A result set containing `IArchiveAuthToken`s.
         """
 
     def getActiveTokenForArchiveAndPerson(archive, person):
-        """Retrieve an active token for the given archive and person.
+        """Retrieve a valid active token for the given archive and person.
 
         :param archive: The archive to which the token corresponds.
         :param person: The person to which the token corresponds.
         :return: An `IArchiveAuthToken` or None.
         """
 
+    def getActiveTokenForArchiveAndPersonName(archive, person_name):
+        """Retrieve a valid active token for the given archive and person name.
+
+        :param archive: The archive to which the token corresponds.
+        :param person_name: The name of the person to which the token
+            corresponds.
+        :return: An `IArchiveAuthToken` or None.
+        """
+
     def getActiveNamedTokenForArchive(archive, name):
-        """Retrieve an active named token for the given archive and name.
+        """Retrieve a valid active named token for the given archive and name.
 
         :param archive: The archive to which the token corresponds.
         :param name: The name of a named authorization token.
@@ -120,9 +130,9 @@
         """
 
     def getActiveNamedTokensForArchive(archive, names=None):
-        """Retrieve a subset of active named tokens for the given archive if
-        `names` is specified, or all active named tokens for the archive if
-        `names` is null.
+        """Retrieve a subset of valid active named tokens for the given
+        archive if `names` is specified, or all valid active named tokens
+        for the archive if `names` is null.
 
         :param archive: The archive to which the tokens correspond.
         :param names: An optional list of token names.
 
=== modified file 'lib/lp/soyuz/model/archiveauthtoken.py'
--- lib/lp/soyuz/model/archiveauthtoken.py	2016-07-14 16:06:01 +0000
+++ lib/lp/soyuz/model/archiveauthtoken.py	2017-11-19 12:35:56 +0000
@@ -21,8 +21,10 @@
 from storm.store import Store
 from zope.interface import implementer
 
+from lp.registry.model.teammembership import TeamParticipation
 from lp.services.database.constants import UTC_NOW
 from lp.services.database.interfaces import IStore
+from lp.soyuz.enums import ArchiveSubscriberStatus
 from lp.soyuz.interfaces.archiveauthtoken import (
     IArchiveAuthToken,
     IArchiveAuthTokenSet,
@@ -85,19 +87,37 @@
         return IStore(ArchiveAuthToken).find(
             ArchiveAuthToken, ArchiveAuthToken.token == token).one()
 
-    def getByArchive(self, archive):
+    def getByArchive(self, archive, valid=False):
         """See `IArchiveAuthTokenSet`."""
+        # Circular import.
+        from lp.soyuz.model.archivesubscriber import ArchiveSubscriber
         store = Store.of(archive)
-        return store.find(
-            ArchiveAuthToken,
+        clauses = [
             ArchiveAuthToken.archive == archive,
-            ArchiveAuthToken.date_deactivated == None)
+            ArchiveAuthToken.date_deactivated == None,
+            ]
+        if valid:
+            clauses.extend([
+                ArchiveAuthToken.archive_id == ArchiveSubscriber.archive_id,
+                ArchiveSubscriber.status == ArchiveSubscriberStatus.CURRENT,
+                ArchiveSubscriber.subscriber_id == TeamParticipation.teamID,
+                TeamParticipation.personID == ArchiveAuthToken.person_id,
+                ])
+        return store.find(ArchiveAuthToken, *clauses)
 
     def getActiveTokenForArchiveAndPerson(self, archive, person):
         """See `IArchiveAuthTokenSet`."""
-        return self.getByArchive(archive).find(
+        return self.getByArchive(archive, valid=True).find(
             ArchiveAuthToken.person == person).one()
 
+    def getActiveTokenForArchiveAndPersonName(self, archive, person_name):
+        """See `IArchiveAuthTokenSet`."""
+        # Circular import.
+        from lp.registry.model.person import Person
+        return self.getByArchive(archive, valid=True).find(
+            ArchiveAuthToken.person == Person.id,
+            Person.name == person_name).one()
+
     def getActiveNamedTokenForArchive(self, archive, name):
         """See `IArchiveAuthTokenSet`."""
         return self.getByArchive(archive).find(
 
=== added directory 'lib/lp/soyuz/wsgi'
=== added file 'lib/lp/soyuz/wsgi/__init__.py'
=== added file 'lib/lp/soyuz/wsgi/archiveauth.py'
--- lib/lp/soyuz/wsgi/archiveauth.py	1970-01-01 00:00:00 +0000
+++ lib/lp/soyuz/wsgi/archiveauth.py	2017-11-19 12:35:56 +0000
@@ -0,0 +1,83 @@
+# Copyright 2017 Canonical Ltd.  This software is licensed under the
+# GNU Affero General Public License version 3 (see the file LICENSE).
+
+"""WSGI archive authorisation provider.
+
+This is as lightweight as possible, as it runs on PPA frontends.
+"""
+
+from __future__ import absolute_import, print_function, unicode_literals
+
+__metaclass__ = type
+__all__ = [
+    'check_password',
+    ]
+
+import crypt
+from random import SystemRandom
+import string
+import time
+try:
+    from xmlrpc.client import (
+        Fault,
+        ServerProxy,
+        )
+except ImportError:
+    from xmlrpclib import (
+        Fault,
+        ServerProxy,
+        )
+
+from lp.services.config import config
+from lp.services.memcache.client import memcache_client_factory
+
+
+def _get_archive_reference(environ):
+    # Reconstruct the relevant part of the URL.  We don't care about where
+    # we're installed.
+    path = environ.get("SCRIPT_NAME") or "/"
+    path_info = environ.get("PATH_INFO", "")
+    path += (path_info if path else path_info[1:])
+    # Extract the first three segments of the path, and rearrange them to
+    # form an archive reference.
+    path_parts = path.lstrip("/").split("/")
+    if len(path_parts) >= 3:
+        return "~%s/%s/%s" % (path_parts[0], path_parts[2], path_parts[1])
+
+
+_sr = SystemRandom()
+
+
+def _crypt_sha256(word):
+    """crypt.crypt(word, crypt.METHOD_SHA256), backported from Python 3.5."""
+    saltchars = string.ascii_letters + string.digits + './'
+    salt = '$5$' + ''.join(_sr.choice(saltchars) for _ in range(16))
+    return crypt.crypt(word, salt)
+
+
+_memcache_client = memcache_client_factory(timeline=False)
+
+
+def check_password(environ, user, password):
+    archive_reference = _get_archive_reference(environ)
+    if archive_reference is None:
+        return None
+    memcache_key = (
+        "archive-auth:%s:%s" % (archive_reference, user)).encode("UTF-8")
+    crypted_password = _memcache_client.get(memcache_key)
+    if (crypted_password and
+            crypt.crypt(password, crypted_password) == crypted_password):
+        return True
+    proxy = ServerProxy(config.personalpackagearchive.archive_api_endpoint)
+    try:
+        proxy.checkArchiveAuthToken(archive_reference, user, password)
+        # Cache positive responses for a minute to reduce database load.
+        _memcache_client.set(
+            memcache_key, _crypt_sha256(password), time.time() + 60)
+        return True
+    except Fault as e:
+        if e.faultCode == 410:  # Unauthorized
+            return False
+        else:
+            # Interpret any other fault as NotFound (320).
+            return None
 
=== added directory 'lib/lp/soyuz/wsgi/tests'
=== added file 'lib/lp/soyuz/wsgi/tests/__init__.py'
=== added file 'lib/lp/soyuz/wsgi/tests/test_archiveauth.py'
--- lib/lp/soyuz/wsgi/tests/test_archiveauth.py	1970-01-01 00:00:00 +0000
+++ lib/lp/soyuz/wsgi/tests/test_archiveauth.py	2017-11-19 12:35:56 +0000
@@ -0,0 +1,151 @@
+# Copyright 2017 Canonical Ltd.  This software is licensed under the
+# GNU Affero General Public License version 3 (see the file LICENSE).
+
+"""Tests for the WSGI archive authorisation provider."""
+
+from __future__ import absolute_import, print_function, unicode_literals
+
+__metaclass__ = type
+
+import crypt
+import os.path
+import subprocess
+import time
+
+from fixtures import MonkeyPatch
+from testtools.matchers import Is
+import transaction
+
+from lp.services.config import config
+from lp.services.memcache.testing import MemcacheFixture
+from lp.soyuz.wsgi import archiveauth
+from lp.testing import TestCaseWithFactory
+from lp.testing.layers import ZopelessAppServerLayer
+from lp.xmlrpc import faults
+
+
+class TestWSGIArchiveAuth(TestCaseWithFactory):
+
+    layer = ZopelessAppServerLayer
+
+    def setUp(self):
+        super(TestWSGIArchiveAuth, self).setUp()
+        self.now = time.time()
+        self.useFixture(MonkeyPatch("time.time", lambda: self.now))
+        self.memcache_fixture = self.useFixture(MemcacheFixture())
+        # The WSGI provider doesn't use Zope, so we can't rely on the
+        # fixture substituting a Zope utility.
+        self.useFixture(MonkeyPatch(
+            "lp.soyuz.wsgi.archiveauth._memcache_client",
+            self.memcache_fixture))
+
+    def test_get_archive_reference_short_url(self):
+        self.assertIsNone(archiveauth._get_archive_reference(
+            {"SCRIPT_NAME": "/foo"}))
+
+    def test_get_archive_reference_archive_base(self):
+        self.assertEqual(
+            "~user/ubuntu/ppa",
+            archiveauth._get_archive_reference(
+                {"SCRIPT_NAME": "/user/ppa/ubuntu"}))
+
+    def test_get_archive_reference_inside_archive(self):
+        self.assertEqual(
+            "~user/ubuntu/ppa",
+            archiveauth._get_archive_reference(
+                {"SCRIPT_NAME": "/user/ppa/ubuntu/dists"}))
+
+    def test_check_password_short_url(self):
+        self.assertIsNone(archiveauth.check_password(
+            {"SCRIPT_NAME": "/foo"}, "user", ""))
+        self.assertEqual({}, self.memcache_fixture._cache)
+
+    def test_check_password_not_found(self):
+        self.assertIsNone(archiveauth.check_password(
+            {"SCRIPT_NAME": "/nonexistent/bad/unknown"}, "user", ""))
+        self.assertEqual({}, self.memcache_fixture._cache)
+
+    def test_crypt_sha256(self):
+        crypted_password = archiveauth._crypt_sha256("secret")
+        self.assertEqual(
+            crypted_password, crypt.crypt("secret", crypted_password))
+
+    def makeArchiveAndToken(self):
+        archive = self.factory.makeArchive(private=True)
+        archive_path = "/%s/%s/ubuntu" % (archive.owner.name, archive.name)
+        subscriber = self.factory.makePerson()
+        archive.newSubscription(subscriber, archive.owner)
+        token = archive.newAuthToken(subscriber)
+        transaction.commit()
+        return archive, archive_path, subscriber.name, token.token
+
+    def test_check_password_unauthorized(self):
+        _, archive_path, username, password = self.makeArchiveAndToken()
+        # Test that this returns False, not merely something falsy (e.g.
+        # None).
+        self.assertThat(
+            archiveauth.check_password(
+                {"SCRIPT_NAME": archive_path}, username, password + "-bad"),
+            Is(False))
+        self.assertEqual({}, self.memcache_fixture._cache)
+
+    def test_check_password_success(self):
+        archive, archive_path, username, password = self.makeArchiveAndToken()
+        self.assertThat(
+            archiveauth.check_password(
+                {"SCRIPT_NAME": archive_path}, username, password),
+            Is(True))
+        crypted_password = self.memcache_fixture.get(
+            ("archive-auth:%s:%s" % (archive.reference, username)).encode(
+                "UTF-8"))
+        self.assertEqual(
+            crypted_password, crypt.crypt(password, crypted_password))
+
+    def test_check_password_considers_cache(self):
+        class FakeProxy:
+            def __init__(self, uri):
+                pass
+
+            def checkArchiveAuthToken(self, archive_reference, username,
+                                      password):
+                raise faults.Unauthorized()
+
+        _, archive_path, username, password = self.makeArchiveAndToken()
+        self.assertThat(
+            archiveauth.check_password(
+                {"SCRIPT_NAME": archive_path}, username, password),
+            Is(True))
+        self.useFixture(
+            MonkeyPatch("lp.soyuz.wsgi.archiveauth.ServerProxy", FakeProxy))
+        # A subsequent check honours the cache.
+        self.assertThat(
+            archiveauth.check_password(
+                {"SCRIPT_NAME": archive_path}, username, password + "-bad"),
+            Is(False))
+        self.assertThat(
+            archiveauth.check_password(
+                {"SCRIPT_NAME": archive_path}, username, password),
+            Is(True))
+        # If we advance time far enough, then the cached result expires.
+        self.now += 60
+        self.assertThat(
+            archiveauth.check_password(
+                {"SCRIPT_NAME": archive_path}, username, password),
+            Is(False))
+
+    def test_script(self):
+        _, archive_path, username, password = self.makeArchiveAndToken()
+        script_path = os.path.join(
+            config.root, "scripts", "wsgi-archive-auth.py")
+
+        def check_via_script(archive_path, username, password):
+            with open(os.devnull, "w") as devnull:
+                return subprocess.call(
+                    [script_path, archive_path, username, password],
+                    stderr=devnull)
+
+        self.assertEqual(0, check_via_script(archive_path, username, password))
+        self.assertEqual(
+            1, check_via_script(archive_path, username, password + "-bad"))
+        self.assertEqual(
+            2, check_via_script("/nonexistent/bad/unknown", "user", ""))
 
=== added file 'lib/lp/soyuz/xmlrpc/archive.py'
--- lib/lp/soyuz/xmlrpc/archive.py	1970-01-01 00:00:00 +0000
+++ lib/lp/soyuz/xmlrpc/archive.py	2017-11-19 12:35:56 +0000
@@ -0,0 +1,62 @@
+# Copyright 2017 Canonical Ltd.  This software is licensed under the
+# GNU Affero General Public License version 3 (see the file LICENSE).
+
+"""Implementations of the XML-RPC APIs for Soyuz archives."""
+
+from __future__ import absolute_import, print_function, unicode_literals
+
+__metaclass__ = type
+__all__ = [
+    'ArchiveAPI',
+    ]
+
+from zope.component import getUtility
+from zope.interface import implementer
+from zope.security.proxy import removeSecurityProxy
+
+from lp.soyuz.interfaces.archive import IArchiveSet
+from lp.soyuz.interfaces.archiveapi import IArchiveAPI
+from lp.soyuz.interfaces.archiveauthtoken import IArchiveAuthTokenSet
+from lp.services.webapp import LaunchpadXMLRPCView
+from lp.xmlrpc import faults
+from lp.xmlrpc.helpers import return_fault
+
+
+BUILDD_USER_NAME = "buildd"
+
+
+@implementer(IArchiveAPI)
+class ArchiveAPI(LaunchpadXMLRPCView):
+    """See `IArchiveAPI`."""
+
+    @return_fault
+    def _checkArchiveAuthToken(self, archive_reference, username, password):
+        archive = getUtility(IArchiveSet).getByReference(archive_reference)
+        if archive is None:
+            raise faults.NotFound(
+                message="No archive found for '%s'." % archive_reference)
+        archive = removeSecurityProxy(archive)
+        token_set = getUtility(IArchiveAuthTokenSet)
+        if username == BUILDD_USER_NAME:
+            secret = archive.buildd_secret
+        else:
+            if username.startswith("+"):
+                token = token_set.getActiveNamedTokenForArchive(
+                    archive, username[1:])
+            else:
+                token = token_set.getActiveTokenForArchiveAndPersonName(
+                    archive, username)
+            if token is None:
+                raise faults.NotFound(
+                    message="No valid tokens for '%s' in '%s'." % (
+                        username, archive_reference))
+            secret = removeSecurityProxy(token).token
+        if password != secret:
+            raise faults.Unauthorized()
+
+    def checkArchiveAuthToken(self, archive_reference, username, password):
+        """See `IArchiveAPI`."""
+        # This thunk exists because you can't use a decorated function as
+        # the implementation of a method exported over XML-RPC.
+        return self._checkArchiveAuthToken(
+            archive_reference, username, password)
 
=== added directory 'lib/lp/soyuz/xmlrpc/tests'
=== added file 'lib/lp/soyuz/xmlrpc/tests/__init__.py'
=== added file 'lib/lp/soyuz/xmlrpc/tests/test_archive.py'
--- lib/lp/soyuz/xmlrpc/tests/test_archive.py	1970-01-01 00:00:00 +0000
+++ lib/lp/soyuz/xmlrpc/tests/test_archive.py	2017-11-19 12:35:56 +0000
@@ -0,0 +1,124 @@
+# Copyright 2017 Canonical Ltd.  This software is licensed under the
+# GNU Affero General Public License version 3 (see the file LICENSE).
+
+"""Tests for the internal Soyuz archive API."""
+
+from __future__ import absolute_import, print_function, unicode_literals
+
+__metaclass__ = type
+
+from zope.security.proxy import removeSecurityProxy
+
+from lp.services.features.testing import FeatureFixture
+from lp.soyuz.interfaces.archive import NAMED_AUTH_TOKEN_FEATURE_FLAG
+from lp.soyuz.xmlrpc.archive import ArchiveAPI
+from lp.testing import TestCaseWithFactory
+from lp.testing.layers import LaunchpadFunctionalLayer
+from lp.xmlrpc import faults
+
+
+class TestArchiveAPI(TestCaseWithFactory):
+
+    layer = LaunchpadFunctionalLayer
+
+    def setUp(self):
+        super(TestArchiveAPI, self).setUp()
+        self.useFixture(FeatureFixture({NAMED_AUTH_TOKEN_FEATURE_FLAG: "on"}))
+        self.archive_api = ArchiveAPI(None, None)
+
+    def assertNotFound(self, archive_reference, username, password, message):
+        """Assert that an archive auth token check returns NotFound."""
+        fault = self.archive_api.checkArchiveAuthToken(
+            archive_reference, username, password)
+        self.assertEqual(faults.NotFound(message), fault)
+
+    def assertUnauthorized(self, archive_reference, username, password):
+        """Assert that an archive auth token check returns Unauthorized."""
+        fault = self.archive_api.checkArchiveAuthToken(
+            archive_reference, username, password)
+        self.assertEqual(faults.Unauthorized("Authorisation required."), fault)
+
+    def test_checkArchiveAuthToken_unknown_archive(self):
+        self.assertNotFound(
+            "~nonexistent/unknown/bad", "user", "",
+            "No archive found for '~nonexistent/unknown/bad'.")
+
+    def test_checkArchiveAuthToken_no_tokens(self):
+        archive = removeSecurityProxy(self.factory.makeArchive(private=True))
+        self.assertNotFound(
+            archive.reference, "nobody", "",
+            "No valid tokens for 'nobody' in '%s'." % archive.reference)
+
+    def test_checkArchiveAuthToken_no_named_tokens(self):
+        archive = removeSecurityProxy(self.factory.makeArchive(private=True))
+        self.assertNotFound(
+            archive.reference, "+missing", "",
+            "No valid tokens for '+missing' in '%s'." % archive.reference)
+
+    def test_checkArchiveAuthToken_buildd_wrong_password(self):
+        archive = removeSecurityProxy(self.factory.makeArchive(private=True))
+        self.assertUnauthorized(
+            archive.reference, "buildd", archive.buildd_secret + "-bad")
+
+    def test_checkArchiveAuthToken_buildd_correct_password(self):
+        archive = removeSecurityProxy(self.factory.makeArchive(private=True))
+        self.assertIsNone(self.archive_api.checkArchiveAuthToken(
+            archive.reference, "buildd", archive.buildd_secret))
+
+    def test_checkArchiveAuthToken_named_token_wrong_password(self):
+        archive = removeSecurityProxy(self.factory.makeArchive(private=True))
+        token = archive.newNamedAuthToken("special")
+        removeSecurityProxy(token).deactivate()
+        self.assertNotFound(
+            archive.reference, "+special", token.token,
+            "No valid tokens for '+special' in '%s'." % archive.reference)
+
+    def test_checkArchiveAuthToken_named_token_deactivated(self):
+        archive = removeSecurityProxy(self.factory.makeArchive(private=True))
+        token = archive.newNamedAuthToken("special")
+        self.assertIsNone(self.archive_api.checkArchiveAuthToken(
+            archive.reference, "+special", token.token))
+
+    def test_checkArchiveAuthToken_named_token_correct_password(self):
+        archive = removeSecurityProxy(self.factory.makeArchive(private=True))
+        token = archive.newNamedAuthToken("special")
+        self.assertIsNone(self.archive_api.checkArchiveAuthToken(
+            archive.reference, "+special", token.token))
+
+    def test_checkArchiveAuthToken_personal_token_wrong_password(self):
+        archive = removeSecurityProxy(self.factory.makeArchive(private=True))
+        subscriber = self.factory.makePerson()
+        archive.newSubscription(subscriber, archive.owner)
+        token = archive.newAuthToken(subscriber)
+        self.assertUnauthorized(
+            archive.reference, subscriber.name, token.token + "-bad")
+
+    def test_checkArchiveAuthToken_personal_token_deactivated(self):
+        archive = removeSecurityProxy(self.factory.makeArchive(private=True))
+        subscriber = self.factory.makePerson()
+        archive.newSubscription(subscriber, archive.owner)
+        token = archive.newAuthToken(subscriber)
+        removeSecurityProxy(token).deactivate()
+        self.assertNotFound(
+            archive.reference, subscriber.name, token.token,
+            "No valid tokens for '%s' in '%s'." % (
+                subscriber.name, archive.reference))
+
+    def test_checkArchiveAuthToken_personal_token_cancelled(self):
+        archive = removeSecurityProxy(self.factory.makeArchive(private=True))
+        subscriber = self.factory.makePerson()
+        subscription = archive.newSubscription(subscriber, archive.owner)
+        token = archive.newAuthToken(subscriber)
+        removeSecurityProxy(subscription).cancel(archive.owner)
+        self.assertNotFound(
+            archive.reference, subscriber.name, token.token,
+            "No valid tokens for '%s' in '%s'." % (
+                subscriber.name, archive.reference))
+
+    def test_checkArchiveAuthToken_personal_token_correct_password(self):
+        archive = removeSecurityProxy(self.factory.makeArchive(private=True))
+        subscriber = self.factory.makePerson()
+        archive.newSubscription(subscriber, archive.owner)
+        token = archive.newAuthToken(subscriber)
+        self.assertIsNone(self.archive_api.checkArchiveAuthToken(
+            archive.reference, subscriber.name, token.token))
 
=== modified file 'lib/lp/systemhomes.py'
--- lib/lp/systemhomes.py	2017-05-16 16:33:53 +0000
+++ lib/lp/systemhomes.py	2017-11-19 12:35:56 +0000
@@ -72,6 +72,7 @@
 from lp.services.webapp.publisher import canonical_url
 from lp.services.webservice.interfaces import IWebServiceApplication
 from lp.services.worlddata.interfaces.language import ILanguageSet
+from lp.soyuz.interfaces.archiveapi import IArchiveApplication
 from lp.testopenid.interfaces.server import ITestOpenIDApplication
 from lp.translations.interfaces.translationgroup import ITranslationGroupSet
 from lp.translations.interfaces.translations import IRosettaApplication
@@ -80,6 +81,12 @@
     )
 
 
+@implementer(IArchiveApplication)
+class ArchiveApplication:
+
+    title = "Archive API"
+
+
 @implementer(ICodehostingApplication)
 class CodehostingApplication:
     """Codehosting End-Point."""
 
=== modified file 'lib/lp/xmlrpc/application.py'
--- lib/lp/xmlrpc/application.py	2015-10-26 14:54:43 +0000
+++ lib/lp/xmlrpc/application.py	2017-11-19 12:35:56 +0000
@@ -31,11 +31,12 @@
 from lp.services.features.xmlrpc import IFeatureFlagApplication
 from lp.services.webapp import LaunchpadXMLRPCView
 from lp.services.webapp.interfaces import ILaunchBag
+from lp.soyuz.interfaces.archiveapi import IArchiveApplication
 from lp.xmlrpc.interfaces import IPrivateApplication
 
 
 # NOTE: If you add a traversal here, you should update
-# the regular expression in utilities/page-performance-report.ini
+# the regular expression in lp:lp-dev-utils page-performance-report.ini.
 @implementer(IPrivateApplication)
 class PrivateApplication:
 
@@ -45,6 +46,11 @@
         return getUtility(IMailingListApplication)
 
     @property
+    def archive(self):
+        """See `IPrivateApplication`."""
+        return getUtility(IArchiveApplication)
+
+    @property
     def authserver(self):
         """See `IPrivateApplication`."""
         return getUtility(IAuthServerApplication)
 
=== modified file 'lib/lp/xmlrpc/configure.zcml'
--- lib/lp/xmlrpc/configure.zcml	2015-05-04 14:56:58 +0000
+++ lib/lp/xmlrpc/configure.zcml	2017-11-19 12:35:56 +0000
@@ -22,6 +22,19 @@
       />
 
   <securedutility
+    class="lp.systemhomes.ArchiveApplication"
+    provides="lp.soyuz.interfaces.archiveapi.IArchiveApplication">
+    <allow interface="lp.soyuz.interfaces.archiveapi.IArchiveApplication"/>
+  </securedutility>
+
+  <xmlrpc:view
+    for="lp.soyuz.interfaces.archiveapi.IArchiveApplication"
+    interface="lp.soyuz.interfaces.archiveapi.IArchiveAPI"
+    class="lp.soyuz.xmlrpc.archive.ArchiveAPI"
+    permission="zope.Public"
+    />
+
+  <securedutility
     class="lp.systemhomes.CodehostingApplication"
     provides="lp.code.interfaces.codehosting.ICodehostingApplication">
     <allow interface="lp.code.interfaces.codehosting.ICodehostingApplication"/>
 
=== modified file 'lib/lp/xmlrpc/interfaces.py'
--- lib/lp/xmlrpc/interfaces.py	2015-05-04 14:56:58 +0000
+++ lib/lp/xmlrpc/interfaces.py	2017-11-19 12:35:56 +0000
@@ -17,6 +17,8 @@
 class IPrivateApplication(ILaunchpadApplication):
     """Launchpad private XML-RPC application root."""
 
+    archive = Attribute("Archive XML-RPC end point.""")
+
     authserver = Attribute("""Old Authserver API end point.""")
 
     codeimportscheduler = Attribute("""Code import scheduler end point.""")
 
=== added file 'scripts/wsgi-archive-auth.py'
--- scripts/wsgi-archive-auth.py	1970-01-01 00:00:00 +0000
+++ scripts/wsgi-archive-auth.py	2017-11-19 12:35:56 +0000
@@ -0,0 +1,71 @@
+#!/usr/bin/python
+#
+# Copyright 2017 Canonical Ltd.  This software is licensed under the
+# GNU Affero General Public License version 3 (see the file LICENSE).
+
+"""WSGI archive authorisation provider entry point.
+
+Unlike most Launchpad scripts, the #! line of this script does not use -S.
+This is because it is only executed (as opposed to imported) for testing,
+and mod_wsgi does not disable the automatic import of the site module when
+importing this script, so we want the test to imitate mod_wsgi's behaviour
+as closely as possible.
+"""
+
+from __future__ import absolute_import, print_function, unicode_literals
+
+__metaclass__ = type
+__all__ = [
+    'check_password',
+    ]
+
+# mod_wsgi imports this file without a useful sys.path, so we need some
+# acrobatics to set ourselves up properly.
+import os.path
+import sys
+
+scripts_dir = os.path.dirname(os.path.abspath(os.path.realpath(__file__)))
+if scripts_dir not in sys.path:
+    sys.path.insert(0, scripts_dir)
+top = os.path.dirname(scripts_dir)
+
+# We can't stop mod_wsgi importing the site module.  Cross fingers and
+# arrange for it to be re-imported.
+sys.modules.pop("site", None)
+sys.modules.pop("sitecustomize", None)
+
+import _pythonpath
+
+from lp.soyuz.wsgi.archiveauth import check_password
+
+
+def main():
+    # Hook for testing, not used by WSGI.
+    from argparse import ArgumentParser
+
+    from lp.services.memcache.testing import MemcacheFixture
+    from lp.soyuz.wsgi import archiveauth
+
+    parser = ArgumentParser()
+    parser.add_argument("archive_path")
+    parser.add_argument("username")
+    parser.add_argument("password")
+    args = parser.parse_args()
+    archiveauth._memcache_client = MemcacheFixture()
+    result = check_password(
+        {"SCRIPT_NAME": args.archive_path}, args.username, args.password)
+    if result is None:
+        print("Archive or user does not exist.", file=sys.stderr)
+        return 2
+    elif result is False:
+        print("Password does not match.", file=sys.stderr)
+        return 1
+    elif result is True:
+        return 0
+    else:
+        print("Unexpected result from check_password: %s" % result)
+        return 3
+
+
+if __name__ == "__main__":
+    sys.exit(main())
 
=== modified file 'utilities/rocketfuel-setup'
--- utilities/rocketfuel-setup	2017-01-10 17:24:08 +0000
+++ utilities/rocketfuel-setup	2017-11-19 12:35:56 +0000
@@ -189,6 +189,12 @@
   exit 1
 fi
 
+sudo a2enmod wsgi > /dev/null
+if [ $? -ne 0 ]; then
+  echo "ERROR: Unable to enable wsgi module in Apache2"
+  exit 1
+fi
+
 if [ $DO_WORKSPACE == 0 ]; then
   cat <<EOT
 Branches have not been created, as requested.  You will need to do some or all