Merge ~ubuntu-release/britney/+git/britney2-ubuntu:sil2100/private-runs into ~ubuntu-release/britney/+git/britney2-ubuntu:master

[Łukasz Zemczak]

Add support for running private tests and using private PPAs.

Add support for running private tests and using private PPAs.

The autopkgtest-cloud counterpart is here: https://code.launchpad.net/~ubuntu-release/autopkgtest-cloud/+git/autopkgtest-cloud/+merge/399668

[Steve Langasek (vorlon)]

Looks good, minor comments.

review: Needs Fixing

[Łukasz Zemczak (sil2100)]

Thank you for the review Steve! On it!

3b1699d... by Łukasz Zemczak on 2021-03-30

Fix some comments.

[Łukasz Zemczak (sil2100)]

Ok, pushed some comment fixes and also commented on the exception handling part. Could you take a look again? Oh, and in previous inline comments I also addressed the question about private PPAs that are not embargoed.

[Łukasz Zemczak (sil2100)]

Actually I almost forgot about one thing and I'm working on it right now: since the test results are private, the britney ADT report will have SWIFT urls to the results which will not works (as the container is private) - but we can have a small SSO protected wrapper to fetch the results for us in webcontrol.

1de71db... by Łukasz Zemczak on 2021-04-28

Add ability to share results with selected people.

a21c904... by Łukasz Zemczak on 2021-04-28

Merge branch 'master' of git+ssh://git.launchpad.net/~ubuntu-release/britney/+git/britney2-ubuntu into sil2100/private-runs

d1549b9... by Łukasz Zemczak on 2021-05-06

Commit the work regarding private-results handling.

7ff150c... by Łukasz Zemczak on 2021-06-09

Switch from swiftclient for private PPAs to using HTTP with X-Auth-Token instead.

This way there's less secrets that need to be shared and less new code to introduce. We also modified the test tooling to be able to check for authentication tokens in the queries.

f79ec3b... by Łukasz Zemczak on 2021-06-16

Revert "Switch from swiftclient for private PPAs to using HTTP with X-Auth-Token instead."

This reverts commit 7ff150ced70d0db565378683909e22dfac5383e1.

Sadly we can't use the X-Auth-Token approach due to implementational details (the token is valid only for an hour). So we need to switch back to using swiftclient.

41b606c... by Łukasz Zemczak on 2021-07-19

Commit fixes from staging testing: add required region name parameter.

090ccdf... by Łukasz Zemczak on 2021-07-19

Add an option to configure whether all ADT test results should be displayed or not.

738cac8... by Łukasz Zemczak on 2021-07-29

Add additional support for additional private test retry capabilities.

1848597... by Łukasz Zemczak on 2023-02-13

Support PPAs which have fingerprint enabled in them.

Unmerged commits

1848597... by Łukasz Zemczak on 2023-02-13

Support PPAs which have fingerprint enabled in them.

738cac8... by Łukasz Zemczak on 2021-07-29

Add additional support for additional private test retry capabilities.

090ccdf... by Łukasz Zemczak on 2021-07-19

Add an option to configure whether all ADT test results should be displayed or not.

41b606c... by Łukasz Zemczak on 2021-07-19

Commit fixes from staging testing: add required region name parameter.

f79ec3b... by Łukasz Zemczak on 2021-06-16

Revert "Switch from swiftclient for private PPAs to using HTTP with X-Auth-Token instead."

This reverts commit 7ff150ced70d0db565378683909e22dfac5383e1.

Sadly we can't use the X-Auth-Token approach due to implementational details (the token is valid only for an hour). So we need to switch back to using swiftclient.

7ff150c... by Łukasz Zemczak on 2021-06-09

Switch from swiftclient for private PPAs to using HTTP with X-Auth-Token instead.

This way there's less secrets that need to be shared and less new code to introduce. We also modified the test tooling to be able to check for authentication tokens in the queries.

d1549b9... by Łukasz Zemczak on 2021-05-06

Commit the work regarding private-results handling.

a21c904... by Łukasz Zemczak on 2021-04-28

Merge branch 'master' of git+ssh://git.launchpad.net/~ubuntu-release/britney/+git/britney2-ubuntu into sil2100/private-runs

1de71db... by Łukasz Zemczak on 2021-04-28

Add ability to share results with selected people.

3b1699d... by Łukasz Zemczak on 2021-03-30

Fix some comments.

diff --git a/britney.conf b/britney.conf
index 52f5b4c..00ffe30 100644
--- a/britney.conf
+++ b/britney.conf
@@ -101,11 +101,26 @@ ADT_SHARED_RESULTS_CACHE =
 # Swift base URL with the results (must be publicly readable and browsable)
 # or file location if results are pre-fetched
 ADT_SWIFT_URL     = https://autopkgtest.ubuntu.com/results/
+# Swift identity with read access to the private result container
+# (this is required whenever a private PPA is used for testing)
+ADT_SWIFT_USER    = 
+ADT_SWIFT_PASS    = 
+ADT_SWIFT_AUTH_URL = 
+ADT_SWIFT_TENANT  = 
+ADT_SWIFT_REGION  = 
+# List of launchpad users/teams that should have read access to any private
+# result logs
+ADT_PRIVATE_SHARED = 
+ADT_PRIVATE_URL    = https://autopkgtest.ubuntu.com/private-results/
+ADT_PRIVATE_RETRY  = 
 # Base URL for autopkgtest site, used for links in the excuses
 ADT_CI_URL        = https://autopkgtest.ubuntu.com/
 # URL for the autopkgtest database, if used
 ADT_DB_URL        = https://autopkgtest.ubuntu.com/static/autopkgtest.db
 ADT_HUGE          = 20
+# Change to 'yes' for excuses to include all ADT results, even those requiring
+# no action (like passing or always-failed)
+ADT_SHOW_IRRELEVANT = no
 
 # Autopkgtest results can be used to influence the aging
 ADT_REGRESSION_PENALTY =
diff --git a/britney.conf.template b/britney.conf.template
index b1df068..f662aa4 100644
--- a/britney.conf.template
+++ b/britney.conf.template
@@ -124,11 +124,26 @@ ADT_SHARED_RESULTS_CACHE =
 # or file location if results are pre-fetched
 #ADT_SWIFT_URL     = https://example.com/some/url
 ADT_SWIFT_URL     = file:///path/to/britney/state/debci.json
+# Swift identity with read access to the private result container
+# (this is required whenever a private PPA is used for testing)
+ADT_SWIFT_USER    = 
+ADT_SWIFT_PASS    = 
+ADT_SWIFT_AUTH_URL = 
+ADT_SWIFT_TENANT  = 
+ADT_SWIFT_REGION  = 
+# List of launchpad users/teams that should have read access to any private
+# result logs
+ADT_PRIVATE_SHARED = 
+ADT_PRIVATE_URL    = 
+ADT_PRIVATE_RETRY  = 
 # Base URL for autopkgtest site, used for links in the excuses
 ADT_CI_URL        = https://example.com/
 # Enable the huge queue for packages that trigger vast amounts of tests to not
 # starve the regular queue
 #ADT_HUGE          = 20
+# Change to 'yes' for excuses to include all ADT results, even those requiring
+# no action (like passing or always-failed)
+ADT_SHOW_IRRELEVANT = no
 
 # Autopkgtest results can be used to influence the aging, leave
 # ADT_REGRESSION_PENALTY empty to have regressions block migration
diff --git a/britney2/policies/autopkgtest.py b/britney2/policies/autopkgtest.py
index 7ff8c1f..d007ba0 100644
--- a/britney2/policies/autopkgtest.py
+++ b/britney2/policies/autopkgtest.py
@@ -150,12 +150,26 @@ class AutopkgtestPolicy(BasePolicy):
 
         try:
             self.options.adt_ppas = self.options.adt_ppas.strip().split()
+            # We also allow, for certain other use-cases, passing the PPA
+            # fingerprint to for each of the PPAs. This however is not
+            # currently used by the ADT policy, so get rid of it.
+            for i, ppa in enumerate(self.options.adt_ppas):
+                if '@' not in ppa and ':' in ppa:
+                    self.options.adt_ppas[i] = ppa.split(':')[0]
         except AttributeError:
             self.options.adt_ppas = []
 
+        try:
+            self.options.adt_private_shared = self.options.adt_private_shared.strip().split()
+        except AttributeError:
+            self.options.adt_private_shared = []
+
         self.swift_container = 'autopkgtest-' + options.series
         if self.options.adt_ppas:
-            self.swift_container += '-' + options.adt_ppas[-1].replace('/', '-')
+            # private PPAs require the auth credentials given + we allow the
+            # PPA fingerprint attached at the end
+            # those need to be removed before the ppa-based name can be used
+            self.swift_container += '-' + options.adt_ppas[-1].rpartition('@')[2].replace('/', '-').partition(':')[0]
 
         # restrict adt_arches to architectures we actually run for
         self.adt_arches = []
@@ -233,6 +247,48 @@ class AutopkgtestPolicy(BasePolicy):
         else:
             self.logger.info('%s does not exist, re-downloading all results from swift', self.results_cache_file)
 
+        # log into swift in case we need to fetch some private results
+        # this is optional - if there are no credentials present, results will
+        # be fetched without authentication
+        if self.options.adt_swift_user:
+            if (not self.options.adt_swift_pass or
+                    not self.options.adt_swift_auth_url or
+                    not self.options.adt_swift_tenant or
+                    not self.options.adt_swift_region):
+                raise RuntimeError('Incomplete swift credentials given')
+
+            # once swift credentials are given, the results will be published
+            # to a private container
+            self.swift_container = 'private-' + self.swift_container
+
+            # check if all private PPAs have a fingerprint provided
+            # private PPAs need to follow the following pattern:
+            # user:token@team/name:fingerprint
+            for ppa in self.options.adt_ppas:
+                # TODO: write a test for this
+                if '@' in ppa and not re.match(r'^.+:.+@.+:.+$', ppa):
+                    raise RuntimeError('Private PPA %s not following required format (user:token@team/name:fingerprint)', ppa)
+
+            import swiftclient
+
+            if '/v2.0' not in self.options.adt_swift_auth_url:
+                raise RuntimeError('Unsupported swift auth version')
+
+            self.logger.info('Creating an authenticated swift connection for user %s', self.options.adt_swift_user)
+            self.swift_conn = swiftclient.Connection(
+                authurl=self.options.adt_swift_auth_url,
+                user=self.options.adt_swift_user,
+                key=self.options.adt_swift_pass,
+                tenant_name=self.options.adt_swift_tenant,
+                auth_version='2.0',
+                os_options={'region_name': self.options.adt_swift_region}
+                )
+        else:
+            if any('@' in ppa for ppa in self.options.adt_ppas):
+                raise RuntimeError('Private PPA configured but no swift credentials given')
+
+            self.swift_conn = None
+
         # read in the new results
         if self.options.adt_swift_url.startswith('file://'):
             debci_file = self.options.adt_swift_url[7:]
@@ -478,7 +534,22 @@ class AutopkgtestPolicy(BasePolicy):
                     if status in ['REGRESSION', 'RUNNING-REFERENCE']:
                         if self.options.adt_retry_url_mech == 'run_id':
                             retry_url = self.options.adt_ci_url + 'api/v1/retry/' + run_id
-                        else:
+                        elif self.options.adt_private_retry:
+                            # if a custom retry url mechanism has been given,
+                            # use that - but instead of passing PPAs with
+                            # sensitive credentials, we pass additional context
+                            # that will help the backend identify what env
+                            # needs to be used
+                            retry_url = self.options.adt_private_retry + \
+                                    urllib.parse.urlencode([('release', self.options.series),
+                                                            ('arch', arch),
+                                                            ('package', testsrc),
+                                                            ('trigger', trigger),
+                                                            ('context', self.swift_container)])
+                        elif not any('@' in ppa for ppa in self.options.adt_ppas):
+                            # otherwise private PPAs currently should not
+                            # display a retry button as we can not guarantee
+                            # that the secrets will not leak
                             retry_url = self.options.adt_ci_url + 'request.cgi?' + \
                                     urllib.parse.urlencode([('release', self.options.series),
                                                             ('arch', arch),
@@ -502,8 +573,10 @@ class AutopkgtestPolicy(BasePolicy):
                     html_archmsg.append(message)
 
                 # render HTML line for testsrc entry, but only when action is
-                # or may be required
-                if r - {'PASS', 'NEUTRAL', 'RUNNING-ALWAYSFAIL', 'ALWAYSFAIL', 'IGNORE-FAIL'}:
+                # or may be required or when britney is configured to print
+                # everything
+                if (self.options.adt_show_irrelevant or
+                        r - {'PASS', 'NEUTRAL', 'RUNNING-ALWAYSFAIL', 'ALWAYSFAIL', 'IGNORE-FAIL'}):
                     results_info.append("autopkgtest for %s: %s" % (testname, ', '.join(html_archmsg)))
 
         if verdict != PolicyVerdict.PASS:
@@ -891,55 +964,93 @@ class AutopkgtestPolicy(BasePolicy):
                 query['marker'] = query['prefix'] + latest_run_id
 
         # request new results from swift
-        url = os.path.join(swift_url, self.swift_container)
-        url += '?' + urllib.parse.urlencode(query)
-        f = None
-        try:
-            f = self.download_retry(url)
-            if f.getcode() == 200:
-                result_paths = f.read().decode().strip().splitlines()
-            elif f.getcode() == 204:  # No content
-                result_paths = []
-            else:
-                # we should not ever end up here as we expect a HTTPError in
-                # other cases; e. g. 3XX is something that tells us to adjust
-                # our URLS, so fail hard on those
-                raise NotImplementedError('fetch_swift_results(%s): cannot handle HTTP code %i' %
-                                          (url, f.getcode()))
-        except IOError as e:
-            # 401 "Unauthorized" is swift's way of saying "container does not exist"
-            if hasattr(e, 'code') and e.code == 401:
-                self.logger.info('fetch_swift_results: %s does not exist yet or is inaccessible', url)
-                return
-            # Other status codes are usually a transient
-            # network/infrastructure failure. Ignoring this can lead to
-            # re-requesting tests which we already have results for, so
-            # fail hard on this and let the next run retry.
-            self.logger.error('Failure to fetch swift results from %s: %s', url, str(e))
-            sys.exit(1)
-        finally:
-            if f is not None:
-                f.close()
+        if self.swift_conn:
+            # when we have an authenticated swift connection, use that to
+            # fetch the result_path list as we might be fetching from an
+            # otherwise unaccessible container
+            from swiftclient.exceptions import ClientException
+
+            try:
+                _, returned_paths = self.swift_conn.get_container(
+                    self.swift_container,
+                    query_string=urllib.parse.urlencode(query))
+            except ClientException as e:
+                # 401 "Unauthorized" is swift's way of saying "container does not exist"
+                if e.http_status == 401 or e.http_status == 404:
+                    self.logger.info('fetch_swift_results: %s does not exist yet or is inaccessible', self.swift_container)
+                    return
+                # Other status codes are usually a transient
+                # network/infrastructure failure. Ignoring this can lead to
+                # re-requesting tests which we already have results for, so
+                # fail hard on this and let the next run retry.
+                self.logger.error('Failure to fetch swift results from %s: %s', self.swift_container, str(e))
+                sys.exit(1)
+            result_paths = [p['subdir'] for p in returned_paths]
+        else:
+            url = os.path.join(swift_url, self.swift_container)
+            url += '?' + urllib.parse.urlencode(query)
+            f = None
+            try:
+                f = self.download_retry(url)
+                if f.getcode() == 200:
+                    result_paths = f.read().decode().strip().splitlines()
+                elif f.getcode() == 204:  # No content
+                    result_paths = []
+                else:
+                    # we should not ever end up here as we expect a HTTPError in
+                    # other cases; e. g. 3XX is something that tells us to adjust
+                    # our URLS, so fail hard on those
+                    raise NotImplementedError('fetch_swift_results(%s): cannot handle HTTP code %i' %
+                                              (url, f.getcode()))
+            except IOError as e:
+                # 401 "Unauthorized" is swift's way of saying "container does not exist"
+                if hasattr(e, 'code') and e.code == 401:
+                    self.logger.info('fetch_swift_results: %s does not exist yet or is inaccessible', url)
+                    return
+                # same as above in the swift authenticated case
+                self.logger.error('Failure to fetch swift results from %s: %s', url, str(e))
+                sys.exit(1)
+            finally:
+                if f is not None:
+                    f.close()
 
         for p in result_paths:
             self.fetch_one_result(
-                os.path.join(swift_url, self.swift_container, p, 'result.tar'), src, arch)
+                swift_url, self.swift_container, p, 'result.tar', src, arch)
 
     fetch_swift_results._done = set()
 
-    def fetch_one_result(self, url, src, arch):
+    def fetch_one_result(self, swift_url, container, path, name, src, arch):
         '''Download one result URL for source/arch
 
         Remove matching pending_tests entries.
         '''
+
         f = None
         try:
-            f = self.download_retry(url)
-            if f.getcode() == 200:
-                tar_bytes = io.BytesIO(f.read())
+            if self.swift_conn:
+                from swiftclient.exceptions import ClientException
+
+                # We don't need any additional retry logic as swiftclient
+                # already performs retries (5 by default).
+                url = os.path.join(path, name)
+                try:
+                    _, contents = self.swift_conn.get_object(container, url)
+                except ClientException as e:
+                    self.logger.error('Failure to fetch %s from container %s: %s',
+                                      url, container, str(e))
+                    if e.http_status == 404:
+                        return
+                    sys.exit(1)
+                tar_bytes = io.BytesIO(contents)
             else:
-                raise NotImplementedError('fetch_one_result(%s): cannot handle HTTP code %i' %
-                                          (url, f.getcode()))
+                url = os.path.join(swift_url, container, path, name)
+                f = self.download_retry(url)
+                if f.getcode() == 200:
+                    tar_bytes = io.BytesIO(f.read())
+                else:
+                    raise NotImplementedError('fetch_one_result(%s): cannot handle HTTP code %i' %
+                                              (url, f.getcode()))
         except IOError as e:
             self.logger.error('Failure to fetch %s: %s', url, str(e))
             # we tolerate "not found" (something went wrong on uploading the
@@ -1122,6 +1233,8 @@ class AutopkgtestPolicy(BasePolicy):
 
         params = {'triggers': triggers}
         if self.options.adt_ppas:
+            # Note: the PPA might be a private PPA, and then the PPA parameter
+            # includes the authorization token.
             params['ppas'] = self.options.adt_ppas
             qname = 'debci-ppa-%s-%s' % (self.options.series, arch)
         elif huge:
@@ -1130,6 +1243,11 @@ class AutopkgtestPolicy(BasePolicy):
             qname = 'debci-%s-%s' % (self.options.series, arch)
         params['submit-time'] = datetime.strftime(datetime.utcnow(), '%Y-%m-%d %H:%M:%S%z')
 
+        if self.swift_conn:
+            params['swiftuser'] = self.options.adt_swift_user
+            if self.options.adt_private_shared:
+                params['readable-by'] = self.options.adt_private_shared
+
         if self.amqp_channel:
             import amqplib.client_0_8 as amqp
             params = json.dumps(params)
@@ -1358,7 +1476,9 @@ class AutopkgtestPolicy(BasePolicy):
                                    run_id,
                                    'log.gz')
             else:
-                url = os.path.join(self.options.adt_swift_url,
+                # Private runs have a different base url
+                results_url = self.options.adt_private_url if self.swift_conn else self.options.adt_swift_url
+                url = os.path.join(results_url,
                                    self.swift_container,
                                    self.options.series,
                                    arch,
diff --git a/tests/__init__.py b/tests/__init__.py
index eaa9436..a277747 100644
--- a/tests/__init__.py
+++ b/tests/__init__.py
@@ -395,9 +395,21 @@ ADT_PPAS          =
 ADT_SHARED_RESULTS_CACHE =
 
 ADT_SWIFT_URL     = http://localhost:18085
+ADT_SWIFT_USER    = 
+ADT_SWIFT_PASS    = 
+ADT_SWIFT_AUTH_URL = 
+ADT_SWIFT_TENANT  = 
+ADT_SWIFT_REGION  = 
+ADT_PRIVATE_SHARED = 
+ADT_PRIVATE_URL    = 
+ADT_PRIVATE_RETRY  = 
 ADT_CI_URL        = https://autopkgtest.ubuntu.com/
 ADT_HUGE          = 20
+<<<<<<< tests/__init__.py
 ADT_DB_URL        = 
+=======
+ADT_SHOW_IRRELEVANT = no
+>>>>>>> tests/__init__.py
 
 ADT_SUCCESS_BOUNTY     =
 ADT_REGRESSION_PENALTY =
diff --git a/tests/mock_swift.py b/tests/mock_swift.py
index b33c65a..0fc1a4c 100644
--- a/tests/mock_swift.py
+++ b/tests/mock_swift.py
@@ -19,6 +19,10 @@ except ImportError:
     from urlparse import urlparse, parse_qs
 
 
+TESTS_DIR = os.path.dirname(os.path.abspath(__file__))
+PROJECT_DIR = os.path.dirname(TESTS_DIR)
+
+
 class SwiftHTTPRequestHandler(BaseHTTPRequestHandler):
     '''Mock swift container with autopkgtest results
 
@@ -124,7 +128,15 @@ class AutoPkgTestSwiftServer:
         '''
         SwiftHTTPRequestHandler.results = results
 
-    def start(self):
+    def start(self, swiftclient=False):
+        if swiftclient:
+            # since we're running britney directly, the only way to reliably
+            # mock out the swiftclient module is to override it in the local
+            # path with the dummy version we created
+            src = os.path.join(TESTS_DIR, 'mock_swiftclient.py')
+            dst = os.path.join(PROJECT_DIR, 'swiftclient.py')
+            os.symlink(src, dst)
+            
         assert self.server_pid is None, 'already started'
         if self.log:
             self.log.close()
@@ -148,12 +160,19 @@ class AutoPkgTestSwiftServer:
         sys.exit(0)
 
     def stop(self):
+        # in case we were 'mocking out' swiftclient, remove the symlink we
+        # created earlier during start()
+        swiftclient_mod = os.path.join(PROJECT_DIR, 'swiftclient.py')
+        if os.path.islink(swiftclient_mod):
+            os.unlink(swiftclient_mod)
+
         assert self.server_pid, 'not running'
         os.kill(self.server_pid, 15)
         os.waitpid(self.server_pid, 0)
         self.server_pid = None
         self.log.close()
 
+
 if __name__ == '__main__':
     srv = AutoPkgTestSwiftServer()
     srv.set_results({'autopkgtest-testing': {
diff --git a/tests/mock_swiftclient.py b/tests/mock_swiftclient.py
new file mode 100644
index 0000000..5f419ac
--- /dev/null
+++ b/tests/mock_swiftclient.py
@@ -0,0 +1,70 @@
+# Mock the swiftclient Python library, the bare minimum for ADT purposes
+# Author: Łukasz 'sil2100' Zemczak <[e-mail]>
+
+import os
+import sys
+
+from urllib.request import urlopen
+
+
+# We want to use this single Python module file to mock out the exception
+# module as well.
+sys.modules["swiftclient.exceptions"] = sys.modules[__name__]
+
+
+class ClientException(Exception):
+    def __init__(self, msg, http_status=''):
+        super(ClientException, self).__init__(msg)
+        self.msg = msg
+        self.http_status = http_status
+
+
+class Connection:
+    def __init__(self, authurl, user, key, tenant_name, auth_version):
+        self._mocked_swift = 'http://localhost:18085'
+
+    def get_container(self, container, marker=None, limit=None, prefix=None,
+                      delimiter=None, end_marker=None, path=None,
+                      full_listing=False, headers=None, query_string=None):
+        url = os.path.join(self._mocked_swift, container) + '?' + query_string
+        req = None
+        try:
+            req = urlopen(url, timeout=30)
+            code = req.getcode()
+            if code == 200:
+                result_paths = req.read().decode().strip().splitlines()
+            elif code == 204:  # No content
+                result_paths = []
+            else:
+                raise ClientException('MockedError', http_status=str(code))
+        except IOError as e:
+            # 401 "Unauthorized" is swift's way of saying "container does not exist"
+            # But here we just assume swiftclient handles this via the usual
+            # ClientException.
+            raise ClientException('MockedError', http_status=str(e.code) if hasattr(e, 'code') else '')
+        finally:
+            if req is not None:
+                req.close()
+
+        return (None, result_paths)
+
+    def get_object(self, container, obj):
+        url = os.path.join(self._mocked_swift, container, obj)
+        req = None
+        try:
+            req = urlopen(url, timeout=30)
+            code = req.getcode()
+            if code == 200:
+                contents = req.read()
+            else:
+                raise ClientException('MockedError', http_status=str(code))
+        except IOError as e:
+            # 401 "Unauthorized" is swift's way of saying "container does not exist"
+            # But here we just assume swiftclient handles this via the usual
+            # ClientException.
+            raise ClientException('MockedError', http_status=str(e.code) if hasattr(e, 'code') else '')
+        finally:
+            if req is not None:
+                req.close()
+
+        return (None, contents)
diff --git a/tests/test_autopkgtest.py b/tests/test_autopkgtest.py
index 5c7bb72..906f729 100644
--- a/tests/test_autopkgtest.py
+++ b/tests/test_autopkgtest.py
@@ -17,6 +17,7 @@ import urllib.parse
 
 import apt_pkg
 import yaml
+from unittest.mock import patch, Mock
 
 PROJECT_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
 sys.path.insert(0, PROJECT_DIR)
@@ -214,7 +215,7 @@ class TestAutopkgtestBase(TestBase):
         with open(email_path, 'w', encoding='utf-8') as email:
             email.write(json.dumps(self.email_cache))
 
-        self.swift.start()
+        self.swift.start(swiftclient=True)
         (excuses_yaml, excuses_html, out) = self.run_britney()
         self.swift.stop()
 
@@ -2666,6 +2667,181 @@ class AT(TestAutopkgtestBase):
         self.assertEqual(self.amqp_requests, set())
         self.assertEqual(self.pending_requests, {})
 
+    def test_ppas_fingerprint(self):
+        '''Run test requests with PPAs where fingerprint is provided'''
+
+        self.data.add_default_packages(lightgreen=False)
+
+        for line in fileinput.input(self.britney_conf, inplace=True):
+            if line.startswith('ADT_PPAS'):
+                print('ADT_PPAS = joe/foo:fingerprint awesome-developers/staging')
+            else:
+                sys.stdout.write(line)
+
+        exc = self.run_it(
+            [('lightgreen', {'Version': '2'}, 'autopkgtest')],
+            {'lightgreen': (True, {'lightgreen': {'amd64': 'RUNNING-ALWAYSFAIL'}})},
+            {'lightgreen': [('old-version', '1'), ('new-version', '2')]}
+        )[1]
+
+        for arch in ['i386', 'amd64']:
+            self.assertTrue(
+                ('debci-ppa-testing-%s:lightgreen {"triggers": ["lightgreen/2"], '
+                 '"ppas": ["joe/foo", "awesome-developers/staging"]}') % arch in self.amqp_requests or
+                ('debci-ppa-testing-%s:lightgreen {"ppas": ["joe/foo", '
+                 '"awesome-developers/staging"], "triggers": ["lightgreen/2"]}') % arch in self.amqp_requests,
+                self.amqp_requests)
+        self.assertEqual(len(self.amqp_requests), 2)
+
+    def test_private_ppas(self):
+        '''Run test requests with an additional private PPA'''
+
+        self.data.add_default_packages(lightgreen=False)
+
+        for line in fileinput.input(self.britney_conf, inplace=True):
+            if line.startswith('ADT_PPAS'):
+                print('ADT_PPAS = first/ppa user:password@joe/foo:DEADBEEF')
+            elif line.startswith('ADT_SWIFT_USER'):
+                print('ADT_SWIFT_USER = user')
+            elif line.startswith('ADT_SWIFT_PASS'):
+                print('ADT_SWIFT_PASS = pass')
+            elif line.startswith('ADT_SWIFT_TENANT'):
+                print('ADT_SWIFT_TENANT = tenant')
+            elif line.startswith('ADT_SWIFT_REGION'):
+                print('ADT_SWIFT_REGION = region')
+            elif line.startswith('ADT_SWIFT_AUTH_URL'):
+                print('ADT_SWIFT_AUTH_URL = http://127.0.0.1:5000/v2.0/')
+            elif line.startswith('ADT_PRIVATE_SHARED'):
+                print('ADT_PRIVATE_SHARED = user1 team2')
+            elif line.startswith('ADT_PRIVATE_URL'):
+                print('ADT_PRIVATE_URL = http://localhost:18085/private-results/')
+            else:
+                sys.stdout.write(line)
+
+        exc = self.run_it(
+            [('lightgreen', {'Version': '2'}, 'autopkgtest')],
+            {'lightgreen': (True, {'lightgreen': {'amd64': 'RUNNING-ALWAYSFAIL'}})},
+            {'lightgreen': [('old-version', '1'), ('new-version', '2')]}
+        )[1]
+
+        # check if the private PPA info is propagated to the AMQP queue
+        self.assertEqual(len(self.amqp_requests), 2)
+        for request in self.amqp_requests:
+            self.assertIn('"triggers": ["lightgreen/2"]', request)
+            self.assertIn('"ppas": ["first/ppa", "user:password@joe/foo:DEADBEEF"]', request)
+            self.assertIn('"swiftuser": "user"', request)
+            self.assertIn('"readable-by": ["user1", "team2"]', request)
+
+        # add results to PPA specific swift container
+        self.swift.set_results({'private-autopkgtest-testing-joe-foo': {
+            'testing/i386/l/lightgreen/20150101_100000@': (0, 'lightgreen 1', tr('passedbefore/1')),
+            'testing/i386/l/lightgreen/20150101_100100@': (4, 'lightgreen 2', tr('lightgreen/2')),
+            'testing/amd64/l/lightgreen/20150101_100101@': (0, 'lightgreen 2', tr('lightgreen/2')),
+        }})
+
+        exc = self.run_it(
+            [],
+            {'lightgreen': (False, {'lightgreen/2': {'i386': 'REGRESSION', 'amd64': 'PASS'}})},
+            {'lightgreen': [('old-version', '1'), ('new-version', '2')]}
+        )[1]
+        # check if the right container name is used and that no secrets are
+        # leaked in the retry url (as it should be None now)
+        self.assertEqual(
+            exc['lightgreen']['policy_info']['autopkgtest'],
+            {'lightgreen/2': {
+                'amd64': [
+                    'PASS',
+                    'http://localhost:18085/private-results/private-autopkgtest-testing-joe-foo/'
+                    'testing/amd64/l/lightgreen/20150101_100101@/log.gz',
+                    None,
+                    'http://localhost:18085/private-results/private-autopkgtest-testing-joe-foo/'
+                    'testing/amd64/l/lightgreen/20150101_100101@/artifacts.tar.gz',
+                    None],
+                'i386': [
+                    'REGRESSION',
+                    'http://localhost:18085/private-results/private-autopkgtest-testing-joe-foo/'
+                    'testing/i386/l/lightgreen/20150101_100100@/log.gz',
+                    None,
+                    'http://localhost:18085/private-results/private-autopkgtest-testing-joe-foo/'
+                    'testing/i386/l/lightgreen/20150101_100100@/artifacts.tar.gz',
+                    None]},
+             'verdict': 'REJECTED_PERMANENTLY'})
+        self.assertEqual(self.amqp_requests, set())
+        self.assertEqual(self.pending_requests, {})
+
+    def test_private_without_ppa(self):
+        '''Run a private test (without using a private PPA)'''
+
+        self.data.add_default_packages(lightgreen=False)
+
+        for line in fileinput.input(self.britney_conf, inplace=True):
+            if line.startswith('ADT_SWIFT_USER'):
+                print('ADT_SWIFT_USER = user')
+            elif line.startswith('ADT_SWIFT_PASS'):
+                print('ADT_SWIFT_PASS = pass')
+            elif line.startswith('ADT_SWIFT_TENANT'):
+                print('ADT_SWIFT_TENANT = tenant')
+            elif line.startswith('ADT_SWIFT_REGION'):
+                print('ADT_SWIFT_REGION = region')
+            elif line.startswith('ADT_SWIFT_AUTH_URL'):
+                print('ADT_SWIFT_AUTH_URL = http://127.0.0.1:5000/v2.0/')
+            elif line.startswith('ADT_PRIVATE_URL'):
+                print('ADT_PRIVATE_URL = http://localhost:18085/private-results/')
+            else:
+                sys.stdout.write(line)
+
+        exc = self.run_it(
+            [('lightgreen', {'Version': '2'}, 'autopkgtest')],
+            {'lightgreen': (True, {'lightgreen': {'amd64': 'RUNNING-ALWAYSFAIL'}})},
+            {'lightgreen': [('old-version', '1'), ('new-version', '2')]}
+        )[1]
+
+        # check if the user info is propagated to the AMQP queue
+        self.assertEqual(len(self.amqp_requests), 2)
+        for request in self.amqp_requests:
+            self.assertIn('"triggers": ["lightgreen/2"]', request)
+            self.assertIn('"swiftuser": "user"', request)
+            # we did not give a list of users to give read-access, so make sure
+            # there are none
+            self.assertNotIn('"readable-by"', request)
+
+        # add results to PPA specific swift container
+        self.swift.set_results({'private-autopkgtest-testing': {
+            'testing/i386/l/lightgreen/20150101_100000@': (0, 'lightgreen 1', tr('passedbefore/1')),
+            'testing/i386/l/lightgreen/20150101_100100@': (4, 'lightgreen 2', tr('lightgreen/2')),
+            'testing/amd64/l/lightgreen/20150101_100101@': (0, 'lightgreen 2', tr('lightgreen/2')),
+        }})
+
+        exc = self.run_it(
+            [],
+            {'lightgreen': (False, {'lightgreen/2': {'i386': 'REGRESSION', 'amd64': 'PASS'}})},
+            {'lightgreen': [('old-version', '1'), ('new-version', '2')]}
+        )[1]
+
+        # check if the right container name is used and that we still have the
+        # retry url (it should only be hidden for private PPAs)
+        self.assertEqual(
+            exc['lightgreen']['policy_info']['autopkgtest'],
+            {'lightgreen/2': {
+                'amd64': [
+                    'PASS',
+                    'http://localhost:18085/private-results/private-autopkgtest-testing/'
+                    'testing/amd64/l/lightgreen/20150101_100101@/log.gz',
+                    'https://autopkgtest.ubuntu.com/packages/l/lightgreen/testing/amd64',
+                    None,
+                    None],
+                'i386': [
+                    'REGRESSION',
+                    'http://localhost:18085/private-results/private-autopkgtest-testing/'
+                    'testing/i386/l/lightgreen/20150101_100100@/log.gz',
+                    'https://autopkgtest.ubuntu.com/packages/l/lightgreen/testing/i386',
+                    None,
+                    'https://autopkgtest.ubuntu.com/request.cgi?release=testing&arch=i386&package=lightgreen&'
+                    'trigger=lightgreen%2F2']},
+             'verdict': 'REJECTED_PERMANENTLY'})
+        self.assertEqual(self.amqp_requests, set())
+        self.assertEqual(self.pending_requests, {})
+
     def test_disable_upgrade_tester(self):
         '''Run without second stage upgrade tester'''
 
diff --git a/tests/test_policy.py b/tests/test_policy.py
index 1952831..c6600c0 100644
--- a/tests/test_policy.py
+++ b/tests/test_policy.py
@@ -43,6 +43,12 @@ def initialize_policy(test_name, policy_class, *args, **kwargs):
         architectures=ARCH,
         adt_swift_url='file://' + debci_data,
         adt_ci_url='',
+        adt_swift_user='',
+        adt_swift_pass='',
+        adt_swift_tenant='',
+        adt_swift_auth_url='',
+        adt_private_shared=[],
+        adt_private_url='',
         adt_success_bounty=3,
         adt_regression_penalty=False,
         adt_retry_url_mech='run_id',