Merge lp:~smoser/software-properties/trunk.lp1667725-https-signing-key into lp:software-properties

[Scott Moser]

Get PPA key signing data from launchpad https rather than keyserver.

Previously adding of a PPA would read the signing key fingerprint from
launchpad and then read the signing key data from keyserver.ubuntu.com
over https.

This changes that to use the newly added getSigningKeyData operation
from launchpad. And uses the launchpad 'devel' api version.
The primary benefit of this is a reduction in the number
of external services that 'add-apt-repository' relies upon.
That means less external services that may be down, and also less
holes poked in egress networking to allow add-apt-repository to work.

There are also 3 other changes/improvements that we found along the way:
 a.) make python2 (pycurl) path follow http redirects.
 b.) use 'mangle_ppa_shortcut' in other paths.
 c.) mangle_ppa_shortcut can now support input with or without 'ppa:' as
     a prefix.

[lp:~smoser/software-properties/trunk.lp1667725-https-signing-key]

1051. By Scott Moser on 2018-08-01

remove staging, launchpad marked released.

1052. By Scott Moser on 2018-08-01

Get PPA key signing data from launchpad https rather than keyserver.
(LP: #1667725)

[Colin Watson (cjwatson)]

review: Approve

[Scott Moser (smoser)]

Addressed your other comments.

Thank you.

[lp:~smoser/software-properties/trunk.lp1667725-https-signing-key]

1053. By Scott Moser on 2018-08-03

debian/changelog: fix version

1054. By Scott Moser on 2018-08-03

address cjwatson's other comments.

1055. By Scott Moser on 2018-08-03

improve docstring, fix a bug in use of 'path', treat response as json.

cjwatson pointed out the response is json so we have to treat it as such.
just use the 'accept_json' and we're good.

1056. By Scott Moser on 2018-08-03

fix bug in calling of _recv_key (thanks cjwatson)

1057. By Scott Moser on 2018-08-03

use mangle_ppa_shortcut consistently.

this fixes a bug/issue in the python2 path with
  python -m softwareproperties.ppa ppa:~smoser/ppa

The issue was that launchpad would return a 301 and pycurl would
not follow the redirect.
  https://launchpad.net/api/devel/~smoser/+archive/ppa
would get a 301 to
  https://launchpad.net/api/devel/~smoser/+archive/ubuntu/ppa

1058. By Scott Moser on 2018-08-03

In python2 (pycurl) path follow redirects.

the pycurl path was not following redirects which made it inconsistent
behavior with python3 path. now they both act the same in this regard.

1059. By Scott Moser on 2018-08-03

Fix tests, support mangle_ppa_shortcut without a leading 'ppa:'

=== modified file 'debian/changelog'
--- debian/changelog	2018-07-26 08:56:14 +0000
+++ debian/changelog	2018-08-03 14:40:33 +0000
@@ -1,3 +1,10 @@
+software-properties (0.96.26) UNRELEASED; urgency=medium
+
+  * Get PPA key signing data from launchpad https rather than keyserver.
+    (LP: #1667725)
+
+ -- Scott Moser <[e-mail]>  Mon, 30 Jul 2018 21:44:24 -0400
+
 software-properties (0.96.25) cosmic; urgency=medium
 
   * Port the KDE frontend to a pure Qt frontend that Lubuntu can use as well.

=== modified file 'softwareproperties/ppa.py'
--- softwareproperties/ppa.py	2018-07-16 17:34:17 +0000
+++ softwareproperties/ppa.py	2018-08-03 14:40:33 +0000
@@ -49,7 +49,7 @@
 
 SKS_KEYSERVER = 'https://keyserver.ubuntu.com/pks/lookup?op=get&options=mr&exact=on&search=0x%s'
 # maintained until 2015
-LAUNCHPAD_PPA_API = 'https://launchpad.net/api/1.0/%s/+archive/%s'
+LAUNCHPAD_PPA_API = 'https://launchpad.net/api/devel/%s/+archive/%s'
 LAUNCHPAD_USER_API = 'https://launchpad.net/api/1.0/%s'
 LAUNCHPAD_USER_PPAS_API = 'https://launchpad.net/api/1.0/%s/ppas'
 LAUNCHPAD_DISTRIBUTION_API = 'https://launchpad.net/api/1.0/%s'
@@ -170,6 +170,7 @@
             curl = pycurl.Curl()
             curl.setopt(pycurl.SSL_VERIFYPEER, 1)
             curl.setopt(pycurl.SSL_VERIFYHOST, 2)
+            curl.setopt(pycurl.FOLLOWLOCATION, 1)
             curl.setopt(pycurl.WRITEFUNCTION, callback.body_callback)
             if LAUNCHPAD_PPA_CERT:
                 curl.setopt(pycurl.CAINFO, LAUNCHPAD_PPA_CERT)
@@ -200,7 +201,10 @@
 
 
 def mangle_ppa_shortcut(shortcut):
-    ppa_shortcut = shortcut.split(":")[1]
+    if ":" in shortcut:
+        ppa_shortcut = shortcut.split(":")[1]
+    else:
+        ppa_shortcut = shortcut
     if ppa_shortcut.startswith("/"):
         ppa_shortcut = ppa_shortcut.lstrip("/")
     user = ppa_shortcut.split("/")[0]
@@ -237,7 +241,8 @@
         cmd = "gpg -q --homedir %s --no-default-keyring --no-options --import --import-options %s" % (self._homedir, args)
         return subprocess.Popen(cmd.split(), stdin=subprocess.PIPE, stdout=subprocess.PIPE)
         
-    def _recv_key(self, signing_key_fingerprint):
+    def _recv_key(self, ppa_info):
+        signing_key_fingerprint = ppa_info["signing_key_fingerprint"]
         try:
             # double check that the signing key is a v4 fingerprint (160bit)
             if not verify_keyid_is_v4(signing_key_fingerprint):
@@ -248,8 +253,7 @@
             print("Error: signing key fingerprint does not exist")
             return False
 
-        return get_info_from_https(SKS_KEYSERVER % signing_key_fingerprint,
-                                   accept_json=False, retry_delays=(1, 2))
+        return get_ppa_signing_key_data(ppa_info)
 
     def _minimize_key(self, key):
         p = self.gpg_cmd("import-minimal,import-export")
@@ -292,7 +296,7 @@
             ppa_path = self.ppa_path
 
         try:
-            ppa_info = get_ppa_info(ppa_path)
+            ppa_info = get_ppa_info(mangle_ppa_shortcut(ppa_path))
         except PPAException as e:
             print(e.value)
             return False
@@ -303,7 +307,7 @@
             return False
         
         #  download the armored_key
-        armored_key = self._recv_key(signing_key_fingerprint)
+        armored_key = self._recv_key(ppa_info)
         if not armored_key:
             return False
 
@@ -394,6 +398,23 @@
               (LAUNCHPAD_PPA_API % (user, ppa)))
 
 
+def get_ppa_signing_key_data(info=None):
+    """Return signing key data in armored ascii format for the provided ppa.
+    
+    If 'info' is a dictionary, it is assumed to be the result
+    of 'get_ppa_info(ppa)'.  If it is a string, it is assumed to
+    be a ppa_path.
+
+    Return value is a text string."""
+    if isinstance(info, dict):
+        link = info["self_link"]
+    else:
+        link = get_ppa_info(mangle_ppa_shortcut(info))["self_link"]
+
+    return get_info_from_https(link + "?ws.op=getSigningKeyData",
+                               accept_json=True, retry_delays=(1, 2, 3))
+
+
 class PPAShortcutHandler(object):
     def __init__(self, shortcut):
         super(PPAShortcutHandler, self).__init__()
@@ -450,5 +471,5 @@
 
 if __name__ == "__main__":
     import sys
-    ppa = sys.argv[1].split(":")[1]
+    ppa = mangle_ppa_shortcut(sys.argv[1])
     print(get_ppa_info(ppa))

=== modified file 'tests/test_lp.py'
--- tests/test_lp.py	2018-03-30 14:14:39 +0000
+++ tests/test_lp.py	2018-08-03 14:40:33 +0000
@@ -25,6 +25,7 @@
     'distribution_link': 'https://launchpad.net/api/1.0/ubuntu',
     'owner_link': 'https://launchpad.net/api/1.0/~mvo',
     'reference': '~mvo/ubuntu/ppa',
+    'self_link': 'https://launchpad.net/api/devel/~mvo/+archive/ubuntu/ppa',
     }
 
 MOCK_KEY="""
@@ -96,6 +97,10 @@
         self.assertEqual("~gottcode/ubuntu/gcppa",
                          mangle_ppa_shortcut("ppa:/gottcode/gcppa"))
 
+    def test_mangle_ppa_supports_no_ppa_colon_prefix(self):
+        """mangle_ppa should also support input without 'ppa:'."""
+        self.assertEqual("~mvo/ubuntu/ppa", mangle_ppa_shortcut("~mvo/ppa"))
+
 
 class AddPPASigningKeyTestCase(unittest.TestCase):