Merge lp:~brian-murray/apport/support-ppa-packages into lp:~apport-hackers/apport/trunk
- support-ppa-packages
- Merge into trunk
Status: | Merged |
---|---|
Merged at revision: | 2980 |
Proposed branch: | lp:~brian-murray/apport/support-ppa-packages |
Merge into: | lp:~apport-hackers/apport/trunk |
Diff against target: |
503 lines (+274/-20) 6 files modified
AUTHORS (+4/-1) apport/packaging.py (+5/-1) apport/sandboxutils.py (+21/-6) backends/packaging-apt-dpkg.py (+151/-7) bin/apport-retrace (+4/-1) test/test_backend_apt_dpkg.py (+89/-4) |
To merge this branch: | bzr merge lp:~brian-murray/apport/support-ppa-packages |
Related bugs: |
Reviewer | Review Type | Date Requested | Status |
---|---|---|---|
Martin Pitt (community) | Approve | ||
Review via email: mp+263437@code.launchpad.net |
Commit message
Description of the change
This branch builds upon Tim Lunn's initial work (https:/
A list of "origins" is created for packages with a foreign origin. This is then passed to install packages so that the sandbox will be created with proper apt data sources for the PPA. Its also possible to setup an apt data source that will be used when creating the sandbox. For consistency with apt these sources are located in /etc/apt/
The following new tests were added to test_backend_
def test_create_
def test_create_
def test_use_
def test_install_
They test the creation of a sources.list entry for two different types of PPAs, using a configured sources.list entry for a PPA, and installing a package from a PPA. Three of the tests are dependent upon two PPAs being available which might be a point of concern.
- 2979. By Brian Murray
-
mess o' modifications based on pitti's feeback
Brian Murray (brian-murray) wrote : | # |
Thanks for all the comments, I think I have addressed all of them and the code should look better now.
Brian Murray (brian-murray) wrote : | # |
Steve indicated there may be some security concerns about installing any package from Launchpad and running that package's binaries inside gdb. Subsequently, I have made the ability to create apt data sources for PPAs an optional one and set the default value to false. Regardless, if there are sources.list entries in the configdir corresponding to a PPA then packages will be used from those PPAs.
- 2980. By Brian Murray
-
Add an option to apport-retrace (and pass it along) so that apt sources data is only created for PPAs if requested.
Steve Langasek (vorlon) wrote : | # |
Security concerns, and also infrastructure load concerns. I don't believe we've made any committment to providing retracing resources for crashes of packages originating from arbitrary ppas.
- 2981. By Brian Murray
-
Fix origins re, pass log_timestamps to make_sandbox, fix a bug in origin_path existence checking.
- 2982. By Brian Murray
-
explicitly close the results from urlopen
Martin Pitt (pitti) wrote : | # |
A couple of inline replies to the previous review round. I'll review the new code in a separate comment.
Martin Pitt (pitti) wrote : | # |
Thanks for the updates! Another round, then this should be perfect.
- 2983. By Brian Murray
-
make testing for gpg keys less fragile
- 2984. By Brian Murray
-
address pitti's feedback and simplify how using origins is specified
Brian Murray (brian-murray) wrote : | # |
Okay, I think I've addressed everything, except for your concern regarding sources.list entries which I've commented on in-line.
Martin Pitt (pitti) wrote : | # |
Responding to sources.list parsing.
Martin Pitt (pitti) wrote : | # |
Some tiny nitpicks left, which I'd also be happy to do myself during merge. Do you want to change the sources.list parsing as discussed? If not I'm okay with leaving it like that and cleaning this up in a separate MP. Thanks!
- 2985. By Brian Murray
-
address final nitpicks
Brian Murray (brian-murray) wrote : | # |
I've fixed the nitpicks and would prefer to sort out the sources.list parsing in a separate merge proposal.
Martin Pitt (pitti) wrote : | # |
The last commit changed the wrong len(components), I fixed that and added a NEWS entry.
The tests fail with PYTHON=python2, with this:
=======
ERROR: test_create_
Add sources.list entries for a named PPA.
-------
Traceback (most recent call last):
File "test/test_
'ubuntu', 'trusty', origins=[ppa])
File "backends/
ppa_info = apport.
File "/home/
return json.loads(content)
File "/usr/lib/
return _default_
File "/usr/lib/
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/lib/
raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded
I checked json_request, and content == "Object: <Person at 0x2b0b217fc8d0 daisy (Daisy Bachmann)>, name: u'+archive'" which is indeed not JSON.
In the "good" python 3 case the URL is https//
This is indeed weird:
$ python3 -c 'from urllib.request import urlopen; print(urlopen("https:/
/~daisy/
[...]
urllib.
$ python -c 'from urllib import urlopen; print(urlopen("https:/
Object: <Person at 0x2b4df96bcbd0 daisy (Daisy Bachmann)>, name: u'+archive'
However, .getcode() says "404" in python 2, so this needs to be checked separately. I added that now.
There is one failure left with Python 2:
=======
ERROR: test_use_
Use a sources.list.d file for a PPA.
-------
Traceback (most recent call last):
File "test/test_
'ubuntu', 'trusty', origins=
File "backends/
ppa_info = apport.
File "/home/
return json.loads(content)
File "/usr/lib/
return _default_
File "/usr/lib/
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/lib/
raise ValueError("No JSON object could be decoded")
Valu...
Preview Diff
1 | === modified file 'AUTHORS' | |||
2 | --- AUTHORS 2011-11-09 08:40:10 +0000 | |||
3 | +++ AUTHORS 2015-07-13 22:32:23 +0000 | |||
4 | @@ -1,7 +1,7 @@ | |||
5 | 1 | Copyright: | 1 | Copyright: |
6 | 2 | --------- | 2 | --------- |
7 | 3 | General: | 3 | General: |
9 | 4 | Copyright (C) 2006 - 2011 Canonical Ltd. | 4 | Copyright (C) 2006 - 2015 Canonical Ltd. |
10 | 5 | 5 | ||
11 | 6 | backends/packaging_rpm.py: | 6 | backends/packaging_rpm.py: |
12 | 7 | Copyright (C) 2007 Red Hat Inc. | 7 | Copyright (C) 2007 Red Hat Inc. |
13 | @@ -32,3 +32,6 @@ | |||
14 | 32 | 32 | ||
15 | 33 | Kees Cook <kees.cook@canonical.com>: | 33 | Kees Cook <kees.cook@canonical.com>: |
16 | 34 | Various fixes, additional GDB output, SEGV parser. | 34 | Various fixes, additional GDB output, SEGV parser. |
17 | 35 | |||
18 | 36 | Brian Murray <brian.murray@canonical.com>: | ||
19 | 37 | Various fixes, installation of packages from Launchpad and PPAs. | ||
20 | 35 | 38 | ||
21 | === modified file 'apport/packaging.py' | |||
22 | --- apport/packaging.py 2015-07-02 16:13:23 +0000 | |||
23 | +++ apport/packaging.py 2015-07-13 22:32:23 +0000 | |||
24 | @@ -192,7 +192,8 @@ | |||
25 | 192 | 192 | ||
26 | 193 | def install_packages(self, rootdir, configdir, release, packages, | 193 | def install_packages(self, rootdir, configdir, release, packages, |
27 | 194 | verbose=False, cache_dir=None, | 194 | verbose=False, cache_dir=None, |
29 | 195 | permanent_rootdir=False, architecture=None): | 195 | permanent_rootdir=False, architecture=None, |
30 | 196 | origins=None): | ||
31 | 196 | '''Install packages into a sandbox (for apport-retrace). | 197 | '''Install packages into a sandbox (for apport-retrace). |
32 | 197 | 198 | ||
33 | 198 | In order to work without any special permissions and without touching | 199 | In order to work without any special permissions and without touching |
34 | @@ -221,6 +222,9 @@ | |||
35 | 221 | the given architecture (as specified in a report's "Architecture" | 222 | the given architecture (as specified in a report's "Architecture" |
36 | 222 | field). If not given it defaults to the host system's architecture. | 223 | field). If not given it defaults to the host system's architecture. |
37 | 223 | 224 | ||
38 | 225 | If origins is given, the sandbox will be created with apt data sources | ||
39 | 226 | for foreign origins. | ||
40 | 227 | |||
41 | 224 | Return a string with outdated packages, or None if all packages were | 228 | Return a string with outdated packages, or None if all packages were |
42 | 225 | installed. | 229 | installed. |
43 | 226 | 230 | ||
44 | 227 | 231 | ||
45 | === modified file 'apport/sandboxutils.py' | |||
46 | --- apport/sandboxutils.py 2015-06-11 05:49:39 +0000 | |||
47 | +++ apport/sandboxutils.py 2015-07-13 22:32:23 +0000 | |||
48 | @@ -10,7 +10,7 @@ | |||
49 | 10 | # option) any later version. See http://www.gnu.org/copyleft/gpl.html for | 10 | # option) any later version. See http://www.gnu.org/copyleft/gpl.html for |
50 | 11 | # the full text of the license. | 11 | # the full text of the license. |
51 | 12 | 12 | ||
53 | 13 | import atexit, os, os.path, shutil, tempfile | 13 | import atexit, os, os.path, re, shutil, tempfile |
54 | 14 | import apport | 14 | import apport |
55 | 15 | 15 | ||
56 | 16 | 16 | ||
57 | @@ -103,7 +103,8 @@ | |||
58 | 103 | 103 | ||
59 | 104 | 104 | ||
60 | 105 | def make_sandbox(report, config_dir, cache_dir=None, sandbox_dir=None, | 105 | def make_sandbox(report, config_dir, cache_dir=None, sandbox_dir=None, |
62 | 106 | extra_packages=[], verbose=False, log_timestamps=False): | 106 | extra_packages=[], verbose=False, log_timestamps=False, |
63 | 107 | dynamic_origins=False): | ||
64 | 107 | '''Build a sandbox with the packages that belong to a particular report. | 108 | '''Build a sandbox with the packages that belong to a particular report. |
65 | 108 | 109 | ||
66 | 109 | This downloads and unpacks all packages from the report's Package and | 110 | This downloads and unpacks all packages from the report's Package and |
67 | @@ -140,8 +141,14 @@ | |||
68 | 140 | are not derived from the report. | 141 | are not derived from the report. |
69 | 141 | 142 | ||
70 | 142 | If verbose is True (False by default), this will write some additional | 143 | If verbose is True (False by default), this will write some additional |
73 | 143 | logging to stdout. If log_timestamps is True, these log messages will be | 144 | logging to stdout. |
74 | 144 | prefixed with the current time. | 145 | |
75 | 146 | If log_timestamps is True, these log messages will be prefixed with the | ||
76 | 147 | current time. | ||
77 | 148 | |||
78 | 149 | If dynamic_origins is True (False by default), the sandbox will be built | ||
79 | 150 | with packages from foreign origins that appear in the report's | ||
80 | 151 | Packages:/Dependencies:. | ||
81 | 145 | 152 | ||
82 | 146 | Return a tuple (sandbox_dir, cache_dir, outdated_msg). | 153 | Return a tuple (sandbox_dir, cache_dir, outdated_msg). |
83 | 147 | ''' | 154 | ''' |
84 | @@ -179,12 +186,20 @@ | |||
85 | 179 | if config_dir == 'system': | 186 | if config_dir == 'system': |
86 | 180 | config_dir = None | 187 | config_dir = None |
87 | 181 | 188 | ||
88 | 189 | origins = None | ||
89 | 190 | if dynamic_origins: | ||
90 | 191 | pkg_list = report.get('Package', '') + '\n' + report.get('Dependencies', '') | ||
91 | 192 | m = re.compile('\[origin: ([a-zA-Z0-9][a-zA-Z0-9\+\.\-]+)\]') | ||
92 | 193 | origins = set(m.findall(pkg_list)) | ||
93 | 194 | if origins: | ||
94 | 195 | apport.log("Origins: %s" % origins) | ||
95 | 196 | |||
96 | 182 | # unpack packages, if any, using cache and sandbox | 197 | # unpack packages, if any, using cache and sandbox |
97 | 183 | try: | 198 | try: |
98 | 184 | outdated_msg = apport.packaging.install_packages( | 199 | outdated_msg = apport.packaging.install_packages( |
99 | 185 | sandbox_dir, config_dir, report['DistroRelease'], pkgs, | 200 | sandbox_dir, config_dir, report['DistroRelease'], pkgs, |
100 | 186 | verbose, cache_dir, permanent_rootdir, | 201 | verbose, cache_dir, permanent_rootdir, |
102 | 187 | architecture=report.get('Architecture')) | 202 | architecture=report.get('Architecture'), origins=origins) |
103 | 188 | except SystemError as e: | 203 | except SystemError as e: |
104 | 189 | apport.fatal(str(e)) | 204 | apport.fatal(str(e)) |
105 | 190 | 205 | ||
106 | @@ -210,7 +225,7 @@ | |||
107 | 210 | outdated_msg += apport.packaging.install_packages( | 225 | outdated_msg += apport.packaging.install_packages( |
108 | 211 | sandbox_dir, config_dir, report['DistroRelease'], pkgs, | 226 | sandbox_dir, config_dir, report['DistroRelease'], pkgs, |
109 | 212 | verbose, cache_dir, permanent_rootdir, | 227 | verbose, cache_dir, permanent_rootdir, |
111 | 213 | architecture=report.get('Architecture')) | 228 | architecture=report.get('Architecture'), origins=origins) |
112 | 214 | except SystemError as e: | 229 | except SystemError as e: |
113 | 215 | apport.fatal(str(e)) | 230 | apport.fatal(str(e)) |
114 | 216 | 231 | ||
115 | 217 | 232 | ||
116 | === modified file 'backends/packaging-apt-dpkg.py' | |||
117 | --- backends/packaging-apt-dpkg.py 2015-07-02 16:13:23 +0000 | |||
118 | +++ backends/packaging-apt-dpkg.py 2015-07-13 22:32:23 +0000 | |||
119 | @@ -16,6 +16,8 @@ | |||
120 | 16 | import hashlib | 16 | import hashlib |
121 | 17 | import json | 17 | import json |
122 | 18 | 18 | ||
123 | 19 | from contextlib import closing | ||
124 | 20 | |||
125 | 19 | import warnings | 21 | import warnings |
126 | 20 | warnings.filterwarnings('ignore', 'apt API not stable yet', FutureWarning) | 22 | warnings.filterwarnings('ignore', 'apt API not stable yet', FutureWarning) |
127 | 21 | import apt | 23 | import apt |
128 | @@ -24,9 +26,10 @@ | |||
129 | 24 | from urllib import urlopen, quote, unquote | 26 | from urllib import urlopen, quote, unquote |
130 | 25 | (pickle, urlopen, quote, unquote) # pyflakes | 27 | (pickle, urlopen, quote, unquote) # pyflakes |
131 | 26 | URLError = IOError | 28 | URLError = IOError |
132 | 29 | HTTPError = IOError | ||
133 | 27 | except ImportError: | 30 | except ImportError: |
134 | 28 | # python 3 | 31 | # python 3 |
136 | 29 | from urllib.error import URLError | 32 | from urllib.error import URLError, HTTPError |
137 | 30 | from urllib.request import urlopen | 33 | from urllib.request import urlopen |
138 | 31 | from urllib.parse import quote, unquote | 34 | from urllib.parse import quote, unquote |
139 | 32 | import pickle | 35 | import pickle |
140 | @@ -47,6 +50,7 @@ | |||
141 | 47 | self._virtual_mapping_obj = None | 50 | self._virtual_mapping_obj = None |
142 | 48 | self._launchpad_base = 'https://api.launchpad.net/devel' | 51 | self._launchpad_base = 'https://api.launchpad.net/devel' |
143 | 49 | self._archive_url = self._launchpad_base + '/%s/main_archive' | 52 | self._archive_url = self._launchpad_base + '/%s/main_archive' |
144 | 53 | self._ppa_archive_url = self._launchpad_base + '/~%(user)s/+archive/%(distro)s/%(ppaname)s' | ||
145 | 50 | 54 | ||
146 | 51 | def __del__(self): | 55 | def __del__(self): |
147 | 52 | try: | 56 | try: |
148 | @@ -88,14 +92,15 @@ | |||
149 | 88 | self._apt_cache = apt.Cache(rootdir='/') | 92 | self._apt_cache = apt.Cache(rootdir='/') |
150 | 89 | return self._apt_cache | 93 | return self._apt_cache |
151 | 90 | 94 | ||
153 | 91 | def _sandbox_cache(self, aptroot, apt_sources, fetchProgress): | 95 | def _sandbox_cache(self, aptroot, apt_sources, fetchProgress, distro_name, release_codename, origins): |
154 | 92 | '''Build apt sandbox and return apt.Cache(rootdir=) (initialized lazily). | 96 | '''Build apt sandbox and return apt.Cache(rootdir=) (initialized lazily). |
155 | 93 | 97 | ||
156 | 94 | Clear the package selection on subsequent calls. | 98 | Clear the package selection on subsequent calls. |
157 | 95 | ''' | 99 | ''' |
158 | 96 | self._apt_cache = None | 100 | self._apt_cache = None |
159 | 97 | if not self._sandbox_apt_cache: | 101 | if not self._sandbox_apt_cache: |
161 | 98 | self._build_apt_sandbox(aptroot, apt_sources) | 102 | self._build_apt_sandbox(aptroot, apt_sources, distro_name, |
162 | 103 | release_codename, origins) | ||
163 | 99 | rootdir = os.path.abspath(aptroot) | 104 | rootdir = os.path.abspath(aptroot) |
164 | 100 | self._sandbox_apt_cache = apt.Cache(rootdir=rootdir) | 105 | self._sandbox_apt_cache = apt.Cache(rootdir=rootdir) |
165 | 101 | try: | 106 | try: |
166 | @@ -660,7 +665,8 @@ | |||
167 | 660 | 665 | ||
168 | 661 | def install_packages(self, rootdir, configdir, release, packages, | 666 | def install_packages(self, rootdir, configdir, release, packages, |
169 | 662 | verbose=False, cache_dir=None, | 667 | verbose=False, cache_dir=None, |
171 | 663 | permanent_rootdir=False, architecture=None): | 668 | permanent_rootdir=False, architecture=None, |
172 | 669 | origins=None): | ||
173 | 664 | '''Install packages into a sandbox (for apport-retrace). | 670 | '''Install packages into a sandbox (for apport-retrace). |
174 | 665 | 671 | ||
175 | 666 | In order to work without any special permissions and without touching | 672 | In order to work without any special permissions and without touching |
176 | @@ -689,6 +695,9 @@ | |||
177 | 689 | the given architecture (as specified in a report's "Architecture" | 695 | the given architecture (as specified in a report's "Architecture" |
178 | 690 | field). If not given it defaults to the host system's architecture. | 696 | field). If not given it defaults to the host system's architecture. |
179 | 691 | 697 | ||
180 | 698 | If origins is given, the sandbox will be created with apt data sources | ||
181 | 699 | for foreign origins. | ||
182 | 700 | |||
183 | 692 | Return a string with outdated packages, or None if all packages were | 701 | Return a string with outdated packages, or None if all packages were |
184 | 693 | installed. | 702 | installed. |
185 | 694 | 703 | ||
186 | @@ -747,9 +756,14 @@ | |||
187 | 747 | else: | 756 | else: |
188 | 748 | fetchProgress = apt.progress.base.AcquireProgress() | 757 | fetchProgress = apt.progress.base.AcquireProgress() |
189 | 749 | if not tmp_aptroot: | 758 | if not tmp_aptroot: |
191 | 750 | cache = self._sandbox_cache(aptroot, apt_sources, fetchProgress) | 759 | cache = self._sandbox_cache(aptroot, apt_sources, fetchProgress, |
192 | 760 | self.get_distro_name(), | ||
193 | 761 | self.current_release_codename, | ||
194 | 762 | origins) | ||
195 | 751 | else: | 763 | else: |
197 | 752 | self._build_apt_sandbox(aptroot, apt_sources) | 764 | self._build_apt_sandbox(aptroot, apt_sources, |
198 | 765 | self.get_distro_name(), | ||
199 | 766 | self.current_release_codename, origins) | ||
200 | 753 | cache = apt.Cache(rootdir=os.path.abspath(aptroot)) | 767 | cache = apt.Cache(rootdir=os.path.abspath(aptroot)) |
201 | 754 | try: | 768 | try: |
202 | 755 | cache.update(fetchProgress) | 769 | cache.update(fetchProgress) |
203 | @@ -1203,7 +1217,67 @@ | |||
204 | 1203 | return None | 1217 | return None |
205 | 1204 | 1218 | ||
206 | 1205 | @classmethod | 1219 | @classmethod |
208 | 1206 | def _build_apt_sandbox(klass, apt_root, apt_sources): | 1220 | def create_ppa_source_from_origin(klass, origin, distro, release_codename): |
209 | 1221 | '''For an origin from a Launchpad PPA create sources.list content. | ||
210 | 1222 | |||
211 | 1223 | distro is the distribution for which content is being created e.g. | ||
212 | 1224 | ubuntu. | ||
213 | 1225 | |||
214 | 1226 | release_codename is the codename of the release for which content is | ||
215 | 1227 | being created e.g. trusty. | ||
216 | 1228 | |||
217 | 1229 | Return a string containing content suitable for writing to a sources.list | ||
218 | 1230 | file, or None if the origin is not a Launchpad PPA. | ||
219 | 1231 | ''' | ||
220 | 1232 | |||
221 | 1233 | if origin.startswith("LP-PPA-"): | ||
222 | 1234 | components = origin.split("-")[2:] | ||
223 | 1235 | # If the PPA is unnamed, it will not appear in origin information | ||
224 | 1236 | # but is named ppa in Launchpad. | ||
225 | 1237 | try_ppa = True | ||
226 | 1238 | if len(components) == 1: | ||
227 | 1239 | components.append('ppa') | ||
228 | 1240 | try_ppa = False | ||
229 | 1241 | |||
230 | 1242 | index = 1 | ||
231 | 1243 | while components[index:]: | ||
232 | 1244 | # For an origin we can't tell where the user name ends and the | ||
233 | 1245 | # PPA name starts, so split on each "-" until we find a PPA | ||
234 | 1246 | # that exists. | ||
235 | 1247 | user = str.join('-', components[0:index]) | ||
236 | 1248 | ppa_name = str.join('-', components[index:len(components)]) | ||
237 | 1249 | try: | ||
238 | 1250 | with closing(urlopen(apport.packaging._ppa_archive_url % | ||
239 | 1251 | {'user': user, 'distro': distro, | ||
240 | 1252 | 'ppaname': ppa_name})) as response: | ||
241 | 1253 | response.read() | ||
242 | 1254 | except (URLError, HTTPError): | ||
243 | 1255 | index += 1 | ||
244 | 1256 | if index == len(components): | ||
245 | 1257 | if try_ppa: | ||
246 | 1258 | components.append('ppa') | ||
247 | 1259 | try_ppa = False | ||
248 | 1260 | index = 2 | ||
249 | 1261 | else: | ||
250 | 1262 | user = None | ||
251 | 1263 | continue | ||
252 | 1264 | break | ||
253 | 1265 | if user and ppa_name: | ||
254 | 1266 | ppa_line = 'deb http://ppa.launchpad.net/%s/%s/%s %s main' % \ | ||
255 | 1267 | (user, ppa_name, distro, release_codename) | ||
256 | 1268 | debug_url = 'http://ppa.launchpad.net/%s/%s/%s/dists/%s/main/debug' % \ | ||
257 | 1269 | (user, ppa_name, distro, release_codename) | ||
258 | 1270 | try: | ||
259 | 1271 | with closing(urlopen(debug_url)) as response: | ||
260 | 1272 | response.read() | ||
261 | 1273 | add_debug = ' main/debug' | ||
262 | 1274 | except (URLError, HTTPError): | ||
263 | 1275 | add_debug = '' | ||
264 | 1276 | return ppa_line + add_debug + '\ndeb-src' + ppa_line[3:] + '\n' | ||
265 | 1277 | return None | ||
266 | 1278 | |||
267 | 1279 | @classmethod | ||
268 | 1280 | def _build_apt_sandbox(klass, apt_root, apt_sources, distro_name, release_codename, origins): | ||
269 | 1207 | # pre-create directories, to avoid apt.Cache() printing "creating..." | 1281 | # pre-create directories, to avoid apt.Cache() printing "creating..." |
270 | 1208 | # messages on stdout | 1282 | # messages on stdout |
271 | 1209 | if not os.path.exists(os.path.join(apt_root, 'var', 'lib', 'apt')): | 1283 | if not os.path.exists(os.path.join(apt_root, 'var', 'lib', 'apt')): |
272 | @@ -1225,6 +1299,46 @@ | |||
273 | 1225 | with open(os.path.join(apt_root, 'etc', 'apt', 'sources.list'), 'w') as dest: | 1299 | with open(os.path.join(apt_root, 'etc', 'apt', 'sources.list'), 'w') as dest: |
274 | 1226 | dest.write(src.read()) | 1300 | dest.write(src.read()) |
275 | 1227 | 1301 | ||
276 | 1302 | if origins: | ||
277 | 1303 | source_list_content = '' | ||
278 | 1304 | # map an origin to a Launchpad username and PPA name | ||
279 | 1305 | origin_data = {} | ||
280 | 1306 | for origin in origins: | ||
281 | 1307 | # apport's report format uses unknown for packages w/o an origin | ||
282 | 1308 | if origin == 'unknown': | ||
283 | 1309 | continue | ||
284 | 1310 | origin_path = None | ||
285 | 1311 | if os.path.isdir(apt_sources + '.d'): | ||
286 | 1312 | # check to see if there is a sources.list file for the origin, | ||
287 | 1313 | # if there isn't try using a sources.list file w/o LP-PPA- | ||
288 | 1314 | origin_path = os.path.join(apt_sources + '.d', origin + '.list') | ||
289 | 1315 | if not os.path.exists(origin_path) and 'LP-PPA' in origin: | ||
290 | 1316 | origin_path = os.path.join(apt_sources + '.d', | ||
291 | 1317 | origin.strip('LP-PPA-') + '.list') | ||
292 | 1318 | if not os.path.exists(origin_path): | ||
293 | 1319 | origin_path = None | ||
294 | 1320 | elif not os.path.exists(origin_path): | ||
295 | 1321 | origin_path = None | ||
296 | 1322 | if origin_path: | ||
297 | 1323 | with open(origin_path) as src_ext: | ||
298 | 1324 | source_list_content = src_ext.read() | ||
299 | 1325 | else: | ||
300 | 1326 | source_list_content = klass.create_ppa_source_from_origin(origin, distro_name, release_codename) | ||
301 | 1327 | if source_list_content: | ||
302 | 1328 | with open(os.path.join(apt_root, 'etc', 'apt', | ||
303 | 1329 | 'sources.list.d', origin + '.list'), 'a') as dest: | ||
304 | 1330 | dest.write(source_list_content) | ||
305 | 1331 | for line in source_list_content.splitlines(): | ||
306 | 1332 | if line.startswith('#'): | ||
307 | 1333 | continue | ||
308 | 1334 | if 'ppa.launchpad.net' not in line: | ||
309 | 1335 | continue | ||
310 | 1336 | user = line.split()[1].split('/')[3] | ||
311 | 1337 | ppa = line.split()[1].split('/')[4] | ||
312 | 1338 | origin_data[origin] = (user, ppa) | ||
313 | 1339 | else: | ||
314 | 1340 | apport.warning("Could not find or create source config for %s" % origin) | ||
315 | 1341 | |||
316 | 1228 | # install apt keyrings; prefer the ones from the config dir, fall back | 1342 | # install apt keyrings; prefer the ones from the config dir, fall back |
317 | 1229 | # to system | 1343 | # to system |
318 | 1230 | trusted_gpg = os.path.join(os.path.dirname(apt_sources), 'trusted.gpg') | 1344 | trusted_gpg = os.path.join(os.path.dirname(apt_sources), 'trusted.gpg') |
319 | @@ -1244,6 +1358,36 @@ | |||
320 | 1244 | else: | 1358 | else: |
321 | 1245 | os.makedirs(trusted_d) | 1359 | os.makedirs(trusted_d) |
322 | 1246 | 1360 | ||
323 | 1361 | # install apt keyrings for PPAs | ||
324 | 1362 | if origins and source_list_content: | ||
325 | 1363 | for origin, (ppa_user, ppa_name) in origin_data.items(): | ||
326 | 1364 | ppa_archive_url = apport.packaging._ppa_archive_url % \ | ||
327 | 1365 | {'user': quote(ppa_user), 'distro': distro_name, | ||
328 | 1366 | 'ppaname': quote(ppa_name)} | ||
329 | 1367 | ppa_info = apport.packaging.json_request(ppa_archive_url) | ||
330 | 1368 | if not ppa_info: | ||
331 | 1369 | continue | ||
332 | 1370 | try: | ||
333 | 1371 | signing_key_fingerprint = ppa_info['signing_key_fingerprint'] | ||
334 | 1372 | except IndexError: | ||
335 | 1373 | apport.warning("Error: can't find signing_key_fingerprint at %s" | ||
336 | 1374 | % ppa_archive_url) | ||
337 | 1375 | continue | ||
338 | 1376 | argv = ['gpg', '--no-options', | ||
339 | 1377 | '--no-default-keyring', | ||
340 | 1378 | '--no-auto-check-trustdb', | ||
341 | 1379 | '--keyring', | ||
342 | 1380 | os.path.join(trusted_d, | ||
343 | 1381 | '%s.gpg' % origin), | ||
344 | 1382 | ] | ||
345 | 1383 | argv += ['--quiet', '--batch', | ||
346 | 1384 | '--keyserver', 'hkp://keyserver.ubuntu.com:80/', | ||
347 | 1385 | '--recv', signing_key_fingerprint] | ||
348 | 1386 | if subprocess.call(argv) != 0: | ||
349 | 1387 | apport.warning('Unable to import key for %s' % | ||
350 | 1388 | ppa_archive_url) | ||
351 | 1389 | pass | ||
352 | 1390 | |||
353 | 1247 | @classmethod | 1391 | @classmethod |
354 | 1248 | def _deb_version(klass, pkg): | 1392 | def _deb_version(klass, pkg): |
355 | 1249 | '''Return the version of a .deb file''' | 1393 | '''Return the version of a .deb file''' |
356 | 1250 | 1394 | ||
357 | === modified file 'bin/apport-retrace' | |||
358 | --- bin/apport-retrace 2015-05-29 21:53:42 +0000 | |||
359 | +++ bin/apport-retrace 2015-07-13 22:32:23 +0000 | |||
360 | @@ -53,6 +53,8 @@ | |||
361 | 53 | help=_('Report download/install progress when installing packages into sandbox')) | 53 | help=_('Report download/install progress when installing packages into sandbox')) |
362 | 54 | argparser.add_argument('--timestamps', action='store_true', | 54 | argparser.add_argument('--timestamps', action='store_true', |
363 | 55 | help=_('Prepend timestamps to log messages, for batch operation')) | 55 | help=_('Prepend timestamps to log messages, for batch operation')) |
364 | 56 | argparser.add_argument('--dynamic-origins', action='store_true', | ||
365 | 57 | help=_('Create and use third-party repositories from origins specified in reports')) | ||
366 | 56 | argparser.add_argument('-C', '--cache', metavar='DIR', | 58 | argparser.add_argument('-C', '--cache', metavar='DIR', |
367 | 57 | help=_('Cache directory for packages downloaded in the sandbox')) | 59 | help=_('Cache directory for packages downloaded in the sandbox')) |
368 | 58 | argparser.add_argument('--sandbox-dir', metavar='DIR', | 60 | argparser.add_argument('--sandbox-dir', metavar='DIR', |
369 | @@ -295,7 +297,8 @@ | |||
370 | 295 | if options.sandbox: | 297 | if options.sandbox: |
371 | 296 | sandbox, cache, outdated_msg = apport.sandboxutils.make_sandbox( | 298 | sandbox, cache, outdated_msg = apport.sandboxutils.make_sandbox( |
372 | 297 | report, options.sandbox, options.cache, options.sandbox_dir, | 299 | report, options.sandbox, options.cache, options.sandbox_dir, |
374 | 298 | options.extra_package, options.verbose) | 300 | options.extra_package, options.verbose, log_timestamps, |
375 | 301 | options.dynamic_origins) | ||
376 | 299 | else: | 302 | else: |
377 | 300 | sandbox = None | 303 | sandbox = None |
378 | 301 | outdated_msg = None | 304 | outdated_msg = None |
379 | 302 | 305 | ||
380 | === modified file 'test/test_backend_apt_dpkg.py' | |||
381 | --- test/test_backend_apt_dpkg.py 2015-06-29 13:31:02 +0000 | |||
382 | +++ test/test_backend_apt_dpkg.py 2015-07-13 22:32:23 +0000 | |||
383 | @@ -953,7 +953,8 @@ | |||
384 | 953 | self._setup_foonux_config() | 953 | self._setup_foonux_config() |
385 | 954 | out_dir = os.path.join(self.workdir, 'out') | 954 | out_dir = os.path.join(self.workdir, 'out') |
386 | 955 | os.mkdir(out_dir) | 955 | os.mkdir(out_dir) |
388 | 956 | impl._build_apt_sandbox(self.rootdir, os.path.join(self.configdir, 'Foonux 1.2', 'sources.list')) | 956 | impl._build_apt_sandbox(self.rootdir, os.path.join(self.configdir, 'Foonux 1.2', 'sources.list'), |
389 | 957 | 'ubuntu', 'trusty', origins=None) | ||
390 | 957 | res = impl.get_source_tree('base-files', out_dir, sandbox=self.rootdir, | 958 | res = impl.get_source_tree('base-files', out_dir, sandbox=self.rootdir, |
391 | 958 | apt_update=True) | 959 | apt_update=True) |
392 | 959 | self.assertTrue(os.path.isdir(os.path.join(res, 'debian'))) | 960 | self.assertTrue(os.path.isdir(os.path.join(res, 'debian'))) |
393 | @@ -967,7 +968,8 @@ | |||
394 | 967 | self._setup_foonux_config() | 968 | self._setup_foonux_config() |
395 | 968 | out_dir = os.path.join(self.workdir, 'out') | 969 | out_dir = os.path.join(self.workdir, 'out') |
396 | 969 | os.mkdir(out_dir) | 970 | os.mkdir(out_dir) |
398 | 970 | impl._build_apt_sandbox(self.rootdir, os.path.join(self.configdir, 'Foonux 1.2', 'sources.list')) | 971 | impl._build_apt_sandbox(self.rootdir, os.path.join(self.configdir, 'Foonux 1.2', 'sources.list'), |
399 | 972 | 'ubuntu', 'trusty', origins=None) | ||
400 | 971 | res = impl.get_source_tree('debian-installer', out_dir, version='20101020ubuntu318.16', | 973 | res = impl.get_source_tree('debian-installer', out_dir, version='20101020ubuntu318.16', |
401 | 972 | sandbox=self.rootdir, apt_update=True) | 974 | sandbox=self.rootdir, apt_update=True) |
402 | 973 | self.assertTrue(os.path.isdir(os.path.join(res, 'debian'))) | 975 | self.assertTrue(os.path.isdir(os.path.join(res, 'debian'))) |
403 | @@ -976,8 +978,86 @@ | |||
404 | 976 | self.assertTrue(res.endswith('/debian-installer-20101020ubuntu318.16'), | 978 | self.assertTrue(res.endswith('/debian-installer-20101020ubuntu318.16'), |
405 | 977 | 'unexpected version: ' + res.split('/')[-1]) | 979 | 'unexpected version: ' + res.split('/')[-1]) |
406 | 978 | 980 | ||
409 | 979 | def _setup_foonux_config(self, updates=False, release='trusty'): | 981 | @unittest.skipUnless(_has_internet(), 'online test') |
410 | 980 | '''Set up directories and configuration for install_packages()''' | 982 | def test_create_sources_for_a_named_ppa(self): |
411 | 983 | '''Add sources.list entries for a named PPA.''' | ||
412 | 984 | ppa = 'LP-PPA-daisy-pluckers-daisy-seeds' | ||
413 | 985 | self._setup_foonux_config() | ||
414 | 986 | impl._build_apt_sandbox(self.rootdir, os.path.join(self.configdir, 'Foonux 1.2', 'sources.list'), | ||
415 | 987 | 'ubuntu', 'trusty', origins=[ppa]) | ||
416 | 988 | with open(os.path.join(self.rootdir, 'etc', 'apt', 'sources.list.d', ppa + '.list')) as f: | ||
417 | 989 | sources = f.read().splitlines() | ||
418 | 990 | self.assertIn('deb http://ppa.launchpad.net/daisy-pluckers/daisy-seeds/ubuntu trusty main main/debug', sources) | ||
419 | 991 | self.assertIn('deb-src http://ppa.launchpad.net/daisy-pluckers/daisy-seeds/ubuntu trusty main', sources) | ||
420 | 992 | |||
421 | 993 | d = subprocess.Popen(['gpg', '--no-options', '--no-default-keyring', | ||
422 | 994 | '--no-auto-check-trustdb', '--trust-model', | ||
423 | 995 | 'always', '--batch', '--list-keys', '--keyring', | ||
424 | 996 | os.path.join(self.rootdir, 'etc', 'apt', 'trusted.gpg.d', 'LP-PPA-daisy-pluckers-daisy-seeds.gpg')], | ||
425 | 997 | stdout=subprocess.PIPE) | ||
426 | 998 | apt_keys = d.communicate()[0].decode() | ||
427 | 999 | assert d.returncode == 0 | ||
428 | 1000 | self.assertIn('Launchpad PPA for Daisy Pluckers', apt_keys) | ||
429 | 1001 | |||
430 | 1002 | @unittest.skipUnless(_has_internet(), 'online test') | ||
431 | 1003 | def test_create_sources_for_an_unnamed_ppa(self): | ||
432 | 1004 | '''Add sources.list entries for an unnamed PPA.''' | ||
433 | 1005 | ppa = 'LP-PPA-brian-murray' | ||
434 | 1006 | self._setup_foonux_config() | ||
435 | 1007 | impl._build_apt_sandbox(self.rootdir, os.path.join(self.configdir, 'Foonux 1.2', 'sources.list'), | ||
436 | 1008 | 'ubuntu', 'trusty', origins=[ppa]) | ||
437 | 1009 | with open(os.path.join(self.rootdir, 'etc', 'apt', 'sources.list.d', ppa + '.list')) as f: | ||
438 | 1010 | sources = f.read().splitlines() | ||
439 | 1011 | self.assertIn('deb http://ppa.launchpad.net/brian-murray/ppa/ubuntu trusty main', sources) | ||
440 | 1012 | self.assertIn('deb-src http://ppa.launchpad.net/brian-murray/ppa/ubuntu trusty main', sources) | ||
441 | 1013 | |||
442 | 1014 | d = subprocess.Popen(['gpg', '--no-options', '--no-default-keyring', | ||
443 | 1015 | '--no-auto-check-trustdb', '--trust-model', | ||
444 | 1016 | 'always', '--batch', '--list-keys', '--keyring', | ||
445 | 1017 | os.path.join(self.rootdir, 'etc', 'apt', 'trusted.gpg.d', 'LP-PPA-brian-murray.gpg')], | ||
446 | 1018 | stdout=subprocess.PIPE) | ||
447 | 1019 | apt_keys = d.communicate()[0].decode() | ||
448 | 1020 | assert d.returncode == 0 | ||
449 | 1021 | self.assertIn('Launchpad PPA for Brian Murray', apt_keys) | ||
450 | 1022 | |||
451 | 1023 | def test_use_sources_for_a_ppa(self): | ||
452 | 1024 | '''Use a sources.list.d file for a PPA.''' | ||
453 | 1025 | ppa = 'fooser-bar-ppa' | ||
454 | 1026 | self._setup_foonux_config(ppa=True) | ||
455 | 1027 | impl._build_apt_sandbox(self.rootdir, os.path.join(self.configdir, 'Foonux 1.2', 'sources.list'), | ||
456 | 1028 | 'ubuntu', 'trusty', origins=['LP-PPA-%s' % ppa]) | ||
457 | 1029 | with open(os.path.join(self.rootdir, 'etc', 'apt', 'sources.list.d', ppa + '.list')) as f: | ||
458 | 1030 | sources = f.read().splitlines() | ||
459 | 1031 | self.assertIn('deb http://ppa.launchpad.net/fooser/bar-ppa/ubuntu trusty main main/debug', sources) | ||
460 | 1032 | self.assertIn('deb-src http://ppa.launchpad.net/fooser/bar-ppa/ubuntu trusty main', sources) | ||
461 | 1033 | |||
462 | 1034 | @unittest.skipUnless(_has_internet(), 'online test') | ||
463 | 1035 | def test_install_package_from_a_ppa(self): | ||
464 | 1036 | '''Install a package from a PPA.''' | ||
465 | 1037 | ppa = 'LP-PPA-brian-murray' | ||
466 | 1038 | self._setup_foonux_config() | ||
467 | 1039 | obsolete = impl.install_packages(self.rootdir, self.configdir, 'Foonux 1.2', | ||
468 | 1040 | [('apport', | ||
469 | 1041 | '2.14.1-0ubuntu3.7~ppa4') | ||
470 | 1042 | ], False, self.cachedir, origins=[ppa]) | ||
471 | 1043 | |||
472 | 1044 | self.assertEqual(obsolete, '') | ||
473 | 1045 | |||
474 | 1046 | def sandbox_ver(pkg): | ||
475 | 1047 | with gzip.open(os.path.join(self.rootdir, 'usr/share/doc', pkg, | ||
476 | 1048 | 'changelog.Debian.gz')) as f: | ||
477 | 1049 | return f.readline().decode().split()[1][1:-1] | ||
478 | 1050 | |||
479 | 1051 | self.assertEqual(sandbox_ver('apport'), | ||
480 | 1052 | '2.14.1-0ubuntu3.7~ppa4') | ||
481 | 1053 | |||
482 | 1054 | def _setup_foonux_config(self, updates=False, release='trusty', ppa=False): | ||
483 | 1055 | '''Set up directories and configuration for install_packages() | ||
484 | 1056 | |||
485 | 1057 | If ppa is True, then a sources.list file for a PPA will be created | ||
486 | 1058 | in sources.list.d used to test copying of a sources.list file to a | ||
487 | 1059 | sandbox. | ||
488 | 1060 | ''' | ||
489 | 981 | 1061 | ||
490 | 982 | self.cachedir = os.path.join(self.workdir, 'cache') | 1062 | self.cachedir = os.path.join(self.workdir, 'cache') |
491 | 983 | self.rootdir = os.path.join(self.workdir, 'root') | 1063 | self.rootdir = os.path.join(self.workdir, 'root') |
492 | @@ -994,6 +1074,11 @@ | |||
493 | 994 | f.write('deb http://archive.ubuntu.com/ubuntu/ %s-updates main\n' % release) | 1074 | f.write('deb http://archive.ubuntu.com/ubuntu/ %s-updates main\n' % release) |
494 | 995 | f.write('deb-src http://archive.ubuntu.com/ubuntu/ %s-updates main\n' % release) | 1075 | f.write('deb-src http://archive.ubuntu.com/ubuntu/ %s-updates main\n' % release) |
495 | 996 | f.write('deb http://ddebs.ubuntu.com/ %s-updates main\n' % release) | 1076 | f.write('deb http://ddebs.ubuntu.com/ %s-updates main\n' % release) |
496 | 1077 | if ppa: | ||
497 | 1078 | os.mkdir(os.path.join(self.configdir, 'Foonux 1.2', 'sources.list.d')) | ||
498 | 1079 | with open(os.path.join(self.configdir, 'Foonux 1.2', 'sources.list.d', 'fooser-bar-ppa.list'), 'w') as f: | ||
499 | 1080 | f.write('deb http://ppa.launchpad.net/fooser/bar-ppa/ubuntu %s main main/debug\n' % release) | ||
500 | 1081 | f.write('deb-src http://ppa.launchpad.net/fooser/bar-ppa/ubuntu %s main\n' % release) | ||
501 | 997 | os.mkdir(os.path.join(self.configdir, 'Foonux 1.2', 'armhf')) | 1082 | os.mkdir(os.path.join(self.configdir, 'Foonux 1.2', 'armhf')) |
502 | 998 | with open(os.path.join(self.configdir, 'Foonux 1.2', 'armhf', 'sources.list'), 'w') as f: | 1083 | with open(os.path.join(self.configdir, 'Foonux 1.2', 'armhf', 'sources.list'), 'w') as f: |
503 | 999 | f.write('deb http://ports.ubuntu.com/ %s main\n' % release) | 1084 | f.write('deb http://ports.ubuntu.com/ %s main\n' % release) |
Thanks Brian! The general idea looks fine; with the restricted "Origin:" fields that we have there's no way around the guesswork.