Merge lp:~rcj/vmbuilder/mfdiff_fixups into lp:~ubuntu-on-ec2/vmbuilder/mfdiff
- mfdiff_fixups
- Merge into mfdiff
Status: | Merged |
---|---|
Merged at revision: | 19 |
Proposed branch: | lp:~rcj/vmbuilder/mfdiff_fixups |
Merge into: | lp:~ubuntu-on-ec2/vmbuilder/mfdiff |
Diff against target: |
903 lines (+581/-281) 2 files modified
README (+9/-10) mfdiff (+572/-271) |
To merge this branch: | bzr merge lp:~rcj/vmbuilder/mfdiff_fixups |
Related bugs: |
Reviewer | Review Type | Date Requested | Status |
---|---|---|---|
Francis Ginther (community) | Approve | ||
Dan Watkins (community) | Approve | ||
Review via email: mp+326375@code.launchpad.net |
Commit message
Description of the change
Modernization and fixes
Break up main(), modernize code, improve documentation and variable
naming, python3 compat, misc fixes, urllib to requests migration,
support non-x86 architectures, and add logging.
This requires python-six version newer than what is shipping in some
supported Ubuntu releases to use six.viewkeys. (>=1.10 is preferable)
[ Robert C Jennings ]
* Strip architecture from binary when searching cache
* Copy rather than rename changelog files
* Force output to UTF-8 at all times
* Output utf-8 rather than ascii
* Change exception rendering for python3 compatibility
* Use the current directory for temp changelog files
* Improve variable naming for changelog_url (was clog_url)
* Support non-x86 architectures
* Fix usage text to correct option order
* Eliminate traceback if binary package has been renamed
(no longer in apt cache)
* Use a real temporary file
* Fixups for code-review
* Improve names in print_changelogs and store more to the sources dict
* Break out changelog processing from main
* More variable renames
* Rearrange summary printout and changelog printing for later refactor
* Name, spelling, and doc cleanup
* Stop processing source package on exception
* Unlink temp file only if it exists
* Add more complete function docstrings
* Additional refactoring to reduce size of main
* isort requests
* Update authors/copyright
* Add logging
* Simplify cmdline --verbose option
* Swap urllib for requests
* Improve variable naming
* Consistent sorted output
* python3 compatibility changes
* Document functions, use 'with' for files, and minor fixes
* [merge] PEP8 cleanup, example updates, & python-apt updates
* Correct vi line for tab expansion
* README: Remove trailing spaces
* Use host apt-secure keychains
* APT version comparison moved from apt to apt_pkg
* Remove unused cmdline option
* Update README to use modern examples
* Move comment to proper function
* remove unused imports/variables
* sort imports
* pep8: autopep8 changes
* pep8: tab -> space conversion
Robert C Jennings (rcj) wrote : | # |
Robert C Jennings (rcj) wrote : | # |
This requires python-six version newer than what is shipping in Trusty to use viewkeys. (1.10 is preferable)
Dan Watkins (oddbloke) : | # |
Francis Ginther (fginther) wrote : | # |
Approve, this MP and resulting mfdiff are identical to a branch I already reviewed and tested.
Preview Diff
1 | === modified file 'README' | |||
2 | --- README 2010-12-13 15:23:43 +0000 | |||
3 | +++ README 2017-06-27 19:03:39 +0000 | |||
4 | @@ -1,15 +1,14 @@ | |||
5 | 1 | "Manifest Diff" (mfdiff) is a tool for determining what changed between | 1 | "Manifest Diff" (mfdiff) is a tool for determining what changed between |
7 | 2 | 2 builds by looking at their manifests. A manifest is simply a | 2 | 2 builds by looking at their manifests. A manifest is simply a |
8 | 3 | list of package and versions in a build, obtainable by: | 3 | list of package and versions in a build, obtainable by: |
9 | 4 | dpkg-query', '-W', '--showformat=${Package} ${Version}\n' | 4 | dpkg-query', '-W', '--showformat=${Package} ${Version}\n' |
10 | 5 | 5 | ||
11 | 6 | Example of running: | 6 | Example of running: |
21 | 7 | mf1=mfs/lucid/release-20100827/ubuntu-10.04-server-uec-amd64.manifest | 7 | mf1=mfs/releases/trusty/release-20160314/ubuntu-14.04-server-cloudimg-amd64.manifest |
22 | 8 | mf2=mfs/lucid/release-20100923/ubuntu-10.04-server-uec-amd64.manifest | 8 | mf2=mfs/releases/trusty/release-20170517/ubuntu-14.04-server-cloudimg-amd64.manifest |
23 | 9 | ./mfdiff i386 lucid "${mf1}" "${mf2}" | 9 | ./mfdiff amd64 trusty "${mf1}" "${mf2}" |
24 | 10 | 10 | ||
25 | 11 | 11 | ||
26 | 12 | 12 | sync manifests from cloud-images.ubuntu.com with: | |
27 | 13 | sync manifests from uec-images.ubuntu.com with: | 13 | rsync -av cloud-images.ubuntu.com::cloud-images/ --prune-empty-dirs mfs \ |
28 | 14 | rsync -av uec-images.ubuntu.com::uec-images/ --prune-empty-dirs mfs \ | 14 | --filter "+ */" --filter "+ *.manifest" --filter "- *" |
20 | 15 | --filter "+ */" --filter "+ *.manifest" --filter "- *" | ||
29 | 16 | 15 | ||
30 | === modified file 'mfdiff' | |||
31 | --- mfdiff 2014-10-03 11:27:32 +0000 | |||
32 | +++ mfdiff 2017-06-27 19:03:39 +0000 | |||
33 | @@ -1,9 +1,17 @@ | |||
34 | 1 | #!/usr/bin/env python | 1 | #!/usr/bin/env python |
36 | 2 | # vi:ts=4 noexpandtab | 2 | """ |
37 | 3 | Given two manifest files for a particular release/arch, find the | ||
38 | 4 | packages which have been added, removed, and changed. Print the | ||
39 | 5 | changelog entries for the changed packages limited to changes between | ||
40 | 6 | the two versions. | ||
41 | 7 | """ | ||
42 | 8 | |||
43 | 9 | # vi:ts=4 expandtab | ||
44 | 3 | # | 10 | # |
46 | 4 | # Copyright (C) 2010 Canonical Ltd. | 11 | # Copyright (C) 2010, 2017 Canonical Ltd. |
47 | 5 | # | 12 | # |
48 | 6 | # Authors: Scott Moser <scott.moser@canonical.com> | 13 | # Authors: Scott Moser <scott.moser@canonical.com> |
49 | 14 | # Robert C Jennings <robert.jennings@canonical.com> | ||
50 | 7 | # | 15 | # |
51 | 8 | # This program is free software: you can redistribute it and/or modify | 16 | # This program is free software: you can redistribute it and/or modify |
52 | 9 | # it under the terms of the GNU General Public License as published by | 17 | # it under the terms of the GNU General Public License as published by |
53 | @@ -17,288 +25,581 @@ | |||
54 | 17 | # You should have received a copy of the GNU General Public License | 25 | # You should have received a copy of the GNU General Public License |
55 | 18 | # along with this program. If not, see <http://www.gnu.org/licenses/>. | 26 | # along with this program. If not, see <http://www.gnu.org/licenses/>. |
56 | 19 | 27 | ||
57 | 28 | import codecs | ||
58 | 29 | import locale | ||
59 | 30 | import logging | ||
60 | 20 | import os | 31 | import os |
62 | 21 | import string | 32 | import os.path |
63 | 33 | import re | ||
64 | 34 | import shutil | ||
65 | 22 | import sys | 35 | import sys |
66 | 36 | import tempfile | ||
67 | 23 | from optparse import OptionParser | 37 | from optparse import OptionParser |
73 | 24 | import urllib | 38 | |
69 | 25 | import logging | ||
70 | 26 | import os.path | ||
71 | 27 | import os | ||
72 | 28 | from debian.changelog import Changelog, ChangeBlock | ||
74 | 29 | import apt | 39 | import apt |
82 | 30 | import re | 40 | import requests |
83 | 31 | 41 | from debian.changelog import Changelog | |
84 | 32 | Usage=""" | 42 | from six import iteritems, viewkeys |
85 | 33 | Usage: %s --apt-sources F arch manifest1 manifest2 | 43 | |
86 | 34 | Compare two manifest files, and show changelog differences. | 44 | try: |
87 | 35 | """ | 45 | from apt import VersionCompare as version_compare |
88 | 36 | logformat="%(asctime)s %(name)s/%(levelname)s: %(message)s" | 46 | except ImportError: |
89 | 47 | from apt_pkg import version_compare as version_compare | ||
90 | 48 | |||
91 | 37 | 49 | ||
92 | 38 | class MissingChangelogError(Exception): | 50 | class MissingChangelogError(Exception): |
94 | 39 | pass | 51 | "The Changelog file could not be found on the server" |
95 | 52 | pass | ||
96 | 53 | |||
97 | 40 | 54 | ||
98 | 41 | class ChangelogMissingVersion(Exception): | 55 | class ChangelogMissingVersion(Exception): |
100 | 42 | pass | 56 | "The Changelog is missing starting and/or ending versions for the search" |
101 | 57 | pass | ||
102 | 58 | |||
103 | 43 | 59 | ||
104 | 44 | class UnknownSourceVersionError(Exception): | 60 | class UnknownSourceVersionError(Exception): |
200 | 45 | pass | 61 | "The binary package did not have a source version listed" |
201 | 46 | 62 | pass | |
202 | 47 | def get_bin2src(changed,cache): | 63 | |
203 | 48 | ret = { } | 64 | |
204 | 49 | for binpkg in changed: | 65 | def get_bin2src(packages, cache): |
205 | 50 | pkg_name = binpkg.split(':')[0] | 66 | """ |
206 | 51 | ret[binpkg]=cache[pkg_name].versions[0].source_name | 67 | Find the source package names for a given binary package list |
207 | 52 | return(ret) | 68 | |
208 | 53 | 69 | :param list packages: List of binary packages | |
209 | 54 | # downloads changelog for src/ver and returns path to that | 70 | :param :class:`apt.Cache` cache: Open/updated apt cache |
210 | 55 | def getchangelog(src,ver,cache_d): | 71 | :return: List mapping binary package name to source package name |
211 | 56 | # http://changelogs.ubuntu.com/changelogs/pool/main/h/hal/hal_0.5.5.1-1ubuntu2/changelog | 72 | :rtype: dict |
212 | 57 | clog_burl="http://changelogs.ubuntu.com/changelogs/pool" | 73 | """ |
213 | 58 | urls=[] | 74 | |
214 | 59 | bsize=2048 | 75 | ret = {} |
215 | 60 | cache_f = "%s/changelog.%s_%s" % (cache_d, src,ver) | 76 | logging.debug('Finding source package names for all binary packages') |
216 | 61 | 77 | for binpkg in packages: | |
217 | 62 | if os.path.isfile(cache_f): | 78 | pkg_name = binpkg.split(':')[0] |
218 | 63 | return(cache_f) | 79 | ret[binpkg] = cache[pkg_name].versions[0].source_name |
219 | 64 | 80 | return ret | |
220 | 65 | furls=[ ] | 81 | |
221 | 66 | num_colon_m=re.compile("[0-9]:") | 82 | |
222 | 67 | 83 | def get_changelog(source, version, changelog_cache_d): | |
223 | 68 | cache_tmp = "%s/.changelog.%s_%s" % (cache_d, src,ver) | 84 | """ |
224 | 69 | for pile in ("main", "universe", "multiverse", "restricted"): | 85 | Download changelog for source / version and returns path to that |
225 | 70 | pre=src[0:1] | 86 | |
226 | 71 | # packages starting with 'lib' are special | 87 | :param str source: Source package name |
227 | 72 | if src.startswith("lib"): pre=src[0:4] | 88 | :param str version: Source package version |
228 | 73 | 89 | :param str changelog_cache_d: path to store cached changelogs | |
229 | 74 | # packages with '1:' versions have different paths | 90 | :raises MissingChangelogError: If changelog file could not be downloaded |
230 | 75 | # update-manager at version 1:0.134.11. | 91 | :return: changelog file for source package & version |
231 | 76 | # would be at main/u/update-manager/update-manager_1:0.134.11 | 92 | :rtype: str |
232 | 77 | # instead at main/u/update-manager/update-manager_0.134.11/ | 93 | """ |
233 | 78 | xver=ver | 94 | |
234 | 79 | if num_colon_m.match(ver): | 95 | cache_f = "%s/changelog.%s_%s" % (changelog_cache_d, source, version) |
235 | 80 | xver=ver[2:] | 96 | |
236 | 81 | 97 | if os.path.isfile(cache_f): | |
237 | 82 | clog_url = "%s/%s/%s/%s/%s_%s/changelog" % \ | 98 | logging.debug("Using cached changelog for %s:%s", source, version) |
238 | 83 | ( clog_burl, pile, pre, src, src, xver) | 99 | return cache_f |
239 | 84 | 100 | ||
240 | 85 | fp = open(cache_tmp,"w") | 101 | furls = [] |
241 | 86 | u_fp = urllib.urlopen(clog_url) | 102 | num_colon_m = re.compile("[0-9]:") |
242 | 87 | while True: | 103 | |
243 | 88 | buf = u_fp.read(bsize) | 104 | cache_tmp_fd, cache_tmp_path = tempfile.mkstemp( |
244 | 89 | fp.write(buf) | 105 | ".changelog.%s_%s" % (source, version), |
245 | 90 | if len(buf) != bsize: break | 106 | prefix="." + tempfile.gettempprefix(), dir=".") |
246 | 91 | fp.close() | 107 | cache_tmp = os.fdopen(cache_tmp_fd, "w") |
247 | 92 | if u_fp.getcode() == 200: | 108 | for pile in ("main", "universe", "multiverse", "restricted"): |
248 | 93 | os.rename(cache_tmp,cache_f) | 109 | pre = source[0:1] |
249 | 94 | return(cache_f) | 110 | # packages starting with 'lib' are special |
250 | 95 | print "missing %s: %s" % (src,clog_url) | 111 | if source.startswith("lib"): |
251 | 96 | furls.append(clog_url) | 112 | pre = source[0:4] |
252 | 97 | os.unlink(cache_tmp) | 113 | |
253 | 98 | raise(MissingChangelogError("Failed to find changelog for %s at version %s.\n tried %s" % (src, ver, ' '.join(furls)))) | 114 | # packages with '1:' versions have different paths |
254 | 99 | 115 | # update-manager at version 1:0.134.11. | |
255 | 100 | def hashmffile(filename): | 116 | # is main/u/update-manager/update-manager_0.134.11/ |
256 | 101 | fp=open(filename,"r") | 117 | # rather than main/u/update-manager/update-manager_1:0.134.11 |
257 | 102 | lines=fp.readlines() | 118 | url_version = version |
258 | 103 | fp.close() | 119 | if num_colon_m.match(version): |
259 | 104 | ret={ } | 120 | url_version = version[2:] |
260 | 105 | for line in lines: | 121 | |
261 | 106 | (pkg,ver)=line.split() | 122 | # Changelog URL example http://changelogs.ubuntu.com/changelogs/\ |
262 | 107 | ret[pkg]=ver | 123 | # pool/main/h/hal/hal_0.5.5.1-1ubuntu2/changelog |
263 | 108 | return(ret) | 124 | changelog_url = "http://changelogs.ubuntu.com/changelogs/pool/" \ |
264 | 109 | 125 | "%s/%s/%s/%s_%s/changelog" % \ | |
265 | 110 | def filecontents(filename): | 126 | (pile, pre, source, source, url_version) |
266 | 111 | fp=open(filename,"r") | 127 | |
267 | 112 | contents=fp.read() | 128 | changelog = requests.get(changelog_url) |
268 | 113 | fp.close() | 129 | if changelog.status_code == 200: |
269 | 114 | return(contents) | 130 | cache_tmp.write(changelog.content) |
270 | 115 | 131 | cache_tmp.close() | |
271 | 116 | def prep_cacheroot(cache_d, release): | 132 | shutil.copy2(cache_tmp_path, cache_f) |
272 | 117 | mirror = "http://archive.ubuntu.com/ubuntu/" | 133 | os.unlink(cache_tmp_path) |
273 | 118 | pockets = ( release, "%s-updates" % release, "%s-security" % release, | 134 | return cache_f |
274 | 119 | "%s-proposed" % release, ) | 135 | else: |
275 | 120 | components = ( "main", "universe" ) | 136 | logging.error("missing %s: %s", source, changelog_url) |
276 | 121 | srclines = [] | 137 | furls.append(changelog_url) |
277 | 122 | for pocket in pockets: | 138 | if os.path.exists(cache_tmp_path): |
278 | 123 | srclines.append("deb %s %s %s" % (mirror, pocket, ' '.join(components))) | 139 | os.unlink(cache_tmp_path) |
279 | 124 | try: | 140 | raise MissingChangelogError("Failed to find changelog for %s at version " |
280 | 125 | os.makedirs( "%s/etc/apt" % cache_d ) | 141 | "%s.\n tried %s" % (source, version, |
281 | 126 | except: | 142 | ' '.join(furls))) |
282 | 127 | pass | 143 | |
283 | 128 | asl = open( "%s/etc/apt/sources.list" % cache_d, "w" ) | 144 | |
284 | 129 | asl.write('\n'.join(srclines)) | 145 | def manifest_to_dict(filename): |
285 | 130 | asl.close() | 146 | """ |
286 | 131 | 147 | Parse manifest file to create a package / version mapping | |
287 | 132 | def get_cache(cache_d, cache=None): | 148 | |
288 | 133 | if cache: return cache | 149 | :param str filename: Name of package manifest file |
289 | 134 | cache = apt.Cache(rootdir=cache_d) | 150 | :return: List of package versions by name |
290 | 135 | cache.update() | 151 | :rtype: dict |
291 | 136 | cache.open() | 152 | """ |
292 | 137 | return cache | 153 | |
293 | 138 | 154 | ret = {} | |
294 | 139 | # render a changelog block to something printable | 155 | logging.debug('Reading package manifest from %s', filename) |
295 | 156 | with open(filename, "r") as manifest: | ||
296 | 157 | for line in manifest: | ||
297 | 158 | (pkg, ver) = line.split() | ||
298 | 159 | ret[pkg] = ver | ||
299 | 160 | return ret | ||
300 | 161 | |||
301 | 162 | |||
302 | 163 | def open_apt_cache(arch, release, cache_d=None): | ||
303 | 164 | """ | ||
304 | 165 | Create, update, and open an apt cache. | ||
305 | 166 | |||
306 | 167 | This creates an apt cache directory and write a sources.list file | ||
307 | 168 | before updating and opening the cache. The caller is responsible | ||
308 | 169 | for closing the cache. | ||
309 | 170 | |||
310 | 171 | :param str arch: Package architecture | ||
311 | 172 | :param str release: Ubuntu release name (e.g. Xenial) | ||
312 | 173 | :param str cache_d: apt cache path | ||
313 | 174 | :returns: tuple of Open/updated apt cache and cache path name | ||
314 | 175 | :rtype: tuple(:class:`apt.Cache`, str) | ||
315 | 176 | """ | ||
316 | 177 | |||
317 | 178 | if not cache_d: | ||
318 | 179 | cache_d = "./cache.%s-%s" % (release, arch) | ||
319 | 180 | logging.info("Using %s as the apt cache directory", cache_d) | ||
320 | 181 | |||
321 | 182 | if arch in ['amd64', 'i386']: | ||
322 | 183 | mirror = "http://archive.ubuntu.com/ubuntu/" | ||
323 | 184 | else: | ||
324 | 185 | mirror = "http://ports.ubuntu.com/ubuntu-ports/" | ||
325 | 186 | logging.debug('Configuring apt cache using mirror %s', mirror) | ||
326 | 187 | |||
327 | 188 | pockets = (release, "%s-updates" % release, "%s-security" % release, | ||
328 | 189 | "%s-proposed" % release, ) | ||
329 | 190 | components = ("main", "universe") | ||
330 | 191 | srclines = [] | ||
331 | 192 | for pocket in pockets: | ||
332 | 193 | srcline = "deb %s %s %s" % (mirror, pocket, ' '.join(components)) | ||
333 | 194 | logging.debug('Adding source: %s', srcline) | ||
334 | 195 | srclines.append(srcline) | ||
335 | 196 | try: | ||
336 | 197 | os.makedirs("%s/etc/apt" % cache_d) | ||
337 | 198 | except OSError as oserror: | ||
338 | 199 | if os.errno.EEXIST != oserror.errno: | ||
339 | 200 | raise | ||
340 | 201 | with open("%s/etc/apt/sources.list" % cache_d, "w") as asl: | ||
341 | 202 | asl.write('\n'.join(srclines)) | ||
342 | 203 | |||
343 | 204 | apt.apt_pkg.config.set("Apt::Architecture", arch) | ||
344 | 205 | |||
345 | 206 | logging.debug('Using host apt keys for signature verification') | ||
346 | 207 | apt.apt_pkg.config.set("Dir::Etc::Trusted", "/etc/apt/trusted.gpg") | ||
347 | 208 | apt.apt_pkg.config.set("Dir::Etc::TrustedParts", | ||
348 | 209 | "/etc/apt/trusted.gpg.d/") | ||
349 | 210 | |||
350 | 211 | cache = apt.Cache(rootdir=cache_d) | ||
351 | 212 | logging.info('Updating apt cache') | ||
352 | 213 | cache.update() | ||
353 | 214 | logging.info('Update of apt cache complete') | ||
354 | 215 | cache.open() | ||
355 | 216 | return cache, cache_d | ||
356 | 217 | |||
357 | 218 | |||
358 | 140 | def render_block(block): | 219 | def render_block(block): |
365 | 141 | return('\n'.join([x.encode('ascii', 'replace') for x in block.changes() if x])) | 220 | """ |
366 | 142 | 221 | Render a changelog block to something printable (dropping blank lines) | |
367 | 143 | def print_blocks(blist): | 222 | |
368 | 144 | # print a blocklist | 223 | :param :class:`debian.changelog.ChangeBlock` block: Changelog block |
369 | 145 | for block in blist: | 224 | :return: String containing the changelog block text |
370 | 146 | print render_block(block) | 225 | :rtype: str |
371 | 226 | """ | ||
372 | 227 | return '\n'.join([x for x in block.changes() if x]) | ||
373 | 228 | |||
374 | 229 | |||
375 | 230 | def print_blocks(block_list): | ||
376 | 231 | """ | ||
377 | 232 | Print a Changelog block list | ||
378 | 233 | |||
379 | 234 | :param list block_list: List of :class:`debian.changelog.ChangeBlock` | ||
380 | 235 | """ | ||
381 | 236 | |||
382 | 237 | for block in block_list: | ||
383 | 238 | print(render_block(block).encode('utf-8').decode('utf-8')) | ||
384 | 239 | |||
385 | 240 | |||
386 | 241 | def kernel_fixups(manifest_from, manifest_to): | ||
387 | 242 | """ | ||
388 | 243 | Fix up kernels so the pkg names match | ||
389 | 244 | |||
390 | 245 | Kernel package names change from release to release so that they are | ||
391 | 246 | co-installable, but we need to find matching package names in the | ||
392 | 247 | two manifests to provide a list of changes between the versions of the | ||
393 | 248 | package in each manifest. | ||
394 | 249 | This function will return an altered version of manifest_from with kernel | ||
395 | 250 | package names changed to match kernel package names in manifest_to. This | ||
396 | 251 | will support later version comparisons. | ||
397 | 252 | |||
398 | 253 | :param dict manifest_from: Dictionary mapping package to version for of | ||
399 | 254 | the starting manifest | ||
400 | 255 | :param dict manifest_to: Dictionary mapping package to version for the | ||
401 | 256 | ending manifest | ||
402 | 257 | :return: Starting manifest dictionary with altered kernel package names | ||
403 | 258 | to match names in ending manifest | ||
404 | 259 | :rtype dict: | ||
405 | 260 | """ | ||
406 | 261 | kfixups = {} | ||
407 | 262 | kmatch = re.compile("linux-image-[0-9]") | ||
408 | 263 | for pkg in manifest_to: | ||
409 | 264 | # if this is a linux-image-* binary package do some hacks to make it | ||
410 | 265 | # look like manifest_from is the same (format like | ||
411 | 266 | # linux-image-2.6.32-32-virtual) | ||
412 | 267 | if kmatch.match(pkg): | ||
413 | 268 | logging.debug('Found kernel %s in manifest #2', pkg) | ||
414 | 269 | img_type = pkg.split("-")[-1] | ||
415 | 270 | if pkg in manifest_from: | ||
416 | 271 | logging.debug('Found same kernel in manifest #1') | ||
417 | 272 | continue | ||
418 | 273 | for fpkg in manifest_from: | ||
419 | 274 | if kmatch.match(fpkg) and fpkg.endswith("-%s" % img_type): | ||
420 | 275 | logging.debug('Found similar kernel %s in manifest #1', | ||
421 | 276 | fpkg) | ||
422 | 277 | kfixups[pkg] = fpkg | ||
423 | 278 | |||
424 | 279 | for pkg_to, pkg_from in iteritems(kfixups): | ||
425 | 280 | logging.debug('Substituting kernel %s for %s in manifest #1 to ' | ||
426 | 281 | 'enable version comparison', pkg_to, pkg_from) | ||
427 | 282 | manifest_from[pkg_to] = manifest_from[pkg_from] | ||
428 | 283 | del manifest_from[pkg_from] | ||
429 | 284 | return manifest_from | ||
430 | 285 | |||
431 | 286 | |||
432 | 287 | def find_added(manifest_from, manifest_to): | ||
433 | 288 | "Find new packages in manifest_to" | ||
434 | 289 | new = {} | ||
435 | 290 | for pkg in sorted(viewkeys(manifest_to) - viewkeys(manifest_from)): | ||
436 | 291 | logging.debug('New package: %s', pkg) | ||
437 | 292 | new[pkg] = manifest_to[pkg] | ||
438 | 293 | return new | ||
439 | 294 | |||
440 | 295 | |||
441 | 296 | def find_removed(manifest_from, manifest_to): | ||
442 | 297 | "Find packages removed from manifest_from" | ||
443 | 298 | removed = {} | ||
444 | 299 | for pkg in sorted(viewkeys(manifest_from) - viewkeys(manifest_to)): | ||
445 | 300 | logging.debug('Removed package: %s', pkg) | ||
446 | 301 | removed[pkg] = manifest_from[pkg] | ||
447 | 302 | return removed | ||
448 | 303 | |||
449 | 304 | |||
450 | 305 | def find_changed(manifest_from, manifest_to): | ||
451 | 306 | "Find modified packages" | ||
452 | 307 | changed = [] | ||
453 | 308 | for pkg in sorted(viewkeys(manifest_from) & viewkeys(manifest_to)): | ||
454 | 309 | if manifest_from[pkg] != manifest_to[pkg]: | ||
455 | 310 | logging.debug('Changed package: %s', pkg) | ||
456 | 311 | changed.append(pkg) | ||
457 | 312 | return changed | ||
458 | 313 | |||
459 | 314 | |||
460 | 315 | def map_source_to_binary(cache, packages): | ||
461 | 316 | "Create a dictionary of source to list of binary packages" | ||
462 | 317 | src2bins = {} | ||
463 | 318 | for bin_pkg in packages: | ||
464 | 319 | bin_name = bin_pkg.split(':')[0] | ||
465 | 320 | if bin_name in cache: | ||
466 | 321 | src2bins.setdefault( | ||
467 | 322 | cache[bin_name].versions[0].source_name, []).append(bin_pkg) | ||
468 | 323 | return src2bins | ||
469 | 324 | |||
470 | 325 | |||
471 | 326 | def get_pkg_versions(cache, binary): | ||
472 | 327 | "Get all known versions from the apt cache" | ||
473 | 328 | pkg_name = binary.split(':')[0] | ||
474 | 329 | try: | ||
475 | 330 | return cache[pkg_name].versions | ||
476 | 331 | except KeyError: | ||
477 | 332 | raise Exception( | ||
478 | 333 | "%s not in cache or did not have version info in cache" % | ||
479 | 334 | pkg_name) | ||
480 | 335 | |||
481 | 336 | |||
482 | 337 | def source_version_for_binary(cache, binary, binary_ver): | ||
483 | 338 | "Find the source version data for a specific binary version" | ||
484 | 339 | versions = get_pkg_versions(cache, binary) | ||
485 | 340 | try: | ||
486 | 341 | return versions[binary_ver].source_version | ||
487 | 342 | except KeyError: | ||
488 | 343 | # Strip the architecture name from the binary | ||
489 | 344 | source_name = cache[binary.split(':')[0]].versions[0].source_name | ||
490 | 345 | msg = ("Unable to determine source version for %s. " | ||
491 | 346 | "Binary package %s/%s not in known source version " | ||
492 | 347 | "list (%s)" % (source_name, binary, binary_ver, versions)) | ||
493 | 348 | raise UnknownSourceVersionError(msg) | ||
494 | 349 | |||
495 | 350 | |||
496 | 351 | def filter_changelog(changelog_path, version_low, version_high): | ||
497 | 352 | """ | ||
498 | 353 | Extract changelog entries within a version range | ||
499 | 354 | |||
500 | 355 | The range of changelog entries returned will include all entries | ||
501 | 356 | after version_low up to, and including, version_high. | ||
502 | 357 | If either the starting or ending version are not found in the | ||
503 | 358 | list of changelog entries the result will be incomplete and | ||
504 | 359 | a non-empty error message is returned to indicate the issue. | ||
505 | 360 | |||
506 | 361 | :param str changelog_path: File name of the changelog to process | ||
507 | 362 | :return: list of changelog blocks and an error_msg if incomplete | ||
508 | 363 | :rtype tuple(list, str): | ||
509 | 364 | """ | ||
510 | 365 | |||
511 | 366 | with open(changelog_path, "r") as fileptr: | ||
512 | 367 | chlog = Changelog(fileptr.read()) | ||
513 | 368 | change_blocks = [] | ||
514 | 369 | start = False | ||
515 | 370 | end = False | ||
516 | 371 | error_msg = '' | ||
517 | 372 | |||
518 | 373 | # The changelog blocks are in reverse order; we'll see high before low. | ||
519 | 374 | for block in chlog: | ||
520 | 375 | if block.version == version_high: | ||
521 | 376 | start = True | ||
522 | 377 | change_blocks = [] | ||
523 | 378 | if block.version == version_low: | ||
524 | 379 | end = True | ||
525 | 380 | break | ||
526 | 381 | change_blocks.append(block) | ||
527 | 382 | if not start: | ||
528 | 383 | error_msg = "Missing starting version {} in {}. " \ | ||
529 | 384 | "Changlelog will be incomplete".format( | ||
530 | 385 | version_high, changelog_path) | ||
531 | 386 | logging.error(error_msg) | ||
532 | 387 | if not end: | ||
533 | 388 | if error_msg: | ||
534 | 389 | # Start and end were not found, put a newline between their | ||
535 | 390 | # error messages | ||
536 | 391 | error_msg += '\n' | ||
537 | 392 | error_msg += "Missing ending version {} in {}. " \ | ||
538 | 393 | "Changelog output truncated".format( | ||
539 | 394 | version_low, changelog_path) | ||
540 | 395 | logging.error(error_msg) | ||
541 | 396 | return change_blocks, error_msg | ||
542 | 397 | |||
543 | 398 | |||
544 | 399 | def print_changelogs(apt_cache, apt_cache_d, manifest_from, manifest_to, | ||
545 | 400 | changed): | ||
546 | 401 | """ | ||
547 | 402 | Print changelog entries for each changed package limited to | ||
548 | 403 | changes in the package between the versions in the two manifests. | ||
549 | 404 | |||
550 | 405 | :param :class:`apt.Cache` apt_cache: Open & up-to-date apt cache | ||
551 | 406 | :param str apt_cache_d: apt cache path | ||
552 | 407 | :param dict manifest_from: Packages and their versions in the | ||
553 | 408 | first manifest file | ||
554 | 409 | :param dict manifest_to: Packages and their versions in the | ||
555 | 410 | second manifest file | ||
556 | 411 | :param list changed: Packages which changed between the two manifests | ||
557 | 412 | """ | ||
558 | 413 | |||
559 | 414 | srcs = {} | ||
560 | 415 | errors = [] | ||
561 | 416 | src2bins = map_source_to_binary(apt_cache, changed) | ||
562 | 417 | |||
563 | 418 | # Generate changelog data per unique source package | ||
564 | 419 | for source_name in src2bins: | ||
565 | 420 | srcs[source_name] = {"changelog_file": "", "changeblocks": []} | ||
566 | 421 | src = srcs[source_name] | ||
567 | 422 | |||
568 | 423 | # Use the first binary listed for a source package | ||
569 | 424 | binary_name = src2bins[source_name][0] | ||
570 | 425 | |||
571 | 426 | # Find the source version data for the binary in manifest #2 | ||
572 | 427 | try: | ||
573 | 428 | src['version_to'] = source_version_for_binary( | ||
574 | 429 | apt_cache, binary_name, manifest_to[binary_name]) | ||
575 | 430 | except UnknownSourceVersionError as excp: | ||
576 | 431 | logging.error(str(excp)) | ||
577 | 432 | errors.append(excp) | ||
578 | 433 | continue | ||
579 | 434 | |||
580 | 435 | # Find the source version data for the binary in manifest #1 | ||
581 | 436 | binver_from = manifest_from[binary_name] | ||
582 | 437 | try: | ||
583 | 438 | src['version_from'] = source_version_for_binary( | ||
584 | 439 | apt_cache, binary_name, binver_from) | ||
585 | 440 | except UnknownSourceVersionError as excp: | ||
586 | 441 | if manifest_to[binary_name] == src['version_to']: | ||
587 | 442 | logging.info('Could not find source version data in apt ' | ||
588 | 443 | 'cache. Assuming source %s version %s from ' | ||
589 | 444 | 'binary %s', source_name, binver_from, | ||
590 | 445 | binary_name) | ||
591 | 446 | src['version_from'] = binver_from | ||
592 | 447 | else: | ||
593 | 448 | logging.error(str(excp)) | ||
594 | 449 | errors.append(excp) | ||
595 | 450 | continue | ||
596 | 451 | |||
597 | 452 | # Check for version regression between manifests | ||
598 | 453 | try: | ||
599 | 454 | if version_compare(src['version_from'], src['version_to']) > 0: | ||
600 | 455 | msg = "Package version regression {} -> {}".format( | ||
601 | 456 | src['version_from'], src['version_to']) | ||
602 | 457 | raise UnknownSourceVersionError(msg) | ||
603 | 458 | except UnknownSourceVersionError as excp: | ||
604 | 459 | errors.append(excp) | ||
605 | 460 | continue | ||
606 | 461 | |||
607 | 462 | # Get the changelog for this source package | ||
608 | 463 | try: | ||
609 | 464 | # Use the apt cache directory to store the changelog cache | ||
610 | 465 | srcs[source_name]["changelog_file"] = get_changelog( | ||
611 | 466 | source_name, src['version_to'], changelog_cache_d=apt_cache_d) | ||
612 | 467 | except MissingChangelogError as excp: | ||
613 | 468 | errors.append(excp) | ||
614 | 469 | continue | ||
615 | 470 | |||
616 | 471 | # Filter the changelog to a list of blocks between the two versions | ||
617 | 472 | try: | ||
618 | 473 | srcs[source_name]["changeblocks"], incomplete = filter_changelog( | ||
619 | 474 | srcs[source_name]["changelog_file"], src['version_from'], | ||
620 | 475 | src['version_to']) | ||
621 | 476 | if incomplete: | ||
622 | 477 | raise ChangelogMissingVersion(incomplete) | ||
623 | 478 | except ChangelogMissingVersion as exp: | ||
624 | 479 | errors.append(exp) | ||
625 | 480 | continue | ||
626 | 481 | |||
627 | 482 | # Print changelog ranges for changed packages | ||
628 | 483 | for source_name in sorted(src2bins): | ||
629 | 484 | binlist = sorted(src2bins[source_name]) | ||
630 | 485 | binary = binlist[0] | ||
631 | 486 | print("==== %s: %s => %s ====" % | ||
632 | 487 | (source_name, manifest_from[binary], manifest_to[binary])) | ||
633 | 488 | print("==== %s" % ' '.join(binlist)) | ||
634 | 489 | print_blocks(srcs[source_name]["changeblocks"]) | ||
635 | 490 | |||
636 | 491 | if errors: | ||
637 | 492 | print("**** Errors ****") | ||
638 | 493 | for error in errors: | ||
639 | 494 | print(error) | ||
640 | 495 | |||
641 | 496 | |||
642 | 497 | def parse_args(): | ||
643 | 498 | """ | ||
644 | 499 | Parse command line arguments | ||
645 | 500 | |||
646 | 501 | :returns: options and remaining arguments from OptionParser.parse_args() | ||
647 | 502 | :rtype list: | ||
648 | 503 | """ | ||
649 | 504 | |||
650 | 505 | parser = OptionParser(usage="Usage: {} arch suite manifest1 manifest2\n" | ||
651 | 506 | "Compare two manifest files, and show " | ||
652 | 507 | "changelog differences." | ||
653 | 508 | .format(os.path.basename(sys.argv[0]))) | ||
654 | 509 | parser.add_option("--cache-dir", dest="cache_d", | ||
655 | 510 | help="cache dir for info", metavar="DIR", type="string", | ||
656 | 511 | default=None) | ||
657 | 512 | parser.add_option("-v", "--verbose", action="count", dest="loglevel", | ||
658 | 513 | help="increase verbosity", default=0) | ||
659 | 514 | |||
660 | 515 | (options, args) = parser.parse_args() | ||
661 | 516 | |||
662 | 517 | if len(args) != 4: | ||
663 | 518 | parser.error('you must provide arch, release, and 2 manifest files') | ||
664 | 519 | |||
665 | 520 | return options, args | ||
666 | 521 | |||
667 | 522 | |||
668 | 523 | def setup_logging(loglevel=0): | ||
669 | 524 | """ | ||
670 | 525 | Configure logging | ||
671 | 526 | |||
672 | 527 | By default, log WARNING and higher messages | ||
673 | 528 | :param int: loglevel 0: Warning, 1: Info, 2: Debug | ||
674 | 529 | """ | ||
675 | 530 | |||
676 | 531 | loglevel = [logging.WARNING, | ||
677 | 532 | logging.INFO, | ||
678 | 533 | logging.DEBUG][min(2, loglevel)] | ||
679 | 534 | |||
680 | 535 | logging.basicConfig( | ||
681 | 536 | level=loglevel, | ||
682 | 537 | format="%(asctime)s %(name)s/%(levelname)s: %(message)s", | ||
683 | 538 | stream=sys.stderr) | ||
684 | 539 | |||
685 | 540 | |||
686 | 541 | def stdout_force_unicode(): | ||
687 | 542 | """ | ||
688 | 543 | Force output to UTF-8 at all times | ||
689 | 544 | |||
690 | 545 | We want to output UTF-8 at all times to preserve Unicode characters | ||
691 | 546 | in the changelog blocks. For Python 2 we will wrap sys.stdout with | ||
692 | 547 | an instance of StreamWriter with our preferred coding. Python3 requires | ||
693 | 548 | no changes. | ||
694 | 549 | |||
695 | 550 | When writing output to the terminal we get the encoding of the | ||
696 | 551 | terminal (utf-8 these days). When we redirect or pipe the output of | ||
697 | 552 | the program it is generally not possible to know what the input | ||
698 | 553 | encoding of the receiving program is, the encoding when redirecting | ||
699 | 554 | to a file will be None (Python 2.7) or UTF-8 (Python 3) | ||
700 | 555 | |||
701 | 556 | $ python2.7 -c "import sys; print sys.stdout.encoding" | cat | ||
702 | 557 | None | ||
703 | 558 | |||
704 | 559 | $ python3.4 -c "import sys; print(sys.stdout.encoding)" | cat | ||
705 | 560 | UTF-8 | ||
706 | 561 | |||
707 | 562 | Source: | ||
708 | 563 | https://wiki.python.org/moin/PrintFails#print.2C_write_and_Unicode_in_pre-3.0_Python | ||
709 | 564 | """ | ||
710 | 565 | |||
711 | 566 | if sys.version_info[0] < 3: | ||
712 | 567 | encoding = codecs.getwriter(locale.getpreferredencoding()) | ||
713 | 568 | sys.stdout = encoding(sys.stdout) | ||
714 | 569 | |||
715 | 147 | 570 | ||
716 | 148 | def main(): | 571 | def main(): |
869 | 149 | parser = OptionParser(usage=Usage) | 572 | """ |
870 | 150 | parser.add_option("--sources-list", dest="sources_list", | 573 | Given two manifest files for a particular release/arch, find the |
871 | 151 | help="sources.list file", metavar="FILE", type="string", | 574 | packages which have been added, removed, and changed. Print the |
872 | 152 | default=None) | 575 | changelog entries for the changed packages limited to changes between |
873 | 153 | parser.add_option("--cache-dir", dest="cache_d", | 576 | the two versions. |
874 | 154 | help="cache dir for info", metavar="DIR", type="string", | 577 | """ |
875 | 155 | default=None) | 578 | |
876 | 156 | parser.add_option("-v", "--verbose", action="count", dest="loglevel", | 579 | stdout_force_unicode() |
877 | 157 | help="increase verbosity", default=0) | 580 | options, (arch, release, manifest_from_filename, |
878 | 158 | 581 | manifest_to_filename) = parse_args() | |
879 | 159 | (options, args) = parser.parse_args() | 582 | |
880 | 160 | 583 | setup_logging(options.loglevel) | |
881 | 161 | if len(args) != 4: | 584 | |
882 | 162 | parser.error('you must provide arch, release, and 2 manifest files') | 585 | # index both manifests |
883 | 163 | 586 | manifest_to = manifest_to_dict(manifest_to_filename) | |
884 | 164 | # logging module levels are 0,10,20,30 ... | 587 | manifest_from = kernel_fixups(manifest_to_dict(manifest_from_filename), |
885 | 165 | loglevel = (6 - options.loglevel) * 10 | 588 | manifest_to) |
886 | 166 | logging.basicConfig(level=loglevel,format=logformat,stream=sys.stderr) | 589 | |
887 | 167 | 590 | new = find_added(manifest_from, manifest_to) | |
888 | 168 | (arch,release,mf_from, mf_to) = args | 591 | removed = find_removed(manifest_from, manifest_to) |
889 | 169 | 592 | changed = find_changed(manifest_from, manifest_to) | |
890 | 170 | cache_d = options.cache_d | 593 | |
891 | 171 | if not options.cache_d: | 594 | print("new: %s" % new) |
892 | 172 | cache_d = "./cache.%s-%s" % (release, arch) | 595 | print("removed: %s" % removed) |
893 | 173 | 596 | print("changed: %s" % changed) | |
894 | 174 | # index both manifests | 597 | |
895 | 175 | h_from=hashmffile(mf_from) | 598 | # if modified packages, download all changelogs from changelogs. |
896 | 176 | h_to=hashmffile(mf_to) | 599 | if changed: |
897 | 177 | 600 | cache, cache_d = open_apt_cache(arch, release, options.cache_d) | |
898 | 178 | # fix up kernels so the pkg names match | 601 | print_changelogs(cache, cache_d, manifest_from, manifest_to, changed) |
747 | 179 | kfixups = { } | ||
748 | 180 | kmatch = re.compile("linux-image-[0-9]") | ||
749 | 181 | for pkg,ver in h_to.iteritems(): | ||
750 | 182 | # if this is a linux-image-* binary package do some hacks to make it | ||
751 | 183 | # look like h_from is the same (format like linux-image-2.6.32-32-virtual) | ||
752 | 184 | if kmatch.match(pkg): | ||
753 | 185 | img_type = pkg[pkg.rfind("-")+1:] | ||
754 | 186 | if pkg in h_from: continue | ||
755 | 187 | for fpkg, fver in h_from.iteritems(): | ||
756 | 188 | if ( kmatch.match(fpkg) and fpkg.endswith("-%s" % img_type)): | ||
757 | 189 | kfixups[pkg] = fpkg | ||
758 | 190 | |||
759 | 191 | for pkg_to, pkg_from in kfixups.iteritems(): | ||
760 | 192 | h_from[pkg_to] = h_from[pkg_from] | ||
761 | 193 | del h_from[pkg_from] | ||
762 | 194 | |||
763 | 195 | # find new packages in mf2 | ||
764 | 196 | new={ } | ||
765 | 197 | for pkg,ver in h_to.iteritems(): | ||
766 | 198 | if pkg not in h_from: new[pkg]=h_to[pkg] | ||
767 | 199 | |||
768 | 200 | # find packages removed from mf1 | ||
769 | 201 | removed={ } | ||
770 | 202 | for pkg,ver in h_from.iteritems(): | ||
771 | 203 | if pkg not in h_to: removed[pkg]=h_from[pkg] | ||
772 | 204 | |||
773 | 205 | # find modified packages | ||
774 | 206 | changed=[] | ||
775 | 207 | for pkg,ver in h_to.iteritems(): | ||
776 | 208 | if pkg in h_from and h_from[pkg] != ver: changed.append(pkg) | ||
777 | 209 | |||
778 | 210 | print "new: %s" % new | ||
779 | 211 | print "removed: %s" % removed | ||
780 | 212 | print "changed: %s" % changed | ||
781 | 213 | |||
782 | 214 | |||
783 | 215 | prep_cacheroot(cache_d,release) | ||
784 | 216 | apt.apt_pkg.config.set("Apt::Architecture",arch) | ||
785 | 217 | cache = None | ||
786 | 218 | |||
787 | 219 | errors = [ ] | ||
788 | 220 | |||
789 | 221 | # if modified packages | ||
790 | 222 | # download all changelogs from changelogs. | ||
791 | 223 | bin2src = { } | ||
792 | 224 | srcs = { } | ||
793 | 225 | if len(changed): | ||
794 | 226 | cache=get_cache(cache_d, cache) | ||
795 | 227 | |||
796 | 228 | b2s_hash=get_bin2src(changed,cache) | ||
797 | 229 | |||
798 | 230 | for pkg,src in b2s_hash.iteritems(): | ||
799 | 231 | if src not in bin2src: | ||
800 | 232 | bin2src[src]=[] | ||
801 | 233 | bin2src[src].append(pkg) | ||
802 | 234 | |||
803 | 235 | missing_cl=[] | ||
804 | 236 | for src,binlist in bin2src.iteritems(): | ||
805 | 237 | if src not in srcs: | ||
806 | 238 | srcs[src] = { "changelog_file": "", "changeblocks": [ ] } | ||
807 | 239 | |||
808 | 240 | try: | ||
809 | 241 | pkg_name = binlist[0].split(':')[0] | ||
810 | 242 | versions = cache[pkg_name].versions | ||
811 | 243 | except KeyError as e: | ||
812 | 244 | raise Exception("%s not in cache did not have versions info in cache") | ||
813 | 245 | binver_to=h_to[binlist[0]] | ||
814 | 246 | try: | ||
815 | 247 | srcver_to=versions[binver_to].source_version | ||
816 | 248 | except KeyError as e: | ||
817 | 249 | raise Exception("bin pkg %s from src %s version %s not in binver list (%s)" % | ||
818 | 250 | (binlist[0],src,binver_to,versions)) | ||
819 | 251 | |||
820 | 252 | binver_from=h_from[binlist[0]] | ||
821 | 253 | try: | ||
822 | 254 | srcver_from=versions[binver_from].source_version | ||
823 | 255 | except KeyError as e: | ||
824 | 256 | if binver_to == srcver_to: | ||
825 | 257 | srcver_from = binver_from | ||
826 | 258 | else: | ||
827 | 259 | raise UnknownSourceVersionError( | ||
828 | 260 | "unable to determine src version for %s/%s" % | ||
829 | 261 | (src,binver_from)) | ||
830 | 262 | |||
831 | 263 | try: | ||
832 | 264 | clogfile=getchangelog(src,srcver_to,cache_d) | ||
833 | 265 | srcs[src]["changelog_file"] = clogfile | ||
834 | 266 | chlog=Changelog(filecontents(clogfile)) | ||
835 | 267 | srcs[src]["changeblocks"] = [ ] | ||
836 | 268 | start = False | ||
837 | 269 | end = False | ||
838 | 270 | for block in chlog: | ||
839 | 271 | if block.version == srcver_to: | ||
840 | 272 | start = True | ||
841 | 273 | if block.version == srcver_from: | ||
842 | 274 | end = True | ||
843 | 275 | break | ||
844 | 276 | if start: | ||
845 | 277 | srcs[src]["changeblocks"].append(block) | ||
846 | 278 | if not (start and end): | ||
847 | 279 | em = ("%s missing %s or %s." % | ||
848 | 280 | (clogfile, srcver_to, binver_from)) | ||
849 | 281 | if apt.VersionCompare(binver_from, srcver_to) > 0: | ||
850 | 282 | em = "%s %s" % (em, "from version > to version") | ||
851 | 283 | raise ChangelogMissingVersion(em) | ||
852 | 284 | |||
853 | 285 | except MissingChangelogError as e: | ||
854 | 286 | errors.append(e) | ||
855 | 287 | except ChangelogMissingVersion as e: | ||
856 | 288 | errors.append(e) | ||
857 | 289 | except UnknownSourceVersionError as e: | ||
858 | 290 | errors.append(e) | ||
859 | 291 | |||
860 | 292 | for src,binlist in bin2src.iteritems(): | ||
861 | 293 | print "==== %s: %s => %s ====" % (src, h_from[binlist[0]], h_to[binlist[0]]) | ||
862 | 294 | print "==== %s" % ' '.join(binlist) | ||
863 | 295 | print_blocks(srcs[src]["changeblocks"]) | ||
864 | 296 | |||
865 | 297 | if errors: | ||
866 | 298 | print "**** Errors ****" | ||
867 | 299 | for e in errors: print e | ||
868 | 300 | # load each changelog | ||
899 | 301 | 602 | ||
900 | 302 | 603 | ||
901 | 303 | if __name__ == "__main__": | 604 | if __name__ == "__main__": |
903 | 304 | main() | 605 | main() |
Break up main(), modernize code, improve documentation and variable naming, python3 compat, misc fixes, urllib to requests migration, support non-x86 architectures, and add logging.
There are a fair number of commits here and this is best reviewed by commit rather than with the MP diff view.