Merge lp:~rcj/vmbuilder/mfdiff_fixups into lp:~ubuntu-on-ec2/vmbuilder/mfdiff

Proposed by Robert C Jennings on 2017-06-27
Status: Merged
Merged at revision: 19
Proposed branch: lp:~rcj/vmbuilder/mfdiff_fixups
Merge into: lp:~ubuntu-on-ec2/vmbuilder/mfdiff
Diff against target: 903 lines (+581/-281)
2 files modified
README (+9/-10)
mfdiff (+572/-271)
To merge this branch: bzr merge lp:~rcj/vmbuilder/mfdiff_fixups
Reviewer Review Type Date Requested Status
Francis Ginther 2017-06-28 Approve on 2017-06-28
Dan Watkins 2017-06-27 Approve on 2017-06-27
Review via email: mp+326375@code.launchpad.net

Description of the change

Modernization and fixes

Break up main(), modernize code, improve documentation and variable
naming, python3 compat, misc fixes, urllib to requests migration,
support non-x86 architectures, and add logging.

This requires python-six version newer than what is shipping in some
supported Ubuntu releases to use six.viewkeys. (>=1.10 is preferable)

 [ Robert C Jennings ]
 * Strip architecture from binary when searching cache
 * Copy rather than rename changelog files
 * Force output to UTF-8 at all times
 * Output utf-8 rather than ascii
 * Change exception rendering for python3 compatibility
 * Use the current directory for temp changelog files
 * Improve variable naming for changelog_url (was clog_url)
 * Support non-x86 architectures
 * Fix usage text to correct option order
 * Eliminate traceback if binary package has been renamed
   (no longer in apt cache)
 * Use a real temporary file
 * Fixups for code-review
 * Improve names in print_changelogs and store more to the sources dict
 * Break out changelog processing from main
 * More variable renames
 * Rearrange summary printout and changelog printing for later refactor
 * Name, spelling, and doc cleanup
 * Stop processing source package on exception
 * Unlink temp file only if it exists
 * Add more complete function docstrings
 * Additional refactoring to reduce size of main
 * isort requests
 * Update authors/copyright
 * Add logging
 * Simplify cmdline --verbose option
 * Swap urllib for requests
 * Improve variable naming
 * Consistent sorted output
 * python3 compatibility changes
 * Document functions, use 'with' for files, and minor fixes
 * [merge] PEP8 cleanup, example updates, & python-apt updates
   * Correct vi line for tab expansion
   * README: Remove trailing spaces
   * Use host apt-secure keychains
   * APT version comparison moved from apt to apt_pkg
   * Remove unused cmdline option
   * Update README to use modern examples
   * Move comment to proper function
   * remove unused imports/variables
   * sort imports
   * pep8: autopep8 changes
   * pep8: tab -> space conversion

To post a comment you must log in.
Robert C Jennings (rcj) wrote :

Break up main(), modernize code, improve documentation and variable naming, python3 compat, misc fixes, urllib to requests migration, support non-x86 architectures, and add logging.

There are a fair number of commits here and this is best reviewed by commit rather than with the MP diff view.

Robert C Jennings (rcj) wrote :

This requires python-six version newer than what is shipping in Trusty to use viewkeys. (1.10 is preferable)

review: Approve
Francis Ginther (fginther) wrote :

Approve, this MP and resulting mfdiff are identical to a branch I already reviewed and tested.

review: Approve

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1=== modified file 'README'
2--- README 2010-12-13 15:23:43 +0000
3+++ README 2017-06-27 19:03:39 +0000
4@@ -1,15 +1,14 @@
5 "Manifest Diff" (mfdiff) is a tool for determining what changed between
6-2 builds by looking at their manifests. A manifest is simply a
7+2 builds by looking at their manifests. A manifest is simply a
8 list of package and versions in a build, obtainable by:
9 dpkg-query', '-W', '--showformat=${Package} ${Version}\n'
10
11 Example of running:
12- mf1=mfs/lucid/release-20100827/ubuntu-10.04-server-uec-amd64.manifest
13- mf2=mfs/lucid/release-20100923/ubuntu-10.04-server-uec-amd64.manifest
14- ./mfdiff i386 lucid "${mf1}" "${mf2}"
15-
16-
17-
18-sync manifests from uec-images.ubuntu.com with:
19- rsync -av uec-images.ubuntu.com::uec-images/ --prune-empty-dirs mfs \
20- --filter "+ */" --filter "+ *.manifest" --filter "- *"
21+ mf1=mfs/releases/trusty/release-20160314/ubuntu-14.04-server-cloudimg-amd64.manifest
22+ mf2=mfs/releases/trusty/release-20170517/ubuntu-14.04-server-cloudimg-amd64.manifest
23+ ./mfdiff amd64 trusty "${mf1}" "${mf2}"
24+
25+
26+sync manifests from cloud-images.ubuntu.com with:
27+ rsync -av cloud-images.ubuntu.com::cloud-images/ --prune-empty-dirs mfs \
28+ --filter "+ */" --filter "+ *.manifest" --filter "- *"
29
30=== modified file 'mfdiff'
31--- mfdiff 2014-10-03 11:27:32 +0000
32+++ mfdiff 2017-06-27 19:03:39 +0000
33@@ -1,9 +1,17 @@
34 #!/usr/bin/env python
35-# vi:ts=4 noexpandtab
36+"""
37+Given two manifest files for a particular release/arch, find the
38+packages which have been added, removed, and changed. Print the
39+changelog entries for the changed packages limited to changes between
40+the two versions.
41+"""
42+
43+# vi:ts=4 expandtab
44 #
45-# Copyright (C) 2010 Canonical Ltd.
46+# Copyright (C) 2010, 2017 Canonical Ltd.
47 #
48 # Authors: Scott Moser <scott.moser@canonical.com>
49+# Robert C Jennings <robert.jennings@canonical.com>
50 #
51 # This program is free software: you can redistribute it and/or modify
52 # it under the terms of the GNU General Public License as published by
53@@ -17,288 +25,581 @@
54 # You should have received a copy of the GNU General Public License
55 # along with this program. If not, see <http://www.gnu.org/licenses/>.
56
57+import codecs
58+import locale
59+import logging
60 import os
61-import string
62+import os.path
63+import re
64+import shutil
65 import sys
66+import tempfile
67 from optparse import OptionParser
68-import urllib
69-import logging
70-import os.path
71-import os
72-from debian.changelog import Changelog, ChangeBlock
73+
74 import apt
75-import re
76-
77-Usage="""
78-Usage: %s --apt-sources F arch manifest1 manifest2
79- Compare two manifest files, and show changelog differences.
80-"""
81-logformat="%(asctime)s %(name)s/%(levelname)s: %(message)s"
82+import requests
83+from debian.changelog import Changelog
84+from six import iteritems, viewkeys
85+
86+try:
87+ from apt import VersionCompare as version_compare
88+except ImportError:
89+ from apt_pkg import version_compare as version_compare
90+
91
92 class MissingChangelogError(Exception):
93- pass
94+ "The Changelog file could not be found on the server"
95+ pass
96+
97
98 class ChangelogMissingVersion(Exception):
99- pass
100+ "The Changelog is missing starting and/or ending versions for the search"
101+ pass
102+
103
104 class UnknownSourceVersionError(Exception):
105- pass
106-
107-def get_bin2src(changed,cache):
108- ret = { }
109- for binpkg in changed:
110- pkg_name = binpkg.split(':')[0]
111- ret[binpkg]=cache[pkg_name].versions[0].source_name
112- return(ret)
113-
114-# downloads changelog for src/ver and returns path to that
115-def getchangelog(src,ver,cache_d):
116- # http://changelogs.ubuntu.com/changelogs/pool/main/h/hal/hal_0.5.5.1-1ubuntu2/changelog
117- clog_burl="http://changelogs.ubuntu.com/changelogs/pool"
118- urls=[]
119- bsize=2048
120- cache_f = "%s/changelog.%s_%s" % (cache_d, src,ver)
121-
122- if os.path.isfile(cache_f):
123- return(cache_f)
124-
125- furls=[ ]
126- num_colon_m=re.compile("[0-9]:")
127-
128- cache_tmp = "%s/.changelog.%s_%s" % (cache_d, src,ver)
129- for pile in ("main", "universe", "multiverse", "restricted"):
130- pre=src[0:1]
131- # packages starting with 'lib' are special
132- if src.startswith("lib"): pre=src[0:4]
133-
134- # packages with '1:' versions have different paths
135- # update-manager at version 1:0.134.11.
136- # would be at main/u/update-manager/update-manager_1:0.134.11
137- # instead at main/u/update-manager/update-manager_0.134.11/
138- xver=ver
139- if num_colon_m.match(ver):
140- xver=ver[2:]
141-
142- clog_url = "%s/%s/%s/%s/%s_%s/changelog" % \
143- ( clog_burl, pile, pre, src, src, xver)
144-
145- fp = open(cache_tmp,"w")
146- u_fp = urllib.urlopen(clog_url)
147- while True:
148- buf = u_fp.read(bsize)
149- fp.write(buf)
150- if len(buf) != bsize: break
151- fp.close()
152- if u_fp.getcode() == 200:
153- os.rename(cache_tmp,cache_f)
154- return(cache_f)
155- print "missing %s: %s" % (src,clog_url)
156- furls.append(clog_url)
157- os.unlink(cache_tmp)
158- raise(MissingChangelogError("Failed to find changelog for %s at version %s.\n tried %s" % (src, ver, ' '.join(furls))))
159-
160-def hashmffile(filename):
161- fp=open(filename,"r")
162- lines=fp.readlines()
163- fp.close()
164- ret={ }
165- for line in lines:
166- (pkg,ver)=line.split()
167- ret[pkg]=ver
168- return(ret)
169-
170-def filecontents(filename):
171- fp=open(filename,"r")
172- contents=fp.read()
173- fp.close()
174- return(contents)
175-
176-def prep_cacheroot(cache_d, release):
177- mirror = "http://archive.ubuntu.com/ubuntu/"
178- pockets = ( release, "%s-updates" % release, "%s-security" % release,
179- "%s-proposed" % release, )
180- components = ( "main", "universe" )
181- srclines = []
182- for pocket in pockets:
183- srclines.append("deb %s %s %s" % (mirror, pocket, ' '.join(components)))
184- try:
185- os.makedirs( "%s/etc/apt" % cache_d )
186- except:
187- pass
188- asl = open( "%s/etc/apt/sources.list" % cache_d, "w" )
189- asl.write('\n'.join(srclines))
190- asl.close()
191-
192-def get_cache(cache_d, cache=None):
193- if cache: return cache
194- cache = apt.Cache(rootdir=cache_d)
195- cache.update()
196- cache.open()
197- return cache
198-
199-# render a changelog block to something printable
200+ "The binary package did not have a source version listed"
201+ pass
202+
203+
204+def get_bin2src(packages, cache):
205+ """
206+ Find the source package names for a given binary package list
207+
208+ :param list packages: List of binary packages
209+ :param :class:`apt.Cache` cache: Open/updated apt cache
210+ :return: List mapping binary package name to source package name
211+ :rtype: dict
212+ """
213+
214+ ret = {}
215+ logging.debug('Finding source package names for all binary packages')
216+ for binpkg in packages:
217+ pkg_name = binpkg.split(':')[0]
218+ ret[binpkg] = cache[pkg_name].versions[0].source_name
219+ return ret
220+
221+
222+def get_changelog(source, version, changelog_cache_d):
223+ """
224+ Download changelog for source / version and returns path to that
225+
226+ :param str source: Source package name
227+ :param str version: Source package version
228+ :param str changelog_cache_d: path to store cached changelogs
229+ :raises MissingChangelogError: If changelog file could not be downloaded
230+ :return: changelog file for source package & version
231+ :rtype: str
232+ """
233+
234+ cache_f = "%s/changelog.%s_%s" % (changelog_cache_d, source, version)
235+
236+ if os.path.isfile(cache_f):
237+ logging.debug("Using cached changelog for %s:%s", source, version)
238+ return cache_f
239+
240+ furls = []
241+ num_colon_m = re.compile("[0-9]:")
242+
243+ cache_tmp_fd, cache_tmp_path = tempfile.mkstemp(
244+ ".changelog.%s_%s" % (source, version),
245+ prefix="." + tempfile.gettempprefix(), dir=".")
246+ cache_tmp = os.fdopen(cache_tmp_fd, "w")
247+ for pile in ("main", "universe", "multiverse", "restricted"):
248+ pre = source[0:1]
249+ # packages starting with 'lib' are special
250+ if source.startswith("lib"):
251+ pre = source[0:4]
252+
253+ # packages with '1:' versions have different paths
254+ # update-manager at version 1:0.134.11.
255+ # is main/u/update-manager/update-manager_0.134.11/
256+ # rather than main/u/update-manager/update-manager_1:0.134.11
257+ url_version = version
258+ if num_colon_m.match(version):
259+ url_version = version[2:]
260+
261+ # Changelog URL example http://changelogs.ubuntu.com/changelogs/\
262+ # pool/main/h/hal/hal_0.5.5.1-1ubuntu2/changelog
263+ changelog_url = "http://changelogs.ubuntu.com/changelogs/pool/" \
264+ "%s/%s/%s/%s_%s/changelog" % \
265+ (pile, pre, source, source, url_version)
266+
267+ changelog = requests.get(changelog_url)
268+ if changelog.status_code == 200:
269+ cache_tmp.write(changelog.content)
270+ cache_tmp.close()
271+ shutil.copy2(cache_tmp_path, cache_f)
272+ os.unlink(cache_tmp_path)
273+ return cache_f
274+ else:
275+ logging.error("missing %s: %s", source, changelog_url)
276+ furls.append(changelog_url)
277+ if os.path.exists(cache_tmp_path):
278+ os.unlink(cache_tmp_path)
279+ raise MissingChangelogError("Failed to find changelog for %s at version "
280+ "%s.\n tried %s" % (source, version,
281+ ' '.join(furls)))
282+
283+
284+def manifest_to_dict(filename):
285+ """
286+ Parse manifest file to create a package / version mapping
287+
288+ :param str filename: Name of package manifest file
289+ :return: List of package versions by name
290+ :rtype: dict
291+ """
292+
293+ ret = {}
294+ logging.debug('Reading package manifest from %s', filename)
295+ with open(filename, "r") as manifest:
296+ for line in manifest:
297+ (pkg, ver) = line.split()
298+ ret[pkg] = ver
299+ return ret
300+
301+
302+def open_apt_cache(arch, release, cache_d=None):
303+ """
304+ Create, update, and open an apt cache.
305+
306+ This creates an apt cache directory and write a sources.list file
307+ before updating and opening the cache. The caller is responsible
308+ for closing the cache.
309+
310+ :param str arch: Package architecture
311+ :param str release: Ubuntu release name (e.g. Xenial)
312+ :param str cache_d: apt cache path
313+ :returns: tuple of Open/updated apt cache and cache path name
314+ :rtype: tuple(:class:`apt.Cache`, str)
315+ """
316+
317+ if not cache_d:
318+ cache_d = "./cache.%s-%s" % (release, arch)
319+ logging.info("Using %s as the apt cache directory", cache_d)
320+
321+ if arch in ['amd64', 'i386']:
322+ mirror = "http://archive.ubuntu.com/ubuntu/"
323+ else:
324+ mirror = "http://ports.ubuntu.com/ubuntu-ports/"
325+ logging.debug('Configuring apt cache using mirror %s', mirror)
326+
327+ pockets = (release, "%s-updates" % release, "%s-security" % release,
328+ "%s-proposed" % release, )
329+ components = ("main", "universe")
330+ srclines = []
331+ for pocket in pockets:
332+ srcline = "deb %s %s %s" % (mirror, pocket, ' '.join(components))
333+ logging.debug('Adding source: %s', srcline)
334+ srclines.append(srcline)
335+ try:
336+ os.makedirs("%s/etc/apt" % cache_d)
337+ except OSError as oserror:
338+ if os.errno.EEXIST != oserror.errno:
339+ raise
340+ with open("%s/etc/apt/sources.list" % cache_d, "w") as asl:
341+ asl.write('\n'.join(srclines))
342+
343+ apt.apt_pkg.config.set("Apt::Architecture", arch)
344+
345+ logging.debug('Using host apt keys for signature verification')
346+ apt.apt_pkg.config.set("Dir::Etc::Trusted", "/etc/apt/trusted.gpg")
347+ apt.apt_pkg.config.set("Dir::Etc::TrustedParts",
348+ "/etc/apt/trusted.gpg.d/")
349+
350+ cache = apt.Cache(rootdir=cache_d)
351+ logging.info('Updating apt cache')
352+ cache.update()
353+ logging.info('Update of apt cache complete')
354+ cache.open()
355+ return cache, cache_d
356+
357+
358 def render_block(block):
359- return('\n'.join([x.encode('ascii', 'replace') for x in block.changes() if x]))
360-
361-def print_blocks(blist):
362- # print a blocklist
363- for block in blist:
364- print render_block(block)
365+ """
366+ Render a changelog block to something printable (dropping blank lines)
367+
368+ :param :class:`debian.changelog.ChangeBlock` block: Changelog block
369+ :return: String containing the changelog block text
370+ :rtype: str
371+ """
372+ return '\n'.join([x for x in block.changes() if x])
373+
374+
375+def print_blocks(block_list):
376+ """
377+ Print a Changelog block list
378+
379+ :param list block_list: List of :class:`debian.changelog.ChangeBlock`
380+ """
381+
382+ for block in block_list:
383+ print(render_block(block).encode('utf-8').decode('utf-8'))
384+
385+
386+def kernel_fixups(manifest_from, manifest_to):
387+ """
388+ Fix up kernels so the pkg names match
389+
390+ Kernel package names change from release to release so that they are
391+ co-installable, but we need to find matching package names in the
392+ two manifests to provide a list of changes between the versions of the
393+ package in each manifest.
394+ This function will return an altered version of manifest_from with kernel
395+ package names changed to match kernel package names in manifest_to. This
396+ will support later version comparisons.
397+
398+ :param dict manifest_from: Dictionary mapping package to version for of
399+ the starting manifest
400+ :param dict manifest_to: Dictionary mapping package to version for the
401+ ending manifest
402+ :return: Starting manifest dictionary with altered kernel package names
403+ to match names in ending manifest
404+ :rtype dict:
405+ """
406+ kfixups = {}
407+ kmatch = re.compile("linux-image-[0-9]")
408+ for pkg in manifest_to:
409+ # if this is a linux-image-* binary package do some hacks to make it
410+ # look like manifest_from is the same (format like
411+ # linux-image-2.6.32-32-virtual)
412+ if kmatch.match(pkg):
413+ logging.debug('Found kernel %s in manifest #2', pkg)
414+ img_type = pkg.split("-")[-1]
415+ if pkg in manifest_from:
416+ logging.debug('Found same kernel in manifest #1')
417+ continue
418+ for fpkg in manifest_from:
419+ if kmatch.match(fpkg) and fpkg.endswith("-%s" % img_type):
420+ logging.debug('Found similar kernel %s in manifest #1',
421+ fpkg)
422+ kfixups[pkg] = fpkg
423+
424+ for pkg_to, pkg_from in iteritems(kfixups):
425+ logging.debug('Substituting kernel %s for %s in manifest #1 to '
426+ 'enable version comparison', pkg_to, pkg_from)
427+ manifest_from[pkg_to] = manifest_from[pkg_from]
428+ del manifest_from[pkg_from]
429+ return manifest_from
430+
431+
432+def find_added(manifest_from, manifest_to):
433+ "Find new packages in manifest_to"
434+ new = {}
435+ for pkg in sorted(viewkeys(manifest_to) - viewkeys(manifest_from)):
436+ logging.debug('New package: %s', pkg)
437+ new[pkg] = manifest_to[pkg]
438+ return new
439+
440+
441+def find_removed(manifest_from, manifest_to):
442+ "Find packages removed from manifest_from"
443+ removed = {}
444+ for pkg in sorted(viewkeys(manifest_from) - viewkeys(manifest_to)):
445+ logging.debug('Removed package: %s', pkg)
446+ removed[pkg] = manifest_from[pkg]
447+ return removed
448+
449+
450+def find_changed(manifest_from, manifest_to):
451+ "Find modified packages"
452+ changed = []
453+ for pkg in sorted(viewkeys(manifest_from) & viewkeys(manifest_to)):
454+ if manifest_from[pkg] != manifest_to[pkg]:
455+ logging.debug('Changed package: %s', pkg)
456+ changed.append(pkg)
457+ return changed
458+
459+
460+def map_source_to_binary(cache, packages):
461+ "Create a dictionary of source to list of binary packages"
462+ src2bins = {}
463+ for bin_pkg in packages:
464+ bin_name = bin_pkg.split(':')[0]
465+ if bin_name in cache:
466+ src2bins.setdefault(
467+ cache[bin_name].versions[0].source_name, []).append(bin_pkg)
468+ return src2bins
469+
470+
471+def get_pkg_versions(cache, binary):
472+ "Get all known versions from the apt cache"
473+ pkg_name = binary.split(':')[0]
474+ try:
475+ return cache[pkg_name].versions
476+ except KeyError:
477+ raise Exception(
478+ "%s not in cache or did not have version info in cache" %
479+ pkg_name)
480+
481+
482+def source_version_for_binary(cache, binary, binary_ver):
483+ "Find the source version data for a specific binary version"
484+ versions = get_pkg_versions(cache, binary)
485+ try:
486+ return versions[binary_ver].source_version
487+ except KeyError:
488+ # Strip the architecture name from the binary
489+ source_name = cache[binary.split(':')[0]].versions[0].source_name
490+ msg = ("Unable to determine source version for %s. "
491+ "Binary package %s/%s not in known source version "
492+ "list (%s)" % (source_name, binary, binary_ver, versions))
493+ raise UnknownSourceVersionError(msg)
494+
495+
496+def filter_changelog(changelog_path, version_low, version_high):
497+ """
498+ Extract changelog entries within a version range
499+
500+ The range of changelog entries returned will include all entries
501+ after version_low up to, and including, version_high.
502+ If either the starting or ending version are not found in the
503+ list of changelog entries the result will be incomplete and
504+ a non-empty error message is returned to indicate the issue.
505+
506+ :param str changelog_path: File name of the changelog to process
507+ :return: list of changelog blocks and an error_msg if incomplete
508+ :rtype tuple(list, str):
509+ """
510+
511+ with open(changelog_path, "r") as fileptr:
512+ chlog = Changelog(fileptr.read())
513+ change_blocks = []
514+ start = False
515+ end = False
516+ error_msg = ''
517+
518+ # The changelog blocks are in reverse order; we'll see high before low.
519+ for block in chlog:
520+ if block.version == version_high:
521+ start = True
522+ change_blocks = []
523+ if block.version == version_low:
524+ end = True
525+ break
526+ change_blocks.append(block)
527+ if not start:
528+ error_msg = "Missing starting version {} in {}. " \
529+ "Changlelog will be incomplete".format(
530+ version_high, changelog_path)
531+ logging.error(error_msg)
532+ if not end:
533+ if error_msg:
534+ # Start and end were not found, put a newline between their
535+ # error messages
536+ error_msg += '\n'
537+ error_msg += "Missing ending version {} in {}. " \
538+ "Changelog output truncated".format(
539+ version_low, changelog_path)
540+ logging.error(error_msg)
541+ return change_blocks, error_msg
542+
543+
544+def print_changelogs(apt_cache, apt_cache_d, manifest_from, manifest_to,
545+ changed):
546+ """
547+ Print changelog entries for each changed package limited to
548+ changes in the package between the versions in the two manifests.
549+
550+ :param :class:`apt.Cache` apt_cache: Open & up-to-date apt cache
551+ :param str apt_cache_d: apt cache path
552+ :param dict manifest_from: Packages and their versions in the
553+ first manifest file
554+ :param dict manifest_to: Packages and their versions in the
555+ second manifest file
556+ :param list changed: Packages which changed between the two manifests
557+ """
558+
559+ srcs = {}
560+ errors = []
561+ src2bins = map_source_to_binary(apt_cache, changed)
562+
563+ # Generate changelog data per unique source package
564+ for source_name in src2bins:
565+ srcs[source_name] = {"changelog_file": "", "changeblocks": []}
566+ src = srcs[source_name]
567+
568+ # Use the first binary listed for a source package
569+ binary_name = src2bins[source_name][0]
570+
571+ # Find the source version data for the binary in manifest #2
572+ try:
573+ src['version_to'] = source_version_for_binary(
574+ apt_cache, binary_name, manifest_to[binary_name])
575+ except UnknownSourceVersionError as excp:
576+ logging.error(str(excp))
577+ errors.append(excp)
578+ continue
579+
580+ # Find the source version data for the binary in manifest #1
581+ binver_from = manifest_from[binary_name]
582+ try:
583+ src['version_from'] = source_version_for_binary(
584+ apt_cache, binary_name, binver_from)
585+ except UnknownSourceVersionError as excp:
586+ if manifest_to[binary_name] == src['version_to']:
587+ logging.info('Could not find source version data in apt '
588+ 'cache. Assuming source %s version %s from '
589+ 'binary %s', source_name, binver_from,
590+ binary_name)
591+ src['version_from'] = binver_from
592+ else:
593+ logging.error(str(excp))
594+ errors.append(excp)
595+ continue
596+
597+ # Check for version regression between manifests
598+ try:
599+ if version_compare(src['version_from'], src['version_to']) > 0:
600+ msg = "Package version regression {} -> {}".format(
601+ src['version_from'], src['version_to'])
602+ raise UnknownSourceVersionError(msg)
603+ except UnknownSourceVersionError as excp:
604+ errors.append(excp)
605+ continue
606+
607+ # Get the changelog for this source package
608+ try:
609+ # Use the apt cache directory to store the changelog cache
610+ srcs[source_name]["changelog_file"] = get_changelog(
611+ source_name, src['version_to'], changelog_cache_d=apt_cache_d)
612+ except MissingChangelogError as excp:
613+ errors.append(excp)
614+ continue
615+
616+ # Filter the changelog to a list of blocks between the two versions
617+ try:
618+ srcs[source_name]["changeblocks"], incomplete = filter_changelog(
619+ srcs[source_name]["changelog_file"], src['version_from'],
620+ src['version_to'])
621+ if incomplete:
622+ raise ChangelogMissingVersion(incomplete)
623+ except ChangelogMissingVersion as exp:
624+ errors.append(exp)
625+ continue
626+
627+ # Print changelog ranges for changed packages
628+ for source_name in sorted(src2bins):
629+ binlist = sorted(src2bins[source_name])
630+ binary = binlist[0]
631+ print("==== %s: %s => %s ====" %
632+ (source_name, manifest_from[binary], manifest_to[binary]))
633+ print("==== %s" % ' '.join(binlist))
634+ print_blocks(srcs[source_name]["changeblocks"])
635+
636+ if errors:
637+ print("**** Errors ****")
638+ for error in errors:
639+ print(error)
640+
641+
642+def parse_args():
643+ """
644+ Parse command line arguments
645+
646+ :returns: options and remaining arguments from OptionParser.parse_args()
647+ :rtype list:
648+ """
649+
650+ parser = OptionParser(usage="Usage: {} arch suite manifest1 manifest2\n"
651+ "Compare two manifest files, and show "
652+ "changelog differences."
653+ .format(os.path.basename(sys.argv[0])))
654+ parser.add_option("--cache-dir", dest="cache_d",
655+ help="cache dir for info", metavar="DIR", type="string",
656+ default=None)
657+ parser.add_option("-v", "--verbose", action="count", dest="loglevel",
658+ help="increase verbosity", default=0)
659+
660+ (options, args) = parser.parse_args()
661+
662+ if len(args) != 4:
663+ parser.error('you must provide arch, release, and 2 manifest files')
664+
665+ return options, args
666+
667+
668+def setup_logging(loglevel=0):
669+ """
670+ Configure logging
671+
672+ By default, log WARNING and higher messages
673+ :param int: loglevel 0: Warning, 1: Info, 2: Debug
674+ """
675+
676+ loglevel = [logging.WARNING,
677+ logging.INFO,
678+ logging.DEBUG][min(2, loglevel)]
679+
680+ logging.basicConfig(
681+ level=loglevel,
682+ format="%(asctime)s %(name)s/%(levelname)s: %(message)s",
683+ stream=sys.stderr)
684+
685+
686+def stdout_force_unicode():
687+ """
688+ Force output to UTF-8 at all times
689+
690+ We want to output UTF-8 at all times to preserve Unicode characters
691+ in the changelog blocks. For Python 2 we will wrap sys.stdout with
692+ an instance of StreamWriter with our preferred coding. Python3 requires
693+ no changes.
694+
695+ When writing output to the terminal we get the encoding of the
696+ terminal (utf-8 these days). When we redirect or pipe the output of
697+ the program it is generally not possible to know what the input
698+ encoding of the receiving program is, the encoding when redirecting
699+ to a file will be None (Python 2.7) or UTF-8 (Python 3)
700+
701+ $ python2.7 -c "import sys; print sys.stdout.encoding" | cat
702+ None
703+
704+ $ python3.4 -c "import sys; print(sys.stdout.encoding)" | cat
705+ UTF-8
706+
707+ Source:
708+ https://wiki.python.org/moin/PrintFails#print.2C_write_and_Unicode_in_pre-3.0_Python
709+ """
710+
711+ if sys.version_info[0] < 3:
712+ encoding = codecs.getwriter(locale.getpreferredencoding())
713+ sys.stdout = encoding(sys.stdout)
714+
715
716 def main():
717- parser = OptionParser(usage=Usage)
718- parser.add_option("--sources-list", dest="sources_list",
719- help="sources.list file", metavar="FILE", type="string",
720- default=None)
721- parser.add_option("--cache-dir", dest="cache_d",
722- help="cache dir for info", metavar="DIR", type="string",
723- default=None)
724- parser.add_option("-v", "--verbose", action="count", dest="loglevel",
725- help="increase verbosity", default=0)
726-
727- (options, args) = parser.parse_args()
728-
729- if len(args) != 4:
730- parser.error('you must provide arch, release, and 2 manifest files')
731-
732- # logging module levels are 0,10,20,30 ...
733- loglevel = (6 - options.loglevel) * 10
734- logging.basicConfig(level=loglevel,format=logformat,stream=sys.stderr)
735-
736- (arch,release,mf_from, mf_to) = args
737-
738- cache_d = options.cache_d
739- if not options.cache_d:
740- cache_d = "./cache.%s-%s" % (release, arch)
741-
742- # index both manifests
743- h_from=hashmffile(mf_from)
744- h_to=hashmffile(mf_to)
745-
746- # fix up kernels so the pkg names match
747- kfixups = { }
748- kmatch = re.compile("linux-image-[0-9]")
749- for pkg,ver in h_to.iteritems():
750- # if this is a linux-image-* binary package do some hacks to make it
751- # look like h_from is the same (format like linux-image-2.6.32-32-virtual)
752- if kmatch.match(pkg):
753- img_type = pkg[pkg.rfind("-")+1:]
754- if pkg in h_from: continue
755- for fpkg, fver in h_from.iteritems():
756- if ( kmatch.match(fpkg) and fpkg.endswith("-%s" % img_type)):
757- kfixups[pkg] = fpkg
758-
759- for pkg_to, pkg_from in kfixups.iteritems():
760- h_from[pkg_to] = h_from[pkg_from]
761- del h_from[pkg_from]
762-
763- # find new packages in mf2
764- new={ }
765- for pkg,ver in h_to.iteritems():
766- if pkg not in h_from: new[pkg]=h_to[pkg]
767-
768- # find packages removed from mf1
769- removed={ }
770- for pkg,ver in h_from.iteritems():
771- if pkg not in h_to: removed[pkg]=h_from[pkg]
772-
773- # find modified packages
774- changed=[]
775- for pkg,ver in h_to.iteritems():
776- if pkg in h_from and h_from[pkg] != ver: changed.append(pkg)
777-
778- print "new: %s" % new
779- print "removed: %s" % removed
780- print "changed: %s" % changed
781-
782-
783- prep_cacheroot(cache_d,release)
784- apt.apt_pkg.config.set("Apt::Architecture",arch)
785- cache = None
786-
787- errors = [ ]
788-
789- # if modified packages
790- # download all changelogs from changelogs.
791- bin2src = { }
792- srcs = { }
793- if len(changed):
794- cache=get_cache(cache_d, cache)
795-
796- b2s_hash=get_bin2src(changed,cache)
797-
798- for pkg,src in b2s_hash.iteritems():
799- if src not in bin2src:
800- bin2src[src]=[]
801- bin2src[src].append(pkg)
802-
803- missing_cl=[]
804- for src,binlist in bin2src.iteritems():
805- if src not in srcs:
806- srcs[src] = { "changelog_file": "", "changeblocks": [ ] }
807-
808- try:
809- pkg_name = binlist[0].split(':')[0]
810- versions = cache[pkg_name].versions
811- except KeyError as e:
812- raise Exception("%s not in cache did not have versions info in cache")
813- binver_to=h_to[binlist[0]]
814- try:
815- srcver_to=versions[binver_to].source_version
816- except KeyError as e:
817- raise Exception("bin pkg %s from src %s version %s not in binver list (%s)" %
818- (binlist[0],src,binver_to,versions))
819-
820- binver_from=h_from[binlist[0]]
821- try:
822- srcver_from=versions[binver_from].source_version
823- except KeyError as e:
824- if binver_to == srcver_to:
825- srcver_from = binver_from
826- else:
827- raise UnknownSourceVersionError(
828- "unable to determine src version for %s/%s" %
829- (src,binver_from))
830-
831- try:
832- clogfile=getchangelog(src,srcver_to,cache_d)
833- srcs[src]["changelog_file"] = clogfile
834- chlog=Changelog(filecontents(clogfile))
835- srcs[src]["changeblocks"] = [ ]
836- start = False
837- end = False
838- for block in chlog:
839- if block.version == srcver_to:
840- start = True
841- if block.version == srcver_from:
842- end = True
843- break
844- if start:
845- srcs[src]["changeblocks"].append(block)
846- if not (start and end):
847- em = ("%s missing %s or %s." %
848- (clogfile, srcver_to, binver_from))
849- if apt.VersionCompare(binver_from, srcver_to) > 0:
850- em = "%s %s" % (em, "from version > to version")
851- raise ChangelogMissingVersion(em)
852-
853- except MissingChangelogError as e:
854- errors.append(e)
855- except ChangelogMissingVersion as e:
856- errors.append(e)
857- except UnknownSourceVersionError as e:
858- errors.append(e)
859-
860- for src,binlist in bin2src.iteritems():
861- print "==== %s: %s => %s ====" % (src, h_from[binlist[0]], h_to[binlist[0]])
862- print "==== %s" % ' '.join(binlist)
863- print_blocks(srcs[src]["changeblocks"])
864-
865- if errors:
866- print "**** Errors ****"
867- for e in errors: print e
868- # load each changelog
869+ """
870+ Given two manifest files for a particular release/arch, find the
871+ packages which have been added, removed, and changed. Print the
872+ changelog entries for the changed packages limited to changes between
873+ the two versions.
874+ """
875+
876+ stdout_force_unicode()
877+ options, (arch, release, manifest_from_filename,
878+ manifest_to_filename) = parse_args()
879+
880+ setup_logging(options.loglevel)
881+
882+ # index both manifests
883+ manifest_to = manifest_to_dict(manifest_to_filename)
884+ manifest_from = kernel_fixups(manifest_to_dict(manifest_from_filename),
885+ manifest_to)
886+
887+ new = find_added(manifest_from, manifest_to)
888+ removed = find_removed(manifest_from, manifest_to)
889+ changed = find_changed(manifest_from, manifest_to)
890+
891+ print("new: %s" % new)
892+ print("removed: %s" % removed)
893+ print("changed: %s" % changed)
894+
895+ # if modified packages, download all changelogs from changelogs.
896+ if changed:
897+ cache, cache_d = open_apt_cache(arch, release, options.cache_d)
898+ print_changelogs(cache, cache_d, manifest_from, manifest_to, changed)
899
900
901 if __name__ == "__main__":
902- main()
903+ main()

Subscribers

People subscribed via source and target branches