Merge lp:~rcj/vmbuilder/mfdiff_fixups into lp:~ubuntu-on-ec2/vmbuilder/mfdiff

[Robert C Jennings]

Modernization and fixes

Break up main(), modernize code, improve documentation and variable
naming, python3 compat, misc fixes, urllib to requests migration,
support non-x86 architectures, and add logging.

This requires python-six version newer than what is shipping in some
supported Ubuntu releases to use six.viewkeys. (>=1.10 is preferable)

 [ Robert C Jennings ]
 * Strip architecture from binary when searching cache
 * Copy rather than rename changelog files
 * Force output to UTF-8 at all times
 * Output utf-8 rather than ascii
 * Change exception rendering for python3 compatibility
 * Use the current directory for temp changelog files
 * Improve variable naming for changelog_url (was clog_url)
 * Support non-x86 architectures
 * Fix usage text to correct option order
 * Eliminate traceback if binary package has been renamed
   (no longer in apt cache)
 * Use a real temporary file
 * Fixups for code-review
 * Improve names in print_changelogs and store more to the sources dict
 * Break out changelog processing from main
 * More variable renames
 * Rearrange summary printout and changelog printing for later refactor
 * Name, spelling, and doc cleanup
 * Stop processing source package on exception
 * Unlink temp file only if it exists
 * Add more complete function docstrings
 * Additional refactoring to reduce size of main
 * isort requests
 * Update authors/copyright
 * Add logging
 * Simplify cmdline --verbose option
 * Swap urllib for requests
 * Improve variable naming
 * Consistent sorted output
 * python3 compatibility changes
 * Document functions, use 'with' for files, and minor fixes
 * [merge] PEP8 cleanup, example updates, & python-apt updates
   * Correct vi line for tab expansion
   * README: Remove trailing spaces
   * Use host apt-secure keychains
   * APT version comparison moved from apt to apt_pkg
   * Remove unused cmdline option
   * Update README to use modern examples
   * Move comment to proper function
   * remove unused imports/variables
   * sort imports
   * pep8: autopep8 changes
   * pep8: tab -> space conversion

[Robert C Jennings (rcj)]

Break up main(), modernize code, improve documentation and variable naming, python3 compat, misc fixes, urllib to requests migration, support non-x86 architectures, and add logging.

There are a fair number of commits here and this is best reviewed by commit rather than with the MP diff view.

[Robert C Jennings (rcj)]

This requires python-six version newer than what is shipping in Trusty to use viewkeys. (1.10 is preferable)

[Dan Watkins (oddbloke)]

review: Approve

[Francis Ginther (fginther)]

Approve, this MP and resulting mfdiff are identical to a branch I already reviewed and tested.

review: Approve

=== modified file 'README'
--- README	2010-12-13 15:23:43 +0000
+++ README	2017-06-27 19:03:39 +0000
@@ -1,15 +1,14 @@
 "Manifest Diff" (mfdiff) is a tool for determining what changed between
-2 builds by looking at their manifests.  A manifest is simply a 
+2 builds by looking at their manifests.  A manifest is simply a
 list of package and versions in a build, obtainable by:
   dpkg-query', '-W', '--showformat=${Package} ${Version}\n'
 
 Example of running:
-  mf1=mfs/lucid/release-20100827/ubuntu-10.04-server-uec-amd64.manifest
-  mf2=mfs/lucid/release-20100923/ubuntu-10.04-server-uec-amd64.manifest
-  ./mfdiff i386 lucid "${mf1}" "${mf2}"
-
-
-
-sync manifests from uec-images.ubuntu.com with:
-   rsync -av uec-images.ubuntu.com::uec-images/ --prune-empty-dirs  mfs \
-      --filter "+ */" --filter "+ *.manifest" --filter "- *"  
+  mf1=mfs/releases/trusty/release-20160314/ubuntu-14.04-server-cloudimg-amd64.manifest
+  mf2=mfs/releases/trusty/release-20170517/ubuntu-14.04-server-cloudimg-amd64.manifest
+  ./mfdiff amd64 trusty "${mf1}" "${mf2}"
+
+
+sync manifests from cloud-images.ubuntu.com with:
+   rsync -av cloud-images.ubuntu.com::cloud-images/ --prune-empty-dirs  mfs \
+      --filter "+ */" --filter "+ *.manifest" --filter "- *"

=== modified file 'mfdiff'
--- mfdiff	2014-10-03 11:27:32 +0000
+++ mfdiff	2017-06-27 19:03:39 +0000
@@ -1,9 +1,17 @@
 #!/usr/bin/env python
-# vi:ts=4 noexpandtab
+"""
+Given two manifest files for a particular release/arch, find the
+packages which have been added, removed, and changed.  Print the
+changelog entries for the changed packages limited to changes between
+the two versions.
+"""
+
+# vi:ts=4 expandtab
 #
-#    Copyright (C) 2010 Canonical Ltd.
+#    Copyright (C) 2010, 2017 Canonical Ltd.
 #
 #    Authors: Scott Moser <[e-mail]>
+#             Robert C Jennings <[e-mail]>
 #
 #    This program is free software: you can redistribute it and/or modify
 #    it under the terms of the GNU General Public License as published by
@@ -17,288 +25,581 @@
 #    You should have received a copy of the GNU General Public License
 #    along with this program.  If not, see <http://www.gnu.org/licenses/>.
 
+import codecs
+import locale
+import logging
 import os
-import string
+import os.path
+import re
+import shutil
 import sys
+import tempfile
 from optparse import OptionParser
-import urllib
-import logging
-import os.path
-import os
-from debian.changelog import Changelog, ChangeBlock
+
 import apt
-import re
-
-Usage="""
-Usage: %s --apt-sources F arch manifest1 manifest2
-   Compare two manifest files, and show changelog differences.
-"""
-logformat="%(asctime)s %(name)s/%(levelname)s: %(message)s"
+import requests
+from debian.changelog import Changelog
+from six import iteritems, viewkeys
+
+try:
+    from apt import VersionCompare as version_compare
+except ImportError:
+    from apt_pkg import version_compare as version_compare
+
 
 class MissingChangelogError(Exception):
-	pass
+    "The Changelog file could not be found on the server"
+    pass
+
 
 class ChangelogMissingVersion(Exception):
-	pass
+    "The Changelog is missing starting and/or ending versions for the search"
+    pass
+
 
 class UnknownSourceVersionError(Exception):
-	pass
-
-def get_bin2src(changed,cache):
-	ret = { }
-	for binpkg in changed:
-		pkg_name = binpkg.split(':')[0]
-		ret[binpkg]=cache[pkg_name].versions[0].source_name
-	return(ret)
-
-# downloads changelog for src/ver and returns path to that
-def getchangelog(src,ver,cache_d):
-    # http://changelogs.ubuntu.com/changelogs/pool/main/h/hal/hal_0.5.5.1-1ubuntu2/changelog
-	clog_burl="http://changelogs.ubuntu.com/changelogs/pool"
-	urls=[]
-	bsize=2048
-	cache_f = "%s/changelog.%s_%s" % (cache_d, src,ver)
-
-	if os.path.isfile(cache_f):
-		return(cache_f)
-
-	furls=[ ]
-	num_colon_m=re.compile("[0-9]:")
-
-	cache_tmp = "%s/.changelog.%s_%s" % (cache_d, src,ver)
-	for pile in ("main", "universe", "multiverse", "restricted"):
-		pre=src[0:1]
-		# packages starting with 'lib' are special
-		if src.startswith("lib"): pre=src[0:4]
-
-		# packages with '1:' versions have different paths
-		# update-manager at version 1:0.134.11.
-		# would be at main/u/update-manager/update-manager_1:0.134.11
-		# instead at  main/u/update-manager/update-manager_0.134.11/
-		xver=ver
-		if num_colon_m.match(ver):
-			xver=ver[2:]
-
-		clog_url = "%s/%s/%s/%s/%s_%s/changelog" % \
-			( clog_burl, pile, pre, src, src, xver)
-
-		fp = open(cache_tmp,"w")
-		u_fp = urllib.urlopen(clog_url)
-		while True:
-			buf = u_fp.read(bsize)
-			fp.write(buf)
-			if len(buf) != bsize: break
-		fp.close()
-		if u_fp.getcode() == 200:
-			os.rename(cache_tmp,cache_f)
-			return(cache_f)
-		print "missing %s: %s" % (src,clog_url)
-		furls.append(clog_url)
-	os.unlink(cache_tmp)
-	raise(MissingChangelogError("Failed to find changelog for %s at version %s.\n tried %s" % (src, ver, ' '.join(furls))))
-
-def hashmffile(filename):
-	fp=open(filename,"r")
-	lines=fp.readlines()
-	fp.close()
-	ret={ }
-	for line in lines:
-		(pkg,ver)=line.split()
-		ret[pkg]=ver
-	return(ret)
-
-def filecontents(filename):
-	fp=open(filename,"r")
-	contents=fp.read()
-	fp.close()
-	return(contents)
-
-def prep_cacheroot(cache_d, release):
-	mirror = "http://archive.ubuntu.com/ubuntu/"
-    pockets = ( release, "%s-updates" % release, "%s-security" % release,
-		"%s-proposed" % release, )
-    components = ( "main", "universe" )
-	srclines = []
-	for pocket in pockets:
-		srclines.append("deb %s %s %s" % (mirror, pocket, ' '.join(components)))
-	try:
-		os.makedirs( "%s/etc/apt" % cache_d )
-	except:
-		pass
-	asl = open( "%s/etc/apt/sources.list" % cache_d, "w" )
-	asl.write('\n'.join(srclines))
-	asl.close()
-
-def get_cache(cache_d, cache=None):
-	if cache: return cache
-	cache = apt.Cache(rootdir=cache_d)
-	cache.update()
-	cache.open()
-	return cache
-
-# render a changelog block to something printable
+    "The binary package did not have a source version listed"
+    pass
+
+
+def get_bin2src(packages, cache):
+    """
+    Find the source package names for a given binary package list
+
+    :param list packages: List of binary packages
+    :param :class:`apt.Cache` cache: Open/updated apt cache
+    :return: List mapping binary package name to source package name
+    :rtype: dict
+    """
+
+    ret = {}
+    logging.debug('Finding source package names for all binary packages')
+    for binpkg in packages:
+        pkg_name = binpkg.split(':')[0]
+        ret[binpkg] = cache[pkg_name].versions[0].source_name
+    return ret
+
+
+def get_changelog(source, version, changelog_cache_d):
+    """
+    Download changelog for source / version and returns path to that
+
+    :param str source: Source package name
+    :param str version: Source package version
+    :param str changelog_cache_d: path to store cached changelogs
+    :raises MissingChangelogError: If changelog file could not be downloaded
+    :return: changelog file for source package & version
+    :rtype: str
+    """
+
+    cache_f = "%s/changelog.%s_%s" % (changelog_cache_d, source, version)
+
+    if os.path.isfile(cache_f):
+        logging.debug("Using cached changelog for %s:%s", source, version)
+        return cache_f
+
+    furls = []
+    num_colon_m = re.compile("[0-9]:")
+
+    cache_tmp_fd, cache_tmp_path = tempfile.mkstemp(
+        ".changelog.%s_%s" % (source, version),
+        prefix="." + tempfile.gettempprefix(), dir=".")
+    cache_tmp = os.fdopen(cache_tmp_fd, "w")
+    for pile in ("main", "universe", "multiverse", "restricted"):
+        pre = source[0:1]
+        # packages starting with 'lib' are special
+        if source.startswith("lib"):
+            pre = source[0:4]
+
+        # packages with '1:' versions have different paths
+        # update-manager at version 1:0.134.11.
+        # is main/u/update-manager/update-manager_0.134.11/
+        # rather than main/u/update-manager/update-manager_1:0.134.11
+        url_version = version
+        if num_colon_m.match(version):
+            url_version = version[2:]
+
+        # Changelog URL example http://changelogs.ubuntu.com/changelogs/\
+        #                       pool/main/h/hal/hal_0.5.5.1-1ubuntu2/changelog
+        changelog_url = "http://changelogs.ubuntu.com/changelogs/pool/" \
+                        "%s/%s/%s/%s_%s/changelog" % \
+            (pile, pre, source, source, url_version)
+
+        changelog = requests.get(changelog_url)
+        if changelog.status_code == 200:
+            cache_tmp.write(changelog.content)
+            cache_tmp.close()
+            shutil.copy2(cache_tmp_path, cache_f)
+            os.unlink(cache_tmp_path)
+            return cache_f
+        else:
+            logging.error("missing %s: %s", source, changelog_url)
+        furls.append(changelog_url)
+    if os.path.exists(cache_tmp_path):
+        os.unlink(cache_tmp_path)
+    raise MissingChangelogError("Failed to find changelog for %s at version "
+                                "%s.\n tried %s" % (source, version,
+                                                    ' '.join(furls)))
+
+
+def manifest_to_dict(filename):
+    """
+    Parse manifest file to create a package / version mapping
+
+    :param str filename: Name of package manifest file
+    :return: List of package versions by name
+    :rtype: dict
+    """
+
+    ret = {}
+    logging.debug('Reading package manifest from %s', filename)
+    with open(filename, "r") as manifest:
+        for line in manifest:
+            (pkg, ver) = line.split()
+            ret[pkg] = ver
+    return ret
+
+
+def open_apt_cache(arch, release, cache_d=None):
+    """
+    Create, update, and open an apt cache.
+
+    This creates an apt cache directory and write a sources.list file
+    before updating and opening the cache.  The caller is responsible
+    for closing the cache.
+
+    :param str arch: Package architecture
+    :param str release: Ubuntu release name (e.g. Xenial)
+    :param str cache_d: apt cache path
+    :returns: tuple of Open/updated apt cache and cache path name
+    :rtype: tuple(:class:`apt.Cache`, str)
+    """
+
+    if not cache_d:
+        cache_d = "./cache.%s-%s" % (release, arch)
+    logging.info("Using %s as the apt cache directory", cache_d)
+
+    if arch in ['amd64', 'i386']:
+        mirror = "http://archive.ubuntu.com/ubuntu/"
+    else:
+        mirror = "http://ports.ubuntu.com/ubuntu-ports/"
+    logging.debug('Configuring apt cache using mirror %s', mirror)
+
+    pockets = (release, "%s-updates" % release, "%s-security" % release,
+               "%s-proposed" % release, )
+    components = ("main", "universe")
+    srclines = []
+    for pocket in pockets:
+        srcline = "deb %s %s %s" % (mirror, pocket, ' '.join(components))
+        logging.debug('Adding source: %s', srcline)
+        srclines.append(srcline)
+    try:
+        os.makedirs("%s/etc/apt" % cache_d)
+    except OSError as oserror:
+        if os.errno.EEXIST != oserror.errno:
+            raise
+    with open("%s/etc/apt/sources.list" % cache_d, "w") as asl:
+        asl.write('\n'.join(srclines))
+
+    apt.apt_pkg.config.set("Apt::Architecture", arch)
+
+    logging.debug('Using host apt keys for signature verification')
+    apt.apt_pkg.config.set("Dir::Etc::Trusted", "/etc/apt/trusted.gpg")
+    apt.apt_pkg.config.set("Dir::Etc::TrustedParts",
+                           "/etc/apt/trusted.gpg.d/")
+
+    cache = apt.Cache(rootdir=cache_d)
+    logging.info('Updating apt cache')
+    cache.update()
+    logging.info('Update of apt cache complete')
+    cache.open()
+    return cache, cache_d
+
+
 def render_block(block):
-	return('\n'.join([x.encode('ascii', 'replace') for x in block.changes() if x]))
-
-def print_blocks(blist):
-	# print a blocklist
-	for block in blist:
-		print render_block(block)
+    """
+    Render a changelog block to something printable (dropping blank lines)
+
+    :param :class:`debian.changelog.ChangeBlock` block: Changelog block
+    :return: String containing the changelog block text
+    :rtype: str
+    """
+    return '\n'.join([x for x in block.changes() if x])
+
+
+def print_blocks(block_list):
+    """
+    Print a Changelog block list
+
+    :param list block_list: List of :class:`debian.changelog.ChangeBlock`
+    """
+
+    for block in block_list:
+        print(render_block(block).encode('utf-8').decode('utf-8'))
+
+
+def kernel_fixups(manifest_from, manifest_to):
+    """
+    Fix up kernels so the pkg names match
+
+    Kernel package names change from release to release so that they are
+    co-installable, but we need to find matching package names in the
+    two manifests to provide a list of changes between the versions of the
+    package in each manifest.
+    This function will return an altered version of manifest_from with kernel
+    package names changed to match kernel package names in manifest_to.  This
+    will support later version comparisons.
+
+    :param dict manifest_from: Dictionary mapping package to version for of
+        the starting manifest
+    :param dict manifest_to: Dictionary mapping package to version for the
+        ending manifest
+    :return: Starting manifest dictionary with altered kernel package names
+        to match names in ending manifest
+    :rtype dict:
+    """
+    kfixups = {}
+    kmatch = re.compile("linux-image-[0-9]")
+    for pkg in manifest_to:
+        # if this is a linux-image-* binary package do some hacks to make it
+        # look like manifest_from is the same (format like
+        # linux-image-2.6.32-32-virtual)
+        if kmatch.match(pkg):
+            logging.debug('Found kernel %s in manifest #2', pkg)
+            img_type = pkg.split("-")[-1]
+            if pkg in manifest_from:
+                logging.debug('Found same kernel in manifest #1')
+                continue
+            for fpkg in manifest_from:
+                if kmatch.match(fpkg) and fpkg.endswith("-%s" % img_type):
+                    logging.debug('Found similar kernel %s in manifest #1',
+                                  fpkg)
+                    kfixups[pkg] = fpkg
+
+    for pkg_to, pkg_from in iteritems(kfixups):
+        logging.debug('Substituting kernel %s for %s in manifest #1 to '
+                      'enable version comparison', pkg_to, pkg_from)
+        manifest_from[pkg_to] = manifest_from[pkg_from]
+        del manifest_from[pkg_from]
+    return manifest_from
+
+
+def find_added(manifest_from, manifest_to):
+    "Find new packages in manifest_to"
+    new = {}
+    for pkg in sorted(viewkeys(manifest_to) - viewkeys(manifest_from)):
+        logging.debug('New package: %s', pkg)
+        new[pkg] = manifest_to[pkg]
+    return new
+
+
+def find_removed(manifest_from, manifest_to):
+    "Find packages removed from manifest_from"
+    removed = {}
+    for pkg in sorted(viewkeys(manifest_from) - viewkeys(manifest_to)):
+        logging.debug('Removed package: %s', pkg)
+        removed[pkg] = manifest_from[pkg]
+    return removed
+
+
+def find_changed(manifest_from, manifest_to):
+    "Find modified packages"
+    changed = []
+    for pkg in sorted(viewkeys(manifest_from) & viewkeys(manifest_to)):
+        if manifest_from[pkg] != manifest_to[pkg]:
+            logging.debug('Changed package: %s', pkg)
+            changed.append(pkg)
+    return changed
+
+
+def map_source_to_binary(cache, packages):
+    "Create a dictionary of source to list of binary packages"
+    src2bins = {}
+    for bin_pkg in packages:
+        bin_name = bin_pkg.split(':')[0]
+        if bin_name in cache:
+            src2bins.setdefault(
+                cache[bin_name].versions[0].source_name, []).append(bin_pkg)
+    return src2bins
+
+
+def get_pkg_versions(cache, binary):
+    "Get  all known versions from the apt cache"
+    pkg_name = binary.split(':')[0]
+    try:
+        return cache[pkg_name].versions
+    except KeyError:
+        raise Exception(
+            "%s not in cache or did not have version info in cache" %
+            pkg_name)
+
+
+def source_version_for_binary(cache, binary, binary_ver):
+    "Find the source version data for a specific binary version"
+    versions = get_pkg_versions(cache, binary)
+    try:
+        return versions[binary_ver].source_version
+    except KeyError:
+        # Strip the architecture name from the binary
+        source_name = cache[binary.split(':')[0]].versions[0].source_name
+        msg = ("Unable to determine source version for %s. "
+               "Binary package %s/%s not in known source version "
+               "list (%s)" % (source_name, binary, binary_ver, versions))
+        raise UnknownSourceVersionError(msg)
+
+
+def filter_changelog(changelog_path, version_low, version_high):
+    """
+    Extract changelog entries within a version range
+
+    The range of changelog entries returned will include all entries
+    after version_low up to, and including, version_high.
+    If either the starting or ending version are not found in the
+    list of changelog entries the result will be incomplete and
+    a non-empty error message is returned to indicate the issue.
+
+    :param str changelog_path: File name of the changelog to process
+    :return: list of changelog blocks and an error_msg if incomplete
+    :rtype tuple(list, str):
+    """
+
+    with open(changelog_path, "r") as fileptr:
+        chlog = Changelog(fileptr.read())
+    change_blocks = []
+    start = False
+    end = False
+    error_msg = ''
+
+    # The changelog blocks are in reverse order; we'll see high before low.
+    for block in chlog:
+        if block.version == version_high:
+            start = True
+            change_blocks = []
+        if block.version == version_low:
+            end = True
+            break
+        change_blocks.append(block)
+    if not start:
+        error_msg = "Missing starting version {} in {}. " \
+                    "Changlelog will be incomplete".format(
+                        version_high, changelog_path)
+        logging.error(error_msg)
+    if not end:
+        if error_msg:
+            # Start and end were not found, put a newline between their
+            # error messages
+            error_msg += '\n'
+        error_msg += "Missing ending version {} in {}. " \
+                     "Changelog output truncated".format(
+                         version_low, changelog_path)
+        logging.error(error_msg)
+    return change_blocks, error_msg
+
+
+def print_changelogs(apt_cache, apt_cache_d, manifest_from, manifest_to,
+                     changed):
+    """
+    Print changelog entries for each changed package limited to
+    changes in the package between the versions in the two manifests.
+
+    :param :class:`apt.Cache` apt_cache: Open & up-to-date apt cache
+    :param str apt_cache_d: apt cache path
+    :param dict manifest_from: Packages and their versions in the
+        first manifest file
+    :param dict manifest_to: Packages and their versions in the
+        second manifest file
+    :param list changed: Packages which changed between the two manifests
+    """
+
+    srcs = {}
+    errors = []
+    src2bins = map_source_to_binary(apt_cache, changed)
+
+    # Generate changelog data per unique source package
+    for source_name in src2bins:
+        srcs[source_name] = {"changelog_file": "", "changeblocks": []}
+        src = srcs[source_name]
+
+        # Use the first binary listed for a source package
+        binary_name = src2bins[source_name][0]
+
+        # Find the source version data for the binary in manifest #2
+        try:
+            src['version_to'] = source_version_for_binary(
+                apt_cache, binary_name, manifest_to[binary_name])
+        except UnknownSourceVersionError as excp:
+            logging.error(str(excp))
+            errors.append(excp)
+            continue
+
+        # Find the source version data for the binary in manifest #1
+        binver_from = manifest_from[binary_name]
+        try:
+            src['version_from'] = source_version_for_binary(
+                apt_cache, binary_name, binver_from)
+        except UnknownSourceVersionError as excp:
+            if manifest_to[binary_name] == src['version_to']:
+                logging.info('Could not find source version data in apt '
+                             'cache. Assuming source %s version %s from '
+                             'binary %s', source_name, binver_from,
+                             binary_name)
+                src['version_from'] = binver_from
+            else:
+                logging.error(str(excp))
+                errors.append(excp)
+                continue
+
+        # Check for version regression between manifests
+        try:
+            if version_compare(src['version_from'], src['version_to']) > 0:
+                msg = "Package version regression {} -> {}".format(
+                    src['version_from'], src['version_to'])
+                raise UnknownSourceVersionError(msg)
+        except UnknownSourceVersionError as excp:
+            errors.append(excp)
+            continue
+
+        # Get the changelog for this source package
+        try:
+            # Use the apt cache directory to store the changelog cache
+            srcs[source_name]["changelog_file"] = get_changelog(
+                source_name, src['version_to'], changelog_cache_d=apt_cache_d)
+        except MissingChangelogError as excp:
+            errors.append(excp)
+            continue
+
+        # Filter the changelog to a list of blocks between the two versions
+        try:
+            srcs[source_name]["changeblocks"], incomplete = filter_changelog(
+                srcs[source_name]["changelog_file"], src['version_from'],
+                src['version_to'])
+            if incomplete:
+                raise ChangelogMissingVersion(incomplete)
+        except ChangelogMissingVersion as exp:
+            errors.append(exp)
+            continue
+
+    # Print changelog ranges for changed packages
+    for source_name in sorted(src2bins):
+        binlist = sorted(src2bins[source_name])
+        binary = binlist[0]
+        print("==== %s: %s => %s ====" %
+              (source_name, manifest_from[binary], manifest_to[binary]))
+        print("====     %s" % ' '.join(binlist))
+        print_blocks(srcs[source_name]["changeblocks"])
+
+    if errors:
+        print("**** Errors ****")
+        for error in errors:
+            print(error)
+
+
+def parse_args():
+    """
+    Parse command line arguments
+
+    :returns: options and remaining arguments from OptionParser.parse_args()
+    :rtype list:
+    """
+
+    parser = OptionParser(usage="Usage: {} arch suite manifest1 manifest2\n"
+                                "Compare two manifest files, and show "
+                                "changelog differences."
+                          .format(os.path.basename(sys.argv[0])))
+    parser.add_option("--cache-dir", dest="cache_d",
+                      help="cache dir for info", metavar="DIR", type="string",
+                      default=None)
+    parser.add_option("-v", "--verbose", action="count", dest="loglevel",
+                      help="increase verbosity", default=0)
+
+    (options, args) = parser.parse_args()
+
+    if len(args) != 4:
+        parser.error('you must provide arch, release, and 2 manifest files')
+
+    return options, args
+
+
+def setup_logging(loglevel=0):
+    """
+    Configure logging
+
+    By default, log WARNING and higher messages
+    :param int: loglevel 0: Warning, 1: Info, 2: Debug
+    """
+
+    loglevel = [logging.WARNING,
+                logging.INFO,
+                logging.DEBUG][min(2, loglevel)]
+
+    logging.basicConfig(
+        level=loglevel,
+        format="%(asctime)s %(name)s/%(levelname)s: %(message)s",
+        stream=sys.stderr)
+
+
+def stdout_force_unicode():
+    """
+    Force output to UTF-8 at all times
+
+    We want to output UTF-8 at all times to preserve Unicode characters
+    in the changelog blocks.  For Python 2 we will wrap sys.stdout with
+    an instance of StreamWriter with our preferred coding.  Python3 requires
+    no changes.
+
+    When writing output to the terminal we get the encoding of the
+    terminal (utf-8 these days).  When we redirect or pipe the output of
+    the program it is generally not possible to know what the input
+    encoding of the receiving program is, the encoding when redirecting
+    to a file will be None (Python 2.7) or UTF-8 (Python 3)
+
+    $ python2.7 -c "import sys; print sys.stdout.encoding" | cat
+    None
+
+    $ python3.4 -c "import sys; print(sys.stdout.encoding)" | cat
+    UTF-8
+
+    Source:
+    https://wiki.python.org/moin/PrintFails#print.2C_write_and_Unicode_in_pre-3.0_Python
+    """
+
+    if sys.version_info[0] < 3:
+        encoding = codecs.getwriter(locale.getpreferredencoding())
+        sys.stdout = encoding(sys.stdout)
+
 
 def main():
-	parser = OptionParser(usage=Usage)
-	parser.add_option("--sources-list", dest="sources_list",
-		help="sources.list file", metavar="FILE", type="string",
-		default=None)
-	parser.add_option("--cache-dir", dest="cache_d",
-		help="cache dir for info", metavar="DIR", type="string",
-		default=None)
-	parser.add_option("-v", "--verbose", action="count", dest="loglevel",
-		help="increase verbosity", default=0)
-
-	(options, args) = parser.parse_args()
-
-	if len(args) != 4:
-		parser.error('you must provide arch, release, and 2 manifest files')
-
-	# logging module levels are 0,10,20,30 ...
-	loglevel = (6 - options.loglevel) * 10
-	logging.basicConfig(level=loglevel,format=logformat,stream=sys.stderr)
-
-	(arch,release,mf_from, mf_to) = args
-
-	cache_d = options.cache_d
-	if not options.cache_d:
-		cache_d = "./cache.%s-%s" % (release, arch)
-
-	# index both manifests
-	h_from=hashmffile(mf_from)
-	h_to=hashmffile(mf_to)
-
-	# fix up kernels so the pkg names match
-	kfixups = { }
-	kmatch = re.compile("linux-image-[0-9]")
-	for pkg,ver in h_to.iteritems():
-		# if this is a linux-image-* binary package do some hacks to make it
-		# look like h_from is the same (format like linux-image-2.6.32-32-virtual)
-		if kmatch.match(pkg):
-			img_type = pkg[pkg.rfind("-")+1:]
-			if pkg in h_from: continue
-			for fpkg, fver in h_from.iteritems():
-				if ( kmatch.match(fpkg) and fpkg.endswith("-%s" % img_type)):
-					kfixups[pkg] = fpkg
-
-	for pkg_to, pkg_from in kfixups.iteritems():
-		h_from[pkg_to] = h_from[pkg_from]
-		del h_from[pkg_from]
-
-	# find new packages in mf2
-	new={ }
-	for pkg,ver in h_to.iteritems():
-		if pkg not in h_from: new[pkg]=h_to[pkg]
-
-	# find packages removed from mf1
-	removed={ }
-	for pkg,ver in h_from.iteritems():
-		if pkg not in h_to: removed[pkg]=h_from[pkg]
-
-	# find modified packages
-	changed=[]
-	for pkg,ver in h_to.iteritems():
-		if pkg in h_from and h_from[pkg] != ver: changed.append(pkg)
-
-	print "new: %s" % new
-	print "removed: %s" % removed
-	print "changed: %s" % changed
-
-
-    prep_cacheroot(cache_d,release)
-    apt.apt_pkg.config.set("Apt::Architecture",arch)
-	cache = None
-
-	errors = [ ]
-
-	# if modified packages
-	#   download all changelogs from changelogs.
-	bin2src = { }
-	srcs = { }
-	if len(changed):
-		cache=get_cache(cache_d, cache)
-
-		b2s_hash=get_bin2src(changed,cache)
-
-		for pkg,src in b2s_hash.iteritems():
-			if src not in bin2src:
-				bin2src[src]=[]
-			bin2src[src].append(pkg)
-
-		missing_cl=[]
-		for src,binlist in bin2src.iteritems():
-			if src not in srcs:
-				srcs[src] = { "changelog_file": "", "changeblocks": [ ] }
-
-			try:
-				pkg_name = binlist[0].split(':')[0]
-				versions = cache[pkg_name].versions
-			except KeyError as e:
-				raise Exception("%s not in cache did not have versions info in cache")
-			binver_to=h_to[binlist[0]]
-			try:
-				srcver_to=versions[binver_to].source_version
-			except KeyError as e:
-				raise Exception("bin pkg %s from src %s version %s not in binver list (%s)" %
-					(binlist[0],src,binver_to,versions))
-
-			binver_from=h_from[binlist[0]]
-			try:
-				srcver_from=versions[binver_from].source_version
-			except KeyError as e:
-				if binver_to == srcver_to:
-					srcver_from = binver_from
-				else:
-					raise UnknownSourceVersionError(
-						"unable to determine src version for %s/%s" %
-						(src,binver_from))
-
-			try:
-				clogfile=getchangelog(src,srcver_to,cache_d)
-				srcs[src]["changelog_file"] = clogfile
-				chlog=Changelog(filecontents(clogfile))
-				srcs[src]["changeblocks"] = [ ]
-				start = False
-				end = False
-				for block in chlog:
-					if block.version == srcver_to:
-						start = True
-					if block.version == srcver_from:
-						end = True
-						break
-					if start:
-						srcs[src]["changeblocks"].append(block)
-				if not (start and end):
-					em = ("%s missing %s or %s." %
-					      (clogfile, srcver_to, binver_from))
-					if apt.VersionCompare(binver_from, srcver_to) > 0:
-						em = "%s %s" % (em, "from version > to version")
-					raise ChangelogMissingVersion(em)
-
-			except MissingChangelogError as e:
-				errors.append(e)
-			except ChangelogMissingVersion as e:
-				errors.append(e)
-			except UnknownSourceVersionError as e:
-				errors.append(e)
-
-		for src,binlist in bin2src.iteritems():
-			print "==== %s: %s => %s ====" % (src, h_from[binlist[0]], h_to[binlist[0]])
-			print "====     %s" % ' '.join(binlist)
-			print_blocks(srcs[src]["changeblocks"])
-
-	if errors:
-		print "**** Errors ****"
-		for e in errors: print e
-	# load each changelog
+    """
+    Given two manifest files for a particular release/arch, find the
+    packages which have been added, removed, and changed.  Print the
+    changelog entries for the changed packages limited to changes between
+    the two versions.
+    """
+
+    stdout_force_unicode()
+    options, (arch, release, manifest_from_filename,
+              manifest_to_filename) = parse_args()
+
+    setup_logging(options.loglevel)
+
+    # index both manifests
+    manifest_to = manifest_to_dict(manifest_to_filename)
+    manifest_from = kernel_fixups(manifest_to_dict(manifest_from_filename),
+                                  manifest_to)
+
+    new = find_added(manifest_from, manifest_to)
+    removed = find_removed(manifest_from, manifest_to)
+    changed = find_changed(manifest_from, manifest_to)
+
+    print("new: %s" % new)
+    print("removed: %s" % removed)
+    print("changed: %s" % changed)
+
+    # if modified packages, download all changelogs from changelogs.
+    if changed:
+        cache, cache_d = open_apt_cache(arch, release, options.cache_d)
+        print_changelogs(cache, cache_d, manifest_from, manifest_to, changed)
 
 
 if __name__ == "__main__":
-	main()
+    main()