Merge lp:~ursinha/ubuntu-cdimage/add-manifest-changelog-libs into lp:ubuntu-cdimage

[Ursula Junque]

This branch adds two new libs to cdimage: manifest.py and changelog.py, required to generate between-images changelogs by parsing their manifests and fetching packages' changelog information. manifest.py also contains the methods previously declared in diff-manifest, that now only calls such methods. Tests were added.

[lp:~ursinha/ubuntu-cdimage/add-manifest-changelog-libs]

1341. By Ursula Junque on 2013-08-30

lib/cdimage/manifest.py: printing all the changed packages in the changed packages session, regardless they have a changelog or not

1342. By Ursula Junque on 2013-08-30

lib/cdimage/manifest.py: sometimes the changelogs are manually (?) changed and versions are removed from it, added a check to return only the newest changelog block in case that happens

1343. By Ursula Junque on 2013-08-30

lib/cdimage/tests/test_manifest.py: testing whenever the old source version
doesn't exist in the current package changelog, which may happen due to a
debian sync for example

1344. By Ursula Junque on 2013-08-30

lib/cdimage/manifest.py: making it clear what are binary versions and source
versions; also adding some docs to the methods

1345. By Ursula Junque on 2013-08-30

lib/cdimage/manifest.py: fixing the changed packages output

[Steve Langasek (vorlon)]

Unfortunately this MP never got reviewed, and in the meantime we are moving the source to git which means this would need to be resubmitted. It is not clear to me what the application of this new code was supposed to be so it is not a priority for me to review and land this change prior to migration to git. Please resubmit against git if it's still relevant.

review: Needs Resubmitting

Unmerged revisions

1345. By Ursula Junque on 2013-08-30

lib/cdimage/manifest.py: fixing the changed packages output

1344. By Ursula Junque on 2013-08-30

lib/cdimage/manifest.py: making it clear what are binary versions and source
versions; also adding some docs to the methods

1343. By Ursula Junque on 2013-08-30

lib/cdimage/tests/test_manifest.py: testing whenever the old source version
doesn't exist in the current package changelog, which may happen due to a
debian sync for example

1342. By Ursula Junque on 2013-08-30

lib/cdimage/manifest.py: sometimes the changelogs are manually (?) changed and versions are removed from it, added a check to return only the newest changelog block in case that happens

1341. By Ursula Junque on 2013-08-30

lib/cdimage/manifest.py: printing all the changed packages in the changed packages session, regardless they have a changelog or not

1340. By Ursula Junque on 2013-07-30

bin/image-changelog: calling manifest.parse_file; lib/cdimage/manifest.py:
adding text to differentiate packages with changelogs (found in
changelogs.ubuntu.com) from the others

1339. By Ursula Junque on 2013-07-30

tests/test_manifest.py: testing manifest_diff.get_changes()

1338. By Ursula Junque on 2013-07-30

lib/cdimage/manifest.py: using correct function to return the changes in the
changelog

1337. By Ursula Junque on 2013-07-30

Fixing pyflakes errors; closing the file when I extract the manifest content.

1336. By Ursula Junque on 2013-07-30

tests/*: fixing pep8 and flakes problems

=== modified file 'bin/diff-manifest'
--- bin/diff-manifest	2013-03-21 08:41:44 +0000
+++ bin/diff-manifest	2013-08-30 23:19:27 +0000
@@ -1,6 +1,6 @@
 #! /usr/bin/python
 #
-# Copyright (C) 2011, 2012 Canonical Ltd.
+# Copyright (C) 2011, 2013 Canonical Ltd.
 # Author: Kate Stewart <[e-mail]>
 # Author: Colin Watson <[e-mail]>
 
@@ -14,48 +14,11 @@
 from __future__ import print_function
 
 from optparse import OptionParser
+import os
 import sys
 
-
-def next_package(manifest):
-    """Get the next package name and version information from a manifest."""
-    while manifest:
-        line = manifest.pop(0)
-        if not line:
-            # silently skip empty lines
-            continue
-        bits = line.split()
-        if len(bits) < 2:
-            print("WARNING: '%s' not in format 'package version'" % line,
-                  file=sys.stderr)
-        else:
-            return bits
-    return None
-
-
-def manifest_pairs(ma, mb):
-    """Generate a current pair of packages, one from each manifest to compare.
-    """
-    left = next_package(ma)
-    right = next_package(mb)
-    while left and right:
-        if left[0] == right[0]:
-            yield left, right
-            left = next_package(ma)
-            right = next_package(mb)
-        elif left[0] < right[0]:
-            yield left, None
-            left = next_package(ma)
-        else:  # left[0] > right[0]
-            yield None, right
-            right = next_package(mb)
-    # drain off whichever file still has packages in it
-    while left:
-        yield left, None
-        left = next_package(ma)
-    while right:
-        yield None, right
-        right = next_package(mb)
+sys.path.insert(0, os.path.join(sys.path[0], os.pardir, "lib"))
+from cdimage.manifest import compare_manifests
 
 
 def main():
@@ -64,23 +27,7 @@
     if len(args) < 2:
         parser.error("must provide two manifest files to diff")
 
-    # create similarly ordered lists from each manifest file
-    with open(args[0]) as mfa:
-        ma = sorted(mfa.readlines())
-    with open(args[1]) as mfb:
-        mb = sorted(mfb.readlines())
-
-    # analyze relation between pairs of packages from manifest lists
-    for a, b in manifest_pairs(ma, mb):
-        if a and b:
-            if a[1] == b[1]:
-                print("= %s\t%s\t%s" % (a[0], a[1], b[1]))
-            else:
-                print("? %s\t%s\t%s" % (a[0], a[1], b[1]))
-        elif a:
-            print("< %s\t%s\t--" % (a[0], a[1]))
-        else:
-            print("> %s\t--\t%s" % (b[0], b[1]))
+    compare_manifests(args[0], args[1])
 
 
 if __name__ == '__main__':

=== added file 'bin/image-changelog'
--- bin/image-changelog	1970-01-01 00:00:00 +0000
+++ bin/image-changelog	2013-08-30 23:19:27 +0000
@@ -0,0 +1,49 @@
+#! /usr/bin/python
+
+# Copyright (C) 2013 Canonical Ltd.
+# Author: Ursula Junque <[e-mail]>
+
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; version 3 of the License.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+"""Generates a changelog of an image comparing its manifest with the previous
+one."""
+
+import os
+import sys
+from optparse import OptionParser
+
+sys.path.insert(0, os.path.join(sys.path[0], os.pardir, "lib"))
+from cdimage.manifest import Manifest, ManifestDiff
+
+
+def main():
+    parser = OptionParser(usage="Usage: %prog manifest1 manifest2")
+    _, args = parser.parse_args()
+    if len(args) < 2:
+        parser.error("must provide two manifest files to diff")
+
+    # TODO: look for the manifest paths and get them directly instead of
+    # passing as an argument
+
+    old_manifest = Manifest(args[0])
+    old_manifest.parse_file()
+    new_manifest = Manifest(args[1])
+    new_manifest.parse_file()
+
+    manifest_diff = ManifestDiff()
+    manifest_diff.compare_manifests(old_manifest, new_manifest)
+    manifest_diff.output_changes()
+
+
+if __name__ == '__main__':
+    main()

=== added file 'lib/cdimage/changelog.py'
--- lib/cdimage/changelog.py	1970-01-01 00:00:00 +0000
+++ lib/cdimage/changelog.py	2013-08-30 23:19:27 +0000
@@ -0,0 +1,160 @@
+# Copyright (C) 2013 Canonical Ltd.
+# Author: Ursula Junque <[e-mail]>
+
+# Utilities to parse and handle ubuntu changelogs.
+#
+# This program is free software: you can redistribute it and/or modify it
+# under the terms of the the GNU General Public License version 3, as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranties of
+# MERCHANTABILITY, SATISFACTORY QUALITY or FITNESS FOR A PARTICULAR
+# PURPOSE.  See the applicable version of the GNU Lesser General Public
+# License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+
+import urllib
+from debian import changelog
+
+
+CHANGELOG_WEBSITE = "http://changelogs.ubuntu.com/changelogs"
+# Currently only most recent series
+
+
+class BinaryToSourceLinkNotFound(Exception):
+    """Raised in case the "binary to source" url returns a 404."""
+
+
+class ChangelogNotFound(Exception):
+    """Raised in case the changelog url returns a 404."""
+    def __init__(self, name, version, url):
+        self.name = name
+        self.version = version
+        self.url = url
+
+    def __str__(self):
+        return "Changelog not found: %s_%s" % (self.name, self.version)
+
+
+class EmptyUrlError(Exception):
+    """Raised in case the url is None."""
+
+
+class VersionNotFound(Exception):
+    """An exception to signal a version isn't found in the changelog."""
+
+    def __init__(self, version):
+        self.version = str(version)
+
+    def __str__(self):
+        return "Version not found in changelog: %s" % self.version
+
+
+def get_source_changelog_url(binary_name, binary_version):
+    """Returns source changelog url given a binary name and version."""
+    if binary_name is None or binary_version is None:
+        return None
+    if binary_name.startswith("lib"):
+        prefix = binary_name[0:4]
+    else:
+        prefix = binary_name[0]
+    url = "%s/binary/%s/%s/%s/changelog" % (CHANGELOG_WEBSITE, prefix,
+                                            binary_name, binary_version)
+    return url
+
+
+def get_changelog_url(source_name, source_version, component):
+    """Returns source changelog url given source name and version."""
+    if source_name is None or source_version is None or component is None:
+        return None
+    if source_name.startswith("lib"):
+        prefix = source_name[0:4]
+    else:
+        prefix = source_name[0]
+    url = ("%s/pool/%s/%s/%s-%s/changelog" % (CHANGELOG_WEBSITE, component,
+           prefix, source_name, source_version))
+    return url
+
+
+def get_changelog(binary_name, binary_version):
+    """Returns a source Changelog object of a given binary and version.
+
+    Looks up the source name and version that originated the binary of version
+    provided. Retrieves the source url and parses it, returning a Changelog
+    object.
+    """
+    url = get_source_changelog_url(binary_name, binary_version)
+    if url is None:
+        raise EmptyUrlError()
+    try:
+        content = urllib.urlopen(url)
+        if content.code == 404:
+            raise ChangelogNotFound(binary_name, binary_version, url)
+        if content.code == 200:
+            return Changelog(content)
+        return None
+    except ChangelogNotFound:
+        raise
+
+
+class Changelog(changelog.Changelog):
+    """Extends deb822.changelog.Changelog.
+
+    Add ability to get a slice of changelog given the versions in between,
+    also get a block given a version.
+    """
+
+    def __init__(self, file_):
+        super(Changelog, self).__init__(file_)
+
+    @property
+    def source_name(self):
+        return self._blocks[0].package
+
+    @property
+    def raw_version(self):
+        """Return the last version string"""
+        return self._blocks[0]._raw_version
+
+    def has_version(self, version):
+        """Returns True if version is found in parsed changelog."""
+        return str(version) in self._raw_versions()
+
+    def get_changes_interval(self, start_version, end_version=None,
+                             include_start_version=True):
+        """Get all changelog blocks on an interval of versions.
+
+        Default is returning all changelog blocks in the given interval,
+        including the start version. If you want to get changes excluding the
+        start version, set include_start_version to False.
+        """
+        if end_version is None:
+            end_version = self.get_version()
+        elif not self.has_version(end_version):
+            raise VersionNotFound(end_version)
+        if not self.has_version(start_version):
+            raise VersionNotFound(start_version)
+
+        start_index = self._raw_versions().index(str(start_version))
+        end_index = self._raw_versions().index(str(end_version))
+
+        if include_start_version:
+            start_index += 1
+
+        return self._blocks[end_index:start_index]
+
+    def get_changes_after_version(self, start_version, up_to=None):
+        """Returns all changelog blocks after given version."""
+        return self.get_changes_interval(start_version, up_to,
+                                         include_start_version=False)
+
+    def get_change_for_version(self, version):
+        """Returns changeblock of a version."""
+        if self.has_version(version):
+            index = self._raw_versions().index(str(version))
+            return self._blocks[index]
+        raise VersionNotFound(version)

=== added file 'lib/cdimage/manifest.py'
--- lib/cdimage/manifest.py	1970-01-01 00:00:00 +0000
+++ lib/cdimage/manifest.py	2013-08-30 23:19:27 +0000
@@ -0,0 +1,318 @@
+# Copyright (C) 2011, 2013 Canonical Ltd.
+# Author: Kate Stewart <[e-mail]>
+# Author: Colin Watson <[e-mail]>
+# Author: Ursula Junque <[e-mail]>
+
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; version 3 of the License.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+"""Manifest handler."""
+
+from __future__ import print_function
+
+__metaclass__ = type
+
+
+from cdimage.log import logger
+from cdimage.changelog import (
+    ChangelogNotFound,
+    get_changelog,
+    VersionNotFound,
+)
+
+
+def compare_manifests(manifest1, manifest2):
+    # create similarly ordered lists from each manifest file
+    with open(manifest1) as mfa:
+        ma = sorted(mfa.readlines())
+    with open(manifest2) as mfb:
+        mb = sorted(mfb.readlines())
+
+    # analyze relation between pairs of packages from manifest lists
+    for a, b in _manifest_pairs(ma, mb):
+        if a and b:
+            if a[1] == b[1]:
+                print("= %s\t%s\t%s" % (a[0], a[1], b[1]))
+            else:
+                print("? %s\t%s\t%s" % (a[0], a[1], b[1]))
+        elif a:
+            print("< %s\t%s\t--" % (a[0], a[1]))
+        else:
+            print("> %s\t--\t%s" % (b[0], b[1]))
+
+
+def _next_package(manifest):
+    """Get the next package name and version information from a manifest."""
+    while manifest:
+        line = manifest.pop(0)
+        if not line:
+            # silently skip empty lines
+            continue
+        bits = line.split()
+        if len(bits) < 2:
+            logger.warning("'%s' not in format 'package version'" % line)
+        else:
+            return bits
+    return None
+
+
+def _manifest_pairs(ma, mb):
+    """Generate a current pair of packages, one from each manifest to compare.
+    """
+    left = _next_package(ma)
+    right = _next_package(mb)
+    while left and right:
+        if left[0] == right[0]:
+            yield left, right
+            left = _next_package(ma)
+            right = _next_package(mb)
+        elif left[0] < right[0]:
+            yield left, None
+            left = _next_package(ma)
+        else:  # left[0] > right[0]
+            yield None, right
+            right = _next_package(mb)
+    # drain off whichever file still has packages in it
+    while left:
+        yield left, None
+        left = _next_package(ma)
+    while right:
+        yield None, right
+        right = _next_package(mb)
+
+
+class Manifest:
+
+    def __init__(self, manifest_name=None):
+        self.filename = manifest_name
+        self.content = {}
+
+    def parse_file(self, manifest_name=None):
+        if manifest_name is None and self.filename is not None:
+            filename = self.filename
+            file_ = open(filename)
+            content = file_.readlines()
+            file_.close()
+            return self.parse(content)
+        return None
+
+    def parse(self, content):
+        if content is None:
+            return None
+        for line in content:
+            parsed_line = line.strip().split()
+            if not parsed_line:
+                continue
+            self.content[parsed_line[0]] = parsed_line[1]
+        return self.content
+
+    @property
+    def packages(self):
+        return set(self.content.keys())
+
+    def get_version(self, package_name):
+        try:
+            return self.content[package_name]
+        except KeyError:
+            return None
+
+
+class PackageDiff:
+
+    def __init__(self, binary_name, old_binary_version, new_binary_version):
+        self.binary_name = binary_name
+        self.old_binary_version = old_binary_version
+        self.new_binary_version = new_binary_version
+        self.old_changelog = self.get_changelog_info(old_binary_version)
+        self.new_changelog = self.get_changelog_info(new_binary_version)
+        self.old_source_info = self._get_source_info(self.old_changelog)
+        self.new_source_info = self._get_source_info(self.new_changelog)
+
+    @property
+    def old_source_name(self):
+        return self.old_source_info[0]
+
+    @property
+    def new_source_name(self):
+        return self.new_source_info[0]
+
+    @property
+    def old_source_version(self):
+        return self.old_source_info[1]
+
+    @property
+    def new_source_version(self):
+        return self.new_source_info[1]
+
+    @property
+    def source_has_changed(self):
+        return (self.old_source_name != self.new_source_name)
+
+    def get_changelog_info(self, binary_version):
+        """Fetches the changelog information for the given binary version.
+
+        This method fetches the changelog specific to this binary
+        version, so the latest source name and version in the changelog are
+        guaranteed to refer to this binary name and version.
+        """
+        return get_changelog(self.binary_name, binary_version)
+
+    @property
+    def source_name(self):
+        return self.new_source_name
+
+    @property
+    def source_info(self):
+        return (self.source_name, self.old_source_version,
+                self.new_source_version)
+
+    def _get_source_info(self, changelog):
+        """Return the latest source name version given a changelog."""
+        if changelog is not None:
+            return changelog.source_name, changelog.raw_version
+        else:
+            return None, None
+
+    @property
+    def changes(self):
+        """List of changelog blocks between two binary versions."""
+        try:
+            if self.source_has_changed:
+                # Return only the latest change of the recent changelog.
+                return [self.new_changelog.get_change_for_version(
+                        self.new_source_version)]
+            changes = self.new_changelog.get_changes_after_version(
+                self.old_source_version, self.new_source_version)
+        except VersionNotFound as e:
+            # Sometimes the old versions disappear from the changelog due to
+            # debian syncs. Whenever that happens, retrieve only the latest
+            # change.
+            if e.version == self.old_source_version:
+                changes = [self.new_changelog.get_change_for_version(
+                    self.new_source_version)]
+            else:
+                # If the problem is the newest version, there's something
+                # really wrong, this is not supposed to happen.
+                raise
+        return changes
+
+
+class ManifestDiff:
+
+    def __init__(self):
+        self.added_pkgs = {}
+        self.removed_pkgs = {}
+        self._changed_pkgs = {}
+        self.changelogs = {}
+        self.missing_pkgs = {}
+
+    def compare_manifests(self, manifest1, manifest2):
+        """Compares two Manifest objects generating the differences."""
+        self.added_pkgs = {pkg: manifest2.get_version(pkg) for pkg in (
+            manifest2.packages - manifest1.packages)}
+        self.removed_pkgs = {pkg: manifest1.get_version(pkg) for pkg in (
+            manifest1.packages - manifest2.packages)}
+
+        for pkg in manifest1.packages.intersection(manifest2.packages):
+            version1 = manifest1.get_version(pkg)
+            version2 = manifest2.get_version(pkg)
+            if version1 != version2:
+                self._changed_pkgs[pkg] = (version1, version2)
+
+    def compare(self, dict1, dict2):
+        """Compares two {pkg: version} dicts generating the differences."""
+        self.added_pkgs = {pkg: dict2[pkg] for pkg in (
+            set(dict2.keys()) - set(dict1.keys()))}
+        self.removed_pkgs = {pkg: dict1[pkg] for pkg in (
+            set(dict1.keys()) - set(dict2.keys()))}
+
+        for pkg in set(dict1.keys()).intersection(set(dict2.keys())):
+            version1 = dict1[pkg]
+            version2 = dict2[pkg]
+            if version1 != version2:
+                self._changed_pkgs[pkg] = (version1, version2)
+
+    def print_new_pkgs(self):
+        for pkg in sorted(list(self.added_pkgs.keys())):
+            print("%s - %s" % (pkg, self.added_pkgs[pkg]))
+
+    def print_removed_pkgs(self):
+        for pkg in sorted(list(self.removed_pkgs.keys())):
+            print("%s - %s" % (pkg, self.removed_pkgs[pkg]))
+
+    def get_changes(self):
+        for pkg, versions in list(self._changed_pkgs.items()):
+            old_binary_version, new_binary_version = versions
+            try:
+                pkg_name = pkg.split(":")[0]
+                pkg_changes = PackageDiff(pkg_name, old_binary_version,
+                                          new_binary_version)
+            except ChangelogNotFound as e:
+                self.missing_pkgs[pkg] = (versions, e)
+                continue
+            if pkg_changes.source_name in self.changelogs:
+                continue
+            self.changelogs[pkg_changes.source_name] = pkg_changes
+        return self.changelogs
+
+    def print_changed_pkgs(self):
+        changed_pkgs = ""
+        changelogs = ""
+        for pkg in sorted(list(self.changed_pkgs.keys())):
+            packagediff = self.changed_pkgs[pkg]
+            changed_pkgs += "%s - %s -> %s\n" % (
+                pkg, packagediff.old_source_version,
+                packagediff.new_source_version)
+            for change in packagediff.changes:
+                changelogs += " %s" % str(change)
+        for pkg in sorted(list(self.missing_pkgs.keys())):
+            versions, error = self.missing_pkgs[pkg]
+            changed_pkgs += "%s (binary) - %s -> %s\n" % (pkg, versions[0],
+                                                          versions[1])
+        print(changed_pkgs)
+        print("== Changelogs ==\n")
+        print(changelogs)
+
+    def print_changed_pkgs_changelogs(self):
+        for pkg in sorted(list(self.changed_pkgs.keys())):
+            for change in self.changed_pkgs[pkg]:
+                print(" %s" % str(change))
+
+    def print_missing_pkgs(self, hide_errors=True):
+        for pkg in sorted(list(self.missing_pkgs.keys())):
+            if hide_errors:
+                print("%s" % pkg)
+            else:
+                print("%s - %s" % (pkg, self.missing_pkgs[pkg][1].version))
+
+    @property
+    def changed_pkgs(self):
+        if self.changelogs:
+            return self.changelogs
+        return self.get_changes()
+
+    def output_changes(self):
+        if self.added_pkgs:
+            print("== Added packages ==")
+            self.print_new_pkgs()
+        if self.removed_pkgs:
+            print("\n== Removed packages ==")
+            self.print_removed_pkgs()
+        if self.changed_pkgs:
+            print("\n== Changed packages ==")
+            self.print_changed_pkgs()
+
+        if self.missing_pkgs:
+            missing = len(self.missing_pkgs)
+            print("== Changed packages not found in the archive (%s) ==" %
+                  missing)
+            self.print_missing_pkgs(hide_errors=False)

=== modified file 'lib/cdimage/tests/helpers.py'
--- lib/cdimage/tests/helpers.py	2013-03-27 17:31:02 +0000
+++ lib/cdimage/tests/helpers.py	2013-08-30 23:19:27 +0000
@@ -130,6 +130,20 @@
         assertRaisesRegex = unittest.TestCase.assertRaisesRegexp
 
 
+class FakeUrlOpenReturn:
+    """Object to mock the return of urllib.urlopen."""
+
+    def __init__(self, code, content):
+        self.code = code
+        self.content = content
+
+    def __repr__(self):
+        return self.content
+
+    def __iter__(self):
+        return (line for line in self.content.split("\n"))
+
+
 @contextlib.contextmanager
 def mkfile(path, mode="w"):
     osextras.ensuredir(os.path.dirname(path))

=== added file 'lib/cdimage/tests/test_changelog.py'
--- lib/cdimage/tests/test_changelog.py	1970-01-01 00:00:00 +0000
+++ lib/cdimage/tests/test_changelog.py	2013-08-30 23:19:27 +0000
@@ -0,0 +1,193 @@
+#! /usr/bin/python
+
+# Copyright (C) 2013 Canonical Ltd.
+# Author: Ursula Junque <[e-mail]>
+
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; version 3 of the License.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+"""Unit tests for cdimage.changelog."""
+
+from __future__ import print_function
+
+__metaclass__ = type
+
+import os
+
+from cdimage.changelog import (Changelog, ChangelogNotFound, EmptyUrlError,
+                               get_source_changelog_url, get_changelog_url,
+                               get_changelog, VersionNotFound)
+from cdimage.tests.helpers import FakeUrlOpenReturn, mkfile, TestCase
+
+try:
+    from unittest import mock
+except ImportError:
+    import mock
+
+
+CHANGELOG_CONTENT = ("""unzip (5.51-1) unstable; urgency=low
+  * New upstream release, improves error message when a zipfile is not
+    readable (Closes: #139331).
+  * Added a newline character to the CannotOpenZipfile string for the
+    previous fix to be really complete.
+
+ -- Santiago Vila <[e-mail]>  Sun,  6 Jun 2004 17:57:46 +0200
+
+unzip (5.51a.0-2) unstable; urgency=low
+
+  * Added unshrinking support (Closes: #252563).
+
+ -- Santiago Vila <[e-mail]>  Tue, 25 May 2004 14:38:26 +0200
+
+unzip (5.50-4) unstable; urgency=low
+
+  * Changed __GNU__ to __GLIBC__ in unix/unxcfg.h to support glibc-based
+    systems not being GNU itself, like GNU/KFreeBSD and GNU/KNetBSD.
+
+ -- Santiago Vila <[e-mail]>  Sun, 16 Nov 2003 14:45:28 +0100
+
+unzip (5.50-3) unstable; urgency=high
+
+  * Fixed "unzip directory traversal revisited" again (Bug #206439).
+    There was still a missing case that the previous patch didn't catch.
+    Patch borrowed from unzip-5.50-33.src.rpm.
+  * For reference, this is (still) CAN-2003-0282.
+
+ -- Santiago Vila <[e-mail]>  Wed, 20 Aug 2003 23:00:42 +0200
+
+unzip (5.50-2) unstable; urgency=high
+
+  * Fixed "unzip directory traversal revisited" problem (Bug #199648).
+    A filename containing ".somenonprintablechar." will not unpack
+    into .. anymore. Patch borrowed from unzip-5.50-11.src.rpm.
+  * For reference, this is CAN-2003-0282.
+  * No more doc symlinks.
+
+ -- Santiago Vila <[e-mail]>  Mon,  7 Jul 2003 20:25:20 +0200
+""")
+
+
+class ChangelogTests(TestCase):
+
+    def setUp(self):
+        super(ChangelogTests, self).setUp()
+        temp_dir = self.use_temp_dir()
+        changelog_path = os.path.join(temp_dir, "changelog")
+        with mkfile(changelog_path) as f:
+            print(CHANGELOG_CONTENT, file=f)
+        file_ = open(changelog_path)
+        self.changelog = Changelog(file_)
+        file_.close()
+
+    def test_changelog_has_version(self):
+        version = "5.51-1"
+        self.assertTrue(self.changelog.has_version(version))
+        version = "foobar"
+        self.assertFalse(self.changelog.has_version(version))
+
+    def test_changelog_raw_version(self):
+        version = self.changelog.get_version()
+        self.assertEqual(str(version), self.changelog.raw_version)
+
+    def test_changelog_get_changes_interval(self):
+        start_version = "5.50-2"
+        end_version = "5.51-1"
+        # 5.50-2 5.50-3 5.50-4 5.51a.0-2 5.51-1
+
+        self.assertEqual(len(self.changelog.get_changes_interval(start_version,
+                         end_version, include_start_version=False)), 4)
+        self.assertEqual(len(self.changelog.get_changes_interval(start_version,
+                         end_version)), 5)
+
+    def test_changelog_get_changes_interval_no_endversion(self):
+        start_version = "5.50-4"
+        end_version = None
+
+        changes = self.changelog.get_changes_interval(
+            start_version, end_version, include_start_version=False)
+        self.assertEqual(len(changes), 2)
+        changes = self.changelog.get_changes_interval(start_version)
+        self.assertEqual(len(changes), 3)
+
+        changes_versions = [change._raw_version for change in changes]
+        self.assertTrue("5.51-1" in changes_versions)
+        self.assertTrue("5.51a.0-2" in changes_versions)
+        self.assertTrue("5.50-4" in changes_versions)
+
+    def test_changelog_get_changes_after_version(self):
+        start_version = "5.50-3"
+
+        changes = self.changelog.get_changes_after_version(start_version)
+        self.assertEqual(len(changes), 3)
+        changes_versions = [change._raw_version for change in changes]
+        self.assertTrue("5.50-4" in changes_versions)
+        self.assertTrue("5.51a.0-2" in changes_versions)
+        self.assertTrue("5.51-1" in changes_versions)
+        self.assertTrue("5.50-3" not in changes_versions)
+
+    def test_changelog_get_changes_interval_version_doesnt_exist(self):
+        start_version = "foobar"
+        self.assertRaises(VersionNotFound,
+                          self.changelog.get_changes_interval,
+                          start_version)
+
+
+class TestGetUrls(TestCase):
+
+    def setUp(self):
+        super(TestGetUrls, self).setUp()
+
+    def test_get_source_changelog_url(self):
+        url = ("http://changelogs.ubuntu.com/changelogs/binary/f/"
+               "foobar/1.0/changelog")
+        self.assertEqual(get_source_changelog_url("foobar", "1.0"), url)
+
+    def test_get_source_changelog_url_missing_argument(self):
+        self.assertEqual(get_source_changelog_url(None, "1.0"), None)
+        self.assertEqual(get_source_changelog_url("foobar", None), None)
+        self.assertEqual(get_source_changelog_url(None, None), None)
+
+    def test_get_changelog_url(self):
+        url = ("http://changelogs.ubuntu.com/changelogs/pool/main/f/"
+               "foobar-1.0/changelog")
+        self.assertEqual(get_changelog_url("foobar", "1.0", "main"), url)
+
+    def test_get_changelog_url_missing_argument(self):
+        self.assertEqual(get_changelog_url("foobar", "1.0", None), None)
+        self.assertEqual(get_changelog_url("foobar", None, "main"), None)
+        self.assertEqual(get_changelog_url(None, "1.0", "main"), None)
+
+
+class TestGetChangelog(TestCase):
+
+    def setUp(self):
+        super(TestGetChangelog, self).setUp()
+
+    @mock.patch("urllib.urlopen",
+                return_value=FakeUrlOpenReturn(200, CHANGELOG_CONTENT))
+    def test_get_changelog(self, mock_urlopen):
+        changelog = get_changelog("foobar", "1.0")
+        self.assertIsNotNone(changelog)
+
+    @mock.patch("urllib.urlopen",
+                return_value=FakeUrlOpenReturn(404, "Not found"))
+    def test_get_changelog_url_not_found(self, mock_urlopen):
+        self.assertRaises(ChangelogNotFound, get_changelog, "foobar", "1.0")
+
+    @mock.patch("urllib.urlopen",
+                return_value=FakeUrlOpenReturn(500, "Forbidden"))
+    def test_get_changelog_othererror(self, mock_urlopen):
+        changelog = get_changelog("foobar", "1.0")
+        self.assertIsNone(changelog)
+
+    def test_get_changelog_urlisNone(self):
+        self.assertRaises(EmptyUrlError, get_changelog, None, None)

=== added file 'lib/cdimage/tests/test_manifest.py'
--- lib/cdimage/tests/test_manifest.py	1970-01-01 00:00:00 +0000
+++ lib/cdimage/tests/test_manifest.py	2013-08-30 23:19:27 +0000
@@ -0,0 +1,384 @@
+#! /usr/bin/python
+
+# Copyright (C) 2013 Canonical Ltd.
+# Author: Ursula Junque <[e-mail]>
+
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; version 3 of the License.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+"""Unit tests for cdimage.manifest."""
+
+from __future__ import print_function
+import os
+
+__metaclass__ = type
+
+try:
+    from unittest import mock
+except ImportError:
+    import mock
+
+import urllib
+urllib.urlopen = mock.Mock()
+
+from cdimage.manifest import Manifest, ManifestDiff, PackageDiff
+from cdimage.tests.helpers import dedent, FakeUrlOpenReturn, mkfile, TestCase
+
+
+def generate_manifest_file(content, temp_dir, filename=None):
+    if filename is None:
+        filename = "manifest"
+    manifest_path = os.path.join(temp_dir, filename)
+    with mkfile(manifest_path) as f:
+        print(content, file=f)
+    return manifest_path
+
+
+class TestManifest(TestCase):
+
+    CONTENT = dedent("""\
+                     apparmor   2.8.0-0ubuntu19.1
+                     at-spi2-core    2.9.4-1ubuntu1
+                     libatk-bridge2.0-0:armhf    2.9.3-1""")
+
+    def setUp(self):
+        super(TestManifest, self).setUp()
+        self.temp_dir = self.use_temp_dir()
+        self.parsed_content = {"apparmor": "2.8.0-0ubuntu19.1",
+                               "at-spi2-core": "2.9.4-1ubuntu1",
+                               "libatk-bridge2.0-0:armhf": "2.9.3-1"}
+
+    def _generate_manifest_from_file(self, content=None):
+        if content is None:
+            content = self.CONTENT
+        return Manifest(generate_manifest_file(content, self.temp_dir))
+
+    def test_parse_regular_file(self):
+        manifest = self._generate_manifest_from_file()
+        self.assertEqual(self.parsed_content, manifest.parse_file())
+
+    def test_parse_empty_file(self):
+        manifest = self._generate_manifest_from_file(content="")
+        self.assertEqual({}, manifest.parse_file())
+
+    def test_parse(self):
+        manifest = Manifest()
+        content = self.CONTENT.split("\n")
+        self.assertEqual(manifest.parse(content), self.parsed_content)
+
+    def test_packages(self):
+        manifest = Manifest()
+        content = self.CONTENT.split("\n")
+        parsed_content = manifest.parse(content)
+        self.assertEqual(set(parsed_content.keys()), manifest.packages)
+
+    def test_get_version(self):
+        manifest = Manifest()
+        content = self.CONTENT.split("\n")
+        manifest.parse(content)
+        package = "apparmor"
+        self.assertEqual(manifest.get_version(package), "2.8.0-0ubuntu19.1")
+
+    def test_get_version_package_not_found(self):
+        manifest = Manifest()
+        content = self.CONTENT.split("\n")
+        manifest.parse(content)
+        package = "foobar"
+        self.assertEqual(manifest.get_version(package), None)
+
+
+OLD_CHANGELOG_CONTENT = ("""\
+unzip (5.51a.0-2) unstable; urgency=low
+
+  * Added unshrinking support (Closes: #252563).
+
+ -- Santiago Vila <[e-mail]>  Tue, 25 May 2004 14:38:26 +0200""")
+
+NEW_CHANGELOG_CONTENT = ("""\
+unzip (5.51-1) unstable; urgency=low
+  * New upstream release, improves error message when a zipfile is not
+    readable (Closes: #139331).
+  * Added a newline character to the CannotOpenZipfile string for the
+    previous fix to be really complete.
+
+ -- Santiago Vila <[e-mail]>  Sun,  6 Jun 2004 17:57:46 +0200
+
+unzip (5.51a.0-3) unstable; urgency=low
+  * One release in the middle to get two changes between manifests.
+
+ -- Santiago Vila <[e-mail]>  Wed,  26 Jun 2004 17:57:46 +0200
+
+unzip (5.51a.0-2) unstable; urgency=low
+
+  * Added unshrinking support (Closes: #252563).
+
+ -- Santiago Vila <[e-mail]>  Tue, 25 May 2004 14:38:26 +0200""")
+
+OLD_CHANGELOG_NO_VERSION_CONTENT = ("""\
+unzip (5.51a.0-1) unstable; urgency=low
+
+  * Added unshrinking support (Closes: #252563).
+
+ -- Santiago Vila <[e-mail]>  Tue, 25 May 2004 14:38:26 +0200""")
+
+NEW_CHANGELOG_CONTENT_NEWSOURCE = ("""\
+unzippo (6.0) unstable; urgency=low
+
+  * Added unshrinking support (Closes: #252563).
+
+ -- Santiago Vila <[e-mail]>  Tue, 25 May 2004 14:38:26 +0200""")
+
+
+class TestPackageDiffSameSource(TestCase):
+
+    def setUp(self):
+        super(TestPackageDiffSameSource, self).setUp()
+        urllib.urlopen.side_effect = [
+            FakeUrlOpenReturn(200, OLD_CHANGELOG_CONTENT),
+            FakeUrlOpenReturn(200, NEW_CHANGELOG_CONTENT)]
+        self.old_version = "5.51a.0-2"
+        self.new_version = "5.51-1"
+        self.packagediff = PackageDiff("unzip", self.old_version,
+                                       self.new_version)
+
+    def test_packagediff_old_source_name(self):
+        self.assertEqual(self.packagediff.old_source_name, "unzip")
+
+    def test_packagediff_new_source_name(self):
+        self.assertEqual(self.packagediff.new_source_name, "unzip")
+
+    def test_packagediff_old_source_version(self):
+        self.assertEqual(self.packagediff.old_source_version, "5.51a.0-2")
+
+    def test_packagediff_new_source_version(self):
+        self.assertEqual(self.packagediff.new_source_version, "5.51-1")
+
+    def test_packagediff_source_hasnt_changed(self):
+        self.assertFalse(self.packagediff.source_has_changed)
+        self.assertEqual(self.packagediff.source_name, "unzip")
+
+    def test_packagediff_source_name_is_new_source_name(self):
+        self.assertEqual(self.packagediff.new_source_name,
+                         self.packagediff.source_name)
+
+    def test_changes(self):
+        self.assertEqual(len(self.packagediff.changes), 2)
+
+
+class TestPackageDiffSameSourceMissingVersion(TestCase):
+    def setUp(self):
+        super(TestPackageDiffSameSourceMissingVersion, self).setUp()
+
+    def test_changes_old_version_is_gone(self):
+        urllib.urlopen.side_effect = [
+            FakeUrlOpenReturn(200, OLD_CHANGELOG_NO_VERSION_CONTENT),
+            FakeUrlOpenReturn(200, NEW_CHANGELOG_CONTENT)]
+        old_version = "5.51a.0-1"
+        new_version = "5.51-1"
+        packagediff = PackageDiff("unzip", old_version, new_version)
+        self.assertEqual(len(packagediff.changes), 1)
+        self.assertEqual(packagediff.changes[0].version, new_version)
+
+
+class TestPackageDiffDifferentSource(TestCase):
+
+    def setUp(self):
+        super(TestPackageDiffDifferentSource, self).setUp()
+        urllib.urlopen.side_effect = [
+            FakeUrlOpenReturn(200, OLD_CHANGELOG_CONTENT),
+            FakeUrlOpenReturn(200, NEW_CHANGELOG_CONTENT_NEWSOURCE)]
+        self.packagediff = PackageDiff("unzip", "5.51a.0-2", "6.0")
+
+    def test_packagediff_source_has_changed(self):
+        self.assertTrue(self.packagediff.source_has_changed)
+        self.assertEqual(self.packagediff.new_source_name, "unzippo")
+
+    def test_packagediff_source_name_is_new_source_name(self):
+        self.assertEqual(self.packagediff.new_source_name,
+                         self.packagediff.source_name)
+        self.assertNotEqual(self.packagediff.old_source_name,
+                            self.packagediff.source_name)
+
+    def test_changes(self):
+        # Check if it's returning only the latest change of the new changelog.
+        self.assertEqual(len(self.packagediff.changes), 1)
+
+
+FOO_NEW_CHANGELOG = """\
+foo (0.1-0ubuntu3) unstable; urgency=low
+  * More foo stuff
+
+ -- John Doe <[e-mail]>  Sun,  6 Jun 2004 17:57:46 +0200
+
+foo (0.1-0ubuntu2) unstable; urgency=low
+  * Initial release
+
+ -- John Doe <[e-mail]>  Tue, 25 May 2004 14:38:26 +0200"""
+
+FOO_OLD_CHANGELOG = """\
+foo (0.1-0ubuntu2) unstable; urgency=low
+  * Initial release
+
+ -- John Doe <[e-mail]>  Tue, 25 May 2004 14:38:26 +0200"""
+
+BAR_NEW_CHANGELOG = """\
+bar (0.2-1ubuntu4) unstable; urgency=low
+  * Baring the thing
+
+ -- John Doe <[e-mail]>  Mon,  7 Jun 2004 17:57:46 +0200
+
+bar (0.2-1ubuntu3) unstable; urgency=low
+  * Release in the middle
+
+ -- John Doe <[e-mail]>  Wed,  26 Jun 2004 17:57:46 +0200
+
+bar (0.2-1ubuntu2) unstable; urgency=low
+  * Initial release
+
+ -- John Doe <[e-mail]>  Mon, 24 May 2004 14:38:26 +0200"""
+
+BAR_OLD_CHANGELOG = """\
+bar (0.2-1ubuntu2) unstable; urgency=low
+  * Initial release
+
+ -- John Doe <[e-mail]>  Mon, 24 May 2004 14:38:26 +0200"""
+
+
+class TestManifestDiff(TestCase):
+
+    # Old manifest: foo, bar, baz, libbaz1, foo5, libbar6
+    OLD_MANIFEST = dedent("""\
+                          foo 0.1-0ubuntu2
+                          bar 0.2-1ubuntu2
+                          libbaz1 0.3-0ubuntu1
+                          libbar6 1.3-2+13.10.20130520-0ubuntu1""")
+
+    # New manifest: foo, bar, baz, libbaz2, foo5, libbar7
+    #  - Added packages: libbaz2, libbar7
+    #  - Removed packages: libbaz1, libbar6
+    #  - Updated packages: foo, bar, baz, foo5
+    NEW_MANIFEST = dedent("""\
+                          foo 0.1-0ubuntu3
+                          bar 0.2-1ubuntu4
+                          libbaz2 0.4-0ubuntu1
+                          libbar7 1.5""")
+
+    def setUp(self):
+        super(TestManifestDiff, self).setUp()
+        self.manifest_diff = ManifestDiff()
+
+    def _generate_manifest_files(self):
+        temp_dir = self.use_temp_dir()
+        self.manifest1_file = generate_manifest_file(self.OLD_MANIFEST,
+                                                     temp_dir, "manifest1")
+        self.manifest2_file = generate_manifest_file(self.NEW_MANIFEST,
+                                                     temp_dir, "manifest2")
+
+    def test_compare_manifests(self):
+        self._generate_manifest_files()
+        manifest1 = Manifest(self.manifest1_file)
+        manifest1.parse_file()
+        manifest2 = Manifest(self.manifest2_file)
+        manifest2.parse_file()
+        manifest_diff = self.manifest_diff
+        manifest_diff.compare_manifests(manifest1, manifest2)
+
+        #  - Added packages: libbaz2, libbar7
+        self.assertEqual(len(manifest_diff.added_pkgs), 2)
+        self.assertTrue("libbaz2" in manifest_diff.added_pkgs)
+        self.assertTrue("libbar7" in manifest_diff.added_pkgs)
+        self.assertTrue("libbaz2" not in manifest_diff._changed_pkgs)
+
+        #  - Removed packages: libbaz1, libbar6
+        self.assertEqual(len(manifest_diff.removed_pkgs), 2)
+        self.assertTrue("libbaz1" in manifest_diff.removed_pkgs)
+        self.assertTrue("libbar6" in manifest_diff.removed_pkgs)
+        self.assertTrue("libbaz1" not in manifest_diff._changed_pkgs)
+
+        #  - Updated packages: foo, bar
+        self.assertEqual(len(manifest_diff._changed_pkgs), 2)
+        self.assertTrue("foo" in manifest_diff._changed_pkgs)
+        self.assertTrue("bar" in manifest_diff._changed_pkgs)
+
+        # All packages in both manifests (ignoring duplicates)
+        all_packages = manifest1.packages.union(manifest2.packages)
+        self.assertEqual((len(manifest_diff.added_pkgs) +
+                          len(manifest_diff.removed_pkgs) +
+                          len(manifest_diff._changed_pkgs)),
+                         len(all_packages))
+
+    def test_compare_dicts(self):
+        manifest1 = Manifest()
+        content1 = self.OLD_MANIFEST.split("\n")
+        manifest1.parse(content1)
+        manifest2 = Manifest()
+        content2 = self.NEW_MANIFEST.split("\n")
+        manifest2.parse(content2)
+
+        manifest_diff = self.manifest_diff
+
+        dict1 = manifest1.content
+        dict2 = manifest2.content
+
+        manifest_diff.compare(dict1, dict2)
+        #  - Added packages: libbaz2, libbar7
+        self.assertEqual(len(manifest_diff.added_pkgs), 2)
+        self.assertTrue("libbaz2" in manifest_diff.added_pkgs)
+        self.assertTrue("libbar7" in manifest_diff.added_pkgs)
+        self.assertTrue("libbaz2" not in manifest_diff._changed_pkgs)
+
+        #  - Removed packages: libbaz1, libbar6
+        self.assertEqual(len(manifest_diff.removed_pkgs), 2)
+        self.assertTrue("libbaz1" in manifest_diff.removed_pkgs)
+        self.assertTrue("libbar6" in manifest_diff.removed_pkgs)
+        self.assertTrue("libbaz1" not in manifest_diff._changed_pkgs)
+
+        #  - Updated packages: foo, bar
+        self.assertEqual(len(manifest_diff._changed_pkgs), 2)
+        self.assertTrue("foo" in manifest_diff._changed_pkgs)
+        self.assertTrue("bar" in manifest_diff._changed_pkgs)
+
+        # All packages in both manifests (ignoring duplicates)
+        all_packages = manifest1.packages.union(manifest2.packages)
+        self.assertEqual((len(manifest_diff.added_pkgs) +
+                          len(manifest_diff.removed_pkgs) +
+                          len(manifest_diff._changed_pkgs)),
+                         len(all_packages))
+
+    def test_get_changes(self):
+        manifest1 = Manifest()
+        content1 = self.OLD_MANIFEST.split("\n")
+        manifest1.parse(content1)
+        manifest2 = Manifest()
+        content2 = self.NEW_MANIFEST.split("\n")
+        manifest2.parse(content2)
+
+        manifest_diff = self.manifest_diff
+
+        dict1 = manifest1.content
+        dict2 = manifest2.content
+
+        manifest_diff.compare(dict1, dict2)
+
+        # Two packages changed, foo and bar. bar had two new versions since old
+        # manifest, foo had one.
+
+        urllib.urlopen.side_effect = [
+            FakeUrlOpenReturn(200, FOO_OLD_CHANGELOG),
+            FakeUrlOpenReturn(200, FOO_NEW_CHANGELOG),
+            FakeUrlOpenReturn(200, BAR_OLD_CHANGELOG),
+            FakeUrlOpenReturn(200, BAR_NEW_CHANGELOG)]
+        changes = self.manifest_diff.get_changes()
+        # One entry for each changed package
+        self.assertIsNotNone(changes)
+        self.assertEqual(len(changes["foo"].changes), 1)
+        self.assertEqual(len(changes["bar"].changes), 2)