Merge lp:~jameinel/bzr/2.1-launchpad-package-freshness-609187 into lp:bzr/2.1

[John A Meinel]

Bug #609187, whenever accessing a Launchpad packaging branch, check that it has the most recent package version.

This is a backport of my Launchpad plugin changes for bug #609187. This makes bzr-2.1 check for the most recent package versions available from Launchpad's database whenever we access a debian or ubuntu branch on Launchpad.

This should cause conflicts when merging up to bzr-2.4 because there were some bzrlib incompatibilities. Namely:

1) TestCase.overrideAttr doesn't exist in bzr-2.1, so I had to go back to manually overriding attributes and cleaning them up.

2) graph.iter_lefthand_ancestry was introduced in 2.4 (IIRC) to replace repository.iter_reverse_revision_history. So I had to write the 2.1 code to use iter_reverse_revision_history.

When we land this, I'll take care to merge up in the most compatible fashion I can. (So as soon as overrideAttr exists in a bzr release, I'll switch back to using it, similarly for iter_reverse_revision_history changes.)

[Jelmer Vernooij (jelmer)]

I have only briefly glanced over this. I've specifically checked the overriding of attributes and iter_reverse_revision_history. That all looks good to me.

It would be useful to test this with 2.4, in case this uses anything that relies on 2.5 or 2.6. I can't find anything when looking manually.

review: Approve

[John A Meinel (jameinel)]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 8/16/2011 4:27 PM, Jelmer Vernooij wrote:
> Review: Approve I have only briefly glanced over this. I've
> specifically checked the overriding of attributes and
> iter_reverse_revision_history. That all looks good to me.
>
> It would be useful to test this with 2.4, in case this uses anything
> that relies on 2.5 or 2.6. I can't find anything when looking
> manually.

I'm not sure what you mean by 2.4. But we already have this landed into
the 2.4 series, using the newer apis.

When I merge this up, I plan on using whatever apis are available at
each stage.

And I did make sure that:

a) The test suite passes
b) Activating the "DONT_*" tests that actually connect to launchpad and
make a query and parse the result also still pass.

John
=:->
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Cygwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk5KfxoACgkQJdeBCYSNAAMr3QCgt8cHcfB33bSHg+tPnNzFKY5I
/p8AnRZpY9lxZZyQ9uNUg/mNCcgSPzqj
=RlmU
-----END PGP SIGNATURE-----

[Jelmer Vernooij (jelmer)]

On 16/08/11 16:30, John Arbash Meinel wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> On 8/16/2011 4:27 PM, Jelmer Vernooij wrote:
>> Review: Approve I have only briefly glanced over this. I've
>> specifically checked the overriding of attributes and
>> iter_reverse_revision_history. That all looks good to me.
>>
>> It would be useful to test this with 2.4, in case this uses anything
>> that relies on 2.5 or 2.6. I can't find anything when looking
>> manually.
> I'm not sure what you mean by 2.4. But we already have this landed into
> the 2.4 series, using the newer apis.
Sorry, that was ambiguous. I meant compatible with python 2.4,
considering we still try to support it for bzr 2.1. I'm also not sure
what PQM uses to land things on lp:bzr/2.1 these days (if it's python2.6
we need to be careful).

Cheers,

Jelmer

[John A Meinel (jameinel)]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 8/16/2011 4:42 PM, Jelmer Vernooij wrote:
> On 16/08/11 16:30, John Arbash Meinel wrote:
>> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
>>
>> On 8/16/2011 4:27 PM, Jelmer Vernooij wrote:
>>> Review: Approve I have only briefly glanced over this. I've
>>> specifically checked the overriding of attributes and
>>> iter_reverse_revision_history. That all looks good to me.
>>>
>>> It would be useful to test this with 2.4, in case this uses
>>> anything that relies on 2.5 or 2.6. I can't find anything when
>>> looking manually.
>> I'm not sure what you mean by 2.4. But we already have this landed
>> into the 2.4 series, using the newer apis.
> Sorry, that was ambiguous. I meant compatible with python 2.4,
> considering we still try to support it for bzr 2.1. I'm also not
> sure what PQM uses to land things on lp:bzr/2.1 these days (if it's
> python2.6 we need to be careful).
>
> Cheers,
>
> Jelmer

Ah, good point. No we don't test with python 2.4 anywhere, and I don't
have it installed on my system anymore.

lp_api_lite is fairly large, but I don't think I have any 2.4-isms in
there. Your point is well made, though, I probably should try to dig
python-2.4 out.

John
=:->

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Cygwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk5KggAACgkQJdeBCYSNAANkkwCeI8IXUzFM60/MppodvoPBsyqn
LG4AnicZX0dS3komAZxTfehQzA/VtQSX
=EtOu
-----END PGP SIGNATURE-----

[John A Meinel (jameinel)]

sent to pqm by email

=== modified file 'NEWS'
--- NEWS	2011-05-17 08:19:08 +0000
+++ NEWS	2011-08-15 10:49:24 +0000
@@ -19,6 +19,31 @@
 Bug Fixes
 *********
 
+* Accessing a packaging branch on Launchpad (eg, ``lp:ubuntu/bzr``) now
+  checks to see if the most recent published source package version for
+  that project is present in the branch tags. This should help developers
+  trust whether the packaging branch is up-to-date and can be used for new
+  changes. The level of verbosity is controlled by the config item
+  ``launchpad.packaging_verbosity``. It can be set to one of
+
+  off
+    disable all checks
+
+
+  minimal
+    only display if the branch is out-of-date
+
+  short
+    also display single-line up-to-date and missing,
+
+
+  all
+    (default) display multi-line content for all states
+
+
+  (John Arbash Meinel, #609187, #812928)
+
+
 Improvements
 ************
 

=== modified file 'bzrlib/plugins/launchpad/__init__.py'
--- bzrlib/plugins/launchpad/__init__.py	2010-11-30 20:42:42 +0000
+++ bzrlib/plugins/launchpad/__init__.py	2011-08-15 10:49:24 +0000
@@ -21,18 +21,20 @@
 
 # see http://bazaar-vcs.org/Specs/BranchRegistrationTool
 
-# Since we are a built-in plugin we share the bzrlib version
-from bzrlib import version_info
-
 from bzrlib.lazy_import import lazy_import
 lazy_import(globals(), """
 from bzrlib import (
-    branch as _mod_branch,
     trace,
     )
 """)
 
 from bzrlib.commands import Command, Option, register_command
+from bzrlib import (
+    branch as _mod_branch,
+    lazy_regex,
+    # Since we are a built-in plugin we share the bzrlib version
+    version_info,
+    )
 from bzrlib.directory_service import directories
 from bzrlib.errors import (
     BzrCommandError,
@@ -283,12 +285,70 @@
                               'Launchpad-based directory service',)
 _register_directory()
 
+# This is kept in __init__ so that we don't load lp_api_lite unless the branch
+# actually matches. That way we can avoid importing extra dependencies like
+# json.
+_package_branch = lazy_regex.lazy_compile(
+    r'bazaar.launchpad.net.*?/'
+    r'(?P<user>~[^/]+/)?(?P<archive>ubuntu|debian)/(?P<series>[^/]+/)?'
+    r'(?P<project>[^/]+)(?P<branch>/[^/]+)?'
+    )
+
+def _get_package_branch_info(url):
+    """Determine the packaging information for this URL.
+
+    :return: If this isn't a packaging branch, return None. If it is, return
+        (archive, series, project)
+    """
+    if url is None:
+        return None
+    m = _package_branch.search(url)
+    if m is None:
+        return None
+    archive, series, project, user = m.group('archive', 'series',
+                                             'project', 'user')
+    if series is not None:
+        # series is optional, so the regex includes the extra '/', we don't
+        # want to send that on (it causes Internal Server Errors.)
+        series = series.strip('/')
+    if user is not None:
+        user = user.strip('~/')
+        if user != 'ubuntu-branches':
+            return None
+    return archive, series, project
+
+
+def _check_is_up_to_date(the_branch):
+    info = _get_package_branch_info(the_branch.base)
+    if info is None:
+        return
+    c = the_branch.get_config()
+    verbosity = c.get_user_option('launchpad.packaging_verbosity')
+    if verbosity is not None:
+        verbosity = verbosity.lower()
+    if verbosity == 'off':
+        trace.mutter('not checking %s because verbosity is turned off'
+                     % (the_branch.base,))
+        return
+    archive, series, project = info
+    from bzrlib.plugins.launchpad import lp_api_lite
+    latest_pub = lp_api_lite.LatestPublication(archive, series, project)
+    lp_api_lite.report_freshness(the_branch, verbosity, latest_pub)
+
+
+def _register_hooks():
+    _mod_branch.Branch.hooks.install_named_hook('open',
+        _check_is_up_to_date, 'package-branch-up-to-date')
+
+
+_register_hooks()
 
 def load_tests(basic_tests, module, loader):
     testmod_names = [
         'test_account',
         'test_register',
         'test_lp_api',
+        'test_lp_api_lite',
         'test_lp_directory',
         'test_lp_login',
         'test_lp_open',

=== added file 'bzrlib/plugins/launchpad/lp_api_lite.py'
--- bzrlib/plugins/launchpad/lp_api_lite.py	1970-01-01 00:00:00 +0000
+++ bzrlib/plugins/launchpad/lp_api_lite.py	2011-08-15 10:49:24 +0000
@@ -0,0 +1,285 @@
+# Copyright (C) 2011 Canonical Ltd
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
+
+"""Tools for dealing with the Launchpad API without using launchpadlib.
+
+The api itself is a RESTful interface, so we can make HTTP queries directly.
+loading launchpadlib itself has a fairly high overhead (just calling
+Launchpad.login_anonymously() takes a 500ms once the WADL is cached, and 5+s to
+get the WADL.
+"""
+
+try:
+    # Use simplejson if available, much faster, and can be easily installed in
+    # older versions of python
+    import simplejson as json
+except ImportError:
+    # Is present since python 2.6
+    try:
+        import json
+    except ImportError:
+        json = None
+
+import time
+import urllib
+import urllib2
+
+from bzrlib import (
+    revision,
+    trace,
+    )
+
+
+class LatestPublication(object):
+    """Encapsulate how to find the latest publication for a given project."""
+
+    LP_API_ROOT = 'https://api.launchpad.net/1.0'
+
+    def __init__(self, archive, series, project):
+        self._archive = archive
+        self._project = project
+        self._setup_series_and_pocket(series)
+
+    def _setup_series_and_pocket(self, series):
+        """Parse the 'series' info into a series and a pocket.
+
+        eg::
+            _setup_series_and_pocket('natty-proposed')
+            => _series == 'natty'
+               _pocket == 'Proposed'
+        """
+        self._series = series
+        self._pocket = None
+        if self._series is not None and '-' in self._series:
+            self._series, self._pocket = self._series.split('-', 1)
+            self._pocket = self._pocket.title()
+        else:
+            self._pocket = 'Release'
+
+    def _archive_URL(self):
+        """Return the Launchpad 'Archive' URL that we will query.
+        This is everything in the URL except the query parameters.
+        """
+        return '%s/%s/+archive/primary' % (self.LP_API_ROOT, self._archive)
+
+    def _publication_status(self):
+        """Handle the 'status' field.
+        It seems that Launchpad tracks all 'debian' packages as 'Pending', while
+        for 'ubuntu' we care about the 'Published' packages.
+        """
+        if self._archive == 'debian':
+            # Launchpad only tracks debian packages as "Pending", it doesn't mark
+            # them Published
+            return 'Pending'
+        return 'Published'
+
+    def _query_params(self):
+        """Get the parameters defining our query.
+        This defines the actions we are making against the archive.
+        :return: A dict of query parameters.
+        """
+        params = {'ws.op': 'getPublishedSources',
+                  'exact_match': 'true',
+                  # If we need to use "" shouldn't we quote the project somehow?
+                  'source_name': '"%s"' % (self._project,),
+                  'status': self._publication_status(),
+                  # We only need the latest one, the results seem to be properly
+                  # most-recent-debian-version sorted
+                  'ws.size': '1',
+        }
+        if self._series is not None:
+            params['distro_series'] = '/%s/%s' % (self._archive, self._series)
+        if self._pocket is not None:
+            params['pocket'] = self._pocket
+        return params
+
+    def _query_URL(self):
+        """Create the full URL that we need to query, including parameters."""
+        params = self._query_params()
+        # We sort to give deterministic results for testing
+        encoded = urllib.urlencode(sorted(params.items()))
+        return '%s?%s' % (self._archive_URL(), encoded)
+
+    def _get_lp_info(self):
+        """Place an actual HTTP query against the Launchpad service."""
+        if json is None:
+            return None
+        query_URL = self._query_URL()
+        try:
+            req = urllib2.Request(query_URL)
+            response = urllib2.urlopen(req)
+            json_info = response.read()
+        # TODO: We haven't tested the HTTPError
+        except (urllib2.URLError, urllib2.HTTPError), e:
+            trace.mutter('failed to place query to %r' % (query_URL,))
+            trace.log_exception_quietly()
+            return None
+        return json_info
+
+    def _parse_json_info(self, json_info):
+        """Parse the json response from Launchpad into objects."""
+        if json is None:
+            return None
+        try:
+            return json.loads(json_info)
+        except Exception:
+            trace.mutter('Failed to parse json info: %r' % (json_info,))
+            trace.log_exception_quietly()
+            return None
+
+    def get_latest_version(self):
+        """Get the latest published version for the given package."""
+        json_info = self._get_lp_info()
+        if json_info is None:
+            return None
+        info = self._parse_json_info(json_info)
+        if info is None:
+            return None
+        try:
+            entries = info['entries']
+            if len(entries) == 0:
+                return None
+            return entries[0]['source_package_version']
+        except KeyError:
+            trace.log_exception_quietly()
+            return None
+
+    def place(self):
+        """Text-form for what location this represents.
+
+        Example::
+            ubuntu, natty => Ubuntu Natty
+            ubuntu, natty-proposed => Ubuntu Natty Proposed
+        :return: A string representing the location we are checking.
+        """
+        place = self._archive
+        if self._series is not None:
+            place = '%s %s' % (place, self._series)
+        if self._pocket is not None and self._pocket != 'Release':
+            place = '%s %s' % (place, self._pocket)
+        return place.title()
+
+
+def get_latest_publication(archive, series, project):
+    """Get the most recent publication for a given project.
+
+    :param archive: Either 'ubuntu' or 'debian'
+    :param series: Something like 'natty', 'sid', etc. Can be set as None. Can
+        also include a pocket such as 'natty-proposed'.
+    :param project: Something like 'bzr'
+    :return: A version string indicating the most-recent version published in
+        Launchpad. Might return None if there is an error.
+    """
+    lp = LatestPublication(archive, series, project)
+    return lp.get_latest_version()
+
+
+def get_most_recent_tag(tag_dict, the_branch):
+    """Get the most recent revision that has been tagged."""
+    # Note: this assumes that a given rev won't get tagged multiple times. But
+    #       it should be valid for the package importer branches that we care
+    #       about
+    reverse_dict = dict((rev, tag) for tag, rev in tag_dict.iteritems())
+    the_branch.lock_read()
+    try:
+        history = the_branch.repository.iter_reverse_revision_history(
+            the_branch.last_revision())
+        for rev_id in history:
+            if rev_id in reverse_dict:
+                return reverse_dict[rev_id]
+    finally:
+        the_branch.unlock()
+
+
+def _get_newest_versions(the_branch, latest_pub):
+    """Get information about how 'fresh' this packaging branch is.
+
+    :param the_branch: The Branch to check
+    :param latest_pub: The LatestPublication used to check most recent
+        published version.
+    :return: (latest_ver, branch_latest_ver)
+    """
+    t = time.time()
+    latest_ver = latest_pub.get_latest_version()
+    t_latest_ver = time.time() - t
+    trace.mutter('LatestPublication.get_latest_version took: %.3fs'
+                 % (t_latest_ver,))
+    if latest_ver is None:
+        return None, None
+    t = time.time()
+    tags = the_branch.tags.get_tag_dict()
+    t_tag_dict = time.time() - t
+    trace.mutter('LatestPublication.get_tag_dict took: %.3fs' % (t_tag_dict,))
+    if latest_ver in tags:
+        # branch might have a newer tag, but we don't really care
+        return latest_ver, latest_ver
+    else:
+        best_tag = get_most_recent_tag(tags, the_branch)
+        return latest_ver, best_tag
+
+
+def _report_freshness(latest_ver, branch_latest_ver, place, verbosity,
+                      report_func):
+    """Report if the branch is up-to-date."""
+    if latest_ver is None:
+        if verbosity == 'all':
+            report_func('Most recent %s version: MISSING' % (place,))
+        elif verbosity == 'short':
+            report_func('%s is MISSING a version' % (place,))
+        return
+    elif latest_ver == branch_latest_ver:
+        if verbosity == 'minimal':
+            return
+        elif verbosity == 'short':
+            report_func('%s is CURRENT in %s' % (latest_ver, place))
+        else:
+            report_func('Most recent %s version: %s\n'
+                       'Packaging branch status: CURRENT'
+                       % (place, latest_ver))
+    else:
+        if verbosity in ('minimal', 'short'):
+            if branch_latest_ver is None:
+                branch_latest_ver = 'Branch'
+            report_func('%s is OUT-OF-DATE, %s has %s'
+                        % (branch_latest_ver, place, latest_ver))
+        else:
+            report_func('Most recent %s version: %s\n'
+                        'Packaging branch version: %s\n'
+                        'Packaging branch status: OUT-OF-DATE'
+                        % (place, latest_ver, branch_latest_ver))
+
+
+def report_freshness(the_branch, verbosity, latest_pub):
+    """Report to the user how up-to-date the packaging branch is.
+
+    :param the_branch: A Branch object
+    :param verbosity: Can be one of:
+        off: Do not print anything, and skip all checks.
+        all: Print all information that we have in a verbose manner, this
+             includes misses, etc.
+        short: Print information, but only one-line summaries
+        minimal: Only print a one-line summary when the package branch is
+                 out-of-date
+    :param latest_pub: A LatestPublication instance
+    """
+    if verbosity == 'off':
+        return
+    if verbosity is None:
+        verbosity = 'all'
+    latest_ver, branch_ver = _get_newest_versions(the_branch, latest_pub)
+    place = latest_pub.place()
+    _report_freshness(latest_ver, branch_ver, place, verbosity,
+                      trace.note)

=== added file 'bzrlib/plugins/launchpad/test_lp_api_lite.py'
--- bzrlib/plugins/launchpad/test_lp_api_lite.py	1970-01-01 00:00:00 +0000
+++ bzrlib/plugins/launchpad/test_lp_api_lite.py	2011-08-15 10:49:24 +0000
@@ -0,0 +1,557 @@
+# Copyright (C) 2011 Canonical Ltd
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
+
+"""Tools for dealing with the Launchpad API without using launchpadlib.
+"""
+
+import doctest
+import socket
+
+from bzrlib import tests
+from bzrlib.plugins import launchpad
+from bzrlib.plugins.launchpad import lp_api_lite
+
+from testtools.matchers import DocTestMatches
+
+
+class _JSONParserFeature(tests.Feature):
+
+    def _probe(self):
+        return lp_api_lite.json is not None
+
+    def feature_name(self):
+        return 'simplejson or json'
+
+JSONParserFeature = _JSONParserFeature()
+
+_example_response = r"""
+{
+    "total_size": 2,
+    "start": 0,
+    "next_collection_link": "https://api.launchpad.net/1.0/ubuntu/+archive/primary?distro_series=%2Fubuntu%2Flucid&exact_match=true&source_name=%22bzr%22&status=Published&ws.op=getPublishedSources&ws.start=1&ws.size=1",
+    "entries": [
+        {
+            "package_creator_link": "https://api.launchpad.net/1.0/~maxb",
+            "package_signer_link": "https://api.launchpad.net/1.0/~jelmer",
+            "source_package_name": "bzr",
+            "removal_comment": null,
+            "display_name": "bzr 2.1.4-0ubuntu1 in lucid",
+            "date_made_pending": null,
+            "source_package_version": "2.1.4-0ubuntu1",
+            "date_superseded": null,
+            "http_etag": "\"9ba966152dec474dc0fe1629d0bbce2452efaf3b-5f4c3fbb3eaf26d502db4089777a9b6a0537ffab\"",
+            "self_link": "https://api.launchpad.net/1.0/ubuntu/+archive/primary/+sourcepub/1750327",
+            "distro_series_link": "https://api.launchpad.net/1.0/ubuntu/lucid",
+            "component_name": "main",
+            "status": "Published",
+            "date_removed": null,
+            "pocket": "Updates",
+            "date_published": "2011-05-30T06:09:58.653984+00:00",
+            "removed_by_link": null,
+            "section_name": "devel",
+            "resource_type_link": "https://api.launchpad.net/1.0/#source_package_publishing_history",
+            "archive_link": "https://api.launchpad.net/1.0/ubuntu/+archive/primary",
+            "package_maintainer_link": "https://api.launchpad.net/1.0/~ubuntu-devel-discuss-lists",
+            "date_created": "2011-05-30T05:19:12.233621+00:00",
+            "scheduled_deletion_date": null
+        }
+    ]
+}"""
+
+_no_versions_response = '{"total_size": 0, "start": 0, "entries": []}'
+
+
+class TestLatestPublication(tests.TestCase):
+
+    def make_latest_publication(self, archive='ubuntu', series='natty',
+                                project='bzr'):
+        return lp_api_lite.LatestPublication(archive, series, project)
+
+    def assertPlace(self, place, archive, series, project):
+        lp = lp_api_lite.LatestPublication(archive, series, project)
+        self.assertEqual(place, lp.place())
+
+    def test_init(self):
+        latest_pub = self.make_latest_publication()
+        self.assertEqual('ubuntu', latest_pub._archive)
+        self.assertEqual('natty', latest_pub._series)
+        self.assertEqual('bzr', latest_pub._project)
+        self.assertEqual('Release', latest_pub._pocket)
+
+    def test__archive_URL(self):
+        latest_pub = self.make_latest_publication()
+        self.assertEqual(
+            'https://api.launchpad.net/1.0/ubuntu/+archive/primary',
+            latest_pub._archive_URL())
+
+    def test__publication_status_for_ubuntu(self):
+        latest_pub = self.make_latest_publication()
+        self.assertEqual('Published', latest_pub._publication_status())
+
+    def test__publication_status_for_debian(self):
+        latest_pub = self.make_latest_publication(archive='debian')
+        self.assertEqual('Pending', latest_pub._publication_status())
+
+    def test_pocket(self):
+        latest_pub = self.make_latest_publication(series='natty-proposed')
+        self.assertEqual('natty', latest_pub._series)
+        self.assertEqual('Proposed', latest_pub._pocket)
+
+    def test_series_None(self):
+        latest_pub = self.make_latest_publication(series=None)
+        self.assertEqual('ubuntu', latest_pub._archive)
+        self.assertEqual(None, latest_pub._series)
+        self.assertEqual('bzr', latest_pub._project)
+        self.assertEqual('Release', latest_pub._pocket)
+
+    def test__query_params(self):
+        latest_pub = self.make_latest_publication()
+        self.assertEqual({'ws.op': 'getPublishedSources',
+                          'exact_match': 'true',
+                          'source_name': '"bzr"',
+                          'status': 'Published',
+                          'ws.size': '1',
+                          'distro_series': '/ubuntu/natty',
+                          'pocket': 'Release',
+                         }, latest_pub._query_params())
+
+    def test__query_params_no_series(self):
+        latest_pub = self.make_latest_publication(series=None)
+        self.assertEqual({'ws.op': 'getPublishedSources',
+                          'exact_match': 'true',
+                          'source_name': '"bzr"',
+                          'status': 'Published',
+                          'ws.size': '1',
+                          'pocket': 'Release',
+                         }, latest_pub._query_params())
+
+    def test__query_params_pocket(self):
+        latest_pub = self.make_latest_publication(series='natty-proposed')
+        self.assertEqual({'ws.op': 'getPublishedSources',
+                          'exact_match': 'true',
+                          'source_name': '"bzr"',
+                          'status': 'Published',
+                          'ws.size': '1',
+                          'distro_series': '/ubuntu/natty',
+                          'pocket': 'Proposed',
+                         }, latest_pub._query_params())
+
+    def test__query_URL(self):
+        latest_pub = self.make_latest_publication()
+        # we explicitly sort params, so we can be sure this URL matches exactly
+        self.assertEqual(
+            'https://api.launchpad.net/1.0/ubuntu/+archive/primary'
+            '?distro_series=%2Fubuntu%2Fnatty&exact_match=true'
+            '&pocket=Release&source_name=%22bzr%22&status=Published'
+            '&ws.op=getPublishedSources&ws.size=1',
+            latest_pub._query_URL())
+
+    def DONT_test__gracefully_handle_failed_rpc_connection(self):
+        # TODO: This test kind of sucks. We intentionally create an arbitrary
+        #       port and don't listen to it, because we want the request to fail.
+        #       However, it seems to take 1s for it to timeout. Is there a way
+        #       to make it fail faster?
+        latest_pub = self.make_latest_publication()
+        s = socket.socket()
+        s.bind(('127.0.0.1', 0))
+        addr, port = s.getsockname()
+        latest_pub.LP_API_ROOT = 'http://%s:%s/' % (addr, port)
+        s.close()
+        self.assertIs(None, latest_pub._get_lp_info())
+
+    def DONT_test__query_launchpad(self):
+        # TODO: This is a test that we are making a valid request against
+        #       launchpad. This seems important, but it is slow, requires net
+        #       access, and requires launchpad to be up and running. So for
+        #       now, it is commented out for production tests.
+        latest_pub = self.make_latest_publication()
+        json_txt = latest_pub._get_lp_info()
+        self.assertIsNot(None, json_txt)
+        if lp_api_lite.json is None:
+            # We don't have a way to parse the text
+            return
+        # The content should be a valid json result
+        content = lp_api_lite.json.loads(json_txt)
+        entries = content['entries'] # It should have an 'entries' field.
+        # ws.size should mean we get 0 or 1, and there should be something
+        self.assertEqual(1, len(entries))
+        entry = entries[0]
+        self.assertEqual('bzr', entry['source_package_name'])
+        version = entry['source_package_version']
+        self.assertIsNot(None, version)
+
+    def disableJSON(self):
+        orig = lp_api_lite.json
+        def cleanup():
+            lp_api_lite.json = orig
+        self.addCleanup(cleanup)
+        lp_api_lite.json = None
+
+    def test__get_lp_info_no_json(self):
+        # If we can't parse the json, we don't make the query.
+        self.disableJSON()
+        latest_pub = self.make_latest_publication()
+        self.assertIs(None, latest_pub._get_lp_info())
+
+    def test__parse_json_info_no_module(self):
+        # If a json parsing module isn't available, we just return None here.
+        self.disableJSON()
+        latest_pub = self.make_latest_publication()
+        self.assertIs(None, latest_pub._parse_json_info(_example_response))
+
+    def test__parse_json_example_response(self):
+        self.requireFeature(JSONParserFeature)
+        latest_pub = self.make_latest_publication()
+        content = latest_pub._parse_json_info(_example_response)
+        self.assertIsNot(None, content)
+        self.assertEqual(2, content['total_size'])
+        entries = content['entries']
+        self.assertEqual(1, len(entries))
+        entry = entries[0]
+        self.assertEqual('bzr', entry['source_package_name'])
+        self.assertEqual("2.1.4-0ubuntu1", entry["source_package_version"])
+
+    def test__parse_json_not_json(self):
+        self.requireFeature(JSONParserFeature)
+        latest_pub = self.make_latest_publication()
+        self.assertIs(None, latest_pub._parse_json_info('Not_valid_json'))
+
+    def test_get_latest_version_no_response(self):
+        latest_pub = self.make_latest_publication()
+        latest_pub._get_lp_info = lambda: None
+        self.assertEqual(None, latest_pub.get_latest_version())
+
+    def test_get_latest_version_no_json(self):
+        self.disableJSON()
+        latest_pub = self.make_latest_publication()
+        self.assertEqual(None, latest_pub.get_latest_version())
+
+    def test_get_latest_version_invalid_json(self):
+        self.requireFeature(JSONParserFeature)
+        latest_pub = self.make_latest_publication()
+        latest_pub._get_lp_info = lambda: "not json"
+        self.assertEqual(None, latest_pub.get_latest_version())
+
+    def test_get_latest_version_no_versions(self):
+        self.requireFeature(JSONParserFeature)
+        latest_pub = self.make_latest_publication()
+        latest_pub._get_lp_info = lambda: _no_versions_response
+        self.assertEqual(None, latest_pub.get_latest_version())
+
+    def test_get_latest_version_missing_entries(self):
+        # Launchpad's no-entries response does have an empty entries value.
+        # However, lets test that we handle other failures without tracebacks
+        self.requireFeature(JSONParserFeature)
+        latest_pub = self.make_latest_publication()
+        latest_pub._get_lp_info = lambda: '{}'
+        self.assertEqual(None, latest_pub.get_latest_version())
+
+    def test_get_latest_version_invalid_entries(self):
+        # Make sure we sanely handle a json response we don't understand
+        self.requireFeature(JSONParserFeature)
+        latest_pub = self.make_latest_publication()
+        latest_pub._get_lp_info = lambda: '{"entries": {"a": 1}}'
+        self.assertEqual(None, latest_pub.get_latest_version())
+
+    def test_get_latest_version_example(self):
+        self.requireFeature(JSONParserFeature)
+        latest_pub = self.make_latest_publication()
+        latest_pub._get_lp_info = lambda: _example_response
+        self.assertEqual("2.1.4-0ubuntu1", latest_pub.get_latest_version())
+
+    def DONT_test_get_latest_version_from_launchpad(self):
+        self.requireFeature(JSONParserFeature)
+        latest_pub = self.make_latest_publication()
+        self.assertIsNot(None, latest_pub.get_latest_version())
+
+    def test_place(self):
+        self.assertPlace('Ubuntu', 'ubuntu', None, 'bzr')
+        self.assertPlace('Ubuntu Natty', 'ubuntu', 'natty', 'bzr')
+        self.assertPlace('Ubuntu Natty Proposed', 'ubuntu', 'natty-proposed',
+                         'bzr')
+        self.assertPlace('Debian', 'debian', None, 'bzr')
+        self.assertPlace('Debian Sid', 'debian', 'sid', 'bzr')
+
+
+class TestIsUpToDate(tests.TestCase):
+
+    def assertPackageBranchRe(self, url, user, archive, series, project):
+        m = launchpad._package_branch.search(url)
+        if m is None:
+            self.fail('package_branch regex did not match url: %s' % (url,))
+        self.assertEqual(
+            (user, archive, series, project),
+            m.group('user', 'archive', 'series', 'project'))
+
+    def assertNotPackageBranch(self, url):
+        self.assertIs(None, launchpad._get_package_branch_info(url))
+
+    def assertBranchInfo(self, url, archive, series, project):
+        self.assertEqual((archive, series, project),
+            launchpad._get_package_branch_info(url))
+
+    def test_package_branch_regex(self):
+        self.assertPackageBranchRe(
+            'http://bazaar.launchpad.net/+branch/ubuntu/foo',
+            None, 'ubuntu', None, 'foo')
+        self.assertPackageBranchRe(
+            'bzr+ssh://bazaar.launchpad.net/+branch/ubuntu/natty/foo',
+            None, 'ubuntu', 'natty/', 'foo')
+        self.assertPackageBranchRe(
+            'sftp://bazaar.launchpad.net/+branch/debian/foo',
+            None, 'debian', None, 'foo')
+        self.assertPackageBranchRe(
+            'http://bazaar.launchpad.net/+branch/debian/sid/foo',
+            None, 'debian', 'sid/', 'foo')
+        self.assertPackageBranchRe(
+            'http://bazaar.launchpad.net/+branch'
+            '/~ubuntu-branches/ubuntu/natty/foo/natty',
+            '~ubuntu-branches/', 'ubuntu', 'natty/', 'foo')
+        self.assertPackageBranchRe(
+            'http://bazaar.launchpad.net/+branch'
+            '/~user/ubuntu/natty/foo/test',
+            '~user/', 'ubuntu', 'natty/', 'foo')
+
+    def test_package_branch_doesnt_match(self):
+        self.assertNotPackageBranch('http://example.com/ubuntu/foo')
+        self.assertNotPackageBranch(
+            'http://bazaar.launchpad.net/+branch/bzr')
+        self.assertNotPackageBranch(
+            'http://bazaar.launchpad.net/+branch/~bzr-pqm/bzr/bzr.dev')
+        # Not a packaging branch because ~user isn't ~ubuntu-branches
+        self.assertNotPackageBranch(
+            'http://bazaar.launchpad.net/+branch'
+            '/~user/ubuntu/natty/foo/natty')
+        # Older versions of bzr-svn/hg/git did not set Branch.base until after
+        # they called Branch.__init__().
+        self.assertNotPackageBranch(None)
+
+    def test__get_package_branch_info(self):
+        self.assertBranchInfo(
+            'bzr+ssh://bazaar.launchpad.net/+branch/ubuntu/natty/foo',
+            'ubuntu', 'natty', 'foo')
+        self.assertBranchInfo(
+            'bzr+ssh://bazaar.launchpad.net/+branch'
+            '/~ubuntu-branches/ubuntu/natty/foo/natty',
+            'ubuntu', 'natty', 'foo')
+        self.assertBranchInfo(
+            'http://bazaar.launchpad.net/+branch'
+            '/~ubuntu-branches/debian/sid/foo/sid',
+            'debian', 'sid', 'foo')
+
+
+class TestGetMostRecentTag(tests.TestCaseWithMemoryTransport):
+
+    def make_simple_builder(self):
+        builder = self.make_branch_builder('tip')
+        builder.build_snapshot('A', None, [
+            ('add', ('', 'root-id', 'directory', None))])
+        b = builder.get_branch()
+        b.tags.set_tag('tip-1.0', 'A')
+        return builder, b, b.tags.get_tag_dict()
+
+    def test_get_most_recent_tag_tip(self):
+        builder, b, tag_dict = self.make_simple_builder()
+        self.assertEqual('tip-1.0',
+                         lp_api_lite.get_most_recent_tag(tag_dict, b))
+
+    def test_get_most_recent_tag_older(self):
+        builder, b, tag_dict = self.make_simple_builder()
+        builder.build_snapshot('B', ['A'], [])
+        self.assertEqual('B', b.last_revision())
+        self.assertEqual('tip-1.0',
+                         lp_api_lite.get_most_recent_tag(tag_dict, b))
+
+
+class StubLatestPublication(object):
+
+    def __init__(self, latest):
+        self.called = False
+        self.latest = latest
+
+    def get_latest_version(self):
+        self.called = True
+        return self.latest
+
+    def place(self):
+        return 'Ubuntu Natty'
+
+
+class TestReportFreshness(tests.TestCaseWithMemoryTransport):
+
+    def setUp(self):
+        super(TestReportFreshness, self).setUp()
+        builder = self.make_branch_builder('tip')
+        builder.build_snapshot('A', None, [
+            ('add', ('', 'root-id', 'directory', None))])
+        self.branch = builder.get_branch()
+
+    def assertFreshnessReports(self, verbosity, latest_version, content):
+        """Assert that lp_api_lite.report_freshness reports the given content.
+
+        :param verbosity: The reporting level
+        :param latest_version: The version reported by StubLatestPublication
+        :param content: The expected content. This should be in DocTest form.
+        """
+        orig_log_len = len(self.get_log())
+        lp_api_lite.report_freshness(self.branch, verbosity,
+            StubLatestPublication(latest_version))
+        new_content = self.get_log()[orig_log_len:]
+        # Strip out lines that have LatestPublication.get_* because those are
+        # timing related lines. While interesting to log for now, they aren't
+        # something we want to be testing
+        new_content = new_content.split('\n')
+        for i in range(2):
+            if (len(new_content) > 0
+                and 'LatestPublication.get_' in new_content[0]):
+                new_content = new_content[1:]
+        new_content = '\n'.join(new_content)
+        self.assertThat(new_content,
+            DocTestMatches(content,
+                doctest.ELLIPSIS | doctest.REPORT_UDIFF))
+
+    def test_verbosity_off_skips_check(self):
+        # We force _get_package_branch_info so that we know it would otherwise
+        # try to connect to launcphad
+        orig_gpbi = launchpad._get_package_branch_info
+        orig_lp = lp_api_lite.LatestPublication
+        def cleanup():
+            launchpad._get_package_branch_info = orig_gpbi
+            lp_api_lite.LatestPublication = orig_lp
+        self.addCleanup(cleanup)
+        launchpad._get_package_branch_info = lambda x: ('ubuntu', 'natty', 'bzr')
+        lp_api_lite.LatestPublication = lambda *args: self.fail('Tried to query launchpad')
+        c = self.branch.get_config()
+        c.set_user_option('launchpad.packaging_verbosity', 'off')
+        orig_log_len = len(self.get_log())
+        launchpad._check_is_up_to_date(self.branch)
+        new_content = self.get_log()[orig_log_len:]
+        self.assertContainsRe(new_content,
+            'not checking memory.*/tip/ because verbosity is turned off')
+
+    def test_verbosity_off(self):
+        latest_pub = StubLatestPublication('1.0-1ubuntu2')
+        lp_api_lite.report_freshness(self.branch, 'off', latest_pub)
+        self.assertFalse(latest_pub.called)
+
+    def test_verbosity_all_out_of_date_smoke(self):
+        self.branch.tags.set_tag('1.0-1ubuntu1', 'A')
+        self.assertFreshnessReports('all', '1.0-1ubuntu2',
+             '    INFO  Most recent Ubuntu Natty version: 1.0-1ubuntu2\n'
+             'Packaging branch version: 1.0-1ubuntu1\n'
+             'Packaging branch status: OUT-OF-DATE\n')
+
+
+class Test_GetNewestVersions(tests.TestCaseWithMemoryTransport):
+
+    def setUp(self):
+        super(Test_GetNewestVersions, self).setUp()
+        builder = self.make_branch_builder('tip')
+        builder.build_snapshot('A', None, [
+            ('add', ('', 'root-id', 'directory', None))])
+        self.branch = builder.get_branch()
+
+    def assertLatestVersions(self, latest_branch_version, pub_version):
+        if latest_branch_version is not None:
+            self.branch.tags.set_tag(latest_branch_version, 'A')
+        latest_pub = StubLatestPublication(pub_version)
+        self.assertEqual((pub_version, latest_branch_version),
+            lp_api_lite._get_newest_versions(self.branch, latest_pub))
+
+    def test_no_tags(self):
+        self.assertLatestVersions(None, '1.0-1ubuntu2')
+
+    def test_out_of_date(self):
+        self.assertLatestVersions('1.0-1ubuntu1', '1.0-1ubuntu2')
+
+    def test_up_to_date(self):
+        self.assertLatestVersions('1.0-1ubuntu2', '1.0-1ubuntu2')
+
+    def test_missing(self):
+        self.assertLatestVersions(None, None)
+
+
+class Test_ReportFreshness(tests.TestCase):
+
+    def assertReportedFreshness(self, verbosity, latest_ver, branch_latest_ver,
+                               content, place='Ubuntu Natty'):
+        """Assert that lp_api_lite.report_freshness reports the given content.
+        """
+        reported = []
+        def report_func(value):
+            reported.append(value)
+        lp_api_lite._report_freshness(latest_ver, branch_latest_ver, place,
+                                      verbosity, report_func)
+        new_content = '\n'.join(reported)
+        self.assertThat(new_content,
+            DocTestMatches(content,
+                doctest.ELLIPSIS | doctest.REPORT_UDIFF))
+
+    def test_verbosity_minimal_no_tags(self):
+        self.assertReportedFreshness('minimal', '1.0-1ubuntu2', None,
+            'Branch is OUT-OF-DATE, Ubuntu Natty has 1.0-1ubuntu2\n')
+
+    def test_verbosity_minimal_out_of_date(self):
+        self.assertReportedFreshness('minimal', '1.0-1ubuntu2', '1.0-1ubuntu1',
+            '1.0-1ubuntu1 is OUT-OF-DATE,'
+            ' Ubuntu Natty has 1.0-1ubuntu2\n')
+
+    def test_verbosity_minimal_up_to_date(self):
+        self.assertReportedFreshness('minimal', '1.0-1ubuntu2', '1.0-1ubuntu2',
+             '')
+
+    def test_verbosity_minimal_missing(self):
+        self.assertReportedFreshness('minimal', None, None,
+             '')
+
+    def test_verbosity_short_out_of_date(self):
+        self.assertReportedFreshness('short', '1.0-1ubuntu2', '1.0-1ubuntu1',
+            '1.0-1ubuntu1 is OUT-OF-DATE,'
+            ' Ubuntu Natty has 1.0-1ubuntu2\n')
+
+    def test_verbosity_short_up_to_date(self):
+        self.assertReportedFreshness('short', '1.0-1ubuntu2', '1.0-1ubuntu2',
+             '1.0-1ubuntu2 is CURRENT in Ubuntu Natty')
+
+    def test_verbosity_short_missing(self):
+        self.assertReportedFreshness('short', None, None,
+             'Ubuntu Natty is MISSING a version')
+
+    def test_verbosity_all_no_tags(self):
+        self.assertReportedFreshness('all', '1.0-1ubuntu2', None,
+             'Most recent Ubuntu Natty version: 1.0-1ubuntu2\n'
+             'Packaging branch version: None\n'
+             'Packaging branch status: OUT-OF-DATE\n')
+
+    def test_verbosity_all_out_of_date(self):
+        self.assertReportedFreshness('all', '1.0-1ubuntu2', '1.0-1ubuntu1',
+             'Most recent Ubuntu Natty version: 1.0-1ubuntu2\n'
+             'Packaging branch version: 1.0-1ubuntu1\n'
+             'Packaging branch status: OUT-OF-DATE\n')
+
+    def test_verbosity_all_up_to_date(self):
+        self.assertReportedFreshness('all', '1.0-1ubuntu2', '1.0-1ubuntu2',
+             'Most recent Ubuntu Natty version: 1.0-1ubuntu2\n'
+             'Packaging branch status: CURRENT\n')
+
+    def test_verbosity_all_missing(self):
+        self.assertReportedFreshness('all', None, None,
+             'Most recent Ubuntu Natty version: MISSING\n')
+
+    def test_verbosity_None_is_all(self):
+        self.assertReportedFreshness(None, '1.0-1ubuntu2', '1.0-1ubuntu2',
+             'Most recent Ubuntu Natty version: 1.0-1ubuntu2\n'
+             'Packaging branch status: CURRENT\n')