Merge lp:~jameinel/bzr/2.4-launchpad-package-freshness-609187 into lp:bzr/2.4

[John A Meinel]

Bug #609187, check that packaging import branches are up-to-date when accessing them.

This backports my bug #609187 fix now that we've settled on the output.

So it will check if "lp:ubuntu/*" and "lp:debian/*" are up-to-date, which can be disabled with the config setting, and the verbosity level adjusted.

[Andrew Bennetts (spiv)]

I haven't read this patch closely, but this is just a direct backport of the patch that landed on lp:bzr, right? In that case, I say: do it!

review: Approve

[Andrew Bennetts (spiv)]

Although it occurs to me that the text you add to release-notes would make a useful help topic.

(Also, at a glance the ReST markup of that release-notes entry doesn't look like it'll render the way you intend.)

[John A Meinel (jameinel)]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 7/27/2011 9:11 AM, Andrew Bennetts wrote:
> Although it occurs to me that the text you add to release-notes would make a useful help topic.
>
> (Also, at a glance the ReST markup of that release-notes entry doesn't look like it'll render the way you intend.)

How do you think I intend it? I was going for a Definition List:
http://docutils.sourceforge.net/docs/user/rst/quickref.html#definition-lists

Which seems to be how it is formatted, though I was expecting the
headers to be bolded, but it is showing up as <dd> and <dt>.

Do you have a different recommendation?

John
=:->

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Cygwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk4v8TsACgkQJdeBCYSNAANwGgCfRvv35DVwN3JVRatehqY9WvrP
5ugAnRZH/XRLSQoAWwzdH4sE2HLZdG7r
=8cZK
-----END PGP SIGNATURE-----

[John A Meinel (jameinel)]

sent to pqm by email

=== modified file 'bzrlib/plugins/launchpad/__init__.py'
--- bzrlib/plugins/launchpad/__init__.py	2011-04-05 01:12:15 +0000
+++ bzrlib/plugins/launchpad/__init__.py	2011-07-26 08:23:04 +0000
@@ -40,19 +40,21 @@
 
 # see http://wiki.bazaar.canonical.com/Specs/BranchRegistrationTool
 
-# Since we are a built-in plugin we share the bzrlib version
-from bzrlib import version_info
-
 from bzrlib.lazy_import import lazy_import
 lazy_import(globals(), """
 from bzrlib import (
-    branch as _mod_branch,
     ui,
     trace,
     )
 """)
 
-from bzrlib import bzrdir
+from bzrlib import (
+    branch as _mod_branch,
+    bzrdir,
+    lazy_regex,
+    # Since we are a built-in plugin we share the bzrlib version
+    version_info,
+    )
 from bzrlib.commands import (
     Command,
     register_command,
@@ -462,12 +464,68 @@
 
 _register_directory()
 
+# This is kept in __init__ so that we don't load lp_api_lite unless the branch
+# actually matches. That way we can avoid importing extra dependencies like
+# json.
+_package_branch = lazy_regex.lazy_compile(
+    r'bazaar.launchpad.net.*?/'
+    r'(?P<user>~[^/]+/)?(?P<archive>ubuntu|debian)/(?P<series>[^/]+/)?'
+    r'(?P<project>[^/]+)(?P<branch>/[^/]+)?'
+    )
+
+def _get_package_branch_info(url):
+    """Determine the packaging information for this URL.
+
+    :return: If this isn't a packaging branch, return None. If it is, return
+        (archive, series, project)
+    """
+    m = _package_branch.search(url)
+    if m is None:
+        return
+    archive, series, project, user = m.group('archive', 'series',
+                                             'project', 'user')
+    if series is not None:
+        # series is optional, so the regex includes the extra '/', we don't
+        # want to send that on (it causes Internal Server Errors.)
+        series = series.strip('/')
+    if user is not None:
+        user = user.strip('~/')
+        if user != 'ubuntu-branches':
+            return None
+    return archive, series, project
+
+
+def _check_is_up_to_date(the_branch):
+    info = _get_package_branch_info(the_branch.base)
+    if info is None:
+        return
+    c = the_branch.get_config()
+    verbosity = c.get_user_option('launchpad.packaging_verbosity')
+    if verbosity is not None:
+        verbosity = verbosity.lower()
+    if verbosity == 'off':
+        trace.mutter('not checking %s because verbosity is turned off'
+                     % (the_branch.base,))
+        return
+    archive, series, project = info
+    from bzrlib.plugins.launchpad import lp_api_lite
+    latest_pub = lp_api_lite.LatestPublication(archive, series, project)
+    lp_api_lite.report_freshness(the_branch, verbosity, latest_pub)
+
+
+def _register_hooks():
+    _mod_branch.Branch.hooks.install_named_hook('open',
+        _check_is_up_to_date, 'package-branch-up-to-date')
+
+
+_register_hooks()
 
 def load_tests(basic_tests, module, loader):
     testmod_names = [
         'test_account',
         'test_register',
         'test_lp_api',
+        'test_lp_api_lite',
         'test_lp_directory',
         'test_lp_login',
         'test_lp_open',

=== added file 'bzrlib/plugins/launchpad/lp_api_lite.py'
--- bzrlib/plugins/launchpad/lp_api_lite.py	1970-01-01 00:00:00 +0000
+++ bzrlib/plugins/launchpad/lp_api_lite.py	2011-07-26 08:23:04 +0000
@@ -0,0 +1,286 @@
+# Copyright (C) 2011 Canonical Ltd
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
+
+"""Tools for dealing with the Launchpad API without using launchpadlib.
+
+The api itself is a RESTful interface, so we can make HTTP queries directly.
+loading launchpadlib itself has a fairly high overhead (just calling
+Launchpad.login_anonymously() takes a 500ms once the WADL is cached, and 5+s to
+get the WADL.
+"""
+
+try:
+    # Use simplejson if available, much faster, and can be easily installed in
+    # older versions of python
+    import simplejson as json
+except ImportError:
+    # Is present since python 2.6
+    try:
+        import json
+    except ImportError:
+        json = None
+
+import time
+import urllib
+import urllib2
+
+from bzrlib import (
+    revision,
+    trace,
+    )
+
+
+class LatestPublication(object):
+    """Encapsulate how to find the latest publication for a given project."""
+
+    LP_API_ROOT = 'https://api.launchpad.net/1.0'
+
+    def __init__(self, archive, series, project):
+        self._archive = archive
+        self._project = project
+        self._setup_series_and_pocket(series)
+
+    def _setup_series_and_pocket(self, series):
+        """Parse the 'series' info into a series and a pocket.
+
+        eg::
+            _setup_series_and_pocket('natty-proposed')
+            => _series == 'natty'
+               _pocket == 'Proposed'
+        """
+        self._series = series
+        self._pocket = None
+        if self._series is not None and '-' in self._series:
+            self._series, self._pocket = self._series.split('-', 1)
+            self._pocket = self._pocket.title()
+        else:
+            self._pocket = 'Release'
+
+    def _archive_URL(self):
+        """Return the Launchpad 'Archive' URL that we will query.
+        This is everything in the URL except the query parameters.
+        """
+        return '%s/%s/+archive/primary' % (self.LP_API_ROOT, self._archive)
+
+    def _publication_status(self):
+        """Handle the 'status' field.
+        It seems that Launchpad tracks all 'debian' packages as 'Pending', while
+        for 'ubuntu' we care about the 'Published' packages.
+        """
+        if self._archive == 'debian':
+            # Launchpad only tracks debian packages as "Pending", it doesn't mark
+            # them Published
+            return 'Pending'
+        return 'Published'
+
+    def _query_params(self):
+        """Get the parameters defining our query.
+        This defines the actions we are making against the archive.
+        :return: A dict of query parameters.
+        """
+        params = {'ws.op': 'getPublishedSources',
+                  'exact_match': 'true',
+                  # If we need to use "" shouldn't we quote the project somehow?
+                  'source_name': '"%s"' % (self._project,),
+                  'status': self._publication_status(),
+                  # We only need the latest one, the results seem to be properly
+                  # most-recent-debian-version sorted
+                  'ws.size': '1',
+        }
+        if self._series is not None:
+            params['distro_series'] = '/%s/%s' % (self._archive, self._series)
+        if self._pocket is not None:
+            params['pocket'] = self._pocket
+        return params
+
+    def _query_URL(self):
+        """Create the full URL that we need to query, including parameters."""
+        params = self._query_params()
+        # We sort to give deterministic results for testing
+        encoded = urllib.urlencode(sorted(params.items()))
+        return '%s?%s' % (self._archive_URL(), encoded)
+
+    def _get_lp_info(self):
+        """Place an actual HTTP query against the Launchpad service."""
+        if json is None:
+            return None
+        query_URL = self._query_URL()
+        try:
+            req = urllib2.Request(query_URL)
+            response = urllib2.urlopen(req)
+            json_info = response.read()
+        # TODO: We haven't tested the HTTPError
+        except (urllib2.URLError, urllib2.HTTPError), e:
+            trace.mutter('failed to place query to %r' % (query_URL,))
+            trace.log_exception_quietly()
+            return None
+        return json_info
+
+    def _parse_json_info(self, json_info):
+        """Parse the json response from Launchpad into objects."""
+        if json is None:
+            return None
+        try:
+            return json.loads(json_info)
+        except Exception:
+            trace.mutter('Failed to parse json info: %r' % (json_info,))
+            trace.log_exception_quietly()
+            return None
+
+    def get_latest_version(self):
+        """Get the latest published version for the given package."""
+        json_info = self._get_lp_info()
+        if json_info is None:
+            return None
+        info = self._parse_json_info(json_info)
+        if info is None:
+            return None
+        try:
+            entries = info['entries']
+            if len(entries) == 0:
+                return None
+            return entries[0]['source_package_version']
+        except KeyError:
+            trace.log_exception_quietly()
+            return None
+
+    def place(self):
+        """Text-form for what location this represents.
+
+        Example::
+            ubuntu, natty => Ubuntu Natty
+            ubuntu, natty-proposed => Ubuntu Natty Proposed
+        :return: A string representing the location we are checking.
+        """
+        place = self._archive
+        if self._series is not None:
+            place = '%s %s' % (place, self._series)
+        if self._pocket is not None and self._pocket != 'Release':
+            place = '%s %s' % (place, self._pocket)
+        return place.title()
+
+
+def get_latest_publication(archive, series, project):
+    """Get the most recent publication for a given project.
+
+    :param archive: Either 'ubuntu' or 'debian'
+    :param series: Something like 'natty', 'sid', etc. Can be set as None. Can
+        also include a pocket such as 'natty-proposed'.
+    :param project: Something like 'bzr'
+    :return: A version string indicating the most-recent version published in
+        Launchpad. Might return None if there is an error.
+    """
+    lp = LatestPublication(archive, series, project)
+    return lp.get_latest_version()
+
+
+def get_most_recent_tag(tag_dict, the_branch):
+    """Get the most recent revision that has been tagged."""
+    # Note: this assumes that a given rev won't get tagged multiple times. But
+    #       it should be valid for the package importer branches that we care
+    #       about
+    reverse_dict = dict((rev, tag) for tag, rev in tag_dict.iteritems())
+    the_branch.lock_read()
+    try:
+        last_rev = the_branch.last_revision()
+        graph = the_branch.repository.get_graph()
+        stop_revisions = (None, revision.NULL_REVISION)
+        for rev_id in graph.iter_lefthand_ancestry(last_rev, stop_revisions):
+            if rev_id in reverse_dict:
+                return reverse_dict[rev_id]
+    finally:
+        the_branch.unlock()
+
+
+def _get_newest_versions(the_branch, latest_pub):
+    """Get information about how 'fresh' this packaging branch is.
+
+    :param the_branch: The Branch to check
+    :param latest_pub: The LatestPublication used to check most recent
+        published version.
+    :return: (latest_ver, branch_latest_ver)
+    """
+    t = time.time()
+    latest_ver = latest_pub.get_latest_version()
+    t_latest_ver = time.time() - t
+    trace.mutter('LatestPublication.get_latest_version took: %.3fs'
+                 % (t_latest_ver,))
+    if latest_ver is None:
+        return None, None
+    t = time.time()
+    tags = the_branch.tags.get_tag_dict()
+    t_tag_dict = time.time() - t
+    trace.mutter('LatestPublication.get_tag_dict took: %.3fs' % (t_tag_dict,))
+    if latest_ver in tags:
+        # branch might have a newer tag, but we don't really care
+        return latest_ver, latest_ver
+    else:
+        best_tag = get_most_recent_tag(tags, the_branch)
+        return latest_ver, best_tag
+
+
+def _report_freshness(latest_ver, branch_latest_ver, place, verbosity,
+                      report_func):
+    """Report if the branch is up-to-date."""
+    if latest_ver is None:
+        if verbosity == 'all':
+            report_func('Most recent %s version: MISSING' % (place,))
+        elif verbosity == 'short':
+            report_func('%s is MISSING a version' % (place,))
+        return
+    elif latest_ver == branch_latest_ver:
+        if verbosity == 'minimal':
+            return
+        elif verbosity == 'short':
+            report_func('%s is CURRENT in %s' % (latest_ver, place))
+        else:
+            report_func('Most recent %s version: %s\n'
+                       'Packaging branch status: CURRENT'
+                       % (place, latest_ver))
+    else:
+        if verbosity in ('minimal', 'short'):
+            if branch_latest_ver is None:
+                branch_latest_ver = 'Branch'
+            report_func('%s is OUT-OF-DATE, %s has %s'
+                        % (branch_latest_ver, place, latest_ver))
+        else:
+            report_func('Most recent %s version: %s\n'
+                        'Packaging branch version: %s\n'
+                        'Packaging branch status: OUT-OF-DATE'
+                        % (place, latest_ver, branch_latest_ver))
+
+
+def report_freshness(the_branch, verbosity, latest_pub):
+    """Report to the user how up-to-date the packaging branch is.
+
+    :param the_branch: A Branch object
+    :param verbosity: Can be one of:
+        off: Do not print anything, and skip all checks.
+        all: Print all information that we have in a verbose manner, this
+             includes misses, etc.
+        short: Print information, but only one-line summaries
+        minimal: Only print a one-line summary when the package branch is
+                 out-of-date
+    :param latest_pub: A LatestPublication instance
+    """
+    if verbosity == 'off':
+        return
+    if verbosity is None:
+        verbosity = 'all'
+    latest_ver, branch_ver = _get_newest_versions(the_branch, latest_pub)
+    place = latest_pub.place()
+    _report_freshness(latest_ver, branch_ver, place, verbosity,
+                      trace.note)

=== added file 'bzrlib/plugins/launchpad/test_lp_api_lite.py'
--- bzrlib/plugins/launchpad/test_lp_api_lite.py	1970-01-01 00:00:00 +0000
+++ bzrlib/plugins/launchpad/test_lp_api_lite.py	2011-07-26 08:23:04 +0000
@@ -0,0 +1,543 @@
+# Copyright (C) 2011 Canonical Ltd
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
+
+"""Tools for dealing with the Launchpad API without using launchpadlib.
+"""
+
+import doctest
+import socket
+
+from bzrlib import tests
+from bzrlib.plugins import launchpad
+from bzrlib.plugins.launchpad import lp_api_lite
+
+from testtools.matchers import DocTestMatches
+
+
+class _JSONParserFeature(tests.Feature):
+
+    def _probe(self):
+        return lp_api_lite.json is not None
+
+    def feature_name(self):
+        return 'simplejson or json'
+
+JSONParserFeature = _JSONParserFeature()
+
+_example_response = r"""
+{
+    "total_size": 2,
+    "start": 0,
+    "next_collection_link": "https://api.launchpad.net/1.0/ubuntu/+archive/primary?distro_series=%2Fubuntu%2Flucid&exact_match=true&source_name=%22bzr%22&status=Published&ws.op=getPublishedSources&ws.start=1&ws.size=1",
+    "entries": [
+        {
+            "package_creator_link": "https://api.launchpad.net/1.0/~maxb",
+            "package_signer_link": "https://api.launchpad.net/1.0/~jelmer",
+            "source_package_name": "bzr",
+            "removal_comment": null,
+            "display_name": "bzr 2.1.4-0ubuntu1 in lucid",
+            "date_made_pending": null,
+            "source_package_version": "2.1.4-0ubuntu1",
+            "date_superseded": null,
+            "http_etag": "\"9ba966152dec474dc0fe1629d0bbce2452efaf3b-5f4c3fbb3eaf26d502db4089777a9b6a0537ffab\"",
+            "self_link": "https://api.launchpad.net/1.0/ubuntu/+archive/primary/+sourcepub/1750327",
+            "distro_series_link": "https://api.launchpad.net/1.0/ubuntu/lucid",
+            "component_name": "main",
+            "status": "Published",
+            "date_removed": null,
+            "pocket": "Updates",
+            "date_published": "2011-05-30T06:09:58.653984+00:00",
+            "removed_by_link": null,
+            "section_name": "devel",
+            "resource_type_link": "https://api.launchpad.net/1.0/#source_package_publishing_history",
+            "archive_link": "https://api.launchpad.net/1.0/ubuntu/+archive/primary",
+            "package_maintainer_link": "https://api.launchpad.net/1.0/~ubuntu-devel-discuss-lists",
+            "date_created": "2011-05-30T05:19:12.233621+00:00",
+            "scheduled_deletion_date": null
+        }
+    ]
+}"""
+
+_no_versions_response = '{"total_size": 0, "start": 0, "entries": []}'
+
+
+class TestLatestPublication(tests.TestCase):
+
+    def make_latest_publication(self, archive='ubuntu', series='natty',
+                                project='bzr'):
+        return lp_api_lite.LatestPublication(archive, series, project)
+
+    def assertPlace(self, place, archive, series, project):
+        lp = lp_api_lite.LatestPublication(archive, series, project)
+        self.assertEqual(place, lp.place())
+
+    def test_init(self):
+        latest_pub = self.make_latest_publication()
+        self.assertEqual('ubuntu', latest_pub._archive)
+        self.assertEqual('natty', latest_pub._series)
+        self.assertEqual('bzr', latest_pub._project)
+        self.assertEqual('Release', latest_pub._pocket)
+
+    def test__archive_URL(self):
+        latest_pub = self.make_latest_publication()
+        self.assertEqual(
+            'https://api.launchpad.net/1.0/ubuntu/+archive/primary',
+            latest_pub._archive_URL())
+
+    def test__publication_status_for_ubuntu(self):
+        latest_pub = self.make_latest_publication()
+        self.assertEqual('Published', latest_pub._publication_status())
+
+    def test__publication_status_for_debian(self):
+        latest_pub = self.make_latest_publication(archive='debian')
+        self.assertEqual('Pending', latest_pub._publication_status())
+
+    def test_pocket(self):
+        latest_pub = self.make_latest_publication(series='natty-proposed')
+        self.assertEqual('natty', latest_pub._series)
+        self.assertEqual('Proposed', latest_pub._pocket)
+
+    def test_series_None(self):
+        latest_pub = self.make_latest_publication(series=None)
+        self.assertEqual('ubuntu', latest_pub._archive)
+        self.assertEqual(None, latest_pub._series)
+        self.assertEqual('bzr', latest_pub._project)
+        self.assertEqual('Release', latest_pub._pocket)
+
+    def test__query_params(self):
+        latest_pub = self.make_latest_publication()
+        self.assertEqual({'ws.op': 'getPublishedSources',
+                          'exact_match': 'true',
+                          'source_name': '"bzr"',
+                          'status': 'Published',
+                          'ws.size': '1',
+                          'distro_series': '/ubuntu/natty',
+                          'pocket': 'Release',
+                         }, latest_pub._query_params())
+
+    def test__query_params_no_series(self):
+        latest_pub = self.make_latest_publication(series=None)
+        self.assertEqual({'ws.op': 'getPublishedSources',
+                          'exact_match': 'true',
+                          'source_name': '"bzr"',
+                          'status': 'Published',
+                          'ws.size': '1',
+                          'pocket': 'Release',
+                         }, latest_pub._query_params())
+
+    def test__query_params_pocket(self):
+        latest_pub = self.make_latest_publication(series='natty-proposed')
+        self.assertEqual({'ws.op': 'getPublishedSources',
+                          'exact_match': 'true',
+                          'source_name': '"bzr"',
+                          'status': 'Published',
+                          'ws.size': '1',
+                          'distro_series': '/ubuntu/natty',
+                          'pocket': 'Proposed',
+                         }, latest_pub._query_params())
+
+    def test__query_URL(self):
+        latest_pub = self.make_latest_publication()
+        # we explicitly sort params, so we can be sure this URL matches exactly
+        self.assertEqual(
+            'https://api.launchpad.net/1.0/ubuntu/+archive/primary'
+            '?distro_series=%2Fubuntu%2Fnatty&exact_match=true'
+            '&pocket=Release&source_name=%22bzr%22&status=Published'
+            '&ws.op=getPublishedSources&ws.size=1',
+            latest_pub._query_URL())
+
+    def DONT_test__gracefully_handle_failed_rpc_connection(self):
+        # TODO: This test kind of sucks. We intentionally create an arbitrary
+        #       port and don't listen to it, because we want the request to fail.
+        #       However, it seems to take 1s for it to timeout. Is there a way
+        #       to make it fail faster?
+        latest_pub = self.make_latest_publication()
+        s = socket.socket()
+        s.bind(('127.0.0.1', 0))
+        addr, port = s.getsockname()
+        latest_pub.LP_API_ROOT = 'http://%s:%s/' % (addr, port)
+        s.close()
+        self.assertIs(None, latest_pub._get_lp_info())
+
+    def DONT_test__query_launchpad(self):
+        # TODO: This is a test that we are making a valid request against
+        #       launchpad. This seems important, but it is slow, requires net
+        #       access, and requires launchpad to be up and running. So for
+        #       now, it is commented out for production tests.
+        latest_pub = self.make_latest_publication()
+        json_txt = latest_pub._get_lp_info()
+        self.assertIsNot(None, json_txt)
+        if lp_api_lite.json is None:
+            # We don't have a way to parse the text
+            return
+        # The content should be a valid json result
+        content = lp_api_lite.json.loads(json_txt)
+        entries = content['entries'] # It should have an 'entries' field.
+        # ws.size should mean we get 0 or 1, and there should be something
+        self.assertEqual(1, len(entries))
+        entry = entries[0]
+        self.assertEqual('bzr', entry['source_package_name'])
+        version = entry['source_package_version']
+        self.assertIsNot(None, version)
+
+    def test__get_lp_info_no_json(self):
+        # If we can't parse the json, we don't make the query.
+        self.overrideAttr(lp_api_lite, 'json', None)
+        latest_pub = self.make_latest_publication()
+        self.assertIs(None, latest_pub._get_lp_info())
+
+    def test__parse_json_info_no_module(self):
+        # If a json parsing module isn't available, we just return None here.
+        self.overrideAttr(lp_api_lite, 'json', None)
+        latest_pub = self.make_latest_publication()
+        self.assertIs(None, latest_pub._parse_json_info(_example_response))
+
+    def test__parse_json_example_response(self):
+        self.requireFeature(JSONParserFeature)
+        latest_pub = self.make_latest_publication()
+        content = latest_pub._parse_json_info(_example_response)
+        self.assertIsNot(None, content)
+        self.assertEqual(2, content['total_size'])
+        entries = content['entries']
+        self.assertEqual(1, len(entries))
+        entry = entries[0]
+        self.assertEqual('bzr', entry['source_package_name'])
+        self.assertEqual("2.1.4-0ubuntu1", entry["source_package_version"])
+
+    def test__parse_json_not_json(self):
+        self.requireFeature(JSONParserFeature)
+        latest_pub = self.make_latest_publication()
+        self.assertIs(None, latest_pub._parse_json_info('Not_valid_json'))
+
+    def test_get_latest_version_no_response(self):
+        latest_pub = self.make_latest_publication()
+        latest_pub._get_lp_info = lambda: None
+        self.assertEqual(None, latest_pub.get_latest_version())
+
+    def test_get_latest_version_no_json(self):
+        self.overrideAttr(lp_api_lite, 'json', None)
+        latest_pub = self.make_latest_publication()
+        self.assertEqual(None, latest_pub.get_latest_version())
+
+    def test_get_latest_version_invalid_json(self):
+        self.requireFeature(JSONParserFeature)
+        latest_pub = self.make_latest_publication()
+        latest_pub._get_lp_info = lambda: "not json"
+        self.assertEqual(None, latest_pub.get_latest_version())
+
+    def test_get_latest_version_no_versions(self):
+        self.requireFeature(JSONParserFeature)
+        latest_pub = self.make_latest_publication()
+        latest_pub._get_lp_info = lambda: _no_versions_response
+        self.assertEqual(None, latest_pub.get_latest_version())
+
+    def test_get_latest_version_missing_entries(self):
+        # Launchpad's no-entries response does have an empty entries value.
+        # However, lets test that we handle other failures without tracebacks
+        self.requireFeature(JSONParserFeature)
+        latest_pub = self.make_latest_publication()
+        latest_pub._get_lp_info = lambda: '{}'
+        self.assertEqual(None, latest_pub.get_latest_version())
+
+    def test_get_latest_version_invalid_entries(self):
+        # Make sure we sanely handle a json response we don't understand
+        self.requireFeature(JSONParserFeature)
+        latest_pub = self.make_latest_publication()
+        latest_pub._get_lp_info = lambda: '{"entries": {"a": 1}}'
+        self.assertEqual(None, latest_pub.get_latest_version())
+
+    def test_get_latest_version_example(self):
+        self.requireFeature(JSONParserFeature)
+        latest_pub = self.make_latest_publication()
+        latest_pub._get_lp_info = lambda: _example_response
+        self.assertEqual("2.1.4-0ubuntu1", latest_pub.get_latest_version())
+
+    def DONT_test_get_latest_version_from_launchpad(self):
+        self.requireFeature(JSONParserFeature)
+        latest_pub = self.make_latest_publication()
+        self.assertIsNot(None, latest_pub.get_latest_version())
+
+    def test_place(self):
+        self.assertPlace('Ubuntu', 'ubuntu', None, 'bzr')
+        self.assertPlace('Ubuntu Natty', 'ubuntu', 'natty', 'bzr')
+        self.assertPlace('Ubuntu Natty Proposed', 'ubuntu', 'natty-proposed',
+                         'bzr')
+        self.assertPlace('Debian', 'debian', None, 'bzr')
+        self.assertPlace('Debian Sid', 'debian', 'sid', 'bzr')
+
+
+class TestIsUpToDate(tests.TestCase):
+
+    def assertPackageBranchRe(self, url, user, archive, series, project):
+        m = launchpad._package_branch.search(url)
+        if m is None:
+            self.fail('package_branch regex did not match url: %s' % (url,))
+        self.assertEqual(
+            (user, archive, series, project),
+            m.group('user', 'archive', 'series', 'project'))
+
+    def assertNotPackageBranch(self, url):
+        self.assertIs(None, launchpad._get_package_branch_info(url))
+
+    def assertBranchInfo(self, url, archive, series, project):
+        self.assertEqual((archive, series, project),
+            launchpad._get_package_branch_info(url))
+
+    def test_package_branch_regex(self):
+        self.assertPackageBranchRe(
+            'http://bazaar.launchpad.net/+branch/ubuntu/foo',
+            None, 'ubuntu', None, 'foo')
+        self.assertPackageBranchRe(
+            'bzr+ssh://bazaar.launchpad.net/+branch/ubuntu/natty/foo',
+            None, 'ubuntu', 'natty/', 'foo')
+        self.assertPackageBranchRe(
+            'sftp://bazaar.launchpad.net/+branch/debian/foo',
+            None, 'debian', None, 'foo')
+        self.assertPackageBranchRe(
+            'http://bazaar.launchpad.net/+branch/debian/sid/foo',
+            None, 'debian', 'sid/', 'foo')
+        self.assertPackageBranchRe(
+            'http://bazaar.launchpad.net/+branch'
+            '/~ubuntu-branches/ubuntu/natty/foo/natty',
+            '~ubuntu-branches/', 'ubuntu', 'natty/', 'foo')
+        self.assertPackageBranchRe(
+            'http://bazaar.launchpad.net/+branch'
+            '/~user/ubuntu/natty/foo/test',
+            '~user/', 'ubuntu', 'natty/', 'foo')
+
+    def test_package_branch_doesnt_match(self):
+        self.assertNotPackageBranch('http://example.com/ubuntu/foo')
+        self.assertNotPackageBranch(
+            'http://bazaar.launchpad.net/+branch/bzr')
+        self.assertNotPackageBranch(
+            'http://bazaar.launchpad.net/+branch/~bzr-pqm/bzr/bzr.dev')
+        # Not a packaging branch because ~user isn't ~ubuntu-branches
+        self.assertNotPackageBranch(
+            'http://bazaar.launchpad.net/+branch'
+            '/~user/ubuntu/natty/foo/natty')
+
+    def test__get_package_branch_info(self):
+        self.assertBranchInfo(
+            'bzr+ssh://bazaar.launchpad.net/+branch/ubuntu/natty/foo',
+            'ubuntu', 'natty', 'foo')
+        self.assertBranchInfo(
+            'bzr+ssh://bazaar.launchpad.net/+branch'
+            '/~ubuntu-branches/ubuntu/natty/foo/natty',
+            'ubuntu', 'natty', 'foo')
+        self.assertBranchInfo(
+            'http://bazaar.launchpad.net/+branch'
+            '/~ubuntu-branches/debian/sid/foo/sid',
+            'debian', 'sid', 'foo')
+
+
+class TestGetMostRecentTag(tests.TestCaseWithMemoryTransport):
+
+    def make_simple_builder(self):
+        builder = self.make_branch_builder('tip')
+        builder.build_snapshot('A', [], [
+            ('add', ('', 'root-id', 'directory', None))])
+        b = builder.get_branch()
+        b.tags.set_tag('tip-1.0', 'A')
+        return builder, b, b.tags.get_tag_dict()
+
+    def test_get_most_recent_tag_tip(self):
+        builder, b, tag_dict = self.make_simple_builder()
+        self.assertEqual('tip-1.0',
+                         lp_api_lite.get_most_recent_tag(tag_dict, b))
+
+    def test_get_most_recent_tag_older(self):
+        builder, b, tag_dict = self.make_simple_builder()
+        builder.build_snapshot('B', ['A'], [])
+        self.assertEqual('B', b.last_revision())
+        self.assertEqual('tip-1.0',
+                         lp_api_lite.get_most_recent_tag(tag_dict, b))
+
+
+class StubLatestPublication(object):
+
+    def __init__(self, latest):
+        self.called = False
+        self.latest = latest
+
+    def get_latest_version(self):
+        self.called = True
+        return self.latest
+
+    def place(self):
+        return 'Ubuntu Natty'
+
+
+class TestReportFreshness(tests.TestCaseWithMemoryTransport):
+
+    def setUp(self):
+        super(TestReportFreshness, self).setUp()
+        builder = self.make_branch_builder('tip')
+        builder.build_snapshot('A', [], [
+            ('add', ('', 'root-id', 'directory', None))])
+        self.branch = builder.get_branch()
+
+    def assertFreshnessReports(self, verbosity, latest_version, content):
+        """Assert that lp_api_lite.report_freshness reports the given content.
+
+        :param verbosity: The reporting level
+        :param latest_version: The version reported by StubLatestPublication
+        :param content: The expected content. This should be in DocTest form.
+        """
+        orig_log_len = len(self.get_log())
+        lp_api_lite.report_freshness(self.branch, verbosity,
+            StubLatestPublication(latest_version))
+        new_content = self.get_log()[orig_log_len:]
+        # Strip out lines that have LatestPublication.get_* because those are
+        # timing related lines. While interesting to log for now, they aren't
+        # something we want to be testing
+        new_content = new_content.split('\n')
+        for i in range(2):
+            if (len(new_content) > 0
+                and 'LatestPublication.get_' in new_content[0]):
+                new_content = new_content[1:]
+        new_content = '\n'.join(new_content)
+        self.assertThat(new_content,
+            DocTestMatches(content,
+                doctest.ELLIPSIS | doctest.REPORT_UDIFF))
+
+    def test_verbosity_off_skips_check(self):
+        # We force _get_package_branch_info so that we know it would otherwise
+        # try to connect to launcphad
+        self.overrideAttr(launchpad, '_get_package_branch_info',
+            lambda x: ('ubuntu', 'natty', 'bzr'))
+        self.overrideAttr(lp_api_lite, 'LatestPublication',
+            lambda *args: self.fail('Tried to query launchpad'))
+        c = self.branch.get_config()
+        c.set_user_option('launchpad.packaging_verbosity', 'off')
+        orig_log_len = len(self.get_log())
+        launchpad._check_is_up_to_date(self.branch)
+        new_content = self.get_log()[orig_log_len:]
+        self.assertContainsRe(new_content,
+            'not checking memory.*/tip/ because verbosity is turned off')
+
+    def test_verbosity_off(self):
+        latest_pub = StubLatestPublication('1.0-1ubuntu2')
+        lp_api_lite.report_freshness(self.branch, 'off', latest_pub)
+        self.assertFalse(latest_pub.called)
+
+    def test_verbosity_all_out_of_date_smoke(self):
+        self.branch.tags.set_tag('1.0-1ubuntu1', 'A')
+        self.assertFreshnessReports('all', '1.0-1ubuntu2',
+             '    INFO  Most recent Ubuntu Natty version: 1.0-1ubuntu2\n'
+             'Packaging branch version: 1.0-1ubuntu1\n'
+             'Packaging branch status: OUT-OF-DATE\n')
+
+
+class Test_GetNewestVersions(tests.TestCaseWithMemoryTransport):
+
+    def setUp(self):
+        super(Test_GetNewestVersions, self).setUp()
+        builder = self.make_branch_builder('tip')
+        builder.build_snapshot('A', [], [
+            ('add', ('', 'root-id', 'directory', None))])
+        self.branch = builder.get_branch()
+
+    def assertLatestVersions(self, latest_branch_version, pub_version):
+        if latest_branch_version is not None:
+            self.branch.tags.set_tag(latest_branch_version, 'A')
+        latest_pub = StubLatestPublication(pub_version)
+        self.assertEqual((pub_version, latest_branch_version),
+            lp_api_lite._get_newest_versions(self.branch, latest_pub))
+
+    def test_no_tags(self):
+        self.assertLatestVersions(None, '1.0-1ubuntu2')
+
+    def test_out_of_date(self):
+        self.assertLatestVersions('1.0-1ubuntu1', '1.0-1ubuntu2')
+
+    def test_up_to_date(self):
+        self.assertLatestVersions('1.0-1ubuntu2', '1.0-1ubuntu2')
+
+    def test_missing(self):
+        self.assertLatestVersions(None, None)
+
+
+class Test_ReportFreshness(tests.TestCase):
+
+    def assertReportedFreshness(self, verbosity, latest_ver, branch_latest_ver,
+                               content, place='Ubuntu Natty'):
+        """Assert that lp_api_lite.report_freshness reports the given content.
+        """
+        reported = []
+        def report_func(value):
+            reported.append(value)
+        lp_api_lite._report_freshness(latest_ver, branch_latest_ver, place,
+                                      verbosity, report_func)
+        new_content = '\n'.join(reported)
+        self.assertThat(new_content,
+            DocTestMatches(content,
+                doctest.ELLIPSIS | doctest.REPORT_UDIFF))
+
+    def test_verbosity_minimal_no_tags(self):
+        self.assertReportedFreshness('minimal', '1.0-1ubuntu2', None,
+            'Branch is OUT-OF-DATE, Ubuntu Natty has 1.0-1ubuntu2\n')
+
+    def test_verbosity_minimal_out_of_date(self):
+        self.assertReportedFreshness('minimal', '1.0-1ubuntu2', '1.0-1ubuntu1',
+            '1.0-1ubuntu1 is OUT-OF-DATE,'
+            ' Ubuntu Natty has 1.0-1ubuntu2\n')
+
+    def test_verbosity_minimal_up_to_date(self):
+        self.assertReportedFreshness('minimal', '1.0-1ubuntu2', '1.0-1ubuntu2',
+             '')
+
+    def test_verbosity_minimal_missing(self):
+        self.assertReportedFreshness('minimal', None, None,
+             '')
+
+    def test_verbosity_short_out_of_date(self):
+        self.assertReportedFreshness('short', '1.0-1ubuntu2', '1.0-1ubuntu1',
+            '1.0-1ubuntu1 is OUT-OF-DATE,'
+            ' Ubuntu Natty has 1.0-1ubuntu2\n')
+
+    def test_verbosity_short_up_to_date(self):
+        self.assertReportedFreshness('short', '1.0-1ubuntu2', '1.0-1ubuntu2',
+             '1.0-1ubuntu2 is CURRENT in Ubuntu Natty')
+
+    def test_verbosity_short_missing(self):
+        self.assertReportedFreshness('short', None, None,
+             'Ubuntu Natty is MISSING a version')
+
+    def test_verbosity_all_no_tags(self):
+        self.assertReportedFreshness('all', '1.0-1ubuntu2', None,
+             'Most recent Ubuntu Natty version: 1.0-1ubuntu2\n'
+             'Packaging branch version: None\n'
+             'Packaging branch status: OUT-OF-DATE\n')
+
+    def test_verbosity_all_out_of_date(self):
+        self.assertReportedFreshness('all', '1.0-1ubuntu2', '1.0-1ubuntu1',
+             'Most recent Ubuntu Natty version: 1.0-1ubuntu2\n'
+             'Packaging branch version: 1.0-1ubuntu1\n'
+             'Packaging branch status: OUT-OF-DATE\n')
+
+    def test_verbosity_all_up_to_date(self):
+        self.assertReportedFreshness('all', '1.0-1ubuntu2', '1.0-1ubuntu2',
+             'Most recent Ubuntu Natty version: 1.0-1ubuntu2\n'
+             'Packaging branch status: CURRENT\n')
+
+    def test_verbosity_all_missing(self):
+        self.assertReportedFreshness('all', None, None,
+             'Most recent Ubuntu Natty version: MISSING\n')
+
+    def test_verbosity_None_is_all(self):
+        self.assertReportedFreshness(None, '1.0-1ubuntu2', '1.0-1ubuntu2',
+             'Most recent Ubuntu Natty version: 1.0-1ubuntu2\n'
+             'Packaging branch status: CURRENT\n')

=== modified file 'doc/en/release-notes/bzr-2.4.txt'
--- doc/en/release-notes/bzr-2.4.txt	2011-07-18 14:43:35 +0000
+++ doc/en/release-notes/bzr-2.4.txt	2011-07-26 08:23:04 +0000
@@ -32,6 +32,31 @@
 .. Fixes for situations where bzr would previously crash or give incorrect
    or undesirable results.
 
+* Accessing a packaging branch on Launchpad (eg, ``lp:ubuntu/bzr``) now
+  checks to see if the most recent published source package version for
+  that project is present in the branch tags. This should help developers
+  trust whether the packaging branch is up-to-date and can be used for new
+  changes. The level of verbosity is controlled by the config item
+  ``launchpad.packaging_verbosity``. It can be set to one of
+
+  off
+    disable all checks
+
+
+  minimal
+    only display if the branch is out-of-date
+
+  short
+    also display single-line up-to-date and missing,
+
+
+  all
+    (default) display multi-line content for all states
+
+
+  (John Arbash Meinel, #609187, #812928)
+
+
 * The fix for bug #513709 caused us to open a new connection when
   switching a lightweight checkout that was pointing at a bound branch.
   This isn't necessary because we know the master URL without opening it,