Merge ~racb/git-ubuntu:fix-publication-list-truncation into git-ubuntu:main

[Robie Basak]

This is to fix the current performance issue where the importer seems to begin importing from the beginning of time instead of resuming from only recent publications.

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:24bf8b20cdd2b589c2ff7b32ba92fcc27a21e321
https://jenkins.canonical.com/server-team/job/git-ubuntu-ci/5/
Executed test runs:
    SUCCESS: VM Setup
    SUCCESS: Build
    SUCCESS: VM Reset
    SUCCESS: Unit Tests
    IN_PROGRESS: Declarative: Post Actions

Click here to trigger a rebuild:
https://jenkins.canonical.com/server-team/job/git-ubuntu-ci/5//rebuild

review: Approve (continuous-integration)

[Bryce Harrington (bryce)]

commit 002e452c9dd413b9a22b7ff8d0d3c277a3ae0a3a
  - Verified list exists at https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel
  - List adopted for git-ubuntu use on https://lists.ubuntu.com/archives/ubuntu-distributed-devel/2018-May/001222.html
  - LGTM

commit 2f0e8556c1a0c36ffefe80cfc3c0f5e3adb49b28
  - Is the name "USD Import Team" still valid as the name for the new list? If not, then the from line in the CHANGELOG_TEMPLATE needs updated.
  - `grep -sir usd-import-team .` shows additional references to the old team in setup.py and a few docs. The urls no longer resolve, so guessing this commit would be a good point to update all those too.
  - There are also other mentions of 'usd-import' in the codebase, such as some man page references to bugs.launchpad.net/usd-importer and git.launchpad.net/usd-importer, etc. which no longer resolve, and to some other email addresses that may no longer be valid. Maybe you want to cleanup those in a different commit in this branch, or in this commit?
  - Needs Fixing

commit 5aad1110b2b849e4704c37751cf656ae14e3b3bb
  - No other references to 'usd-importer-bot' in the codebase
  - There is still the 'usd-importer' service, but afaict that'll keep the name, and in any case is unrelated to the bot.
  - LGTM

commit b312ec9a2de2c0835fb19bae6527f59fb8ea6b97
  - Verified no other references to get_heads_and_versions()
  - Are you using namedtuple instead of an ordinary class for performance reasons? Or for test parameterization convenience? You've used namedtuples elsewhere in the codebase so wonder if there's a reason or just a style thing. Just curious, not suggesting it be changed.
  - In the :returns: for get_head_info() it documents the parameters in the namedtuple; would be better to just put those parameter docs with the HeadInfoItem namedtuple definition.
  - Needs Fixing

commit 8bace066b8c96d214d8f2bb04f93c791ee82e402
  - Since this is a bug fix (and the principle change for this branch), there should be a bug report filed for the issue for tracking.
  - The irclog is useful for reference, however I would be concerned that irc logs might be ephemeral, so please include a synopsis/summary of the discussion in the bug report.
  - I notice this drops checking if the date is defined before using it; presumably you're counting on date_created always being defined, is that true?
  - I don't know if 100 chars is considered too long a line length, but storing spi.head_name(namespace) to a local var could help shorten it.
  - This commit might be a good point to add code docs for _head_version_is_equal(). (Not sure if we bother with code docs for internal methods though.)
  - Needs Fixing

commit 2be6d7d8bef25b03a850b519402eef7cfa2bac5a
  - Better code doc for the test might be "get_head_info() extracts expected commit and version info from the head commit."
  - Otherwise, looks good to me.

commit 172c33290f7faa2d8d6ff903213df3ae140c9f51
  - LGTM

commit 793e8e3dcbf57df0e01f3c6c967e96a32b981761
  - LGTM

commit 5fd2d23b89b13240cc711f80a4d438632632981d
  - LGTM

commit 24bf8b20cdd2b589c2ff7b32ba92fcc27a21e321
  - The parameterization here is very clean, well documented, and easy to f...

review: Needs Fixing

[Robie Basak (racb)]

On Tue, Jun 21, 2022 at 05:08:56PM -0000, Bryce Harrington wrote:
> commit 002e452c9dd413b9a22b7ff8d0d3c277a3ae0a3a
> - Verified list exists at https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel
> - List adopted for git-ubuntu use on https://lists.ubuntu.com/archives/ubuntu-distributed-devel/2018-May/001222.html
> - LGTM
>
> commit 2f0e8556c1a0c36ffefe80cfc3c0f5e3adb49b28
> - Is the name "USD Import Team" still valid as the name for the new list? If not, then the from line in the CHANGELOG_TEMPLATE needs updated.
> - `grep -sir usd-import-team .` shows additional references to the old team in setup.py and a few docs. The urls no longer resolve, so guessing this commit would be a good point to update all those too.
> - There are also other mentions of 'usd-import' in the codebase, such as some man page references to bugs.launchpad.net/usd-importer and git.launchpad.net/usd-importer, etc. which no longer resolve, and to some other email addresses that may no longer be valid. Maybe you want to cleanup those in a different commit in this branch, or in this commit?
> - Needs Fixing
>
> commit 5aad1110b2b849e4704c37751cf656ae14e3b3bb
> - No other references to 'usd-importer-bot' in the codebase
> - There is still the 'usd-importer' service, but afaict that'll keep the name, and in any case is unrelated to the bot.
> - LGTM

These three commits aren't actually part of this MP. I'm not sure how they
ended up in your review. Your points sound valid but I'll address them
separately from this MP.

> commit b312ec9a2de2c0835fb19bae6527f59fb8ea6b97
> - Verified no other references to get_heads_and_versions()
> - Are you using namedtuple instead of an ordinary class for performance reasons? Or for test parameterization convenience? You've used namedtuples elsewhere in the codebase so wonder if there's a reason or just a style thing. Just curious, not suggesting it be changed.

I find myself naturally graduating from a single value to a 2-tuple to a
namedtuple.

I don't go all the way to a full class because I think that constraining
the structure to a namedtuple conveys to the reader (and my future self)
that it's purely data that's being passed around, with no attached
special behaviour. The immutability helps avoid surprises too. There
can't be any accidental modifications made by a called function, which
is a class of bug that is painful to detect and pin down were it to
happen.

If you're curious, dataclasses (new since Python 3.7) are interesting as
a normal powerful version of namedtuples that are more easily extended.
But I don't usually need that power, so I've been sticking to
namedtuples most of the time.

> - In the :returns: for get_head_info() it documents the parameters in the namedtuple; would be better to just put those parameter docs with the HeadInfoItem namedtuple definition.
> - Needs Fixing

Fixed - I squashed this into the original commit (now f5dc43c).

> commit 8bace066b8c96d214d8f2bb04f93c791ee82e402
> - Since this is a bug fix (and the principle change for this branch), there should be a bug report filed for the issue for tracking.

Done: LP: #1979650.

> - The irclog is usef...

[Robie Basak (racb)]

Here's the range-diff of what I just pushed: https://paste.ubuntu.com/p/jcxbVgVCPT/

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:21338706797920b251bce5da6ff131e15dab60f3
https://jenkins.canonical.com/server-team/job/git-ubuntu-ci/6/
Executed test runs:
    SUCCESS: VM Setup
    SUCCESS: Build
    SUCCESS: VM Reset
    SUCCESS: Unit Tests
    IN_PROGRESS: Declarative: Post Actions

Click here to trigger a rebuild:
https://jenkins.canonical.com/server-team/job/git-ubuntu-ci/6//rebuild

review: Approve (continuous-integration)

[Bryce Harrington (bryce)]

Looks good, +1

review: Approve

diff --git a/gitubuntu/git_repository.py b/gitubuntu/git_repository.py
index d73972b..1ef6548 100644
--- a/gitubuntu/git_repository.py
+++ b/gitubuntu/git_repository.py
@@ -1026,6 +1026,25 @@ def datetime_to_signature_spec(datetime):
     return int(datetime.timestamp()), offset_mins
 
 
+class HeadInfoItem(collections.namedtuple(
+    'HeadInfoItem',
+    [
+        'version',
+        'commit_time',
+        'commit_id',
+    ],
+)):
+    """Information associated with a single branch head
+
+    :ivar str version: the package version found in debian/changelog at the
+        branch head.
+    :ivar int commit_time: the timestamp of the commit at the branch head,
+        expressed as seconds since the Unix epoch.
+    :ivar pygit2.Oid commit_id: the hash of the commit at the branch head.
+    """
+    pass
+
+
 class GitUbuntuRepository:
     """A class for interacting with an importer git repository
 
@@ -1888,33 +1907,37 @@ class GitUbuntuRepository:
                 'Cannot get changelog source package name'
             )
 
-    def get_heads_and_versions(self, head_prefix, namespace):
-        """Extract the last version in debian/changelog of all
-        '<namespace>/<head_prefix>/debian/*' and
-        '<namespace>/<head_prefix>/ubuntu/*' branches.
+    def get_head_info(self, head_prefix, namespace):
+        """Extract package versions at branch heads
 
-        :rtype: dict
-        :returns: a dictionary keyed by branch name (without a 'refs/heads/'
-            prefix). The value is a dictionary with two keys: 'head' and
-            'version'. The 'head' value is a pygit2.Branch object. The
-            'version' value is the package version string found at that branch
-            head.
+        Extract the version from debian/changelog of all
+        f'{namespace}/{head_prefix>/*' branches, excluding any branch that
+        contains 'ubuntu/devel'.
+
+        :param str namespace: the namespace under which git refs are found
+        :param str head_prefix: the prefix to look for
+        :rtype: dict(str, HeadInfoItem)
+        :returns: a dictionary keyed by the namespaced branch name (ie. without
+            a 'refs/heads/' prefix but with the namespace prefix, eg.
+            'importer/ubuntu/focal-devel').
         """
-        versions = dict()
-        errors = False
+        head_info = dict()
         for head in self.local_branches:
             prefix = '%s/%s' % (namespace, head_prefix)
             if not head.branch_name.startswith(prefix):
                 continue
             if 'ubuntu/devel' in head.branch_name:
                 continue
-            versions[head.branch_name] = dict()
-            versions[head.branch_name]['head'] = head
-            versions[head.branch_name]['version'], _ = (
+            version, _ = (
                 self.get_changelog_versions_from_treeish(str(head.peel().id))
             )
+            head_info[head.branch_name] = HeadInfoItem(
+                version=version,
+                commit_time=head.peel().commit_time,
+                commit_id=head.peel().id,
+            )
 
-        return versions
+        return head_info
 
     def treeishs_identical(self, treeish_string1, treeish_string2):
         if treeish_string1 is None or treeish_string2 is None:
diff --git a/gitubuntu/git_repository_test.py b/gitubuntu/git_repository_test.py
index c822ff1..838c31e 100644
--- a/gitubuntu/git_repository_test.py
+++ b/gitubuntu/git_repository_test.py
@@ -12,6 +12,7 @@ import pygit2
 import pytest
 
 import gitubuntu.git_repository as target
+from gitubuntu.git_repository import HeadInfoItem
 from gitubuntu.repo_builder import (
     Blob,
     Commit,
@@ -1157,3 +1158,31 @@ def test_get_all_reimport_tags(repo):
         'refs/tags/importer/reimport/import/1/0',
         'refs/tags/importer/reimport/import/1/1',
     }
+
+
+def test_get_head_info(repo):
+    """get_head_info() extracts expected commit and version info from the head
+    commit
+    """
+    Repo(
+        commits=[
+            Commit(name='foo', tree=SourceTree(source=Source())),
+        ],
+        branches={
+            'importer/ubuntu/foo': Placeholder('foo'),
+            'importer/debian/bar': Placeholder('foo'),
+            'other/ubuntu/baz': Placeholder('foo'),
+        },
+    ).write(repo.raw_repo)
+    foo_commit_id = (
+        repo.raw_repo
+        .lookup_reference('refs/heads/importer/ubuntu/foo')
+        .peel(pygit2.Commit).id
+    )
+    assert repo.get_head_info('ubuntu', 'importer') == {
+        'importer/ubuntu/foo': HeadInfoItem(
+            version='1-1',
+            commit_time=0,
+            commit_id=foo_commit_id,
+        ),
+    }
diff --git a/gitubuntu/importer.py b/gitubuntu/importer.py
index 8850233..0bbc718 100644
--- a/gitubuntu/importer.py
+++ b/gitubuntu/importer.py
@@ -251,53 +251,53 @@ def _main_with_repo(
         retry_backoffs,
     )
 
-    debian_head_versions = (
-        repo.get_heads_and_versions('debian', namespace)
+    debian_head_info = (
+        repo.get_head_info('debian', namespace)
     )
-    for head_name in sorted(debian_head_versions):
+    for head_name in sorted(debian_head_info):
         _, _, pretty_head_name = head_name.partition('%s/' % namespace)
         logging.debug('Last imported %s version is %s',
             pretty_head_name,
-            debian_head_versions[head_name]['version']
+            debian_head_info[head_name].version
         )
 
-    ubuntu_head_versions = (
-        repo.get_heads_and_versions('ubuntu', namespace)
+    ubuntu_head_info = (
+        repo.get_head_info('ubuntu', namespace)
     )
-    for head_name in sorted(ubuntu_head_versions):
+    for head_name in sorted(ubuntu_head_info):
         _, _, pretty_head_name = head_name.partition('%s/' % namespace)
         logging.debug('Last imported %s version is %s',
             pretty_head_name,
-            ubuntu_head_versions[head_name]['version']
+            ubuntu_head_info[head_name].version
         )
 
     if not skip_applied:
-        applied_debian_head_versions = (
-            repo.get_heads_and_versions(
+        applied_debian_head_info = (
+            repo.get_head_info(
                 'applied/debian', namespace
             )
         )
-        for head_name in sorted(applied_debian_head_versions):
+        for head_name in sorted(applied_debian_head_info):
             _, _, pretty_head_name = head_name.partition(
                 '%s/' % namespace
             )
             logging.debug('Last applied %s version is %s',
                 pretty_head_name,
-                applied_debian_head_versions[head_name]['version']
+                applied_debian_head_info[head_name].version
             )
 
-        applied_ubuntu_head_versions = (
-            repo.get_heads_and_versions(
+        applied_ubuntu_head_info = (
+            repo.get_head_info(
                 'applied/ubuntu', namespace
             )
         )
-        for head_name in sorted(applied_ubuntu_head_versions):
+        for head_name in sorted(applied_ubuntu_head_info):
             _, _, pretty_head_name = head_name.partition(
                 '%s/' % namespace
             )
             logging.debug('Last applied %s version is %s',
                 pretty_head_name,
-                applied_ubuntu_head_versions[head_name]['version']
+                applied_ubuntu_head_info[head_name].version
             )
 
     oldcwd = os.getcwd()
@@ -321,8 +321,8 @@ def _main_with_repo(
         pkgname=pkgname,
         namespace=namespace,
         patches_applied=False,
-        debian_head_versions=debian_head_versions,
-        ubuntu_head_versions=ubuntu_head_versions,
+        debian_head_info=debian_head_info,
+        ubuntu_head_info=ubuntu_head_info,
         debian_sinfo=debian_sinfo,
         ubuntu_sinfo=ubuntu_sinfo,
         active_series_only=active_series_only,
@@ -345,8 +345,8 @@ def _main_with_repo(
             pkgname=pkgname,
             namespace=namespace,
             patches_applied=True,
-            debian_head_versions=applied_debian_head_versions,
-            ubuntu_head_versions=applied_ubuntu_head_versions,
+            debian_head_info=applied_debian_head_info,
+            ubuntu_head_info=applied_ubuntu_head_info,
             debian_sinfo=debian_sinfo,
             ubuntu_sinfo=ubuntu_sinfo,
             active_series_only=active_series_only,
@@ -1102,7 +1102,7 @@ def override_parents(parent_overrides, repo, version, namespace):
 
 def _devel_branch_updates(
     ref_prefix,
-    head_versions,
+    head_info,
     series_name_list,
 ):
     """Calculate devel branch pointer commits.
@@ -1113,11 +1113,10 @@ def _devel_branch_updates(
     Also yield a further generic 'ubuntu/devel' ref_name with its target.
 
     :param: ref_prefix: what to stick on the front of all refs yielded and to
-        look for in head_versions
+        look for in head_info
 
-    :param: head_versions: a dictionary
-        {ref_name: (hash_string, version_string)} describing which package
-        version each ref is currently on.
+    :param: head_info: a dictionary as returned by
+        GitUbuntuRepository.get_head_info()
 
     :param: series_name_list: a list of strings of series names to consider
         with most recent series first.
@@ -1135,12 +1134,11 @@ def _devel_branch_updates(
         # order of pockets
         for suff in ['-proposed', '-updates', '-security', '']:
             name = "%subuntu/%s%s" % (ref_prefix, series_name, suff)
-            if name in head_versions:
-                head_commit, head_version = head_versions[name]
-                if version_compare(head_version, series_devel_version) > 0:
+            if name in head_info:
+                if version_compare(head_info[name].version, series_devel_version) > 0:
                     series_devel_name = name
-                    series_devel_version = head_version
-                    series_devel_hash = head_commit
+                    series_devel_version = head_info[name].version
+                    series_devel_hash = str(head_info[name].commit_id)
         if series_devel_hash:
             series_devel_branch_name = '%subuntu/%s-devel' % (
                 ref_prefix,
@@ -1187,16 +1185,12 @@ def update_devel_branches(
     archive
     """
     for applied_prefix in ['', 'applied/']:
-        head_versions = {
-            ref: (str(target['head'].peel().id), target['version'])
-            for ref, target in repo.get_heads_and_versions(
-                    '%subuntu' % applied_prefix,
-                    namespace=namespace,
-                ).items()
-        }
         for ref, commit in _devel_branch_updates(
             ref_prefix=('%s/%s' % (namespace, applied_prefix)),
-            head_versions=head_versions,
+            head_info=repo.get_head_info(
+                head_prefix='%subuntu' % applied_prefix,
+                namespace=namespace,
+            ),
             series_name_list=ubuntu_sinfo.all_series_name_list,
         ):
             if commit is None:
@@ -2521,8 +2515,8 @@ def import_publishes(
     pkgname,
     namespace,
     patches_applied,
-    debian_head_versions,
-    ubuntu_head_versions,
+    debian_head_info,
+    ubuntu_head_info,
     debian_sinfo,
     ubuntu_sinfo,
     active_series_only,
@@ -2559,8 +2553,8 @@ def import_publishes(
             changelog_date_overrides=changelog_date_overrides,
         )
     # Always look in Ubuntu for new publications
-    gusi_head_versions_tuple_list = [
-        (ubuntu_sinfo, ubuntu_head_versions),
+    gusi_head_info_tuple_list = [
+        (ubuntu_sinfo, ubuntu_head_info),
     ]
     # Unless --active-series-only was specified, also look in Debian for new
     # publications.
@@ -2568,16 +2562,16 @@ def import_publishes(
         # For hash stability, we must process new Debian publications with the
         # same date_created before new Ubuntu publications, so we insert the
         # Debian entry before the Ubuntu entry.
-        gusi_head_versions_tuple_list.insert(
+        gusi_head_info_tuple_list.insert(
             0,
-            (debian_sinfo, debian_head_versions),
+            (debian_sinfo, debian_head_info),
         )
     try:
         # We must process new publications in order of date_created interleaved
         # across both distributions (Debian first) in order to maintain hash
         # stability according to the spec.
         for srcpkg_information in GitUbuntuSourceInformation.interleave_launchpad_versions_published_after(
-            gusi_head_versions_tuple_list,
+            gusi_head_info_tuple_list,
             namespace=namespace,
             workdir=workdir,
             active_series_only=active_series_only,
diff --git a/gitubuntu/importer_test.py b/gitubuntu/importer_test.py
index 05cafb0..433450f 100644
--- a/gitubuntu/importer_test.py
+++ b/gitubuntu/importer_test.py
@@ -17,6 +17,7 @@ import pkg_resources
 import pygit2
 import tenacity
 
+from gitubuntu.git_repository import HeadInfoItem
 from gitubuntu.patch_state import PatchState
 import gitubuntu.repo_builder as repo_builder
 from gitubuntu.repo_builder import Placeholder
@@ -33,43 +34,50 @@ from gitubuntu.test_fixtures import pygit2_repo, repo
 @pytest.mark.parametrize('head_versions,expected', [
     (
         {
-            'prefix/ubuntu/trusty': (sentinel.a, '1.0-1'),
+            'prefix/ubuntu/trusty': ('a', '1.0-1'),
         },
         {
-            'prefix/ubuntu/trusty-devel': sentinel.a,
-            'prefix/ubuntu/devel': sentinel.a,
+            'prefix/ubuntu/trusty-devel': 'a',
+            'prefix/ubuntu/devel': 'a',
         }
     ),
     (
         {
-            'prefix/ubuntu/trusty': (sentinel.t, '1.0-1'),
-            'prefix/ubuntu/xenial': (sentinel.x, '1.0-2'),
-            'prefix/ubuntu/yakkety': (sentinel.y, '1.0-3'),
+            'prefix/ubuntu/trusty': ('t', '1.0-1'),
+            'prefix/ubuntu/xenial': ('x', '1.0-2'),
+            'prefix/ubuntu/yakkety': ('y', '1.0-3'),
         },
         {
-            'prefix/ubuntu/trusty-devel': sentinel.t,
-            'prefix/ubuntu/xenial-devel': sentinel.x,
-            'prefix/ubuntu/yakkety-devel': sentinel.y,
-            'prefix/ubuntu/devel': sentinel.y,
+            'prefix/ubuntu/trusty-devel': 't',
+            'prefix/ubuntu/xenial-devel': 'x',
+            'prefix/ubuntu/yakkety-devel': 'y',
+            'prefix/ubuntu/devel': 'y',
         }
     ),
     (
         {
-            'prefix/ubuntu/trusty': (sentinel.t, '1.0-1'),
-            'prefix/ubuntu/trusty-security': (sentinel.ts, '1.0-1.2'),
-            'prefix/ubuntu/trusty-proposed': (sentinel.tp, '1.0-1.1'),
+            'prefix/ubuntu/trusty': ('t', '1.0-1'),
+            'prefix/ubuntu/trusty-security': ('ts', '1.0-1.2'),
+            'prefix/ubuntu/trusty-proposed': ('tp', '1.0-1.1'),
         },
         {
-            'prefix/ubuntu/trusty-devel': sentinel.ts,
-            'prefix/ubuntu/devel': sentinel.ts,
+            'prefix/ubuntu/trusty-devel': 'ts',
+            'prefix/ubuntu/devel': 'ts',
         }
     ),
 ])
 def test_devel_branch_updates(head_versions, expected):
     result = target._devel_branch_updates(
-        'prefix/',
-        head_versions,
-        ['yakkety', 'xenial', 'trusty'],
+        ref_prefix='prefix/',
+        head_info={
+            head: HeadInfoItem(
+                commit_id=commit_id,
+                version=version,
+                commit_time=None
+            )
+            for head, (commit_id, version) in head_versions.items()
+        },
+        series_name_list=['yakkety', 'xenial', 'trusty'],
     )
     assert set(result) == set(expected.items())
 
diff --git a/gitubuntu/source_information.py b/gitubuntu/source_information.py
index fbe2ee5..da79c5c 100644
--- a/gitubuntu/source_information.py
+++ b/gitubuntu/source_information.py
@@ -394,9 +394,10 @@ class GitUbuntuSourceInformation(object):
     def __init__(self, dist_name, pkgname=None,
                  pull_overrides_filename='/dev/null',
                  retries=0,
-                 retry_backoffs=[]
+                 retry_backoffs=[],
+                 lp=None,
                 ):
-        self.launchpad = launchpad_login()
+        self.launchpad = lp or launchpad_login()
         self.dist_name = dist_name
         if self.dist_name.startswith('ppa:'):
             _, ppa = dist_name.split(':')
@@ -421,16 +422,36 @@ class GitUbuntuSourceInformation(object):
         self.retry_backoffs = retry_backoffs
 
     @staticmethod
-    def _head_version_is_equal(head_versions, namespace, spi):
+    def _spi_found_in_relevant_head(head_info, namespace, spi):
+        """Determine if a branch head matches the given source publication
+
+        Consider the branch head that is expected to be updated were the source
+        publication to be imported. If the source publication appears to
+        already be imported there, then this method returns True.
+
+        This is assumed if the commit timestamp at the branch head matches the
+        date_created timestamp of the source publication, and the package
+        version strings also match.
+
+        :param dict(str, HeadInfoItem) head_info: the state of the relevant
+            branch heads, as returned by GitUbuntuRepository.get_head_info().
+        :param str namespace: the namespace prefix used in the git repository.
+        :param GitUbuntuSourcePackageInformation spi: the Launchpad publication
+            to consider.
+        :rtype: bool
+        :returns: True if the spi matches the relevant branch head
+        """
         try:
-            if (head_versions[spi.head_name(namespace)]['version'] == spi.version and
-                (spi.date_published is None or
-                 int(spi.date_published.timestamp()) == head_versions[spi.head_name(namespace)]['head'].peel().commit_time)
-                ):
-                 return True
-            return False
+            head_info_item = head_info[spi.head_name(namespace)]
         except KeyError:
+            # We don't have information for the relevant branch. Probably
+            # because it doesn't exist. So our answer is no: we don't appear to
+            # have this source publication imported.
             return False
+        return (
+            head_info_item.version == spi.version
+            and int(spi.date_created.timestamp()) == head_info_item.commit_time
+        )
 
     @property
     def current_series(self):
@@ -552,7 +573,7 @@ class GitUbuntuSourceInformation(object):
 
     def launchpad_versions_published_after(
         self,
-        head_versions,
+        head_info,
         namespace,
         workdir=None,
         active_series_only=False,
@@ -561,13 +582,13 @@ class GitUbuntuSourceInformation(object):
 
         Return a sequence of GitUbuntuSourcePackageInformation instances
         representing Launchpad publications created in a Launchpad distribution
-        after a particular point, as determined by the head_versions parameter.
+        after a particular point, as determined by the head_info parameter.
 
-        :param dict head_versions: as returned by
-            GitUbuntuRepository.get_heads_and_versions(). This may be empty, in
+        :param dict head_info: as returned by
+            GitUbuntuRepository.get_head_info(). This may be empty, in
             which case all publications are returned. Otherwise, publications
-            are skipped that match a version already found in head_versions, or
-            with a date_created older than what is found in head_versions, is
+            are skipped that match a version already found in head_info, or
+            with a date_created older than what is found in head_info, is
             skipped, as well as any publications that have a prior date_created
             date.
         :param str namespace: the namespace prefix used in the git repository.
@@ -577,11 +598,6 @@ class GitUbuntuSourceInformation(object):
             an active series, as determined by self.active_series_name_list.
         :rtype: sequence(GitUbuntuSourcePackageInformation)
         """
-        args = {
-            'exact_match': True,
-            'source_name': self.pkgname,
-            'order_by_date': True,
-        }
 
         # we have the date of the commit too, so we can double-check
         # that it matches
@@ -598,7 +614,11 @@ class GitUbuntuSourceInformation(object):
         # versions, as there are cases (clamav 0.91.2-3ubuntu2.2~feisty1
         # in feisty-backports of a 'next' version being prior in the
         # ordering)
-        spph = self.archive.getPublishedSources(**args)
+        spph = self.archive.getPublishedSources(
+            exact_match=True,
+            source_name=self.pkgname,
+            order_by_date=True,
+        )
         # Coherence check that the passed in srcpkg name has a publication
         # history
         if len(spph) == 0:
@@ -606,47 +626,24 @@ class GitUbuntuSourceInformation(object):
                 self.pkgname, self.dist_name
             )
             return
-        if len(head_versions) > 0:
-            _spph = list()
-            for spphr in spph:
-                spi = GitUbuntuSourcePackageInformation(spphr, self.dist_name,
-                                                        workdir=workdir)
-                if self._head_version_is_equal(head_versions, namespace, spi):
-                    break
-                _spph.append(spphr)
-            spph = _spph
-        spph_iter = reversed(spph)
-
-        seen_versions = list()
-        for srcpkg in spph_iter:
-            # skip carried forward versions between series/pocket
-            if (
-                srcpkg.source_package_version,
-                # Access through cache to avoid terrible performance
-                _get_cached_lp_link(srcpkg, 'distro_series'),
-                srcpkg.pocket,
-                srcpkg.date_published
-               ) \
-                 in seen_versions:
-                continue
-            spi = self.get_corrected_spi(srcpkg, workdir)
+        truncated_spph = list()
+        for spphr in spph:
+            spi = GitUbuntuSourcePackageInformation(
+                spphr, self.dist_name, workdir=workdir
+            )
+            if self._spi_found_in_relevant_head(head_info, namespace, spi):
+                break
+            truncated_spph.append(spphr)
+
+        for spphr in reversed(truncated_spph):
+            spi = self.get_corrected_spi(spphr, workdir)
             if active_series_only and spi.series.name.lower() not in self.active_series_name_list:
                 continue
-            seen_versions.append(
-                (
-                srcpkg.source_package_version,
-                # Access through cache to avoid terrible performance
-                _get_cached_lp_link(srcpkg, 'distro_series'),
-                srcpkg.pocket,
-                srcpkg.date_published
-                )
-               )
             yield spi
-        return
 
     @staticmethod
     def interleave_launchpad_versions_published_after(
-        gusi_head_versions_tuple_list,
+        gusi_head_info_tuple_list,
         namespace,
         workdir=None,
         active_series_only=False,
@@ -662,16 +659,16 @@ class GitUbuntuSourceInformation(object):
         simultaneously and interleaves the results so that the caller sees a
         single combined sequence in ascending order of date_created. If
         date_created is the same, results are provided in the order that the
-        distributions are specified in gusi_head_versions_tuple_list. In other
+        distributions are specified in gusi_head_info_tuple_list. In other
         words, the distribution ordering is the secondary sort key.
 
         :param list(tuple(GitUbuntuSourceInformation, dict))
-            gusi_head_versions_tuple_list: a list of the parameters to use for
-            the underlying launchpad_versions_published_after() method call.
-            The GitUbuntuSourceInformation instance is the object to call the
-            method against. The second element of the tuple is the
-            head_versions parameter to pass to that method. The other
-            parameters are passed through to the underlying call as-is.
+            gusi_head_info_tuple_list: a list of the parameters to use for the
+            underlying launchpad_versions_published_after() method call. The
+            GitUbuntuSourceInformation instance is the object to call the
+            method against. The second element of the tuple is the head_info
+            parameter to pass to that method. The other parameters are passed
+            through to the underlying call as-is.
         :param namespace: passed through to
             launchpad_versions_published_after().
         :param workdir: passed through to launchpad_versions_published_after().
@@ -687,11 +684,11 @@ class GitUbuntuSourceInformation(object):
         # be able to sort according to priority as a secondary key to ensure
         # that if timestamps of a publication are the same across
         # distributions, the distribution mentioned first in
-        # gusi_head_versions_tuple_list will appear in the results first.
+        # gusi_head_info_tuple_list will appear in the results first.
         dist_priority = {
             gusi.dist.self_link: i
-            for i, (gusi, head_version)
-            in enumerate(gusi_head_versions_tuple_list)
+            for i, (gusi, head_info)
+            in enumerate(gusi_head_info_tuple_list)
         }
 
         # Now that we have a mapping of distribution_link to priority, the sort
@@ -706,15 +703,15 @@ class GitUbuntuSourceInformation(object):
             distribution_link = distro_series.distribution_link
             return spi._spphr.date_created, dist_priority[distribution_link]
 
-        # Create one generator per gusi_head_versions_tuple_list entry.
+        # Create one generator per gusi_head_info_tuple_list entry.
         spi_generators_to_interleave = [
             gusi.launchpad_versions_published_after(
-                head_versions=head_versions,
+                head_info=head_info,
                 namespace=namespace,
                 workdir=workdir,
                 active_series_only=active_series_only,
             )
-            for gusi, head_versions in gusi_head_versions_tuple_list
+            for gusi, head_info in gusi_head_info_tuple_list
         ]
 
         # heapq.merge() will now do the interleaving. This relies on the
diff --git a/gitubuntu/source_information_test.py b/gitubuntu/source_information_test.py
index 7e88d77..e88b072 100644
--- a/gitubuntu/source_information_test.py
+++ b/gitubuntu/source_information_test.py
@@ -4,6 +4,7 @@ from unittest.mock import Mock, sentinel
 
 import keyring
 import pytest
+from gitubuntu.git_repository import HeadInfoItem
 import gitubuntu.source_information as target
 from gitubuntu.source_information import derive_codename_from_series
 from gitubuntu import source_information
@@ -255,12 +256,12 @@ def test_interleave_launchpad_versions_published_after(
         ])
     )
 
-    gusi_head_versions_tuple_list = [
+    gusi_head_info_tuple_list = [
         (debian, None),
         (ubuntu, None),
     ]
     result = target.GitUbuntuSourceInformation.interleave_launchpad_versions_published_after(
-        gusi_head_versions_tuple_list=gusi_head_versions_tuple_list,
+        gusi_head_info_tuple_list=gusi_head_info_tuple_list,
         namespace=None,
     )
     expanded_result = list(result)
@@ -274,3 +275,226 @@ def test_interleave_launchpad_versions_published_after(
         )
         for date_created, dist in expected
     ]
+
+
[e-mail](['dist_name', 'input_head_info', 'input_spph'], [
+    # Empty case
+    (
+        'debian', {}, [],
+    ),
+    # Single publication, nothing imported yet
+    (
+        'debian',
+        {},
+        [
+            ('sid', '1', 1, True),
+        ],
+    ),
+    # Single publication, already imported
+    (
+        'debian',
+        {'sid': ('1', 1)},
+        [
+            ('sid', '1', 1, False),
+        ],
+    ),
+    # Two identical publications, nothing imported yet
+    (
+        'debian',
+        {},
+        [
+            ('sid', '1', 2, True),
+            ('bullseye', '1', 1, True),
+        ],
+    ),
+    # Two identical publications, already imported
+    (
+        'debian',
+        {
+            'sid': ('1', 1),
+            'bullseye': ('1', 1),
+        },
+        [
+            ('sid', '1', 2, True),
+            ('bullseye', '1', 1, False),
+        ],
+    ),
+    # Two versions in the normal proposed migration way, one imported
+    (
+        'debian',
+        {
+            'sid': ('1', 1),
+            'bullseye': ('1', 1),
+        },
+        [
+            ('bullseye', '2', 4, True),
+            ('sid', '2', 3, True),
+            ('bullseye', '1', 2, True),
+            ('sid', '1', 1, False),
+        ],
+    ),
+    # Two versions in the normal proposed migration way, both imported
+    (
+        'debian',
+        {
+            'sid': ('2', 3),
+            'bullseye': ('2', 3),
+        },
+        [
+            ('bullseye', '2', 4, True),
+            ('sid', '2', 3, False),
+            ('bullseye', '1', 2, False),
+            ('sid', '1', 1, False),
+        ],
+    ),
+    # Empty case
+    (
+        'ubuntu', {}, [],
+    ),
+    # Single publication, nothing imported yet
+    (
+        'ubuntu',
+        {},
+        [
+            ('kinetic-proposed', '1', 1, True),
+        ],
+    ),
+    # Single publication, already imported
+    (
+        'ubuntu',
+        {'kinetic-proposed': ('1', 1)},
+        [
+            ('kinetic-proposed', '1', 1, False),
+        ],
+    ),
+    # Two identical publications, nothing imported yet
+    (
+        'ubuntu',
+        {},
+        [
+            ('kinetic', '1', 2, True),
+            ('kinetic-proposed', '1', 1, True),
+        ],
+    ),
+    # Two identical publications, already imported
+    (
+        'ubuntu',
+        {
+            'kinetic': ('1', 1),
+            'kinetic-proposed': ('1', 1),
+        },
+        [
+            ('kinetic', '1', 2, True),
+            ('kinetic-proposed', '1', 1, False),
+        ],
+    ),
+    # Two versions in the normal proposed migration way, one imported
+    (
+        'ubuntu',
+        {
+            'kinetic-proposed': ('1', 1),
+            'kinetic': ('1', 1),
+        },
+        [
+            ('kinetic', '2', 4, True),
+            ('kinetic-proposed', '2', 3, True),
+            ('kinetic', '1', 2, True),
+            ('kinetic-proposed', '1', 1, False),
+        ],
+    ),
+    # Two versions in the normal proposed migration way, both imported
+    (
+        'ubuntu',
+        {
+            'kinetic-proposed': ('2', 3),
+            'kinetic': ('2', 3),
+        },
+        [
+            ('kinetic', '2', 4, True),
+            ('kinetic-proposed', '2', 3, False),
+            ('kinetic', '1', 2, False),
+            ('kinetic-proposed', '1', 1, False),
+        ],
+    ),
+])
+def test_launchpad_versions_published_after(
+    dist_name,
+    input_head_info,
+    input_spph,
+):
+    """If launchpad_versions_published_after() is shown some particular
+    head_info and some set of publications in Launchpad, then it should present
+    the expected subset of those publications for import.
+
+    :param str dist_name: either 'debian' or 'ubuntu'
+    :param dict input_head_info: a dictionary. Keyed by the branch name without
+        any namespace or prefix. The value is a tuple(str, int) of
+        (package_version, timestamp).
+    :param list input_spph: a list of mock publications from Launchpad in
+        reverse chronological order by date_created. Each entry is a 4-tuple
+        with the following fields:
+            str: the name of the apt suite the publication was made in
+            str: the package version string
+            int: date_created
+            bool: True if this entry should be included in the expected result
+    """
+    namespaced_input_head_info = {
+        f'importer/{dist_name}/{suite}':
+            HeadInfoItem(
+                version=version,
+                commit_time=commit_time,
+                commit_id=None,
+            )
+        for suite, (version, commit_time) in input_head_info.items()
+    }
+
+    dist = Mock()
+    lp = Mock(distributions={dist_name: dist})
+    getPublishedSources = dist.main_archive.getPublishedSources
+
+    getPublishedSources_return_value = []
+    for i, (suite, version, date_created, _) in enumerate(input_spph):
+        # Create a minimal fake record that would be returned by Launchpad's
+        # getPublishedSources method on the Launchpad archive object. We have
+        # to do this imperatively in order to set the name attributes
+        # (https://docs.python.org/3/library/unittest.mock.html#mock-names-and-the-name-attribute).
+        distribution_mock = Mock()
+        distribution_mock.name = dist_name
+        distro_series_mock = Mock(
+            distribution=distribution_mock,
+            distribution_link=dist_name,
+        )
+        distro_series_mock.name = suite.split('-')[0]
+        spphr_mock = Mock(
+            source_package_version=version,
+            date_created=Mock(timestamp=Mock(return_value=date_created)),
+            pocket=suite.split('-')[1] if '-' in suite else 'release',
+            distro_series=distro_series_mock,
+            distro_series_link=suite.split('-')[0],
+            test_record_id=i,
+        )
+        getPublishedSources_return_value.append(spphr_mock)
+    getPublishedSources.return_value = getPublishedSources_return_value
+
+    gusi = target.GitUbuntuSourceInformation(
+        dist_name=dist_name,
+        lp=lp,
+        pkgname=sentinel.pkgname,
+    )
+
+    actual_result = list(gusi.launchpad_versions_published_after(
+        head_info=namespaced_input_head_info,
+        namespace='importer',
+    ))
+
+    getPublishedSources.assert_called_once_with(
+        exact_match=True,
+        source_name=sentinel.pkgname,
+        order_by_date=True,
+    )
+
+    assert list(reversed([
+        i
+        for i, (_, _, _, expected_in_result) in enumerate(input_spph)
+        if expected_in_result
+    ])) == [m._spphr.test_record_id for m in actual_result]