Merge ~nacc/git-ubuntu:modernize-scripts-v2 into git-ubuntu:master

[Nish Aravamudan]

Make jenkins happy.

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:a341a5c452dde92ab1d0a319c8aac0f729cbd90f
https://jenkins.ubuntu.com/server/job/git-ubuntu-ci/195/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Style Check
    SUCCESS: Unit Tests
    SUCCESS: Integration Tests
    IN_PROGRESS: Declarative: Post Actions

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/git-ubuntu-ci/195/rebuild

review: Approve (continuous-integration)

[Robie Basak (racb)]

Why not put all the code into the gitubuntu module, with thin wrappers in bin/, like git-ubuntu, rather than have a separate infrastructure in scripts/ with a different additional sys.path hack? Or is there some kind of snap-related difficulty with this?

[Robie Basak (racb)]

If we're going with pylint and treating these as new, then none of this is currently pylint clean. I'm not saying it necessarily should be. What's your opinion?

[Nish Aravamudan (nacc)]

On Tue, Nov 7, 2017 at 6:24 AM, Robie Basak <email address hidden> wrote:
> Why not put all the code into the gitubuntu module, with thin wrappers in bin/, like git-ubuntu,

I can do this, but I don't want these scripts in the snap. Every
default change would imply a a new snap version, which forces everyone
to download a new update. While the xdelta gunk from the store should
make that minimal, in practice it does not.

> rather than have a separate infrastructure in scripts/ with a different additional sys.path hack?

I can move it to bin/, but it was mostly for keeping the code isolated
and clean. It's not something everyone is going to need to run. I
realize, though, the alias script, at least, should be in the snap
maybe.

[Robie Basak (racb)]

On Tue, Nov 07, 2017 at 04:04:53PM -0000, Nish Aravamudan wrote:
> On Tue, Nov 7, 2017 at 6:24 AM, Robie Basak <email address hidden> wrote:
> > Why not put all the code into the gitubuntu module, with thin wrappers in bin/, like git-ubuntu,
>
> I can do this, but I don't want these scripts in the snap. Every
> default change would imply a a new snap version, which forces everyone
> to download a new update. While the xdelta gunk from the store should
> make that minimal, in practice it does not.

Is this because edge is automatically generated? Could we start using
"beta" instead, and push to beta only when there's some other real
change? Or does this make other pain worse?

[Nish Aravamudan (nacc)]

On Tue, Nov 7, 2017 at 6:28 AM, Robie Basak <email address hidden> wrote:
> If we're going with pylint and treating these as new, then none of this is currently pylint clean. I'm not saying it necessarily should be. What's your opinion?

I'm fine with targetting clean runs, but we need a pylintrc, i think,
in order to have it agree to our formatting (4 space always indented).
And I think the 'too many arguments' and 'too many variables' rules
are dumb, but I'll read why the exist first. I pushed a few cleanups
that I had missed previously.

[Nish Aravamudan (nacc)]

On Tue, Nov 7, 2017 at 8:12 AM, Robie Basak <email address hidden> wrote:
> On Tue, Nov 07, 2017 at 04:04:53PM -0000, Nish Aravamudan wrote:
>> On Tue, Nov 7, 2017 at 6:24 AM, Robie Basak <email address hidden> wrote:
>> > Why not put all the code into the gitubuntu module, with thin wrappers in bin/, like git-ubuntu,
>>
>> I can do this, but I don't want these scripts in the snap. Every
>> default change would imply a a new snap version, which forces everyone
>> to download a new update. While the xdelta gunk from the store should
>> make that minimal, in practice it does not.
>
> Is this because edge is automatically generated? Could we start using
> "beta" instead, and push to beta only when there's some other real
> change? Or does this make other pain worse?

Well, and also, these are not really part of the application, they are
part of the wrapper of the application.

Adding another branch for the beta channel would be fine, but then we
have even more management to deal with (merge from edge to beta, merge
from beta to stable).

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:5b794876d4e94f6fbcbf44de018535c94f692e9b
https://jenkins.ubuntu.com/server/job/git-ubuntu-ci/198/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Style Check
    SUCCESS: Unit Tests
    SUCCESS: Integration Tests
    IN_PROGRESS: Declarative: Post Actions

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/git-ubuntu-ci/198/rebuild

review: Approve (continuous-integration)

[Robie Basak (racb)]

Looks good in general! I like the multiprocess.Pool.map use as it is now.

See my review branch for review comments for which it's easier just to describe in code. Individual explanations are in individual commit messages: https://code.launchpad.net/~racb/usd-importer/+git/usd-importer/+ref/modernize-scripts-review

I've left a couple of inline comments.

Some other general comments:

I'm not keen on the return spec of import_srcpkg. It seems rather arbitrary and unnatural, and it's not even convenient as you still have to massage it in the caller. That massaging is also duplicated twice in the two callers. I suggest you change the return spec to (pkgname, success_bool) instead. That would save you from having to zip it back up in the caller.

If you still want a (success_list, fail_list) tuple to process further up the stack more easily, then that can't be done by changing the return type of import_srcpkg further. So you'd still end up having some duplication. To fix that (if you still want success_list, fail_list), I suggest wrapping import_srcpkg and putting the wrapper in scriptutils. The wrapper could to the multiprocess pool, call the real import_srcpkg, and generate the (success_list, fail_list) result.

An example of getting from a (pkgname, success_bool) list to a (success_list, fail_list):

return (
    [pkg for pkg, success in results if success],
    [pkg for pkg, succces in results if not success],
)

General comments on source-package-walker.py:

Parsing Sources.xz alarmed me; perhaps add a comment that it's a cjwatson suggestion because the API way will be very slow and there's no better API for it :)

Can we make the components a list, because we'll be adding more later? Otherwise there's code duplication already due to the hardcoding like this.

We also need to import from other pockets and other series (at least the supported ones). Eg. source packages that have been deleted before this cycle, and source packages that were added to older stable releases in the other pockets directly such as HWE stacks.

Please parse using debian.deb822 or similar:

10:42 <rbasak> While you're here (I'm review nacc's MP), any opinion on the actual parsing, as implemented in Python? nacc is doing some Python-based parsing of the Sources file. Which feels ugly, but shelling out to grep-dctrl would also be ugly.
10:42 <rbasak> What would you do?
10:42 <cjwatson> rbasak: I'd use python-debian
10:42 <cjwatson> The stuff in debian.deb822 is generally fine for this
10:43 <rbasak> Thanks!
10:43 <cjwatson> rbasak: (python-apt is also fine; python-debian sometimes makes use of that for speed. Use whichever interface is more comfortable.)
10:44 <cjwatson> rbasak: germinate uses python-apt, so possibly I preferred that at one point. I think when I wrote that python-debian was significantly less good.

review: Needs Fixing

[Robie Basak (racb)]

PS. all my suggestions are entirely untested :)

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:56ea91693c5d49d6f52ef05b41329b0fa55f4c03
https://jenkins.ubuntu.com/server/job/git-ubuntu-ci/204/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Style Check
    SUCCESS: Unit Tests
    SUCCESS: Integration Tests
    IN_PROGRESS: Declarative: Post Actions

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/git-ubuntu-ci/204/rebuild

review: Approve (continuous-integration)

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:325412302b10decc4dd3ec714d814092d2e2669c
https://jenkins.ubuntu.com/server/job/git-ubuntu-ci/208/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Style Check
    SUCCESS: Unit Tests
    SUCCESS: Integration Tests
    IN_PROGRESS: Declarative: Post Actions

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/git-ubuntu-ci/208/rebuild

review: Approve (continuous-integration)

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:9ed2e8a2ac18665cbc7d966a74b1cf0deee32680
https://jenkins.ubuntu.com/server/job/git-ubuntu-ci/210/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Style Check
    SUCCESS: Unit Tests
    SUCCESS: Integration Tests
    IN_PROGRESS: Declarative: Post Actions

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/git-ubuntu-ci/210/rebuild

review: Approve (continuous-integration)

[Robie Basak (racb)]

import_all_published_sources looks really neat now, thanks! Minor fixups inline, but apart from those, this looks good now.

review: Approve

[Nish Aravamudan (nacc)]

On Mon, Nov 13, 2017 at 11:36 PM, Robie Basak <email address hidden> wrote:
> Review: Approve
>
> import_all_published_sources looks really neat now, thanks! Minor fixups inline, but apart from those, this looks good now.

Responses inline.

>
> Diff comments:
>
>> diff --git a/scripts/scriptutils.py b/scripts/scriptutils.py
>> new file mode 100644
>> index 0000000..c89507e
>> --- /dev/null
>> +++ b/scripts/scriptutils.py
>> @@ -0,0 +1,191 @@
>> +from collections import namedtuple
>> +import functools
>> +import hashlib
>> +import multiprocessing
>> +import os
>> +import sys
>> +import subprocess
>> +import time
>> +
>> +import pkg_resources
>> +
>> +# We expect to be running from a git repository in master for this
>> +# script, because the snap's python code is not exposed except within
>> +# the snap
>> +try:
>> + REALPATH = os.readlink(__file__)
>> +except OSError:
>> + REALPATH = __file__
>> +sys.path.insert(
>> + 0,
>> + os.path.abspath(
>> + os.path.join(os.path.dirname(REALPATH), os.path.pardir)
>> + )
>> +)
>> +
>> +from gitubuntu.run import run
>> +
>> +Defaults = namedtuple(
>> + 'Defaults',
>> + [
>> + 'num_workers',
>> + 'whitelist',
>> + 'blacklist',
>> + 'phasing_universe',
>> + 'phasing_main',
>> + 'dry_run',
>> + 'use_whitelist',
>> + ],
>> +)
>> +
>> +DEFAULTS = Defaults(
>> + num_workers=10,
>> + whitelist=pkg_resources.resource_filename(
>> + 'gitubuntu',
>> + 'source-package-whitelist.txt',
>> + ),
>> + blacklist=pkg_resources.resource_filename(
>> + 'gitubuntu',
>> + 'source-package-blacklist.txt',
>> + ),
>> + phasing_universe=0,
>> + phasing_main=1,
>> + dry_run=False,
>> + use_whitelist=True,
>> +)
>> +
>> +
>> +def should_import_srcpkg(
>> + pkgname,
>> + component,
>> + whitelist,
>> + blacklist,
>> + phasing_main,
>> + phasing_universe,
>> +):
>> + """should_import_srcpkg - indicate if a given source package should be imported
>> +
>> + The phasing is implemented similarly to update-manager. If the
>> + md5sum of the source package name is less than the (appropriate
>> + percentage * 2^128) (the maximum representable md5sum), the source
>> + package name is in the appropriate phasing set.
>> +
>> + Arguments:
>> + pkgname - string name of a source package
>> + component - string archive component of @pgkname
>> + whitelist - a list of of packages to always import
>> + blacklist - a list of packages to never import
>> + phasing_main - a integer percentage of all packages in main to import
>> + phasing_universe - a integer percentage of all packages in universe to import
>> +
>> + Returns:
>> + True if @pkgname should be imported, False if not.
>> + """
>> + if pkgname in blacklist:
>> + return False
>> + if pkgname in whitelist:
>> + return True
>> + md5sum = int(
>> + hashlib.md5(
>> + pkgname.encode('utf-8')
>> + ).hexdigest(),
>> + 16,
>> + )
>> + if component == 'main':
>> + if md5sum <= (phasing_main / 100) * (2**128):
>> + return Tru...

[Robie Basak (racb)]

On Tue, Nov 14, 2017 at 03:30:56PM -0000, Nish Aravamudan wrote:
> On Mon, Nov 13, 2017 at 11:36 PM, Robie Basak <email address hidden> wrote:
> Heh, and I apparently have an aversion to the below method. In C, the
> following can result in less efficient code (iirc), on some
> architectures.
>
> I'm happy to make your change, though.

IMHO, extra state is more error-prone, and so are bindings that get
redefined (ie. "variables" that change value). OTOH, Python is already
pretty non-performant, and we don't care much for that, in favour of
readability and less-error-prone-ness. Unless we actually hit a
performance issue, or if there are two equivalent ways of doing things
and one is more performant without harming anything else.

> > Style: no trailing comma
>
> I find this a bit confusing, if I had done
>
> pocket_suffixes = [
> '-proposed',
> '-updates',
> '-security',
> ''
> ]
>
> I think our style would have indicated it was wrong because it was missing
> a trailing comma. Why does it matter if it's inline versus multiline for the
> style rule to be applied? Can this be clarified in the style document (it also
> means moving from inline to multi-line is more prone to missing this, IMO).

Python convention is to use [x, y, z] (with no other whitespace or
commas) AIUI. I'm not sure about multi-line, but Python allows the
trailing comma specifically (AIUI) to allow the style of not having to
edit the previous line to add a new one. So in my head, the trailing
comma only applies to the multi-line case.

I'm interested to know what others on the team think. But yeah, I'll
happily edit the style document if you agree.

diff --git a/bin/import-cron b/bin/import-cron
deleted file mode 100755
index 0044b74..0000000
--- a/bin/import-cron
+++ /dev/null
@@ -1,199 +0,0 @@
-#!/usr/bin/env python3
-
-from copy import copy
-import os
-import subprocess
-import sys
-import tempfile
-import time
-# we know that our relative path doesn't include the module
-try:
-    realpath = os.readlink(__file__)
-except OSError:
-    realpath = __file__
-sys.path.insert(
-    0,
-    os.path.abspath(
-        os.path.join(os.path.dirname(realpath), os.path.pardir)
-    )
-)
-from gitubuntu.source_information import launchpad_login
-from gitubuntu.run import run
-try:
-    pkg = 'python3-pkg-resources'
-    import pkg_resources
-except ImportError:
-    logging.error('Is %s installed?', pkg)
-    sys.exit(1)
-
-# we might want this to be contentfully stored in the importer's git
-# repository?
-import_scan_timestamps = dict()
-import_scan_timestamps['debian'] = time.time() - (24 * 60 * 60)
-import_scan_timestamps['ubuntu'] = time.time() - (24 * 60 * 60)
-import_scan_links = dict()
-import_scan_links['debian'] = None
-import_scan_links['ubuntu'] = None
-
-import_cron_log = os.path.join(tempfile.gettempdir(), 'import-cron-log')
-import_cron_packages = pkg_resources.resource_filename('gitubuntu',
-                                                       'import-cron-packages.txt')
-
-def write_timestamps():
-    global import_scan_timestamps
-    global import_scan_links
-
-    with open(import_cron_log, 'w+') as f:
-        for dist_name in ['debian', 'ubuntu']:
-            f.write('%s timestamp: %f\n' % (dist_name, import_scan_timestamps[dist_name]))
-            f.write('%s link: %s\n' % (dist_name, import_scan_links[dist_name]))
-
-def read_timestamps():
-    global import_scan_timestamps
-    global import_scan_links
-
-    # This could be more defensive
-    try:
-        with open(import_cron_log, 'r') as f:
-            for line in f:
-                dist_name, log_type, value = line.split()
-                if 'timestamp' in log_type:
-                    import_scan_timestamps[dist_name] = float(value)
-                elif 'link' in log_type:
-                    import_scan_links[dist_name] = value
-    except IOError:
-        pass
-
-def update_timestamp(dist_name, spphr):
-    global import_scan_timestamps
-    global import_scan_links
-
-    if spphr.date_published:
-        timestamp = spphr.date_published.timestamp()
-        link = spphr.self_link
-        import_scan_timestamps[dist_name] = timestamp
-        import_scan_links[dist_name] = str(link)
-        return True
-    return False
-
-def import_new_published_sources(launchpad, packages, imported_srcpkgs, failed_srcpkgs):
-    global import_scan_timestamps
-    global import_scan_links
-    args = {'order_by_date':True}
-
-    for dist_name in ['debian', 'ubuntu']:
-        print('Examining publishes in %s since %f' % (dist_name, import_scan_timestamps[dist_name]))
-        dist = launchpad.distributions[dist_name]
-        spph = dist.main_archive.getPublishedSources(**args)
-        if len(spph) == 0:
-            print('No publishing data found in %s' % dist_name)
-            continue
-        if import_scan_timestamps[dist_name]:
-            _spph = list()
-            for spphr in spph:
-                # only check if we should stop iterating if there is a
-                # timestamp to compare to
-                if spphr.date_published:
-                    # stop iterating (backwards chronologically) when we see
-                    # a publish timestamp before the last cron run
-                    if spphr.date_published.timestamp() < import_scan_timestamps[dist_name]:
-                        break
-                if spphr.source_package_name not in packages:
-                    continue
-                _spph.append(spphr)
-            # if no packages in our whitelist have been updated in the
-            # scan window, then bump the last scan date manually to the
-            # last valid timestamp in the window
-            if len(_spph) == 0:
-                print('No new relevant publishes found in %s relative to %f' % (dist_name, import_scan_timestamps[dist_name]))
-                for spphr in spph:
-                    if update_timestamp(dist_name, spphr):
-                        break
-            spph = _spph
-        spph_iter = reversed(spph)
-        caught_up = False
-        for spphr in spph_iter:
-            # find the matching upload to the last publish seen
-            if not caught_up:
-                if not import_scan_links[dist_name] or str(spphr.self_link) == import_scan_links[dist_name]:
-                    caught_up = True
-                continue
-
-            if spphr.source_package_name in failed_srcpkgs:
-                update_timestamp(dist_name, spphr)
-                continue
-
-            # try up to 3 times before declaring failure, in case of
-            # racing with the publisher/transient download failure
-            success = False
-            for i in range(3):
-                if spphr.source_package_name in imported_srcpkgs:
-                    success = True
-                    break
-                print('git ubuntu import -l usd-importer-bot %s' % spphr.source_package_name)
-                try:
-                    run(['git', 'ubuntu', 'import', '-l', 'usd-importer-bot', spphr.source_package_name], check=True)
-                    imported_srcpkgs.append(spphr.source_package_name)
-                except subprocess.CalledProcessError:
-                    print('failed to import %s (attempt %d/3)' % (spphr.source_package_name, i+1))
-                    time.sleep(10)
-                    continue
-
-            if not success:
-                print('failed to import %s' % spphr.source_package_name)
-                failed_srcpkgs.append(spphr.source_package_name)
-            # we want to bump the timestamp for every source package
-            # published, regardless of whether we have already run the
-            # importer for it in this cron run, since it might be published
-            # multiple times (e.g., different series for an SRU).
-            update_timestamp(dist_name, spphr)
-    return (imported_srcpkgs, failed_srcpkgs)
-
-def main():
-    try:
-        run(['git', 'config', '--global', 'user.name'], check=True)
-    except subprocess.CalledProcessError:
-        run(['git', 'config', '--global', 'user.name', 'Ubuntu Git Importer'])
-    try:
-        run(['git', 'config', '--global', 'user.email'], check=True)
-    except subprocess.CalledProcessError:
-        run(['git', 'config', '--global', 'user.email', '[e-mail]'])
-    try:
-        with open(import_cron_packages, 'r') as f:
-            packages = [line.strip() for line in f if not line.startswith('#')]
-    except:
-        packages = list()
-    read_timestamps()
-    # minutes
-    sleep_interval = 20
-    # hours
-    mail_interval = 1
-    launchpad = launchpad_login()
-    imported_srcpkgs = list()
-    failed_srcpkgs = list()
-    last_mail_timestamp = time.time()
-    while True:
-        if (time.time() - last_mail_timestamp >= (mail_interval * 60 * 60) and
-            (len(imported_srcpkgs) != 0 or len(failed_srcpkgs) != 0)
-           ):
-            i = b'Subject: Importer report\n'
-            if len(imported_srcpkgs) != 0:
-                i += b'Successfully imported the following source packages\n' + b'\n'.join(map(lambda x : x.encode('utf-8'), imported_srcpkgs))
-            if len(failed_srcpkgs) != 0:
-                i += b'\nFailed to import the following source packages\n' + b'\n'.join(map(lambda x : x.encode('utf-8'), failed_srcpkgs))
-            run(['sendmail', '-F', 'Ubuntu Git Importer', '-f', '[e-mail]', '[e-mail]'], input=i)
-            last_mail_timestamp = time.time()
-        orig_imported_srcpkgs = copy(imported_srcpkgs)
-        imported_srcpkgs, failed_srcpkgs = import_new_published_sources(launchpad, packages, imported_srcpkgs, failed_srcpkgs)
-        print('Imported %d source packages' % (len(imported_srcpkgs) - len(orig_imported_srcpkgs)))
-        if len(imported_srcpkgs) == len(orig_imported_srcpkgs):
-            write_timestamps()
-            time.sleep(sleep_interval * 60)
-            imported_srcpkgs = list()
-            failed_srcpkgs = list()
-            # I have seen transient network issues lead to timeouts and stalls
-            # so reset the connection on each iteration
-            launchpad = launchpad_login()
-
-if __name__ == '__main__':
-    main()
diff --git a/gitubuntu/importer.py b/gitubuntu/importer.py
index fb5cd77..f7dfe92 100644
--- a/gitubuntu/importer.py
+++ b/gitubuntu/importer.py
@@ -26,7 +26,6 @@
 
 import argparse
 import atexit
-import dbm
 import functools
 import getpass
 import logging
@@ -151,7 +150,6 @@ def main(
     parentfile=top_level_defaults.parentfile,
     retries=top_level_defaults.retries,
     retry_backoffs=top_level_defaults.retry_backoffs,
-    db_cache_dir=None,
 ):
     """Main entry point to the importer
 
@@ -180,9 +178,6 @@ def main(
     @parentfile: string path to file specifying parent overrides
     @retries: integer number of download retries to attempt
     @retry_backoffs: list of backoff durations to use between retries
-    @db_cache_dir: string fileystem directory containing 'ubuntu' and
-    'debian' dbm database files, which store the progress of prior
-    importer runs
 
     If directory is None, a temporary directory is created and used.
 
@@ -192,11 +187,6 @@ def main(
     If dl_cache is None, CACHE_PATH in the local repository will be
     used.
 
-    If db_cache_dir is None, no database lookups are performed which makes
-    it possible that the import will attempt to re-import already
-    imported publishes. This should be fine, although less efficient
-    than possible.
-
     Returns 0 on successful import (which includes non-fatal failures);
     1 otherwise.
     """
@@ -324,9 +314,6 @@ def main(
     else:
         workdir = dl_cache
 
-    if db_cache_dir:
-        os.makedirs(db_cache_dir, exist_ok=True)
-
     os.makedirs(workdir, exist_ok=True)
 
     # now sets a global _PARENT_OVERRIDES
@@ -339,7 +326,6 @@ def main(
         patches_applied=False,
         debian_head_versions=debian_head_versions,
         ubuntu_head_versions=ubuntu_head_versions,
-        db_cache_dir=db_cache_dir,
         debian_sinfo=debian_sinfo,
         ubuntu_sinfo=ubuntu_sinfo,
         active_series_only=active_series_only,
@@ -361,7 +347,6 @@ def main(
             patches_applied=True,
             debian_head_versions=applied_debian_head_versions,
             ubuntu_head_versions=applied_ubuntu_head_versions,
-            db_cache_dir=db_cache_dir,
             debian_sinfo=debian_sinfo,
             ubuntu_sinfo=ubuntu_sinfo,
             active_series_only=active_series_only,
@@ -1367,7 +1352,6 @@ def import_publishes(
     patches_applied,
     debian_head_versions,
     ubuntu_head_versions,
-    db_cache_dir,
     debian_sinfo,
     ubuntu_sinfo,
     active_series_only,
@@ -1378,8 +1362,6 @@ def import_publishes(
     history_found = False
     only_debian = False
     srcpkg_information = None
-    last_debian_spphr = None
-    last_ubuntu_spphr = None
     if patches_applied:
         _namespace = namespace
         namespace = '%s/applied' % namespace
@@ -1395,28 +1377,15 @@ def import_publishes(
             import_unapplied_spi,
             skip_orig=skip_orig,
         )
-
-    for distname, versions, dist_sinfo, last_spphr in (
-        ("debian", debian_head_versions, debian_sinfo, last_debian_spphr),
-        ("ubuntu", ubuntu_head_versions, ubuntu_sinfo, last_ubuntu_spphr),
-    ):
+    for distname, versions, dist_sinfo in (
+            ("debian", debian_head_versions, debian_sinfo),
+            ("ubuntu", ubuntu_head_versions, ubuntu_sinfo)):
         if active_series_only and distname == "debian":
             continue
-
-        last_spphr = None
-        if db_cache_dir:
-             with dbm.open(os.path.join(db_cache_dir, distname), 'c') as cache:
-                try:
-                    last_spphr = decode_binary(cache[pkgname])
-                except KeyError:
-                    pass
-
         try:
-            last_spi = None
             for srcpkg_information in dist_sinfo.launchpad_versions_published_after(
                 versions,
                 namespace,
-                last_spphr,
                 workdir=workdir,
                 active_series_only=active_series_only
             ):
@@ -1427,11 +1396,6 @@ def import_publishes(
                     namespace=_namespace,
                     ubuntu_sinfo=ubuntu_sinfo,
                 )
-                last_spi = srcpkg_information
-            if last_spi:
-                if db_cache_dir:
-                    with dbm.open(os.path.join(db_cache_dir, distname), 'w') as db_cache:
-                        db_cache[pkgname] = str(last_spi.spphr)
         except NoPublicationHistoryException:
             logging.warning("No publication history found for %s in %s.",
                 pkgname, distname
@@ -1507,11 +1471,6 @@ def parse_args(subparsers=None, base_subparsers=None):
         action='store_true',
         help=argparse.SUPPRESS,
     )
-    parser.add_argument(
-        '--db-cache',
-        type=str,
-        help=argparse.SUPPRESS,
-    )
     if not subparsers:
         return parser.parse_args()
     return 'import - %s' % kwargs['description']
@@ -1540,11 +1499,6 @@ def cli_main(args):
     except AttributeError:
         dl_cache = None
 
-    try:
-        db_cache = args.db_cache
-    except AttributeError:
-        db_cache = None
-
     return main(
         pkgname=args.package,
         owner=args.lp_owner,
@@ -1564,5 +1518,4 @@ def cli_main(args):
         parentfile=args.parentfile,
         retries=args.retries,
         retry_backoffs=args.retry_backoffs,
-        db_cache_dir=args.db_cache,
     )
diff --git a/gitubuntu/source-package-blacklist.txt b/gitubuntu/source-package-blacklist.txt
new file mode 100644
index 0000000..ac8aef9
--- /dev/null
+++ b/gitubuntu/source-package-blacklist.txt
@@ -0,0 +1,7 @@
+linux
+linux-base
+linux-firmware
+linux-meta
+lxc
+lxcfs
+lxd
diff --git a/gitubuntu/import-cron-packages.txt b/gitubuntu/source-package-whitelist.txt
index 74df660..61b6f35 100644
--- a/gitubuntu/import-cron-packages.txt
+++ b/gitubuntu/source-package-whitelist.txt
@@ -435,11 +435,7 @@ libxml-security-java
 libxml-xpath-perl
 libxmu
 libyaml
-#linux
 linux-atm
-#linux-base
-#linux-firmware
-#linux-meta
 lm-sensors
 lockfile-progs
 logcheck
@@ -454,9 +450,6 @@ ltrace
 lua5.2
 lua-lpeg
 lvm2
-#lxc
-#lxcfs
-#lxd
 lz4
 lzo2
 m2300w
diff --git a/gitubuntu/source_information.py b/gitubuntu/source_information.py
index 6399460..3fef4ec 100644
--- a/gitubuntu/source_information.py
+++ b/gitubuntu/source_information.py
@@ -421,14 +421,7 @@ class GitUbuntuSourceInformation(object):
         for srcpkg in spph:
             yield self.get_corrected_spi(srcpkg, workdir)
 
-    def launchpad_versions_published_after(
-       self,
-       head_versions,
-       namespace,
-       last_spphr=None,
-       workdir=None,
-       active_series_only=False,
-    ):
+    def launchpad_versions_published_after(self, head_versions, namespace, workdir=None, active_series_only=False):
         args = {
                 'exact_match':True,
                 'source_name':self.pkgname,
@@ -456,14 +449,7 @@ class GitUbuntuSourceInformation(object):
         if len(spph) == 0:
             raise NoPublicationHistoryException("Is %s published in %s?" %
                                                 (self.pkgname, self.dist_name))
-        if last_spphr:
-            _spph = list()
-            for spphr in spph:
-                if str(spphr) == last_spphr:
-                    break
-                _spph.append(spphr)
-            spph = _spph
-        elif head_versions:
+        if len(head_versions) > 0:
             _spph = list()
             for spphr in spph:
                 spi = GitUbuntuSourcePackageInformation(spphr, self.dist_name,
diff --git a/man/man1/git-ubuntu-import.1 b/man/man1/git-ubuntu-import.1
index 74bfe83..dd7fd9e 100644
--- a/man/man1/git-ubuntu-import.1
+++ b/man/man1/git-ubuntu-import.1
@@ -1,4 +1,4 @@
-.TH "GIT-UBUNTU-IMPORT" "1" "2017-11-08" "Git-Ubuntu 0.6.2" "Git-Ubuntu Manual"
+.TH "GIT-UBUNTU-IMPORT" "1" "2017-07-19" "Git-Ubuntu 0.2" "Git-Ubuntu Manual"
 
 .SH "NAME"
 git-ubuntu import \- Import Launchpad publishing history to Git
@@ -9,8 +9,7 @@ git-ubuntu import \- Import Launchpad publishing history to Git
 <user>] [\-\-dl-cache <dl_cache>] [\-\-no-fetch] [\-\-no-push]
 [\-\-no-clean] [\-d | \-\-directory <directory>]
 [\-\-active-series-only] [\-\-skip-applied] [\-\-skip-orig]
-[\-\-reimport] [\-\-allow-applied-failures] [\-\-db-cache <db_cache>]
-<package>
+[\-\-reimport] [\-\-allow-applied-failures] <package>
 .FI
 .SP
 .SH "DESCRIPTION"
@@ -198,21 +197,6 @@ After investigation, this flag can be used to indicate the importer is
 allowed to ignore such a failure\&.
 .RE
 .PP
-\-\-db-cache <db_cache>
-.RS 4
-The path to a directory containing Python dbm database disk files for
-importer metadata\&.
-If \fB<db_cache>\fR does not exist, it will be created\&.
-Two files in \fB<db_cache>\fR are used, "ubuntu" and "debian', which are
-created if not already present\&.
-The cache files provide information to the importer about prior imports
-of \fB<package>\fR and which Launchpad publishing record was last
-imported\&.
-This is necessary because the imported Git repository does not
-necessarily maintain any metadata about Launchpad publishing
-information\&.
-.RE
-.PP
 <package>
 .RS 4
 The name of the source package to import\&.
diff --git a/scripts/import-source-packages.py b/scripts/import-source-packages.py
new file mode 100755
index 0000000..698ae35
--- /dev/null
+++ b/scripts/import-source-packages.py
@@ -0,0 +1,377 @@
+#!/usr/bin/env python3
+
+# General design:
+# Infinite loop:
+#   now = time()
+#   let new_publishes be the set of unique srcpkg names in publishes between last run and now
+#   for srcpkg in new_publishes:
+#       If srcpkg in blacklist: skip
+#       If srcpkg not in PHASING_{component}: skip
+#       try to import srcpkg
+#   Report on successful and failed imports
+#   If no srcpkgs: sleep for some time to let publisher run
+
+import argparse
+import collections
+import datetime
+import os
+import sys
+import tempfile
+import time
+
+# We expect to be running from a git repository in master for this
+# script, because the snap's python code is not exposed except within
+# the snap
+try:
+    REALPATH = os.readlink(__file__)
+except OSError:
+    REALPATH = __file__
+sys.path.insert(
+    0,
+    os.path.abspath(
+        os.path.join(os.path.dirname(REALPATH), os.path.pardir)
+    )
+)
+
+from gitubuntu.source_information import launchpad_login
+from gitubuntu.run import run
+import scriptutils
+
+# The 'time' attribute is the publication date, as a timestamp, of the SPPHR
+# corresponding to the URL stored in the 'link' attribute.
+Timestamp = collections.namedtuple('Timestamp', ['time', 'link'])
+
+LOG_PATH = os.path.join(tempfile.gettempdir(), 'import-source-packages-log')
+
+def import_new_published_sources(
+    num_workers,
+    whitelist,
+    blacklist,
+    phasing_main,
+    phasing_universe,
+    dry_run,
+):
+    """import_new_published_source - Import all new publishes since a prior execution
+
+    Arguments:
+    num_workers - integer number of worker processes to use
+    whitelist - a list of of packages to always import
+    blacklist - a list of packages to never import
+    phasing_main - a integer percentage of all packages in main to import
+    phasing_universe - a integer percentage of all packages in universe to import
+    dry_run - a boolean to indicate a dry-run operation
+
+    Returns:
+    A tuple of two lists, the first containing the names of all
+    successfully imported source packages, the second containing the
+    names of all source packages that failed to import.
+    """
+    timestamps = read_timestamps()
+    launchpad = launchpad_login()
+
+    # filtered_pkgnames is the list of source package names across all
+    # distributions we will want to process, based upon
+    # scriptutils.should_import_srcpkg
+    filtered_pkgnames = set()
+
+    for dist_name in ['debian', 'ubuntu']:
+        timestamp = timestamps[dist_name]
+
+        # dist_newest_spphr is the most recent publication record with a
+        # valid published date in the distribution
+        dist_newest_spphr = None
+
+        # dist_filtered_pkgnames is the set of source package names in
+        # the dist_name distribution that qualify for import according to our
+        # whitelists, blacklists and phasing requirements
+        dist_filtered_pkgnames = set()
+
+        print(
+            "Examining publishes in %s since %s" % (
+                dist_name,
+                datetime.datetime.fromtimestamp(
+                    timestamp.time,
+                ).strftime("%Y-%m-%d %H:%M:%S"),
+            )
+        )
+
+        # spph is the raw publication history for a distribution from
+        # Launchpad, sorted by publication date in reverse chronological order
+        dist = launchpad.distributions[dist_name]
+        spph = dist.main_archive.getPublishedSources(order_by_date=True)
+        if not spph:
+            print("No publishing data found in %s" % dist_name)
+            continue
+
+        for spphr in spph:
+            # this is the matching upload to the last publish seen
+            if str(spphr) == timestamp.link:
+                break
+
+            # only check if we should stop iterating due to the
+            # timestamps if there is a timestamp to compare to, which
+            # means the source package is actually published.
+            if spphr.date_published:
+                if not dist_newest_spphr:
+                    dist_newest_spphr = spphr
+                # stop iterating (backwards chronologically) when we see
+                # a publish timestamp before the last run
+                if spphr.date_published.timestamp() < timestamp.time:
+                    break
+                if scriptutils.should_import_srcpkg(
+                    spphr.source_package_name,
+                    spphr.component_name,
+                    whitelist,
+                    blacklist,
+                    phasing_main,
+                    phasing_universe,
+                ):
+                    dist_filtered_pkgnames.add(spphr.source_package_name)
+
+        if not dist_filtered_pkgnames:
+            print(
+                "No new relevant publishes found in %s relative to %s" % (
+                    dist_name,
+                    datetime.datetime.fromtimestamp(
+                        timestamp.time
+                    ).strftime("%Y-%m-%d %H:%M:%S"),
+                )
+            )
+
+        filtered_pkgnames = filtered_pkgnames | dist_filtered_pkgnames
+        if dist_newest_spphr:
+            # Update timestamp
+            timestamps[dist_name] = timestamp = Timestamp(
+                time=dist_newest_spphr.date_published.timestamp(),
+                link=str(dist_newest_spphr),
+            )
+
+
+    ret = scriptutils.pool_map_import_srcpkg(
+        num_workers=num_workers,
+        dry_run=dry_run,
+        pkgnames=filtered_pkgnames,
+    )
+
+    write_timestamps(timestamps)
+
+    return ret
+
+def read_timestamps():
+    """read_timestamps - Read saved timestamp values from LOG_PATH
+
+    If the log file is not readable (e.g., does not exist), the
+    timestamps will be set to 24 hours before now.
+
+    This method is symmetrical to write_timestamps.
+
+    Returns:
+    A dictionary with 'debian' and 'ubuntu' keys, each of which is a
+    Timestamp namedtuple.
+    """
+    try:
+        timestamps = dict()
+        with open(LOG_PATH, 'r') as log_file:
+            for line in log_file:
+                dist_name, log_type, value = line.split()
+                if dist_name not in timestamps:
+                    timestamps[dist_name] = dict()
+                assert log_type in ['time', 'link']
+                assert log_type not in timestamps[dist_name]
+                if log_type == 'link':
+                    timestamps[dist_name][log_type] = value
+                else:
+                    timestamps[dist_name][log_type] = float(value)
+        return {
+            dist: Timestamp(**dict_form)
+            for dist, dict_form in timestamps
+        }
+    except IOError:
+        _start = time.time() - (24 * 60 * 60)
+        return dict(
+            ubuntu=Timestamp(time=_start, link=None),
+            debian=Timestamp(time=_start, link=None),
+        )
+
+def write_timestamps(timestamps):
+    """write_timestamps - Write timestamp values to LOG_PATH
+
+    Arguments:
+    timestamps - a dictionary with 'debian' and 'ubuntu' keys, each of which is
+    a Timestamp namedtuple.
+
+    This method is symmetrical to read_timestamps.
+    """
+    new_log = os.path.join(LOG_PATH, '.new')
+    with open(new_log, 'w+') as log_file:
+        log_file.write(
+            'debian time %f\n' %
+            timestamps['debian'].time
+        )
+        log_file.write(
+            'debian link %s\n' %
+            timestamps['debian'].link
+        )
+        log_file.write(
+            'ubuntu time %f\n' %
+            timestamps['ubuntu'].time
+        )
+        log_file.write(
+            'ubuntu link %s\n' %
+            timestamps['ubuntu'].link
+        )
+    os.replace(new_log, LOG_PATH)
+
+def main(
+    num_workers=scriptutils.DEFAULTS.num_workers,
+    whitelist_path=scriptutils.DEFAULTS.whitelist,
+    blacklist_path=scriptutils.DEFAULTS.blacklist,
+    phasing_main=scriptutils.DEFAULTS.phasing_main,
+    phasing_universe=scriptutils.DEFAULTS.phasing_universe,
+    dry_run=scriptutils.DEFAULTS.dry_run,
+):
+    """main - Main entry point to the script
+
+    Arguments:
+    num_workers - integer number of worker threads to use
+    whitelist_path - string filesystem path to a text file of packages
+    to always import
+    blacklist_path - string filesystem path to a text file of packages
+    to never import
+    phasing_main - a integer percentage of all packages in main to
+    import
+    phasing_universe - a integer percentage of all packages in universe
+    to import
+    dry_run - a boolean to indicate a dry-run operation
+    """
+    scriptutils.setup_git_config()
+
+    try:
+        with open(whitelist_path, 'r') as whitelist_file:
+            whitelist = [
+                line.strip() for line in whitelist_file
+                if not line.startswith('#')
+            ]
+    except (FileNotFoundError, IOError):
+        whitelist = list()
+
+    try:
+        with open(blacklist_path, 'r') as blacklist_file:
+            blacklist = [
+                line.strip() for line in blacklist_file
+                if not line.startswith('#')
+            ]
+    except (FileNotFoundError, IOError):
+        blacklist = list()
+
+    sleep_interval_minutes = 20
+    mail_interval_hours = 1
+
+    # pretend we sent an e-mail recently
+    last_mail_timestamp = time.time()
+    mail_imported_srcpkgs = set()
+    mail_failed_srcpkgs = set()
+
+    while True:
+        imported_srcpkgs, failed_srcpkgs = import_new_published_sources(
+            num_workers,
+            whitelist,
+            blacklist,
+            phasing_main,
+            phasing_universe,
+            dry_run,
+        )
+        print("Imported %d source packages" % len(imported_srcpkgs))
+        mail_imported_srcpkgs |= set(imported_srcpkgs)
+        mail_failed_srcpkgs |= set(failed_srcpkgs)
+        secs_since_last_mail = time.time() - last_mail_timestamp
+        if (
+            secs_since_last_mail >= (mail_interval_hours * 60 * 60) and
+            (mail_imported_srcpkgs or mail_failed_srcpkgs)
+        ):
+            msg = b"Subject: Importer report\n"
+            if mail_imported_srcpkgs:
+                msg += b"Successfully imported the following source packages:\n"
+                msg += b"\n".join(
+                    map(lambda x: x.encode('utf-8'), mail_imported_srcpkgs)
+                )
+            if mail_failed_srcpkgs:
+                msg += b"\nFailed to import the following source packages:\n"
+                msg += b"\n".join(
+                    map(lambda x: x.encode('utf-8'), mail_failed_srcpkgs)
+                )
+            if dry_run:
+                print("Would send email with contents:\n%s" % msg.decode())
+            else:
+                run(
+                    [
+                        'sendmail',
+                        '-F', 'Ubuntu Git Importer',
+                        '-f', '[e-mail]',
+                        '[e-mail]',
+                    ],
+                    input=msg,
+                )
+            last_mail_timestamp = time.time()
+            mail_imported_srcpkgs = set()
+            mail_failed_srcpkgs = set()
+        # if we have caught up to the publisher, go to sleep
+        if not imported_srcpkgs:
+            time.sleep(sleep_interval_minutes * 60)
+
+def cli_main():
+    """cli_main - CLI entry point to script
+    """
+    parser = argparse.ArgumentParser(
+        description='Script to import all source packages with phasing',
+    )
+    parser.add_argument(
+        '--num-workers',
+        type=int,
+        help="Number of worker threads to use",
+        default=scriptutils.DEFAULTS.num_workers,
+    )
+    parser.add_argument(
+        '--whitelist',
+        type=str,
+        help="Path to whitelist file",
+        default=scriptutils.DEFAULTS.whitelist,
+    )
+    parser.add_argument(
+        '--blacklist',
+        type=str,
+        help="Path to blacklist file",
+        default=scriptutils.DEFAULTS.blacklist,
+    )
+    parser.add_argument(
+        '--phasing-universe',
+        type=int,
+        help="Percentage of universe packages to phase",
+        default=scriptutils.DEFAULTS.phasing_universe,
+    )
+    parser.add_argument(
+        '--phasing-main',
+        type=int,
+        help="Percentage of main packages to phase",
+        default=scriptutils.DEFAULTS.phasing_main,
+    )
+    parser.add_argument(
+        '--dry-run',
+        action='store_true',
+        help="Simulate operation but do not actually do anything",
+        default=scriptutils.DEFAULTS.dry_run,
+    )
+
+    args = parser.parse_args()
+
+    main(
+        num_workers=args.num_workers,
+        whitelist_path=args.whitelist,
+        blacklist_path=args.blacklist,
+        phasing_main=args.phasing_main,
+        phasing_universe=args.phasing_universe,
+        dry_run=args.dry_run,
+    )
+
+if __name__ == '__main__':
+    cli_main()
diff --git a/scripts/scriptutils.py b/scripts/scriptutils.py
new file mode 100644
index 0000000..c89507e
--- /dev/null
+++ b/scripts/scriptutils.py
@@ -0,0 +1,191 @@
+from collections import namedtuple
+import functools
+import hashlib
+import multiprocessing
+import os
+import sys
+import subprocess
+import time
+
+import pkg_resources
+
+# We expect to be running from a git repository in master for this
+# script, because the snap's python code is not exposed except within
+# the snap
+try:
+    REALPATH = os.readlink(__file__)
+except OSError:
+    REALPATH = __file__
+sys.path.insert(
+    0,
+    os.path.abspath(
+        os.path.join(os.path.dirname(REALPATH), os.path.pardir)
+    )
+)
+
+from gitubuntu.run import run
+
+Defaults = namedtuple(
+    'Defaults',
+    [
+        'num_workers',
+        'whitelist',
+        'blacklist',
+        'phasing_universe',
+        'phasing_main',
+        'dry_run',
+        'use_whitelist',
+    ],
+)
+
+DEFAULTS = Defaults(
+    num_workers=10,
+    whitelist=pkg_resources.resource_filename(
+        'gitubuntu',
+        'source-package-whitelist.txt',
+    ),
+    blacklist=pkg_resources.resource_filename(
+        'gitubuntu',
+        'source-package-blacklist.txt',
+    ),
+    phasing_universe=0,
+    phasing_main=1,
+    dry_run=False,
+    use_whitelist=True,
+)
+
+
+def should_import_srcpkg(
+    pkgname,
+    component,
+    whitelist,
+    blacklist,
+    phasing_main,
+    phasing_universe,
+):
+    """should_import_srcpkg - indicate if a given source package should be imported
+
+    The phasing is implemented similarly to update-manager. If the
+    md5sum of the source package name is less than the (appropriate
+    percentage * 2^128) (the maximum representable md5sum), the source
+    package name is in the appropriate phasing set.
+
+    Arguments:
+    pkgname - string name of a source package
+    component - string archive component of @pgkname
+    whitelist - a list of of packages to always import
+    blacklist - a list of packages to never import
+    phasing_main - a integer percentage of all packages in main to import
+    phasing_universe - a integer percentage of all packages in universe to import
+
+    Returns:
+    True if @pkgname should be imported, False if not.
+    """
+    if pkgname in blacklist:
+        return False
+    if pkgname in whitelist:
+        return True
+    md5sum = int(
+        hashlib.md5(
+            pkgname.encode('utf-8')
+        ).hexdigest(),
+        16,
+    )
+    if component == 'main':
+        if md5sum <= (phasing_main / 100) * (2**128):
+            return True
+    elif component == 'universe':
+        if md5sum <= (phasing_universe / 100) * (2**128):
+            return True
+    # skip partner and multiverse for now
+    return False
+
+
+def import_srcpkg(pkgname, dry_run):
+    """import_srcpkg - Invoke git ubuntu import on @pkgname
+
+    Arguments:
+    pkgname - string name of a source package
+    dry_run - a boolean to indicate a dry-run operation
+
+    Returns:
+    A tuple of boolean and a string, where the boolean is the success or
+    failure of the import and the string is the package name.
+    """
+    ret = False
+
+    # try up to 3 times before declaring failure, in case of
+    # racing with the publisher finalizing files and/or
+    # transient download failure
+
+    for attempt in range(3):
+        cmd = [
+            'git',
+            'ubuntu',
+            'import',
+            '-l',
+            'usd-importer-bot',
+            pkgname,
+        ]
+        try:
+            print(' '.join(cmd))
+            if not dry_run:
+                run(cmd, check=True)
+            ret = True
+            break
+        except subprocess.CalledProcessError:
+            print(
+                "Failed to import %s (attempt %d/3)" % (
+                    pkgname,
+                    attempt+1,
+                )
+            )
+            time.sleep(10)
+
+    return pkgname, ret
+
+def setup_git_config(
+    name='Ubuntu Git Importer',
+    email='[e-mail]',
+):
+    """setup_git_config - Ensure global required Git configuration values are set
+
+    Arguments:
+    name - string name to set as user.name in Git config
+    email - string email to set as user.email in Git config
+    """
+    try:
+        run(['git', 'config', '--global', 'user.name'], check=True)
+    except subprocess.CalledProcessError:
+        run(['git', 'config', '--global', 'user.name', name])
+    try:
+        run(['git', 'config', '--global', 'user.email'], check=True)
+    except subprocess.CalledProcessError:
+        run(['git', 'config', '--global', 'user.email', email])
+
+def pool_map_import_srcpkg(
+    num_workers,
+    dry_run,
+    pkgnames,
+):
+    """pool_map_import_srcpkg - Use a multiprocessing.Pool to parallel
+    import source packages
+
+    Arguments:
+    num_workers - integer number of worker processes to use
+    dry_run - a boolean to indicate a dry-run operation
+    pkgnames - a list of string names of source packages
+    """
+    with multiprocessing.Pool(processes=num_workers) as pool:
+        results = pool.map(
+            functools.partial(
+                import_srcpkg,
+                dry_run=dry_run,
+            ),
+            pkgnames,
+        )
+
+    return (
+        [pkg for pkg, success in results if success],
+        [pkg for pkg, success in results if not success],
+    )
diff --git a/scripts/source-package-walker.py b/scripts/source-package-walker.py
new file mode 100755
index 0000000..aaa8ef1
--- /dev/null
+++ b/scripts/source-package-walker.py
@@ -0,0 +1,272 @@
+#!/usr/bin/env python3
+
+# General design:
+# let publishes be the set of srcpkg names
+# for srcpkg in publishes:
+#    If srcpkg in blacklist: skip
+#    If srcpkg not in PHASING_{component}: skip
+#    try to import srcpkg
+# Report on successful and failed imports
+
+import argparse
+import bz2
+import itertools
+import gzip
+import lzma
+import os
+import sys
+import urllib.request
+
+import scriptutils
+
+from debian.deb822 import Sources
+
+# We expect to be running from a git repository in master for this
+# script, because the snap's python code is not exposed except within
+# the snap
+try:
+    REALPATH = os.readlink(__file__)
+except OSError:
+    REALPATH = __file__
+sys.path.insert(
+    0,
+    os.path.abspath(
+        os.path.join(os.path.dirname(REALPATH), os.path.pardir)
+    )
+)
+
+from gitubuntu.source_information import GitUbuntuSourceInformation
+
+def import_all_published_sources(
+    num_workers,
+    whitelist,
+    blacklist,
+    phasing_main,
+    phasing_universe,
+    dry_run,
+):
+    """import_all_published_sources - Import all publishes satisfying a
+    {white,black}list and phasing
+
+    Arguments:
+    num_workers - integer number of worker processes to use
+    whitelist - a list of of packages to always import
+    blacklist - a list of packages to never import
+    phasing_main - a integer percentage of all packages in main to import
+    phasing_universe - a integer percentage of all packages in universe to import
+    dry_run - a boolean to indicate a dry-run operation
+
+    Returns:
+    A tuple of two lists, the first containing the names of all
+    successfully imported source packages, the second containing the
+    names of all source packages that failed to import.
+    """
+    serieses = GitUbuntuSourceInformation('ubuntu').active_series_name_list
+    components = ['main', 'universe',]
+    pocket_suffixes = ['-proposed', '-updates', '-security', '',]
+    compressions = {
+        '.xz': lzma.open,
+        '.bz2': bz2.open,
+        '.gz': gzip.open,
+    }
+
+    base_sources_url = (
+        'http://archive.ubuntu.com/ubuntu/dists/%s%s/%s/source/Sources'
+    )
+    pkgnames = set()
+    for component in components:
+        for series, pocket_suffix in itertools.product(
+            serieses,
+            pocket_suffixes,
+        ):
+            url = base_sources_url % (
+                series,
+                pocket_suffix,
+                component,
+            )
+            for compression, opener in compressions.items():
+                try:
+                    with urllib.request.urlopen(
+                        url + compression
+                    ) as source_url_file:
+                        with opener(source_url_file, mode='r') as sources:
+                            print(url + compression)
+                            for src in Sources.iter_paragraphs(
+                                sources,
+                                #fields=['Package,'],
+                                use_apt_pkg=False,
+                            ):
+                                pkgname = src['Package']
+                                if scriptutils.should_import_srcpkg(
+                                    pkgname,
+                                    component,
+                                    whitelist,
+                                    blacklist,
+                                    phasing_main,
+                                    phasing_universe,
+                                ):
+                                    pkgnames.add(pkgname)
+                        break
+                except urllib.error.HTTPError:
+                    pass
+            else:
+                print(
+                    "Unable to find any Sources file for component=%s, "
+                    "series=%s, pocket=%s" % (
+                        component,
+                        series,
+                        pocket_suffix,
+                    )
+                )
+                sys.exit(1)
+
+    if not pkgnames:
+        print("No relevant publishes found")
+        return [], []
+
+    return scriptutils.pool_map_import_srcpkg(
+        num_workers=num_workers,
+        dry_run=dry_run,
+        pkgnames=pkgnames,
+    )
+
+def main(
+    num_workers=scriptutils.DEFAULTS.num_workers,
+    whitelist_path=scriptutils.DEFAULTS.whitelist,
+    blacklist_path=scriptutils.DEFAULTS.blacklist,
+    phasing_main=scriptutils.DEFAULTS.phasing_main,
+    phasing_universe=scriptutils.DEFAULTS.phasing_universe,
+    dry_run=scriptutils.DEFAULTS.dry_run,
+    use_whitelist=scriptutils.DEFAULTS.use_whitelist,
+):
+    """main - Main entry point to the script
+
+    Arguments:
+    num_workers - integer number of worker threads to use
+    whitelist_path - string filesystem path to a text file of packages
+    to always import
+    blacklist_path - string filesystem path to a text file of packages
+    to never import
+    phasing_main - a integer percentage of all packages in main to
+    import
+    phasing_universe - a integer percentage of all packages in universe
+    to import
+    dry_run - a boolean to indicate a dry-run operation
+    use_whitelist - a boolean to control whether the whitelist data is
+    used
+
+    use_whitelist exists because during the rampup of imports, we want
+    to import the whitelist packages and the phased packages. But after
+    that first operation to import (or possibly reimport), we do not
+    want to keep hitting the whitelist set (we only want to adjust the
+    phasing). This is more important in other scripts, but is relevant
+    here too.
+    """
+    scriptutils.setup_git_config()
+
+    if use_whitelist:
+        try:
+            with open(whitelist_path, 'r') as whitelist_file:
+                whitelist = [
+                    line.strip() for line in whitelist_file
+                    if not line.startswith('#')
+                ]
+        except (FileNotFoundError, IOError):
+            whitelist = list()
+    else:
+        whitelist = list()
+
+    try:
+        with open(blacklist_path, 'r') as blacklist_file:
+            blacklist = [
+                line.strip() for line in blacklist_file
+                if not line.startswith('#')
+            ]
+    except (FileNotFoundError, IOError):
+        blacklist = list()
+
+    imported_srcpkgs, failed_srcpkgs = import_all_published_sources(
+        num_workers,
+        whitelist,
+        blacklist,
+        phasing_main,
+        phasing_universe,
+        dry_run,
+    )
+    print(
+        "Imported %d source packages:\n%s" % (
+            len(imported_srcpkgs),
+            '\n'.join(imported_srcpkgs),
+        )
+    )
+    print(
+        "Failed to import %d source packages:\n%s" % (
+            len(failed_srcpkgs),
+            '\n'.join(failed_srcpkgs),
+        )
+    )
+
+def cli_main():
+    """cli_main - CLI entry point to script
+    """
+    parser = argparse.ArgumentParser(
+        description='Script to import all source packages with phasing',
+    )
+    parser.add_argument(
+        '--num-workers',
+        type=int,
+        help="Number of worker threads to use",
+        default=scriptutils.DEFAULTS.num_workers,
+    )
+    parser.add_argument(
+        '--no-whitelist',
+        action='store_false',
+        dest='use_whitelist',
+        help="Do not process packages in the whitelist",
+        default=not scriptutils.DEFAULTS.use_whitelist,
+    )
+    parser.add_argument(
+        '--whitelist',
+        type=str,
+        help="Path to whitelist file",
+        default=scriptutils.DEFAULTS.whitelist,
+    )
+    parser.add_argument(
+        '--blacklist',
+        type=str,
+        help="Path to blacklist file",
+        default=scriptutils.DEFAULTS.blacklist,
+    )
+    parser.add_argument(
+        '--phasing-universe',
+        type=int,
+        help="Percentage of universe packages to phase",
+        default=scriptutils.DEFAULTS.phasing_universe,
+    )
+    parser.add_argument(
+        '--phasing-main',
+        type=int,
+        help="Percentage of main packages to phase",
+        default=scriptutils.DEFAULTS.phasing_main,
+    )
+    parser.add_argument(
+        '--dry-run',
+        action='store_true',
+        help="Simulate operation but do not actually do anything",
+        default=scriptutils.DEFAULTS.dry_run,
+    )
+
+    args = parser.parse_args()
+
+    main(
+        num_workers=args.num_workers,
+        whitelist_path=args.whitelist,
+        blacklist_path=args.blacklist,
+        phasing_main=args.phasing_main,
+        phasing_universe=args.phasing_universe,
+        dry_run=args.dry_run,
+        use_whitelist=args.use_whitelist,
+    )
+
+if __name__ == '__main__':
+    cli_main()
diff --git a/scripts/update-repository-alias.py b/scripts/update-repository-alias.py
new file mode 100755
index 0000000..5f2c8ae
--- /dev/null
+++ b/scripts/update-repository-alias.py
@@ -0,0 +1,268 @@
+#!/usr/bin/env python3
+
+import argparse
+import functools
+import lzma
+import multiprocessing
+import os
+import sys
+import urllib
+
+import scriptutils
+
+# We expect to be running from a git repository in master for this
+# script, because the snap's python code is not exposed except within
+# the snap
+try:
+    REALPATH = os.readlink(__file__)
+except OSError:
+    REALPATH = __file__
+sys.path.insert(
+    0,
+    os.path.abspath(
+        os.path.join(os.path.dirname(REALPATH), os.path.pardir)
+    )
+)
+
+from gitubuntu.source_information import launchpad_login_auth
+
+def update_git_repository(package, dry_run, unset):
+    """update_git_repository - set the default Git Repository on Launchpad for a source package
+
+    Arguments:
+    package - string name of a source package
+    dry_run - a boolean to indicate a dry-run operation
+    unset - a boolean to indicate the URL should be set to None instead
+    of the usd-import-team repository
+    """
+    launchpad = launchpad_login_auth()
+    quoted_package = urllib.parse.quote(package)
+    target = launchpad.load('ubuntu/+source/%s' % quoted_package)
+    current_default = launchpad.git_repositories.getDefaultRepository(
+        target=target,
+    )
+    logmsg = list()
+    if current_default:
+        logmsg.append(
+            "Current default Git repository for %s: %s" % (
+                package,
+                current_default.git_https_url,
+            )
+        )
+    else:
+        logmsg.append("No default Git repository set for %s" % package)
+
+    if unset:
+        logmsg.append("Unsetting default Git repository for %s" % package)
+        if not dry_run:
+            launchpad.git_repositories.setDefaultRepository(
+                repository=None,
+                target=target,
+            )
+    else:
+        path = '~usd-import-team/ubuntu/+source/%s/+git/%s' % (
+            quoted_package,
+            quoted_package,
+        )
+        repository = launchpad.git_repositories.getByPath(path=path)
+        if repository:
+            logmsg.append(
+                "Setting default Git repository for %s to %s" % (
+                    package,
+                    path,
+                )
+            )
+            if not dry_run:
+                launchpad.git_repositories.setDefaultRepository(
+                    repository=repository,
+                    target=target,
+                )
+        else:
+            logmsg.append("No usd-import-team repository for %s" % package)
+
+    print('\n'.join(logmsg))
+
+# add whitelist, blacklist, phasing and loop
+def main(
+    num_workers=scriptutils.DEFAULTS.num_workers,
+    whitelist_path=scriptutils.DEFAULTS.whitelist,
+    blacklist_path=scriptutils.DEFAULTS.blacklist,
+    phasing_main=scriptutils.DEFAULTS.phasing_main,
+    phasing_universe=scriptutils.DEFAULTS.phasing_universe,
+    dry_run=scriptutils.DEFAULTS.dry_run,
+    unset=False,
+    use_whitelist=scriptutils.DEFAULTS.use_whitelist,
+):
+    """main - Main entry point to the script
+
+    Set the default Git Repository target on Launchpad for all source
+    packages imported to usd-import-team to the usd-import-team
+    repository.
+
+    Arguments:
+    num_workers - integer number of worker threads to use
+    whitelist - a list of of packages to always import
+    blacklist - a list of packages to never import
+    phasing_main - a integer percentage of all packages in main to import
+    phasing_universe - a integer percentage of all packages in universe to import
+    dry_run - a boolean to indicate a dry-run operation
+    unset - a boolean to indicate the URL should be set to None instead
+    of the usd-import-team repository
+    use_whitelist - a boolean to control whether the whitelist data is
+    used
+
+    use_whitelist exists because during the rampup of imports, we want
+    to import the whitelist packages and the phased packages. But after
+    that first operation to import (or possibly reimport), we do not
+    want to keep hitting the whitelist set (we only want to adjust the
+    phasing). This is more important in other scripts, but is relevant
+    here too.
+    """
+    if use_whitelist:
+        try:
+            with open(whitelist_path, 'r') as whitelist_file:
+                whitelist = [
+                    line.strip() for line in whitelist_file
+                    if not line.startswith('#')
+                ]
+        except (FileNotFoundError, IOError):
+            whitelist = list()
+    else:
+        whitelist = list()
+
+    try:
+        with open(blacklist_path, 'r') as blacklist_file:
+            blacklist = [
+                line.strip() for line in blacklist_file
+                if not line.startswith('#')
+            ]
+    except (FileNotFoundError, IOError):
+        blacklist = list()
+
+    main_packages = list()
+    with urllib.request.urlopen(
+        'http://archive.ubuntu.com/ubuntu/dists/devel/main/source/Sources.xz'
+    ) as source_url_file:
+        with lzma.open(source_url_file, mode='rt') as main_sources:
+            for line in main_sources:
+                if line.startswith('Package:'):
+                    _, pkgname = line.split(':')
+                    main_packages.append(pkgname.strip())
+
+    universe_packages = list()
+    with urllib.request.urlopen(
+        'http://archive.ubuntu.com/ubuntu/dists/devel/universe/source/Sources.xz'
+    ) as source_url_file:
+        with lzma.open(source_url_file, mode='rt') as universe_sources:
+            for line in universe_sources:
+                if line.startswith('Package:'):
+                    _, pkgname = line.split(':')
+                    universe_packages.append(pkgname.strip())
+
+    filtered_pkgnames = set()
+    for pkgname in main_packages:
+        if scriptutils.should_import_srcpkg(
+            pkgname,
+            'main',
+            whitelist,
+            blacklist,
+            phasing_main,
+            phasing_universe,
+        ):
+            filtered_pkgnames.add(pkgname)
+    for pkgname in universe_packages:
+        if scriptutils.should_import_srcpkg(
+            pkgname,
+            'universe',
+            whitelist,
+            blacklist,
+            phasing_main,
+            phasing_universe,
+        ):
+            filtered_pkgnames.add(pkgname)
+
+    if not filtered_pkgnames:
+        print("No relevant publishes found")
+        return
+
+    with multiprocessing.Pool(processes=num_workers) as pool:
+        pool.map(
+            functools.partial(
+                update_git_repository,
+                dry_run=dry_run,
+                unset=unset,
+            ),
+            filtered_pkgnames,
+        )
+
+def cli_main():
+    """cli_main - main entry point for CLI
+    """
+    parser = argparse.ArgumentParser(
+        description='Update the default Git repository for imported source packages',
+    )
+    parser.add_argument(
+        '--num-workers',
+        type=int,
+        help="Number of worker threads to use",
+        default=scriptutils.DEFAULTS.num_workers,
+    )
+    parser.add_argument(
+        '--no-whitelist',
+        action='store_false',
+        dest='use_whitelist',
+        help="Do not process packages in the whitelist",
+        default=not scriptutils.DEFAULTS.use_whitelist,
+    )
+    parser.add_argument(
+        '--whitelist',
+        type=str,
+        help="Path to whitelist file",
+        default=scriptutils.DEFAULTS.whitelist,
+    )
+    parser.add_argument(
+        '--blacklist',
+        type=str,
+        help="Path to blacklist file",
+        default=scriptutils.DEFAULTS.blacklist,
+    )
+    parser.add_argument(
+        '--phasing-universe',
+        type=int,
+        help="Percentage of universe packages to phase",
+        default=scriptutils.DEFAULTS.phasing_universe,
+    )
+    parser.add_argument(
+        '--phasing-main',
+        type=int,
+        help="Percentage of main packages to phase",
+        default=scriptutils.DEFAULTS.phasing_main,
+    )
+    parser.add_argument(
+        '--dry-run',
+        action='store_true',
+        help="Simulate operation but do not actually do anything",
+        default=False,
+    )
+    parser.add_argument(
+        '--unset',
+        action='store_true',
+        help="Unset default repository (for testing)",
+        default=False,
+    )
+
+    args = parser.parse_args()
+
+    main(
+        num_workers=args.num_workers,
+        whitelist_path=args.whitelist,
+        blacklist_path=args.blacklist,
+        phasing_main=args.phasing_main,
+        phasing_universe=args.phasing_universe,
+        dry_run=args.dry_run,
+        unset=args.unset,
+        use_whitelist=args.use_whitelist,
+    )
+
+if __name__ == '__main__':
+    cli_main()