Merge ~nacc/git-ubuntu:lp1730734-cache-importer-progress into git-ubuntu:master

[Nish Aravamudan]

Make jenkins happy.

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:16af9e00af3e93c4fcba5a49cd660d01d6d10f61
https://jenkins.ubuntu.com/server/job/git-ubuntu-ci/205/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Style Check
    SUCCESS: Unit Tests
    SUCCESS: Integration Tests
    IN_PROGRESS: Declarative: Post Actions

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/git-ubuntu-ci/205/rebuild

review: Approve (continuous-integration)

[Nish Aravamudan (nacc)]

As a local test, I built the git-ubuntu snap with this change in place and ran the following:

# 1) prime the cache
$ git ubuntu import --no-push ipsec-tools --no-clean --db-cache /tmp/git-ubuntu-db-cache
11/10/2017 11:58:19 - INFO:Ubuntu Server Team importer v0.6.2
11/10/2017 11:58:19 - INFO:Using git repository at /tmp/tmpso3vrstw
11/10/2017 11:59:25 - INFO:Importing patches-unapplied 1:0.8.2+20140711-10 to ubuntu/bionic
11/10/2017 11:59:39 - WARNING:ubuntu/bionic is identical to 1:0.8.2+20140711-10
11/10/2017 12:00:15 - INFO:Not pushing to remote as specified
11/10/2017 12:00:15 - INFO:Leaving /tmp/tmpso3vrstw as directed

# 2) setup test repositories
$ cp -R /tmp/tmpso3vrstw /tmp/cache-test
$ cp -R /tmp/tmpso3vrstw /tmp/no-cache-test

# 3) Run again using the same cache, no new uploads processed (cache worked)
$ git ubuntu import --no-push ipsec-tools --no-clean --db-cache /tmp/git-ubuntu-db-cache --no-fetch -d /tmp/cache-test
11/10/2017 12:01:45 - INFO:Ubuntu Server Team importer v0.6.2
11/10/2017 12:02:49 - INFO:Not pushing to remote as specified
11/10/2017 12:02:49 - INFO:Leaving /tmp/cache-test as directed

# 4) Run again not using the cache (default operation, as well), we end up processing already seen records
$ git ubuntu import --no-push ipsec-tools --no-clean --no-fetch -d /tmp/no-cache-test
11/10/2017 12:03:33 - INFO:Ubuntu Server Team importer v0.6.2
11/10/2017 12:04:17 - INFO:Importing patches-unapplied 1:0.8.2+20140711-10 to ubuntu/bionic
11/10/2017 12:04:22 - WARNING:ubuntu/bionic is identical to 1:0.8.2+20140711-10
11/10/2017 12:04:37 - INFO:Importing patches-applied 1:0.8.2+20140711-10 to ubuntu/bionic
11/10/2017 12:04:40 - WARNING:ubuntu/bionic is identical to 1:0.8.2+20140711-10
11/10/2017 12:05:07 - INFO:Not pushing to remote as specified
11/10/2017 12:05:07 - INFO:Leaving /tmp/no-cache-test as directed

[Nish Aravamudan (nacc)]

I am fairly confident in the actual changes here. What I think needs deciding is how the linear import script and publisher-watching script should interact with a shared DBM cache. I think it is relatively safe within one process, but from what I can tell, there is no guarantee the two won't race and the DBM implementation itself is not concurrency safe.

Possibly we should switch to sqlite3 for that, then, even though it is more overhead to configure and query.

Finally, I'm wondering how to describe the race in question. Basically, we'd see two processes read different (possibly unset in one case) values for the last SPPHR seen in the cache. They would then iterate a different set of Launchpad records, but the result would be the same, relative to the actual contents of the imports. The question is the branches, I think. They both start with the same git repository (effectively, under the assumption neither has pushed) and move the branches. In the case of one processing more (older) records, they would move the branches more, but the end result *should* be the same, or an ancestor of what will occur. But that's the concern (ancestor) as we are now timing-sensitive to what gets pushed (since we force push branches). I wonder if we should treat it more like RCU, in that we can always read or write, but the read data may be stale. So it needs to be verified before and after our loop. I imagine we could read the data, iterate our records and import them, and before we write our last_spphr back into the database, check to see if the current value is still the same? That would shrink the race to between re-reading the value and writing the new value (and the linear importer is a one-time operation, in theory).

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:e087180218bfe4276957a93c9375d875c6652a3d
https://jenkins.ubuntu.com/server/job/git-ubuntu-ci/207/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Style Check
    SUCCESS: Unit Tests
    SUCCESS: Integration Tests
    IN_PROGRESS: Declarative: Post Actions

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/git-ubuntu-ci/207/rebuild

review: Approve (continuous-integration)

[Nish Aravamudan (nacc)]

Aother thought, does the cache need to be distinct for debian/ubuntu X unapplied/applied? Given that applied failures may be fixable, they can be at a different version than the unapplied?

[Christian Ehrhardt  (paelzer)]

I'm not sure I can help you with the "implications" as you asked on IRC without thinking much more into it. But I have a few questions that might help to clarify at least.

To do so I want to hear more details on the actual use-case that lets us run into this race:
1) Do we even want that two importers run concurrently on the same system?
   If not lets just lock the DB to be of single use - problem solved.

2) If we want multiple processes to be allowed we could make the use of
   the cache exclusive per package. (I beg your pardon if I don't see the point why this would
   fail). So only one worker can be in said [package] at any time.
   Processes working on different packages should never conflict right?
   Another worker with a different [package] can continue and we know there won't be a race.
   Essentially locking on [package] (yes this might need a better db to support
   rollback/timeouts on aborted processes if you want to lock in the DB).
   The tail of the processing would be updating sphhr and then unlocking the given package.

In general I think you should add tests along these commits that e.g. does "import-no-cache-used == import into cache == import with cache".

[Nish Aravamudan (nacc)]

On 08.01.2018 [20:17:52 -0000], ChristianEhrhardt wrote:
> I'm not sure I can help you with the "implications" as you asked on
> IRC without thinking much more into it. But I have a few questions
> that might help to clarify at least.

Understood and thank you!

> To do so I want to hear more details on the actual use-case that lets
> us run into this race:

The race is internal, in some ways, to the importer, but you do bring up
an interesting second case.

`git ubuntu import` is managed by a looping script,
scripts/import-source-packages.py, which is operating in a keepup
fashion. Roughly:

Obtain a LIST of publishing events in reverse chronological order
Walk LIST backwards until we find a PUBLISH that we have
already imported (or earlier)
   Walk LIST foward after PUBLISH and IMPORT each unique source package
name

IMPORT itself is algorithmically:
   Get current REPOSITORY
   Get Launchpad publishing DATA for a given source package in reverse
chronological order
   Walk DATA backwards until we find a PUBLISH that we have already
imported (or earlier)
      Walk DATA forwards and import each publish record

The issue we are trying to resolve here is the "until we find a PUBLISH
that we have already imported" in IMPORT.

In the prior importer code, we had a unique Git commit for every
publication event, because the targetted <series>-<pocket> was part of
the Git commit (via the commit message and parenting). So as we did
IMPORT's reverse walk, we could look at the branch tips and compare
their commit data (where we stored the publishing timestamp) to the
publishing records to find the last imported publishing record.

We dropped that ability, by dropping publishing parents altogether. We
now just import all published versions once, tying them together only by
their changelogs. We then forcibly move the branch tips.

So now if we use an unmodified importer, we will end up going back in
IMPORT to the last publish that was the first time we saw a given
version (typically in Debian, therefore) and walking forward from there.
This will be unnecessary iteration of publishing data, at least, and
unnecessary moving of the branch pointers.

My branch modifies the catch-up to move the storage of the publication
event to an external cache, currently DBM.

> 1) Do we even want that two importers run concurrently on the same system?
> If not lets just lock the DB to be of single use - problem solved.

Right, so we have a mode we know will exist at some point, where there
is a linear script walking all 'to-import' source packages getting them
loaded to Launchpad, and the keep-up script which is ensuring that
publishes that are happeninng while the linear walker is going get
integrated. So we do expect to see (and it shouldn't just break) if
there are two imports of the same source package going on. What we don't
want to happen is for a slower 'old' import (where the list of events to
import is older) to somehow trump a faster 'new' import and thus end up
setting the branch pointers to the wrong place(s).

The problem is, that if we just outright lock the DB, then the linear
script can't work if it's also using the DB. And the point of it is that
they...

[Nish Aravamudan (nacc)]

> Aother thought, does the cache need to be distinct for debian/ubuntu X
> unapplied/applied? Given that applied failures may be fixable, they can be at
> a different version than the unapplied?

I do think we actually need 4 caches, since the unapplied / applied heads can easily be at a different state. I'll work on this now.

I'll also add some changes to the importer to clean up that code.

[Nish Aravamudan (nacc)]

> Aother thought, does the cache need to be distinct for debian/ubuntu X
> unapplied/applied? Given that applied failures may be fixable, they can be at
> a different version than the unapplied?

I do think we actually need 4 caches, since the unapplied / applied heads can easily be at a different state. I'll work on this now.

I'll also add some changes to the importer to clean up that code.

[Christian Ehrhardt  (paelzer)]

[...]

> So now if we use an unmodified importer, we will end up going back in
> IMPORT to the last publish that was the first time we saw a given
> version (typically in Debian, therefore) and walking forward from there.

To confirm - you go to "the first time we saw a given version" because without parenting history that is the only safe place to walk forward from?

> This will be unnecessary iteration of publishing data

because some might already have been handled.

>, at least, and
> unnecessary moving of the branch pointers.

For the walking

Thanks for the details, much clearer to me now what your case actually is.

>
> My branch modifies the catch-up to move the storage of the publication
> event to an external cache, currently DBM.

[...]

> The more I think about it, I think I am spinning myself on this problem
> for no reason :)
>
> I think we can let the linear walker not use the dbm [we can keep the
> code, just not pass anything] (or sqlite) and it will just mean if there
> is a race between when the linear walker hits a source package and when
> the keep-up script does, there will be some no-op looping (unless new
> SPRs are found).

Sounds good, and then you can actually lock the DB to be fully safe against unintentional concurrent use of it by the catch up (if one manually starts it twice or so).

[...]

> I do think we actually need 4 caches, since the unapplied / applied heads can easily be at a
> different state. I'll work on this now.

> I'll also add some changes to the importer to clean up that code.

Since you will end up only accessing it by code and never manually I think you could even implement as debian/ubuntu x unapplied/applied x per-package. If you end up locking the DBs in any way you will still have a better granularity on those locks.
And for your code it doesn't matter how split the file structure is.

Unmerged commits

a9960c8... by Nish Aravamudan on 2017-11-08

git ubuntu import: use a dbm cache to store importer progress

With the recent changes to the importer algorithm, we no longer can
identify a Launchpad publishing event by the commit information in the
repository -- a given SourcePackageRelease (source package name and
version) is only imported once (presuming all future publishes of the
same name and version match exactly). We rely on this in
launchpad_versions_published_after, which iterates the Launchpad data
backwards until we either match the commit data for a branch head, or
see Launchpad data from before one of the branch heads.

Change the importer code to take a --db-cache argument as a directory
containing a DBM cache for debian and ubuntu (this is needed, because
DBM are single-leve string-indexed string storage databases). Lookup the
source package name in the relevant cache before we lookup Launchpad
data, to obtain the last SPPHR used. Store the latest processed SPPHR
after iterating the Launchpad data.

Also update the scripts to support passing a persistent/consistent value
to the import for the cache.

LP: #1730734

diff --git a/gitubuntu/importer.py b/gitubuntu/importer.py
index 2063f83..6abad7d 100644
--- a/gitubuntu/importer.py
+++ b/gitubuntu/importer.py
@@ -26,6 +26,7 @@
 
 import argparse
 import atexit
+import dbm
 import functools
 import getpass
 import logging
@@ -150,6 +151,7 @@ def main(
     parentfile=top_level_defaults.parentfile,
     retries=top_level_defaults.retries,
     retry_backoffs=top_level_defaults.retry_backoffs,
+    db_cache_dir=None,
 ):
     """Main entry point to the importer
 
@@ -178,6 +180,9 @@ def main(
     @parentfile: string path to file specifying parent overrides
     @retries: integer number of download retries to attempt
     @retry_backoffs: list of backoff durations to use between retries
+    @db_cache_dir: string fileystem directory containing 'ubuntu' and
+    'debian' dbm database files, which store the progress of prior
+    importer runs
 
     If directory is None, a temporary directory is created and used.
 
@@ -187,6 +192,11 @@ def main(
     If dl_cache is None, CACHE_PATH in the local repository will be
     used.
 
+    If db_cache_dir is None, no database lookups are performed which makes
+    it possible that the import will attempt to re-import already
+    imported publishes. This should be fine, although less efficient
+    than possible.
+
     Returns 0 on successful import (which includes non-fatal failures);
     1 otherwise.
     """
@@ -314,6 +324,9 @@ def main(
     else:
         workdir = dl_cache
 
+    if db_cache_dir:
+        os.makedirs(db_cache_dir, exist_ok=True)
+
     os.makedirs(workdir, exist_ok=True)
 
     # now sets a global _PARENT_OVERRIDES
@@ -326,6 +339,7 @@ def main(
         patches_applied=False,
         debian_head_versions=debian_head_versions,
         ubuntu_head_versions=ubuntu_head_versions,
+        db_cache_dir=db_cache_dir,
         debian_sinfo=debian_sinfo,
         ubuntu_sinfo=ubuntu_sinfo,
         active_series_only=active_series_only,
@@ -347,6 +361,7 @@ def main(
             patches_applied=True,
             debian_head_versions=applied_debian_head_versions,
             ubuntu_head_versions=applied_ubuntu_head_versions,
+            db_cache_dir=db_cache_dir,
             debian_sinfo=debian_sinfo,
             ubuntu_sinfo=ubuntu_sinfo,
             active_series_only=active_series_only,
@@ -1401,6 +1416,7 @@ def import_publishes(
     patches_applied,
     debian_head_versions,
     ubuntu_head_versions,
+    db_cache_dir,
     debian_sinfo,
     ubuntu_sinfo,
     active_series_only,
@@ -1411,6 +1427,8 @@ def import_publishes(
     history_found = False
     only_debian = False
     srcpkg_information = None
+    last_debian_spphr = None
+    last_ubuntu_spphr = None
     if patches_applied:
         _namespace = namespace
         namespace = '%s/applied' % namespace
@@ -1426,15 +1444,28 @@ def import_publishes(
             import_unapplied_spi,
             skip_orig=skip_orig,
         )
-    for distname, versions, dist_sinfo in (
-            ("debian", debian_head_versions, debian_sinfo),
-            ("ubuntu", ubuntu_head_versions, ubuntu_sinfo)):
+
+    for distname, versions, dist_sinfo, last_spphr in (
+        ("debian", debian_head_versions, debian_sinfo, last_debian_spphr),
+        ("ubuntu", ubuntu_head_versions, ubuntu_sinfo, last_ubuntu_spphr),
+    ):
         if active_series_only and distname == "debian":
             continue
+
+        last_spphr = None
+        if db_cache_dir:
+             with dbm.open(os.path.join(db_cache_dir, distname), 'c') as cache:
+                try:
+                    last_spphr = decode_binary(cache[pkgname])
+                except KeyError:
+                    pass
+
         try:
+            last_spi = None
             for srcpkg_information in dist_sinfo.launchpad_versions_published_after(
                 versions,
                 namespace,
+                last_spphr,
                 workdir=workdir,
                 active_series_only=active_series_only
             ):
@@ -1445,6 +1476,11 @@ def import_publishes(
                     namespace=_namespace,
                     ubuntu_sinfo=ubuntu_sinfo,
                 )
+                last_spi = srcpkg_information
+            if last_spi:
+                if db_cache_dir:
+                    with dbm.open(os.path.join(db_cache_dir, distname), 'w') as db_cache:
+                        db_cache[pkgname] = str(last_spi.spphr)
         except NoPublicationHistoryException:
             logging.warning("No publication history found for %s in %s.",
                 pkgname, distname
@@ -1520,6 +1556,11 @@ def parse_args(subparsers=None, base_subparsers=None):
         action='store_true',
         help=argparse.SUPPRESS,
     )
+    parser.add_argument(
+        '--db-cache',
+        type=str,
+        help=argparse.SUPPRESS,
+    )
     if not subparsers:
         return parser.parse_args()
     return 'import - %s' % kwargs['description']
@@ -1548,6 +1589,11 @@ def cli_main(args):
     except AttributeError:
         dl_cache = None
 
+    try:
+        db_cache = args.db_cache
+    except AttributeError:
+        db_cache = None
+
     return main(
         pkgname=args.package,
         owner=args.lp_owner,
@@ -1567,4 +1613,5 @@ def cli_main(args):
         parentfile=args.parentfile,
         retries=args.retries,
         retry_backoffs=args.retry_backoffs,
+        db_cache_dir=args.db_cache,
     )
diff --git a/gitubuntu/source_information.py b/gitubuntu/source_information.py
index b97bc8a..050d60a 100644
--- a/gitubuntu/source_information.py
+++ b/gitubuntu/source_information.py
@@ -443,7 +443,14 @@ class GitUbuntuSourceInformation(object):
         for srcpkg in spph:
             yield self.get_corrected_spi(srcpkg, workdir)
 
-    def launchpad_versions_published_after(self, head_versions, namespace, workdir=None, active_series_only=False):
+    def launchpad_versions_published_after(
+       self,
+       head_versions,
+       namespace,
+       last_spphr=None,
+       workdir=None,
+       active_series_only=False,
+    ):
         args = {
                 'exact_match':True,
                 'source_name':self.pkgname,
@@ -471,7 +478,14 @@ class GitUbuntuSourceInformation(object):
         if len(spph) == 0:
             raise NoPublicationHistoryException("Is %s published in %s?" %
                                                 (self.pkgname, self.dist_name))
-        if len(head_versions) > 0:
+        if last_spphr:
+            _spph = list()
+            for spphr in spph:
+                if str(spphr) == last_spphr:
+                    break
+                _spph.append(spphr)
+            spph = _spph
+        elif head_versions:
             _spph = list()
             for spphr in spph:
                 spi = GitUbuntuSourcePackageInformation(spphr, self.dist_name,
diff --git a/man/man1/git-ubuntu-import.1 b/man/man1/git-ubuntu-import.1
index dd7fd9e..74bfe83 100644
--- a/man/man1/git-ubuntu-import.1
+++ b/man/man1/git-ubuntu-import.1
@@ -1,4 +1,4 @@
-.TH "GIT-UBUNTU-IMPORT" "1" "2017-07-19" "Git-Ubuntu 0.2" "Git-Ubuntu Manual"
+.TH "GIT-UBUNTU-IMPORT" "1" "2017-11-08" "Git-Ubuntu 0.6.2" "Git-Ubuntu Manual"
 
 .SH "NAME"
 git-ubuntu import \- Import Launchpad publishing history to Git
@@ -9,7 +9,8 @@ git-ubuntu import \- Import Launchpad publishing history to Git
 <user>] [\-\-dl-cache <dl_cache>] [\-\-no-fetch] [\-\-no-push]
 [\-\-no-clean] [\-d | \-\-directory <directory>]
 [\-\-active-series-only] [\-\-skip-applied] [\-\-skip-orig]
-[\-\-reimport] [\-\-allow-applied-failures] <package>
+[\-\-reimport] [\-\-allow-applied-failures] [\-\-db-cache <db_cache>]
+<package>
 .FI
 .SP
 .SH "DESCRIPTION"
@@ -197,6 +198,21 @@ After investigation, this flag can be used to indicate the importer is
 allowed to ignore such a failure\&.
 .RE
 .PP
+\-\-db-cache <db_cache>
+.RS 4
+The path to a directory containing Python dbm database disk files for
+importer metadata\&.
+If \fB<db_cache>\fR does not exist, it will be created\&.
+Two files in \fB<db_cache>\fR are used, "ubuntu" and "debian', which are
+created if not already present\&.
+The cache files provide information to the importer about prior imports
+of \fB<package>\fR and which Launchpad publishing record was last
+imported\&.
+This is necessary because the imported Git repository does not
+necessarily maintain any metadata about Launchpad publishing
+information\&.
+.RE
+.PP
 <package>
 .RS 4
 The name of the source package to import\&.
diff --git a/scripts/import-source-packages.py b/scripts/import-source-packages.py
index 698ae35..7dd534e 100755
--- a/scripts/import-source-packages.py
+++ b/scripts/import-source-packages.py
@@ -50,6 +50,7 @@ def import_new_published_sources(
     phasing_main,
     phasing_universe,
     dry_run,
+    db_cache,
 ):
     """import_new_published_source - Import all new publishes since a prior execution
 
@@ -60,6 +61,10 @@ def import_new_published_sources(
     phasing_main - a integer percentage of all packages in main to import
     phasing_universe - a integer percentage of all packages in universe to import
     dry_run - a boolean to indicate a dry-run operation
+    db_cache - string filesystem path containing DBM files for storing
+    importer progress, will be created if it does not exist
+
+    If db_cache is None, no cache is used.
 
     Returns:
     A tuple of two lists, the first containing the names of all
@@ -150,6 +155,7 @@ def import_new_published_sources(
     ret = scriptutils.pool_map_import_srcpkg(
         num_workers=num_workers,
         dry_run=dry_run,
+        db_cache=db_cache,
         pkgnames=filtered_pkgnames,
     )
 
@@ -229,6 +235,7 @@ def main(
     phasing_main=scriptutils.DEFAULTS.phasing_main,
     phasing_universe=scriptutils.DEFAULTS.phasing_universe,
     dry_run=scriptutils.DEFAULTS.dry_run,
+    db_cache=scriptutils.DEFAULTS.db_cache,
 ):
     """main - Main entry point to the script
 
@@ -243,6 +250,8 @@ def main(
     phasing_universe - a integer percentage of all packages in universe
     to import
     dry_run - a boolean to indicate a dry-run operation
+    db_cache - string filesystem path containing DBM files for storing
+    importer progress, will be created if it does not exist
     """
     scriptutils.setup_git_config()
 
@@ -280,6 +289,7 @@ def main(
             phasing_main,
             phasing_universe,
             dry_run,
+            db_cache,
         )
         print("Imported %d source packages" % len(imported_srcpkgs))
         mail_imported_srcpkgs |= set(imported_srcpkgs)
@@ -361,6 +371,13 @@ def cli_main():
         help="Simulate operation but do not actually do anything",
         default=scriptutils.DEFAULTS.dry_run,
     )
+    parser.add_argument(
+        '--db-cache',
+        type=str,
+        help="Directory containing Python DBM files which store importer "
+             "progress",
+        default=scriptutils.DEFAULTS.db_cache,
+    )
 
     args = parser.parse_args()
 
@@ -371,6 +388,7 @@ def cli_main():
         phasing_main=args.phasing_main,
         phasing_universe=args.phasing_universe,
         dry_run=args.dry_run,
+        db_cache=args.db_cache,
     )
 
 if __name__ == '__main__':
diff --git a/scripts/scriptutils.py b/scripts/scriptutils.py
index 912efd1..e24c394 100644
--- a/scripts/scriptutils.py
+++ b/scripts/scriptutils.py
@@ -5,6 +5,7 @@ import multiprocessing
 import os
 import sys
 import subprocess
+import tempfile
 import time
 
 import pkg_resources
@@ -35,6 +36,7 @@ Defaults = namedtuple(
         'phasing_main',
         'dry_run',
         'use_whitelist',
+        'db_cache',
     ],
 )
 
@@ -52,6 +54,7 @@ DEFAULTS = Defaults(
     phasing_main=1,
     dry_run=False,
     use_whitelist=True,
+    db_cache=os.path.join(tempfile.gettempdir(), 'git-ubuntu-db-cache'),
 )
 
 
@@ -101,12 +104,16 @@ def should_import_srcpkg(
     return False
 
 
-def import_srcpkg(pkgname, dry_run):
+def import_srcpkg(pkgname, dry_run, db_cache):
     """import_srcpkg - Invoke git ubuntu import on @pkgname
 
     Arguments:
     pkgname - string name of a source package
     dry_run - a boolean to indicate a dry-run operation
+    db_cache - string filesystem path containing DBM files for storing
+    importer progress, will be created if it does not exist
+
+    If db_cache is None, no cache is used.
 
     Returns:
     A tuple of a string and a boolean, where the boolean is the success
@@ -125,6 +132,8 @@ def import_srcpkg(pkgname, dry_run):
             'usd-importer-bot',
             pkgname,
         ]
+        if db_cache:
+            cmd.extend(['--db-cache', db_cache,])
         try:
             print(' '.join(cmd))
             if not dry_run:
@@ -163,6 +172,7 @@ def setup_git_config(
 def pool_map_import_srcpkg(
     num_workers,
     dry_run,
+    db_cache,
     pkgnames,
 ):
     """pool_map_import_srcpkg - Use a multiprocessing.Pool to parallel
@@ -171,13 +181,18 @@ def pool_map_import_srcpkg(
     Arguments:
     num_workers - integer number of worker processes to use
     dry_run - a boolean to indicate a dry-run operation
+    db_cache - string filesystem path containing DBM files for storing
+    importer progress, will be created if it does not exist
     pkgnames - a list of string names of source packages
+
+    If db_cache is None, no cache is used.
     """
     with multiprocessing.Pool(processes=num_workers) as pool:
         results = pool.map(
             functools.partial(
                 import_srcpkg,
                 dry_run=dry_run,
+                db_cache=db_cache,
             ),
             pkgnames,
         )
diff --git a/scripts/source-package-walker.py b/scripts/source-package-walker.py
index 58c3d5a..f7b9629 100755
--- a/scripts/source-package-walker.py
+++ b/scripts/source-package-walker.py
@@ -44,6 +44,7 @@ def import_all_published_sources(
     phasing_main,
     phasing_universe,
     dry_run,
+    db_cache,
 ):
     """import_all_published_sources - Import all publishes satisfying a
     {white,black}list and phasing
@@ -55,6 +56,10 @@ def import_all_published_sources(
     phasing_main - a integer percentage of all packages in main to import
     phasing_universe - a integer percentage of all packages in universe to import
     dry_run - a boolean to indicate a dry-run operation
+    db_cache - string filesystem path containing DBM files for storing
+    importer progress, will be created if it does not exist
+
+    If db_cache is None, no cache is used.
 
     Returns:
     A tuple of two lists, the first containing the names of all
@@ -126,6 +131,7 @@ def import_all_published_sources(
     return scriptutils.pool_map_import_srcpkg(
         num_workers=num_workers,
         dry_run=dry_run,
+        db_cache=db_cache,
         pkgnames=pkgnames,
     )
 
@@ -137,6 +143,7 @@ def main(
     phasing_universe=scriptutils.DEFAULTS.phasing_universe,
     dry_run=scriptutils.DEFAULTS.dry_run,
     use_whitelist=scriptutils.DEFAULTS.use_whitelist,
+    db_cache=scriptutils.DEFAULTS.db_cache,
 ):
     """main - Main entry point to the script
 
@@ -153,6 +160,8 @@ def main(
     dry_run - a boolean to indicate a dry-run operation
     use_whitelist - a boolean to control whether the whitelist data is
     used
+    db_cache - string filesystem path containing DBM files for storing
+    importer progress, will be created if it does not exist
 
     use_whitelist exists because during the rampup of imports, we want
     to import the whitelist packages and the phased packages. But after
@@ -191,6 +200,7 @@ def main(
         phasing_main,
         phasing_universe,
         dry_run,
+        db_cache,
     )
     print(
         "Imported %d source packages:\n%s" % (
@@ -254,6 +264,13 @@ def cli_main():
         help="Simulate operation but do not actually do anything",
         default=scriptutils.DEFAULTS.dry_run,
     )
+    parser.add_argument(
+        '--db-cache',
+        type=str,
+        help="Directory containing Python DBM files which store importer "
+             "progress",
+        default=scriptutils.DEFAULTS.db_cache,
+    )
 
     args = parser.parse_args()
 
@@ -265,6 +282,7 @@ def cli_main():
         phasing_universe=args.phasing_universe,
         dry_run=args.dry_run,
         use_whitelist=args.use_whitelist,
+        db_cache=args.db_cache,
     )
 
 if __name__ == '__main__':