Problems in applied patches imports cause import failures which are not ignored

Traceback (most recent call last):
  File "/home/robie/Code/Canonical/usd-importer/bin/usd", line 15, in <module>
    main()
  File "/home/robie/Code/Canonical/usd-importer/usd/__main__.py", line 160, in main
    args.func(args)
  File "/home/robie/Code/Canonical/usd-importer/usd/importer.py", line 934, in main
    self.import_spi(srcpkg_information)
  File "/home/robie/Code/Canonical/usd-importer/usd/importer.py", line 816, in import_spi
    for (applied_import_tree_hash, patch_name, patch_desc) in self.import_patches_applied_tree(spi):
  File "/home/robie/Code/Canonical/usd-importer/usd/importer.py", line 420, in import_patches_applied_tree
    cp = run(['dpkg-source', '--print-format', extracted_dir])
  File "/home/robie/Code/Canonical/usd-importer/usd/run.py", line 58, in run
    raise e
  File "/home/robie/Code/Canonical/usd-importer/usd/run.py", line 44, in run
    stdout=stdout, stderr=stderr, stdin=stdin)
  File "/usr/lib/python3.5/subprocess.py", line 708, in run
    output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['dpkg-source', '--print-format', '/tmp/tmp3y515osf/util-linux-2.13~rc3']' returned non-zero exit status 25

Crude workaround in https://code.launchpad.net/~racb/usd-importer/+git/usd-importer/+ref/disable-applied

Robie,

Here is a fairly dumb version of what you asked for (but smaller than your workaround), that orphans if patches-applied fails. I think this will do the right thing, and does successfully import util-linux again, but I'm waiting for the import to finish before verifying fully. Holidays and all oncoming, probably won't commit this until the new year and after discussing with you.

For reference, here is the output you get:

12/23/2016 15:24:01 - ERROR:Unable to import patches-applied 2.13~rc3-5
Traceback (most recent call last):
  File "/home/nacc/work/usd-importer/usd/importer.py", line 817, in import_spi
    for (applied_import_tree_hash, patch_name, patch_desc) in self.import_patches_applied_tree(spi):
  File "/home/nacc/work/usd-importer/usd/importer.py", line 420, in import_patches_applied_tree
    cp = run(['dpkg-source', '--print-format', extracted_dir])
  File "/home/nacc/work/usd-importer/usd/run.py", line 58, in run
    raise e
  File "/home/nacc/work/usd-importer/usd/run.py", line 44, in run
    stdout=stdout, stderr=stderr, stdin=stdin)
  File "/usr/lib/python3.5/subprocess.py", line 708, in run
    output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['dpkg-source', '--print-format', '/tmp/tmpa_3xs9wb/util-linux-2.13~rc3']' returned non-zero exit status 25

I think the trace is only being spat out becuase I used loggign.exception rather than logging.error.

Thanks. I'll try using this next time. I wonder if the behaviour is correct for the next successful/expected successful patches-applied import after the orphan?

Relevant publishes:

Importing 2.13~rc3-2ubuntu2 to ubuntu/gutsy
Importing 2.13~rc3-5 to ubuntu/gutsy
01/05/2017 10:31:33 - ERROR:Unable to import patches-applied 2.13~rc3-5
Importing 2.13-3ubuntu1 to ubuntu/gutsy

With some fixes applied onto the previous diff, here's how the applied branch looks:

commit 9d1a7ce049feb2ff62e782ec02556c8999b36261 (tag: lpusip/applied/2.13-3ubunt
u1, lpusip/applied/ubuntu/gutsy)
Merge: de659b6 910da7e
Author: LaMont Jones <email address hidden>
Date: Thu Sep 6 14:38:14 2007 -0600

Where the parents are:

    Publish parent: de659b6722d7c06097b3fa20f648f92e8401c657
    Unapplied parent: 910da7ea18fd62a43fa723aad744c81fa687c1c2

commit de659b6722d7c06097b3fa20f648f92e8401c657 (tag: lpusip/applied/orphan/2.13
_rc3-5)

commit 910da7ea18fd62a43fa723aad744c81fa687c1c2 (tag: lpusip/import/2.13-3ubuntu1, lpusip/ubuntu/gutsy)

I think this is actually incorrect, though? Do we want three top-level namespaces in lpusip?

lpusip/applied
lpusip/import
lpusip/orphan/applied
lpusip/orphan/import

Or do we want two top-level namespaces:

lpusip/applied
lpusip/applied/orphan
lpusip/import
lpusip/import/orphan

?

And for unapplied, it's ok to use an orphaned commit as a parent, but for applied it's not, as the applied orphan may be an orphan for the reason that it does not correspond to the upload at all?

Put another way, perhaps it's incorrect overload 'orphan' to mean two things:

1) failed to find any parents for an import (regardless of type)

2) failed to derive a patches-applied import

The latter should really be a distinct namespace (or maybe would be what orphan under lpusip/applied means? I think that would still be confusing...), perhaps lpusip/failed or lpusip/applied/failed

I would need to adjust the algorithm for finding nearest parent to not look at anything tagged lpusip/applied/* but lpusip/applied/[^/]*, I think?

That makes me feel like we should perhaps have the following namespaces:

lpusip/applied/* -- all fully imported patches-applied versions
lpusip/import/* -- all fully imported patches-unapplied versions
lpusip/orphan/applied/* -- successfully imported patches-applied version, but no parents found
lpusip/orphan/import/* -- successfully imported patches-unapplied version, but no parents found
lpusip/failed/applied/* -- unsuccessfully patches-applied import
lpusip/failed/import/* -- unsuccessfully patches-unapplied import (empty set, by defintion?)

To find parents, we would look in lpusip/applied/* or lpusip/import/* as appropriate, but if none are found there, look in the orphans?

I would think we would never have any failed-tagged objects in any branch's history. At which point, given what they represent, I'm not entirely sure why we would even put them in the tree? They are derive-able frmo the patches-unapplied equivalent (orphan or otherwise).

For now, we are simply going to punt the problem and disable patches-applied imports in the automatic importer.