Copied a package twice in a PPA

This indeed resulted in publishing for PPAs to screech to a halt. After deleting canonical-oem-keyring from the destination archive, canonical-oem-keyring was once again published to the archive and normal publishing for all PPAs resumed. I'll update the incident report with the technical details.

2010-04-06 16:36:45 ERROR Failure processing queue_item 1745827
 -> http://launchpadlibrarian.net/43260391/kBVUA6vVrJcxC4CvkBSskBMreZ7.txt (canonical-oem-keyring_2009.07.23+build2.tar.gz is
already published in archive for lucid)
2010-04-06 16:36:45 DEBUG Rolling back any remaining transactions.
2010-04-06 16:36:49 DEBUG Removing lock file: /var/lock/launchpad-upload-queue.lock
Traceback (most recent call last):
  File "/srv/launchpad.net/codelines/current/scripts/process-accepted.py", line 26, in <module>
    script.lock_and_run(isolation=ISOLATION_LEVEL_READ_COMMITTED)
  File "/srv/launchpad.net/codelines/ppa-rev-9186/lib/lp/services/scripts/base.py", line 290, in lock_and_run
    implicit_begin=implicit_begin, isolation=isolation)
  File "/srv/launchpad.net/codelines/ppa-rev-9186/lib/lp/services/scripts/base.py", line 248, in run
    self.main()
  File "/srv/launchpad.net/codelines/ppa-rev-9186/lib/lp/soyuz/scripts/processaccepted.py", line 246, in main
    queue_item.realiseUpload(self.logger)
  File "/srv/launchpad.net/codelines/ppa-rev-9186/lib/lp/soyuz/model/queue.py", line 588, in realiseUpload
    queue_source.verifyBeforePublish()
  File "/srv/launchpad.net/codelines/ppa-rev-9186/lib/lp/soyuz/model/queue.py", line 1553, in verifyBeforePublish
    filename, self.packageupload.distroseries.name))
lp.soyuz.interfaces.queue.QueueInconsistentStateError: canonical-oem-keyring_2009.07.23+build2.tar.gz is already published in archive for lucid

Just some background - during the check of the second copy, it looks like _checkArchiveConflicts() is ignoring the conflicting source due to the following code: (lib/lp/soyuz/scripts/packagecopier.py:281)

            # If the conflicting candidate (which we already know refer to
            # the same sourcepackagerelease) was found in the copy
            # destination series we don't have to check its building status
            # if binaries are included. It's not going to change in terms of
            # new builds and the resulting binaries will match. See more
            # details in `ISourcePackageRelease.getBuildsByArch`.
            if (candidate.distroseries.id == series.id and
                self.archive.id == source.archive.id and
                self.include_binaries):
                continue

So the (second) copy with binaries succeeds but *doesn't* create a new source publishing record (afaics locally, and from _do_direct_copy).

Jelmer will continue looking into why this lead to the already published tar.gz during process accepted.

I'm about to submit a patch that makes process-accepted catch the exception and file an OOPS but still continue with the other items in the queue, rather than aborting completely.

This almost certainly happened when hitting the "copy" button twice in quick succession on the form. We don't deal with that at all and it's quite hard to fix.

For the record, I did re-create this with a double tap on the "copy" button. It will halt the publishing pipeline when it happens so should probably be marked critical priority. We're just lucky avoiding it so far.

The reason it happens is because there's no database constraint to prevent this, just some myriad checks in the CopyChecker. If both requests cross paths in separate appserver instances the checks cannot prevent the second one getting in.

William and I discussed a hack where you force a change to an unrelated table somewhere, which would cause PG to reject the second attempt. Adding a real DB constraint would be extremely hard.

This could only happen with delayed copies. RIP.