Launchpad itself

archive publisher needs to learn how to publish from a test-rebuild archive copy

Bug #520520 reported by LaMont Jones
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Launchpad itself
Fix Released
High
Julian Edwards
Launchpad itself 10.02

Bug Description

Now that we can create a copy archive and not have it show up, well... we need to put a publisher instance on another machine that specifically _just_ publishes exactly that copy archive, and does it in a way that doesn't confuse the mainline archive publisher.

This will allow us to publish the rebuilt archive for a second-round testing, so that we can find things like "when rebuilt from current source, gcc doesn't successfully build packages any more" or "when we rebuilt libc6, everything broke"

thanks,
lamont

Related branches

lp:~julian-edwards/launchpad/publish-copy-archives-bug-520520
Paul Hummer (community): Approve (code)
Diff: 289 lines (+71/-64)
2 files modified
lib/lp/soyuz/adapters/archivedependencies.py (+35/-28)
lib/lp/soyuz/doc/archive-dependencies.txt (+36/-36)
lp:~julian-edwards/launchpad/publish-copy-archives-bug-520520-copy-url
Paul Hummer (community): Approve (code)
Diff: 67 lines (+17/-4)
3 files modified
lib/canonical/config/schema-lazr.conf (+1/-0)
lib/lp/soyuz/doc/archive.txt (+11/-4)
lib/lp/soyuz/model/archive.py (+5/-0)
lp:~julian-edwards/launchpad/publish-copy-archives-bug-520520-getPubConfig
Guilherme Salgado (community): Approve (code)
Diff: 59 lines (+41/-0)
2 files modified
lib/lp/archivepublisher/config.py (+12/-0)
lib/lp/archivepublisher/tests/publisher-config.txt (+29/-0)
lp:~julian-edwards/launchpad/publish-copy-archives-bug-520520-getPubConfig-proc-accept
Graham Binns (community): Approve (code)
Diff: 438 lines (+234/-110)
4 files modified
lib/lp/soyuz/scripts/processaccepted.py (+112/-2)
lib/lp/soyuz/tests/test_processaccepted.py (+85/-0)
lib/lp/soyuz/tests/test_publishing.py (+29/-6)
scripts/process-accepted.py (+8/-102)
lp:~julian-edwards/launchpad/publish-copy-archives-bug-520520-publish-distro
Jeroen T. Vermeulen (community): Approve (code)
Diff: 142 lines (+72/-7)
2 files modified
lib/lp/soyuz/scripts/publishdistro.py (+20/-6)
lib/lp/soyuz/scripts/tests/test_publishdistro.py (+52/-1)
Rejected for merging into lp:launchpad/db-devel
Canonical Launchpad Engineering: Pending requested
Diff: 1553 lines (+343/-460)
33 files modified
lib/canonical/launchpad/doc/product-update-remote-product.txt (+1/-6)
lib/lp/bugs/doc/bugzilla-import.txt (+2/-6)
lib/lp/bugs/doc/externalbugtracker.txt (+3/-27)
lib/lp/code/browser/branch.py (+3/-3)
lib/lp/code/doc/xmlrpc-branch-puller.txt (+1/-16)
lib/lp/code/enums.py (+1/-1)
lib/lp/code/interfaces/branchpuller.py (+0/-6)
lib/lp/code/interfaces/codehosting.py (+0/-14)
lib/lp/code/model/branchpuller.py (+0/-22)
lib/lp/code/model/tests/test_branchjob.py (+5/-24)
lib/lp/code/model/tests/test_branchpuller.py (+0/-82)
lib/lp/code/model/tests/test_codeimport.py (+9/-0)
lib/lp/code/model/tests/test_codeimportjob.py (+2/-1)
lib/lp/code/xmlrpc/codehosting.py (+1/-43)
lib/lp/code/xmlrpc/tests/test_codehosting.py (+0/-100)
lib/lp/codehosting/codeimport/tests/test_worker.py (+25/-0)
lib/lp/codehosting/codeimport/worker.py (+15/-5)
lib/lp/codehosting/inmemory.py (+0/-23)
lib/lp/soyuz/scripts/publishdistro.py (+19/-6)
lib/lp/soyuz/scripts/tests/test_publishdistro.py (+52/-1)
lib/lp/testing/faketransaction.py (+35/-0)
lib/lp/translations/browser/poexportrequest.py (+35/-0)
lib/lp/translations/browser/tests/test_baseexportview.py (+68/-2)
lib/lp/translations/doc/distroseries-translations-copy.txt (+2/-7)
lib/lp/translations/doc/gettext-check-messages.txt (+14/-22)
lib/lp/translations/doc/poexport-queue.txt (+6/-4)
lib/lp/translations/doc/poexport-request-productseries.txt (+2/-5)
lib/lp/translations/doc/poexport-request.txt (+3/-6)
lib/lp/translations/doc/poimport.txt (+2/-10)
lib/lp/translations/interfaces/poexportrequest.py (+7/-1)
lib/lp/translations/model/poexportrequest.py (+23/-7)
lib/lp/translations/scripts/tests/test_copy_distroseries_translations.py (+2/-10)
lib/lp/translations/templates/translations-export.pt (+5/-0)
lp:~julian-edwards/launchpad/archive-path-bug-520520
Michael Nelson (community): Approve (code)
Diff: 61 lines (+12/-10)
3 files modified
lib/lp/archivepublisher/config.py (+2/-1)
lib/lp/archivepublisher/tests/publisher-config.txt (+8/-8)
lib/lp/soyuz/scripts/tests/test_publishdistro.py (+2/-1)
lp:~julian-edwards/launchpad/publish-copy-archives-bug-520520-cron-job
Michael Nelson (community): Approve (code)
Diff: 88 lines (+84/-0)
1 file modified
cronscripts/publishing/cron.publish-copy-archives (+84/-0)
lp:~julian-edwards/launchpad/enable-publish-ui-bug-520520
Michael Nelson (community): Approve (code)
Diff: 100 lines (+19/-8)
4 files modified
lib/lp/soyuz/browser/archive.py (+1/-1)
lib/lp/soyuz/configure.zcml (+1/-1)
lib/lp/soyuz/interfaces/archive.py (+4/-4)
lib/lp/soyuz/stories/ppa/xx-ppa-workflow.txt (+13/-2)
lp:~julian-edwards/launchpad/auto-accept-copy-archive-binaries-bug-520520
Abel Deuring (community): Approve (code)
Diff: 67 lines (+31/-2)
2 files modified
lib/lp/archiveuploader/nascentupload.py (+9/-2)
lib/lp/archiveuploader/tests/test_uploadprocessor.py (+22/-0)
Julian Edwards (julian-edwards)
Changed in soyuz:
status: New → Triaged
importance: Undecided → High
tags: added: soyuz-publish
Revision history for this message
Julian Edwards (julian-edwards) wrote :

Steps to fixing:

* IArchive.archive_url:
  needs to cope with ArchivePurpose.COPY

* lib/lp/soyuz/scripts/publishdistro.py
  New option --copy-archives
  copy archives still must use a-f, change the logic at the bottom of the file

* lp/archivepublisher/config.py - getPubConfig
  needs to cope with ArchivePurpose.COPY

If the publishing location needs to be different, then a new canonical.config block is required for copy archives.

Revision history for this message
William Grant (wgrant) wrote :

You also need to fix process-accepted.py.

Revision history for this message
Julian Edwards (julian-edwards) wrote :

I don't think you do - copy archives don't go through the accepted state in the queue.

Changed in soyuz:
milestone: none → 10.02
assignee: nobody → Julian Edwards (julian-edwards)
status: Triaged → In Progress
Revision history for this message
William Grant (wgrant) wrote :

realiseUpload is all that can cause a binary to move from ACCEPTED to DONE, and for non-source uploads that's only called by process-accepted.py. https://edge.launchpad.net/api/beta/ubuntu/+archive/test-rebuild-20090513/?ws.op=getPublishedBinaries suggests that there are no binary publishing records.

Revision history for this message
Julian Edwards (julian-edwards) wrote :

Right - I'd forgotten about the binaries.

Revision history for this message
Julian Edwards (julian-edwards) wrote :

We will build in 2 phases, each with different dependency requirements:

phase1: depends phase1, main_archive
phase2: depends phase2, phase1, main_archive

The +admin form on the copy archive page can then set up to include itself and other archives (phase1, phase2 in this case). archivedependencies.py/get_sources_list_for_building() needs altering so that it puts these dependencies in the right order.

This means we need to publish rebuilds in real-time, so will need a cron job on mcmurdo (where they're being published). We also need a new config item to complement base_url, e.g. copy_archive_base_url)

When fixing IArchive.archive_url, it should make sure the copy archive name is in the path so there can be more than one of them on the same machine.

Revision history for this message
Diogo Matsubara (matsubara) wrote : Bug fixed by a commit

Fixed in devel r10343 <http://bazaar.launchpad.net/~launchpad-pqm/launchpad/devel/revision/10343>

Changed in soyuz:
status: In Progress → Fix Committed
Julian Edwards (julian-edwards)
Changed in soyuz:
status: Fix Committed → In Progress
Revision history for this message
Diogo Matsubara (matsubara) wrote :

Fixed in devel r10347 <http://bazaar.launchpad.net/~launchpad-pqm/launchpad/devel/revision/10347>

Changed in soyuz:
status: In Progress → Fix Committed
Julian Edwards (julian-edwards)
Changed in soyuz:
status: Fix Committed → In Progress
Revision history for this message
Julian Edwards (julian-edwards) wrote :

We also need to add a UI change to enable someone to change the "publish" flag on the archive. We can put this on the +edit form.

Revision history for this message
Diogo Matsubara (matsubara) wrote :

Fixed in devel r10395 <http://bazaar.launchpad.net/~launchpad-pqm/launchpad/devel/revision/10395>

Changed in soyuz:
status: In Progress → Fix Committed
Revision history for this message
Julian Edwards (julian-edwards) wrote :

The binaries that get uploaded go right into the NEW queue - they need to be auto-accepted for COPY archives. More changes to the upload processor needed.

Julian Edwards (julian-edwards)
Changed in soyuz:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.