Ubuntu publisher is taking more than an hour to complete
Bug #884649 reported by
Julian Edwards
This bug affects 1 person
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Launchpad itself |
Fix Released
|
Critical
|
Jeroen T. Vermeulen |
Bug Description
Due to the recent domination changes to allow arch-all binaries to persist until their counterpart arch-any get dominated, the domination part of the publisher is now taking 30 minutes or more which pushes the whole run over 1 hour.
Related branches
lp:~jtv/launchpad/bug-884649-branch-1
- Julian Edwards (community): Approve
-
Diff: 502 lines (+276/-89)3 files modifiedlib/lp/archivepublisher/domination.py (+104/-75)
lib/lp/archivepublisher/tests/test_dominator.py (+170/-2)
lib/lp/soyuz/model/publishing.py (+2/-12)
lp:~jtv/launchpad/getBinariesForDomination-bulk
- Raphaël Badin (community): Approve
-
Diff: 21 lines (+10/-1)1 file modifiedlib/lp/archivepublisher/domination.py (+10/-1)
lp:~jtv/launchpad/bug-884649-branch-2
- Julian Edwards (community): Approve
- j.c.sackett (community): Approve
-
Diff: 676 lines (+325/-121)2 files modifiedlib/lp/archivepublisher/domination.py (+183/-93)
lib/lp/archivepublisher/tests/test_dominator.py (+142/-28)
lp:~jtv/launchpad/bug-884649-branch-3
- Julian Edwards (community): Approve
-
Diff: 110 lines (+57/-9)2 files modifiedlib/lp/archivepublisher/domination.py (+31/-9)
lib/lp/archivepublisher/tests/test_dominator.py (+26/-0)
lp:~jtv/launchpad/bug-884649-branch-4
- Brad Crittenden (community): Approve (code)
- Diff: 0 lines
lp:~jtv/launchpad/bug-884649-branch-5
- Julian Edwards (community): Approve
-
Diff: 292 lines (+67/-91)2 files modifiedlib/lp/archivepublisher/domination.py (+67/-78)
lib/lp/archivepublisher/tests/test_dominator.py (+0/-13)
Changed in launchpad: | |
status: | New → Triaged |
importance: | Undecided → Critical |
tags: | added: lp-soyuz soyuz-publish |
Changed in launchpad: | |
status: | Triaged → In Progress |
assignee: | nobody → Jeroen T. Vermeulen (jtv) |
tags: |
added: qa-ok removed: qa-needstesting |
To post a comment you must log in.
See lib/lp/ archepublisher/ domination. py for background, and especially the comment in dominateBinaries.
Some notes about our planned approach follow, from IRC (times in UTC):
(11:54:23) jtv:
But the big change I'm hoping for is that we might find that the first binary domination pass can simply keep all arch-all pubs alive.
(11:54:55) bigjools:
That's how I originally did it.
(11:55:47) jtv:
Doesn't work?
(11:56:43) bigjools:
The two corner cases mentioned in the comment ...
The arch-all pubs were never getting dominated.
(12:03:59) jtv:
Looking at the comment, I see why the second pass is needed. Accepting that we need the second pass anyway, there's nothing per se against further increasing the need for it. Which means: legroom for optimization!
About the "keep arch-all bpphs alive" thing: are you saying you tried that with, or without the double domination?
(12:09:18) bigjools:
Without.
Well — 2 sets of queries
rather than a full domination run
so it did arch-any first then arch-all
but that has the schizo problem.
(12:15:30) jtv:
Quite.
(12:15:53) jtv:
So that means that _with_ double domination, this might just work.
Dominate twice, but on the first pass, don't supersede arch-all at all. ons-from- same-source.
(So consider them, but keep them live)
I wonder if we could do the first pass for all architectures, before doing the second pass.
Because that way, we get to supersede all non-live arch-specific pubs before we even start looking at the other-publicati
The second pass could group by SPRs. There'd be only one getOtherPublica tionsForSameSou rce for each.
(12:20:44) bigjools:
Isn't this what we do right now?
(12:20:59) jtv:
Slightly different loop nesting.
Right now we loop over DASes, and for each, do 2 domination passes.
(12:21:15) bigjools:
Actually one way to speed it is to get the list of sources that had arch-all binaries that we left live.
(12:21:23) jtv:
Indeed.
Should be very easy to collect that information in the per-package domination loop.
That's a nice touch.
In fact, it eliminates a whole lot of work that I thought we were going to need!
I guess for the second pass, the algorithm could be something like:
Keep the latest version alive. Keep the remaining arch-specific versions alive. Keep any arch-indep versions alive if there are still arch-specific pubs for their SPR.
“What's the difference with what we do now?” I hear you ask.
(12:26:32) bigjools:
:)
(12:26:57) jtv:
The difference is that there's no need to dominate arch-specific BPPHs at all in that pass.
(12:27:06) bigjools:
Indeed.
(12:27:48) jtv:
AFAICS we can pass just the arch-specific BPPHs to the inner domination method.
Och aye, no we can't quite do that.
Because if the latest version is arch-specific, we'd end up deleting the last arch-all version instead of superseding it.
(Unless we allow the caller to "pre-seed" the dominant)
Anyway, that's going into needless detail.
First I'll apply some loop fission to separate the two passes into separate DAS loops.
Then I add code to collect the SPRs-with- live-arch- indep-BPPHs.
Then I rearrange the second-pass loop to iterate over those, with an inner loop that iterates DAS.
(12:33:15)...