livefs builds fail without logs (possibly due to multiple large artifacts)
| Affects | Status | Importance | Assigned to | Milestone | |
|---|---|---|---|---|---|
| launchpad-buildd |
Fix Released
|
High
|
Colin Watson | ||
Bug Description
CPC is seeing livefs build failures without any artifacts or logs to indicate what the problem is. Examples:
* https:/
* https:/
One of these was believed to be just a network glitch at the wrong time, but we've seen this in other builds as well (will try to get a better list).
These livefs builds create a number of cloud images and filesystems as artifacts which total multiple GBs.
The problem we are currently seeing only occurs on cosmic builds. The suspicion is that the recent addition of the lxd snap to all images has increased image size across the entire set and pushed the time it takes to copy artifacts past a threshold that causes them to get terminated.
Related branches
- William Grant: Approve (code)
-
Diff: 930 lines (+161/-75)12 files modifieddebian/changelog (+2/-0)
lpbuildd/binarypackage.py (+2/-9)
lpbuildd/debian.py (+37/-2)
lpbuildd/livefs.py (+2/-2)
lpbuildd/snap.py (+2/-2)
lpbuildd/sourcepackagerecipe.py (+4/-2)
lpbuildd/tests/test_binarypackage.py (+52/-28)
lpbuildd/tests/test_livefs.py (+14/-6)
lpbuildd/tests/test_snap.py (+12/-8)
lpbuildd/tests/test_sourcepackagerecipe.py (+19/-9)
lpbuildd/tests/test_translationtemplatesbuildmanager.py (+13/-5)
lpbuildd/translationtemplates.py (+2/-2)
| Colin Watson (cjwatson) wrote | #1 |
| affects: | launchpad → launchpad-buildd |
| Changed in launchpad-buildd: | |
| importance: | Undecided → High |
| status: | New → Triaged |
| Changed in launchpad-buildd: | |
| status: | Triaged → Fix Committed |
| assignee: | nobody → Colin Watson (cjwatson) |
| Changed in launchpad-buildd: | |
| status: | Fix Committed → Fix Released |
We think this should probably be fixed by deferring gatherResults calls to a thread rather than running them in the reactor thread (which relies on it being possible to gather all the files from the backend before buildd-manager gets bored of the lack of status responses and times out).