MagicProxy broken when repo-snapshot-stamp is used

Bug #1944906 reported by Thomas Bechtold
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
livecd-rootfs (Ubuntu)
Fix Released
Undecided
Thomas Bechtold
Focal
Fix Released
Undecided
Unassigned
Hirsute
Fix Released
Undecided
Unassigned

Bug Description

The CPC team wants to reenable the repo-snapshot-stamp feature in magic proxy to get similar package sets when building multiple images. But that is currently broken.
When building in launchpad with something like:

RUN: /usr/share/launchpad-buildd/bin/in-target buildlivefs --backend=lxd --series=impish --arch=arm64 LIVEFSBUILD-299827 --project ubuntu-cpc --datestamp 20210924 --image-format ext4 --image-target disk-image --image-target kvm --image-target qcow2 --image-target squashfs --image-target tarball --image-target vagrant --image-target vmdk --image-target azure --image-target azure-pro --image-target hyperv-gcs --image-target wsl --repo-snapshot-stamp 1632467595 --cohort-key MSAqIDE2MzI0Njc4OTcgMTVjNTk1ZWQ1NjFlMjM5OWEzNmQ5ZTE0MDlmZjlmMDdiOGNhM2JlMGI0N2UyMzBjZGVjMGUzOWEyNjk0NTA1Mw==

the build fails with:

REDIRECT tcp opt -- in * out * 0.0.0.0/0 -> 0.0.0.0/0 tcp dpt:80 ! owner UID match 1 redir ports 8080
add password for /cloudware/cpc-livecd-rootfs/ubuntu
Ign:1 http://ftpmaster.internal/ubuntu impish InRelease
Ign:2 http://ftpmaster.internal/ubuntu impish-security InRelease
Ign:3 http://ftpmaster.internal/ubuntu impish-updates InRelease
Ign:4 http://private-ppa.buildd/cloudware/cpc-livecd-rootfs/ubuntu impish InRelease
Ign:1 http://ftpmaster.internal/ubuntu impish InRelease
Ign:2 http://ftpmaster.internal/ubuntu impish-security InRelease
Ign:3 http://ftpmaster.internal/ubuntu impish-updates InRelease
Ign:4 http://private-ppa.buildd/cloudware/cpc-livecd-rootfs/ubuntu impish InRelease
Ign:1 http://ftpmaster.internal/ubuntu impish InRelease
Ign:2 http://ftpmaster.internal/ubuntu impish-security InRelease
Ign:3 http://ftpmaster.internal/ubuntu impish-updates InRelease
Ign:4 http://private-ppa.buildd/cloudware/cpc-livecd-rootfs/ubuntu impish InRelease
Err:1 http://ftpmaster.internal/ubuntu impish InRelease
  Connection failed [IP: 10.189.129.95 80]
Err:2 http://ftpmaster.internal/ubuntu impish-security InRelease
  Connection failed [IP: 10.189.129.95 80]
Err:3 http://ftpmaster.internal/ubuntu impish-updates InRelease
  Connection failed [IP: 10.189.129.95 80]
Err:4 http://private-ppa.buildd/cloudware/cpc-livecd-rootfs/ubuntu impish InRelease
  Connection failed [IP: 91.189.95.85 80]
Reading package lists...
W: Failed to fetch http://private-ppa.buildd/cloudware/cpc-livecd-rootfs/ubuntu/dists/impish/InRelease Connection failed [IP: 91.189.95.85 80]
W: Failed to fetch http://ftpmaster.internal/ubuntu/dists/impish/InRelease Connection failed [IP: 10.189.129.95 80]
W: Failed to fetch http://ftpmaster.internal/ubuntu/dists/impish-security/InRelease Connection failed [IP: 10.189.129.95 80]
W: Failed to fetch http://ftpmaster.internal/ubuntu/dists/impish-updates/InRelease Connection failed [IP: 10.189.129.95 80]
W: Some index files failed to download. They have been ignored, or old ones used instead.
P: Setting up cleanup function
[2021-09-24 07:23:31] lb_bootstrap
P: Setting up cleanup function
[2021-09-24 07:23:31] lb_bootstrap_cache restore
P: Restoring bootstrap stage from cache...
[2021-09-24 07:23:31] lb_bootstrap_copy
[2021-09-24 07:23:32] lb_bootstrap_cdebootstrap
[2021-09-24 07:23:32] lb_bootstrap_debootstrap
P: Begin bootstrapping system...
[2021-09-24 07:23:32] lb_testroot
P: If the following stage fails, the most likely cause of the problem is with your mirror configuration or a caching proxy.
P: Running debootstrap (download-only)...
I: Retrieving InRelease
I: Retrieving Release
E: Failed getting release file http://ftpmaster.internal/ubuntu/dists/impish/Release
P: Begin unmounting filesystems...
P: Saving caches...
================= Magic proxy log (start) =================
----------------------------------------
Exception occurred during processing of request from ('10.10.10.2', 39554)
Traceback (most recent call last):
  File "/usr/lib/python3.9/socketserver.py", line 683, in process_request_thread
    self.finish_request(request, client_address)
  File "/usr/lib/python3.9/socketserver.py", line 360, in finish_request
    self.RequestHandlerClass(request, client_address, self)
  File "/usr/lib/python3.9/socketserver.py", line 747, in __init__
    self.handle()
  File "/usr/lib/python3.9/http/server.py", line 427, in handle
    self.handle_one_request()
  File "/usr/lib/python3.9/http/server.py", line 415, in handle_one_request
    method()
  File "/usr/share/livecd-rootfs/magic-proxy", line 787, in do_GET
    self.__get_request()
  File "/usr/share/livecd-rootfs/magic-proxy", line 829, in __get_request
    mirror = get_uri(parsed.hostname, m.group("base"))
  File "/usr/share/livecd-rootfs/magic-proxy", line 84, in get_uri
    return "http://" + host + path
TypeError: can only concatenate str (not "NoneType") to str

See attached build log (from https://i559944962.restricted.launchpadlibrarian.net/559944962/buildlog_ubuntu_impish_arm64_arm64-disk-image_kvm_qcow2_squashfs_tarball_vagrant_vmdk_azure_azure-pro_hyperv-gcs_wsl_cpc-development_BUILDING.txt.gz?token=8pdlp0RtcWhsl3N1tdd0GhtdhDwWt4hd )

Revision history for this message
Thomas Bechtold (toabctl) wrote :
Changed in livecd-rootfs (Ubuntu):
assignee: nobody → Thomas Bechtold (toabctl)
status: New → In Progress
Changed in livecd-rootfs (Ubuntu):
status: In Progress → Fix Released
Revision history for this message
Robie Basak (racb) wrote :

The livecd-rootfs SRU for Bionic, Focal and Hirsute is currently blocked by another SRU in progress.

Revision history for this message
Brian Murray (brian-murray) wrote : Please test proposed package

Hello Thomas, or anyone else affected,

Accepted livecd-rootfs into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/livecd-rootfs/2.664.32 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in livecd-rootfs (Ubuntu Focal):
status: New → Fix Committed
tags: added: verification-needed verification-needed-focal
Revision history for this message
Ubuntu SRU Bot (ubuntu-sru-bot) wrote : Autopkgtest regression report (livecd-rootfs/2.664.32)

All autopkgtests for the newly accepted livecd-rootfs (2.664.32) for focal have finished running.
The following regressions have been reported in tests triggered by the package:

ubuntu-image/1.11+20.04ubuntu1 (amd64)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/focal/update_excuses.html#livecd-rootfs

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

Revision history for this message
Brian Murray (brian-murray) wrote : Please test proposed package

Hello Thomas, or anyone else affected,

Accepted livecd-rootfs into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/livecd-rootfs/2.664.33 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Revision history for this message
Ubuntu SRU Bot (ubuntu-sru-bot) wrote : Autopkgtest regression report (livecd-rootfs/2.664.33)

All autopkgtests for the newly accepted livecd-rootfs (2.664.33) for focal have finished running.
The following regressions have been reported in tests triggered by the package:

ubuntu-image/1.11+20.04ubuntu1 (armhf)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/focal/update_excuses.html#livecd-rootfs

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

Revision history for this message
Thomas Bechtold (toabctl) wrote :

I did a testbuild with repo-stamp enabled against the livecd-rootfs version from proposed (2.664.33). See
- https://launchpadlibrarian.net/566970666/buildlog_ubuntu_focal_amd64_amd64-tarball_proposed_BUILDING.txt.gz
- https://launchpad.net/~toabctl/+livefs/ubuntu/focal/proposed/+build/307101

That looks good.

tags: added: verification-done-focal
removed: verification-needed-focal
Revision history for this message
Steve Langasek (vorlon) wrote : Please test proposed package

Hello Thomas, or anyone else affected,

Accepted livecd-rootfs into hirsute-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/livecd-rootfs/2.719.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-hirsute to verification-done-hirsute. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-hirsute. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in livecd-rootfs (Ubuntu Hirsute):
status: New → Fix Committed
tags: added: verification-needed-hirsute
Revision history for this message
Łukasz Zemczak (sil2100) wrote : Update Released

The verification of the Stable Release Update for livecd-rootfs has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package livecd-rootfs - 2.664.33

---------------
livecd-rootfs (2.664.33) focal; urgency=medium

  * Install cloud-initramfs-growroot to actually enable rootfs resize.
  * Fix a grub error by making sure the unicode.pf2 font is installed in the
    right path for preinstalled amd64 desktop images.

livecd-rootfs (2.664.32) focal; urgency=medium

  * 099-ubuntu-image-customization.chroot: fix a typo in it.

livecd-rootfs (2.664.31) focal; urgency=medium

  [ Łukasz 'sil2100' Zemczak ]
  * Add the 099-ubuntu-image-customization.chroot for
    desktop-preinstalled images similar to what we have in groovy+ (for the pi
    desktop), but improved for amd64 platforms. We need it to generate a valid
    grub.cfg on the rootfs (similar to ubuntu-cpc) and then use that instead
    of a static configuration locked on the boot partition (LP: #1949102).

  [ Brian Murray ]
  * Properly check ARCH when setting the intel-iot model.

livecd-rootfs (2.664.30) focal; urgency=medium

  [ Thomas Bechtold ]
  * magic-proxy: Replace http.client with urllib calls. live-build/auto/build:
    change iptables calls to query rules and quickly check that connectivity
    works after transparent proxy has been installed. (LP: #1917920)
  * magic-proxy: fix TypeError when trying to call get_uri() (LP: #1944906)

 -- Łukasz 'sil2100' Zemczak <email address hidden> Fri, 29 Oct 2021 15:33:34 +0200

Changed in livecd-rootfs (Ubuntu Focal):
status: Fix Committed → Fix Released
Revision history for this message
Thomas Bechtold (toabctl) wrote :

SRU verification for hirsute:

I did a testbuild with repo-stamp enabled against the livecd-rootfs version from proposed (2.719.3). See
- https://launchpad.net/~toabctl/+livefs/ubuntu/hirsute/proposed/+build/308173
- https://launchpadlibrarian.net/567803112/buildlog_ubuntu_hirsute_amd64_amd64-tarball_proposed_BUILDING.txt.gz

That looks good.

tags: added: verification-done-hirsute
removed: verification-needed-hirsute
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package livecd-rootfs - 2.719.3

---------------
livecd-rootfs (2.719.3) hirsute; urgency=medium

  [ Thomas Bechtold ]
  * magic-proxy: Replace http.client with urllib calls. live-build/auto/build:
    change iptables calls to query rules and quickly check that connectivity
    works after transparent proxy has been installed. (LP: #1917920)
  * magic-proxy: fix TypeError when trying to call get_uri() (LP: #1944906)

 -- Brian Murray <email address hidden> Thu, 21 Oct 2021 11:17:58 -0700

Changed in livecd-rootfs (Ubuntu Hirsute):
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.