preseeded snap installs fail in images

Bug #1864252 reported by Robert C Jennings
60
This bug affects 10 people
Affects Status Importance Assigned to Milestone
livecd-rootfs (Ubuntu)
Fix Released
High
Robert C Jennings
Bionic
Fix Released
High
Unassigned
Eoan
Fix Released
High
Unassigned
Focal
Fix Released
High
Robert C Jennings

Bug Description

[Impact]

Images built with pre-seeded snaps contain insufficient assertion data causing boot to fail.

The snaps for seeding are downloaded with a custom snap tool for an earlier cohort API (now deprecated). The assertions that it pulls are incomplete. We could update that list and move to the new API but at this time the snap-tool provides no value compared to use of the snap CLI (cohort support has moved to the cli as well). The development overhead of maintaining snap-tool in livecd-rootfs are not warranted.

This patch removes the bespoke snap-tool and relies on the snap CLI instead.

[Test Case]

 * Produce images that include preseeded snaps (in /var/lib/snapd/seed/*)

 * Produce an image which does not include snaps by default from the ubuntu package seed (ubuntu-cpc:minimized does this) but add a snap in a binary hook. Ensure the base snap for the specified snap is installed.

 * Boot the resulting image and ensure that the snapd.seeded unit is successful and the snaps (from the correct channels) show up in 'snap list'

[Regression Potential]

 * The interface for these two tools is consistent and the output should be the same. There's always a chance that snap-tool had quirks which a move to the snap CLI uncovers, where the result would be different snaps seeded from before the change. An example would be channel differences before and after this change. I haven't seen issues in my testing and I do think it's unlikely, mostly I'm suspicious of SRUs that don't list any regression potentials.

[Other Info]

* The snap-tool that is being removed is used only in livecd-rootfs through live-build/functions. The replacement with the upstream snap cli is equivalent the 'info' and 'download' commands that are used. The snap-tool code is using deprecated APIs which do not provide the data needed for pre-seeding.
* The attached MP for Xenial is simpler as it never had snap-tool. The Xenial MP is a minor change to give us parity between releases for use of the snap cohort key during download.

Related branches

Robert C Jennings (rcj)
description: updated
Changed in livecd-rootfs (Ubuntu):
assignee: nobody → Robert C Jennings (rcj)
Robert C Jennings (rcj)
description: updated
Revision history for this message
Robert C Jennings (rcj) wrote :

All of the MPs have been tested per the plan in the description. For eoan (where we have lxd seeded in the base image) the ubuntu-cpc project was built and the qcow2 image booted in kvm to verify the snaps. For bionic and xenial (where no snaps are seeded in the image set) I tested with the ubuntu-cpc project with a binary hook that seeded lxd.

Revision history for this message
Robert C Jennings (rcj) wrote :

The merged code in focal has been tested in production and resolved this issue. We have images failing with this issue at least in eoan so a short SRU cycle is desired.

Revision history for this message
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in livecd-rootfs (Ubuntu):
status: New → Confirmed
Revision history for this message
Iain Lane (laney) wrote :

After the dropping of snap-tool, Ubuntu desktop ISOs are failing to build:

error: cannot validate seed:
- cannot use snap "gnome-3-28-1804": base "core18" is missing
- cannot use snap "gnome-calculator": base "core18" is missing
- cannot use snap "gnome-characters": base "core18" is missing
- cannot use snap "gnome-logs": base "core18" is missing
- cannot use snap "gtk-common-themes": base "core18" is missing

https://launchpadlibrarian.net/466333748/buildlog_ubuntu_focal_amd64_ubuntu_BUILDING.txt.gz

Ah, OK:

$ ./snap-tool info gnome-logs | egrep '^base:'
$ base: core18
$ snap info gnome-logs | egrep '^base:'
$

but there is:

$ snap info --verbose gnome-logs | egrep '^base:'
$ base: core18

so I think we just need to add --verbose.

Revision history for this message
Iain Lane (laney) wrote :

> so I think we just need to add --verbose.

(I've done that in focal now. Will trigger a rebuild if & when this migrates out.)

Revision history for this message
Robert C Jennings (rcj) wrote :

Thank laney, I've fixed up the SRUs

Robert C Jennings (rcj)
description: updated
Changed in livecd-rootfs (Ubuntu Xenial):
status: New → In Progress
assignee: nobody → Brian Murray (brian-murray)
Changed in livecd-rootfs (Ubuntu Bionic):
status: New → Confirmed
Changed in livecd-rootfs (Ubuntu Eoan):
status: New → Confirmed
Mathew Hodson (mhodson)
Changed in livecd-rootfs (Ubuntu):
importance: Undecided → High
Changed in livecd-rootfs (Ubuntu Xenial):
importance: Undecided → High
Changed in livecd-rootfs (Ubuntu Bionic):
importance: Undecided → High
Changed in livecd-rootfs (Ubuntu Eoan):
importance: Undecided → High
Revision history for this message
corrado venturini (corradoventu) wrote :

On my Ubuntu 20.04 installed from Ubuntu 20.04 LTS "Focal Fossa" - Alpha amd64 (20200220)
doing update+upgrade the upgrade process hang with message
snapd.failure.service is a disabled or a static unit, not starting it.
snapd.snap-repair.service is a disabled or a static unit, not starting it.

Revision history for this message
Steve Langasek (vorlon) wrote : Re: [Bug 1864252] Re: preseeded snap installs fail in images

On Wed, Feb 26, 2020 at 06:20:29PM -0000, corrado venturini wrote:
> On my Ubuntu 20.04 installed from Ubuntu 20.04 LTS "Focal Fossa" - Alpha amd64 (20200220)
> doing update+upgrade the upgrade process hang with message
> snapd.failure.service is a disabled or a static unit, not starting it.
> snapd.snap-repair.service is a disabled or a static unit, not starting it.

That does not sound like it's related to this bug, but if it is, the remedy
would be for you to reinstall from a later image that has the fixed
preseeding behavior, or to manually fix the state of your snap installs.

Robert C Jennings (rcj)
Changed in livecd-rootfs (Ubuntu):
status: Confirmed → Fix Released
Revision history for this message
Steve Langasek (vorlon) wrote : Proposed package upload rejected

An upload of livecd-rootfs to eoan-proposed has been rejected from the upload queue for the following reason: "needs another bugfix".

Revision history for this message
Brian Murray (brian-murray) wrote : Please test proposed package

Hello Robert, or anyone else affected,

Accepted livecd-rootfs into eoan-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/livecd-rootfs/2.620.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-eoan to verification-done-eoan. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-eoan. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in livecd-rootfs (Ubuntu Eoan):
status: Confirmed → Fix Committed
tags: added: verification-needed verification-needed-eoan
Changed in livecd-rootfs (Ubuntu Bionic):
status: Confirmed → Fix Committed
tags: added: verification-needed-bionic
Revision history for this message
Brian Murray (brian-murray) wrote :

Hello Robert, or anyone else affected,

Accepted livecd-rootfs into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/livecd-rootfs/2.525.41 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Robert C Jennings (rcj)
description: updated
Revision history for this message
Robert C Jennings (rcj) wrote :

Brian, I've performed SRU testing per the test plan for both eoan and bionic and I've marked them verification-done. Can we get this promoted to -updates now?

tags: added: verification-done verification-done-bionic verification-done-eoan
removed: verification-needed verification-needed-bionic verification-needed-eoan
Revision history for this message
Robert C Jennings (rcj) wrote :

Okay, that verification comment was insufficient, here are the full details.

I tested 2.620.1 from eoan-proposed and 2.525.41 from bionic-proposed.

For each suite I produce images that include preseeded snaps (in /var/lib/snapd/seed/*) using the ubuntu-cpc project (unminimized) and images that did not include snaps by default using the ubuntu-cpc project with the minimized subproject but adding a snap in a binary hook (and not specifying a core snap beforehand).

Review of the build logs showed that the snaps were downloaded and when 'core' was not present in the image previously the core snap would be downloaded as a dependency.

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package livecd-rootfs - 2.620.1

---------------
livecd-rootfs (2.620.1) eoan; urgency=medium

  * Use snap cli rather than custom snap-tool (LP: #1864252)

 -- Robert C Jennings <email address hidden> Tue, 25 Feb 2020 16:15:48 -0600

Changed in livecd-rootfs (Ubuntu Eoan):
status: Fix Committed → Fix Released
Revision history for this message
Brian Murray (brian-murray) wrote : Update Released

The verification of the Stable Release Update for livecd-rootfs has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package livecd-rootfs - 2.525.41

---------------
livecd-rootfs (2.525.41) bionic; urgency=medium

  * Use snap cli rather than custom snap-tool (LP: #1864252)
  * Address snap base regression after snap-tool removal

 -- Robert C Jennings <email address hidden> Fri, 28 Feb 2020 08:29:57 -0600

Changed in livecd-rootfs (Ubuntu Bionic):
status: Fix Committed → Fix Released
Changed in livecd-rootfs (Ubuntu Xenial):
status: In Progress → Invalid
assignee: Brian Murray (brian-murray) → nobody
Revision history for this message
Paride Legovini (paride) wrote :

I have to reopen a Focal task for this bug as the problem is happening again with the live-server ISOs with serial 20200309: the snapd seeding never completes (see screenshot). The latest Bionic image boots fine. The Focal live-server images with serial 20200306 were working fine too. This affects multiple architectures.

Changed in livecd-rootfs (Ubuntu Focal):
status: Fix Released → New
Revision history for this message
Paride Legovini (paride) wrote :

Steps to reproduce:

1. Download the latest focal-live-server-amd64.iso
2. qemu-img create -f raw disk.img 8G
3. kvm -m 2048 -boot d -cdrom focal-live-server-amd64.iso -drive file=disk.img,if=virtio

Revision history for this message
Michael Hudson-Doyle (mwhudson) wrote :

That was my fault, should be better now though.

Changed in livecd-rootfs (Ubuntu Focal):
status: New → Fix Released
Revision history for this message
Jean-Baptiste Lallement (jibel) wrote :

I'm still having this issue with Ubuntu Desktop 20200312. Is this really fixed?

Revision history for this message
Michael Hudson-Doyle (mwhudson) wrote :

Snap seeding completed for me using 20200312 when I was testing https://bugs.launchpad.net/ubuntu/+source/casper/+bug/1867065 so if someone is going to claim it's not we need logs and evidence I think.

Revision history for this message
Jean-Baptiste Lallement (jibel) wrote :

it is not a question of claiming or not claiming, it's a fact.

$ cat /var/log/installer/media-info
Ubuntu 20.04 LTS "Focal Fossa" - Alpha amd64 (20200312)

$ systemd-analyze
Bootup is not yet finished (org.freedesktop.systemd1.Manager.FinishTimestampMonotonic=0).

$ systemctl list-jobs
JOB UNIT TYPE STATE
147 snapd.seeded.service start running
170 zsys-commit.service start waiting
  1 graphical.target start waiting
  2 multi-user.target start waiting
136 systemd-update-utmp-runlevel.service start waiting
109 snapd.autoimport.service start waiting

6 jobs listed.

$ sudo snap install firefox
[sudo] password for u:
error: too early for operation, device not yet seeded or device model not acknowledged

from the journal:
mars 12 14:31:57 u-Inspiron-13-5378 systemd[1]: Starting Snappy daemon...
mars 12 14:31:58 u-Inspiron-13-5378 snapd[1637]: AppArmor status: apparmor is enabled and all features are available
mars 12 14:31:58 u-Inspiron-13-5378 snapd[1637]: daemon.go:343: started snapd/2.44~pre1+20.04 (series 16; classic) ubuntu/20.04 (amd64) linux/5.4.0-14-generic.
mars 12 14:31:58 u-Inspiron-13-5378 snapd[1637]: daemon.go:436: adjusting startup timeout by 30s (pessimistic estimate of 30s plus 5s per snap)
mars 12 14:31:58 u-Inspiron-13-5378 snapd[1637]: helpers.go:105: error trying to compare the snap system key: system-key missing on disk
mars 12 14:31:58 u-Inspiron-13-5378 systemd[1]: Started Snappy daemon.
mars 12 14:32:02 u-Inspiron-13-5378 snapd[1637]: stateengine.go:150: state ensure error: devicemgr: cannot find signatures with metadata for snap "gnome-3-34-1804" ("/var/lib/snapd/seed/snaps/gnome-3-34-1804_12.snap")

What other log do you need?

tags: added: id-5e5051a4e00bad81f435391c
tags: added: id-5e4efeb8f00ff60ed69d009d
Revision history for this message
Michael Hudson-Doyle (mwhudson) wrote :

Sorry I phrased that badly :( But I don't think your issue is caused by the problem that this bug was about. I'll try to reproduce again but it might be time to file a new bug. Can you include the output of "snap changes" and "snap tasks $id" where $id is the id of any change that is not Done?

Revision history for this message
Michael Hudson-Doyle (mwhudson) wrote :

I tried with 20200315 and snap seeding completed. It did take 57s though per systemd-analyze blame, and as with that other bug I was booting off pretty fast install media. I wonder if something is making it excessively slow?

Mathew Hodson (mhodson)
no longer affects: livecd-rootfs (Ubuntu Xenial)
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.