snapd fails always in Optimised Ubuntu Desktop images available in Microsoft Hyper-V gallery

To my limited understanding system appears to have not completed its start:

peterg@xenial-Virtual-Machine:~$ systemd-analyze blame
Bootup is not yet finished. Please try again later.

Status changed to 'Confirmed' because the bug affects multiple users.

'systemd-analyze blame' suggests to run 'systemctl list-jobs', this is its output:

JOB UNIT TYPE STATE
148 snapd.autoimport.service start waiting
 89 systemd-update-utmp-runlevel.service start waiting
  2 multi-user.target start waiting
129 snapd.seeded.service start running
  1 graphical.target start waiting

5 jobs listed.

After a long and fruitless digging around systemd unit and seed.loaded I have removed (purged) and installed snapd package and it fixed this issue for me.

Something along the lines of:
$ sudo apt purge snapd
$ sudo apt install snapd
$ sudo snap install core

I've had the same problem and purging/re-installing via apt package manager is the only way I could figure out to fix it.

This is a pretty bad first experience for 19.04 gallery image as its supposed to be 'out of the box'. Must be lots of people trying this first time in hyper-v gallery and giving up. Took few hours to find this bug post. I can verify that the #5 commands corrected problem. purge was needed. uninstall didn't work.

I can confirm this bug is still present in the optimised 18.04.2 image, and also verify that the commands from post #5 above worked for me to fix it. I actually had to purge snapd twice, as the first time it failed due to being unable to delete a folder with `rm` since it was a folder, not a file. `sudo rm -r /var/cache/snapd/aux` fixed it for me, and then the second purge completed properly. I had no issues after that with reinstalling snap or the core snap package, or a subsequent dist-upgrade, which had failed completely on a previous spinup of the 18.04.2 LTS image.

I suspect these images have a broken seed.yaml. Could somebody attach that?

@chipaca: seed.yaml from the 18.04.2 image in the Hyper-V Gallery

$ cat /var/lib/snapd/seed/seed.yaml
snaps:
  -
    name: core
    channel: stable
    file: core_6673.snap
  -
    name: gnome-3-26-1604
    channel: stable/ubuntu-18.04
    file: gnome-3-26-1604_82.snap
  -
    name: gnome-calculator
    channel: stable/ubuntu-18.04
    file: gnome-calculator_352.snap
  -
    name: gnome-characters
    channel: stable/ubuntu-18.04
    file: gnome-characters_206.snap
  -
    name: gnome-logs
    channel: stable/ubuntu-18.04
    file: gnome-logs_57.snap
  -
    name: gnome-system-monitor
    channel: stable/ubuntu-18.04
    file: gnome-system-monitor_70.snap
  -
    name: gtk-common-themes
    channel: stable/ubuntu-18.04
    file: gtk-common-themes_1198.snap

yeah. That seed.yaml is missing core18, and those snaps need it AFAIK.

To confirm, (from memory), sudo snap info --verbose /var/lib/snapd/seed/snaps/*.snap | grep base:

followup:

$ sudo snap info --verbose /var/lib/snapd/seed/snaps/*.snap | grep base:
[sudo] password for dllewellyn:
base: core18
base: core18
base: core18
base: core18

Yeah. The images are broken, because they ship a broken seed.yaml. Who creates the images? That's the project this bug needs to be assigned to.

I think we need to add validation of the seed.yaml to livecd-rootfs:
https://code.launchpad.net/~mvo/livecd-rootfs/+git/livecd-rootfs/+merge/370065

We also need to do more validation in "snap debug validate-seed":
https://github.com/snapcore/snapd/pull/7105

I have no understanding of the test process used to check a system livecd build, but I imagine installing a snap that confirmed itself to the test-system would be useful on each platform.

I ran the following test on the current version of livecd-rootfs for Bionic:

https://pastebin.ubuntu.com/p/wQknctr6ys/

with a positive result:

https://pastebin.ubuntu.com/p/N8PXRJJV9b/

So I think in principle it should work, but maybe there are silent download failures. We definitely need mvo's sanity checks on seed.yaml.

My mistake that I didn't do proper error checking while determining the base of a snap. This here

https://code.launchpad.net/~tobijk/livecd-rootfs/+git/livecd-rootfs/+merge/370096

will hopefully prevent this from happening again.

Thanks everyone!

Just as a heads up: the seed.yaml for the azure image also need gnome-3-28-1804 in the seed. The `snap debug validate-image` will validate all this once its landed so hopefully this will at least be easy to detect fix (build-time instead of runtime).

Maybe also useful: modifications to snap-tool to make it more resilient to connection failures of all sorts:

https://code.launchpad.net/~tobijk/livecd-rootfs/+git/livecd-rootfs/+merge/370139

The attachment "Backport for Bionic" seems to be a debdiff. The ubuntu-sponsors team has been subscribed to the bug report so that they can review and hopefully sponsor the debdiff. If the attachment isn't a patch, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are member of the ~ubuntu-sponsors, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issue please contact him.]

Hello Peter, or anyone else affected,

Accepted livecd-rootfs into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/livecd-rootfs/2.525.28 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Hello Peter, or anyone else affected,

Accepted livecd-rootfs into disco-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/livecd-rootfs/2.578.6 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-disco to verification-done-disco. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-disco. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

All autopkgtests for the newly accepted livecd-rootfs (2.578.6) for disco have finished running.
There have been regressions in tests triggered by the package. Please visit the sru report page and investigate the failures.

https://people.canonical.com/~ubuntu-archive/pending-sru.html#disco

Comment #27 is not correct.

I tested a bionic desktop ISO built with this livecd-rootfs and built successfully and got core, so snaps work.

Not sure if that is sufficient validation for this bug, so I won't (yet) change the tags.

I believe that comment #27 is correct, because after that the autopkgtests for ubuntu-image succeeded (triggered by a qemu update) using the previous revision of livecd-rootfs on disco. But I'm not able to tell how the changes made to livecd-rootfs affect the operation/testing of ubuntu-image.

I should have been more explicit. When I wrote comment #28, the tests *had not finished running yet* so it was false at that point.

I don't know about this regression, let me retry it and if it happens again, please investigate.

It passed now.

FYI: Comment #8 (with comment #5) fixed the problem for me :-)

I have verified (bionic 2.525.28 and disco 2.578.6) that the changes to snap_preseed do not break existing functionality and that an error is thrown when the helper _snap_preseed is called with an invalid snap name, resulting in snap-info failing to retrieve the base.

This bug was fixed in the package livecd-rootfs - 2.525.28

---------------
livecd-rootfs (2.525.28) bionic; urgency=medium

  [ Tobias Koch ]
  * Do proper error checking when calling snap-tool info to determine
    the base of a snap. (LP: #1828500)

  [ Michael Vogt ]
  * Run "snap debug validate-seed" during preseeding to do basic validation of
    the generated seed.yaml

  [ Iain Lane ]
  * Seed core for non minimized builds, as it is still required (LP:
    #1836594).

 -- Iain Lane <email address hidden> Tue, 16 Jul 2019 13:20:18 +0100

The verification of the Stable Release Update for livecd-rootfs has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

This bug was fixed in the package livecd-rootfs - 2.578.7

---------------
livecd-rootfs (2.578.7) disco; urgency=medium

  [ Tobias Koch ]
  * Add retry logic to snap-tool to make downloads more resilient.
    (LP: #1837871)

 -- Tobias Koch <email address hidden> Mon, 26 Aug 2019 13:36:36 +0200