cloud-images

Users are prompted on upgrade of cloud images

Bug #1485685 reported by Robie Basak
52
This bug affects 11 people
Affects Status Importance Assigned to Milestone
cloud-images
Fix Released
High
Unassigned

Bug Description

Steps to reproduce (use case):

1. Fire up a Trusty cloud image.
2. Run: sudo apt-get update && sudo apt-get -y dist-upgrade

Steps to reproduce (developer):

1. Fire up an older image that does not contain the newest grub2 SRU. Example: http://cloud-images.ubuntu.com/trusty/20150706/trusty-server-cloudimg-amd64-disk1.img. I'm using "uvt-kvm create --backing-image-file trusty-server-cloudimg-amd64-disk1.img testvm" to do this.
2. Run: sudo apt-get update && sudo apt-get -y dist-upgrade

Expected results: completes without prompts.

Actual results: prompts the user for input to resolve a "manually" changed /boot/grub/menu.lst.

User impact: this messes up automation; it is a common use of cloud images for users to automate their server deployments which generally involves running a dist-upgrade and then bootstrapping some kind of configuration management system.

This means that we cannot SRU grub2 without breaking some set of users in a fairly common use case of cloud images until users start using newer images with the new SRU'd grub2 rolled in.

Workaround: users can use something like: apt-get update && UCF_FORCE_CONFOLD=1 apt-get -y dist-upgrade. Note that to avoid prompts in various other cases users should also set DEBIAN_FRONTEND=noninteractive, run apt-get with -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" and consider redirecting stdin from /dev/null.

Investigation:

I noticed that running "sudo update-grub-legacy-ec2" before dist-upgrade works around the problem. This is because ucf updates the entry for /run/grub/menu.lst in /var/lib/ucf/hashfile, at which point it doesn't believe that the configurable snippet in menu.lst has changed locally in a future run.

I suspect that menu.lst is being changed without ucf being updated in the image build process. Line 421 "_xchroot "${rootd}" /usr/bin/ucfr --purge grub /var/run/grub/menu.lst" in vmbuilder-cloudimg-fixes (r699) looks like it works around the issue, but I don't think the workaround is working because calling "/usr/bin/ucfr --purge grub /var/run/grub/menu.lst" doesn't clear the entry from /var/lib/ucf/registry after the instance is launched either. It seems that ucfr is normalising the path to /run/grub/menu.lst but the registry stores the denormalised path so it fails to match.

See original description
Robie Basak (racb)
tags: added: bot-stop-nagging
Revision history for this message
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in ubuntu:
status: New → Confirmed
Revision history for this message
Ben Howard (darkmuggle-deactivatedaccount) wrote :

rcj, can you take a look at this?

Changed in ubuntu:
assignee: nobody → Robert C Jennings (rcj)
Revision history for this message
Robie Basak (racb) wrote :

Note bug 1323772 (marked as duplicate) with more commentary, description of other workarounds and links to other bugs which may all be the same issue.

description: updated
Revision history for this message
Robert C Jennings (rcj) wrote :

At the root of this issue is ucf. The ucfr command will store a path that contains symlinks during insertion but will only match the symlink-resolved path when attempting removal. Cloud image builds modify grub's menu.lst but have attempted to purge the ufc registry entry. This bug in ucf causes the registry removal to fail.

This was partially fixed upstream in 3.0028[1] so that future additions to the registry would be symlink-resolved as well; however it does not address upgrade such that the user still can not remove an entry containing a symlink.

For our current cloud image users, we can ensure the entry is properly removed during image creation for new images, ideally with a fixed ucf package that benefits everyone, but for images in the field the user would need to remove the registry entry[2] and this is a manual edit of the registry file without further patches to upstream ucf.

[1] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=724457
[2] sed -i -r "/grub[[:space:]]+\/var\/run\/grub\/menu\.lst/d" /var/lib/ucf/registry

Revision history for this message
Robert C Jennings (rcj) wrote :

I spoke with racb and he thought that "the right thing to do might be to debconf-ize grub-legacy-ec2 so that you don't need to change the [grub's menu.lst] manually." rather than changing things outside of packaging within the cloud image build.

Revision history for this message
Robert C Jennings (rcj) wrote :

To add clarity, the debconf proposal in comment #5 is the long-term solution. In the interim, the workaround in comment #4 to remove the ucf registry entry is needed. For existing users, this will need to be done by hand. For future cloud images the ufc entry will be removed (in the short-term) with debconf-ized grub-legacy-ec2 as a long-term solution.

Changed in ubuntu:
status: Confirmed → Triaged
affects: ubuntu → cloud-init (Ubuntu)
Robert C Jennings (rcj)
Changed in cloud-init (Ubuntu):
importance: Undecided → Medium
assignee: Robert C Jennings (rcj) → nobody
Mathew Hodson (mhodson)
tags: removed: bot-stop-nagging
Dan Watkins (oddbloke)
Changed in cloud-images:
status: New → Confirmed
importance: Undecided → High
Revision history for this message
Dan Watkins (oddbloke) wrote :

So the reason that this regressed in xenial is that we've changed the way in which we build images, and the workaround for this was only being applied to PV images. I'm now building test images with the workaround applied to HVM and PV images, to confirm that my reasoning is sound.

Changed in cloud-images:
status: Confirmed → In Progress
Revision history for this message
Dan Watkins (oddbloke) wrote :

The latest xenial daily no longer exhibits this issue.

Changed in cloud-images:
status: In Progress → Fix Released
Revision history for this message
Andy (g-launchpad-z) wrote :

I think this is still an issue. At least, I made a new Xenial box from ami-c1167eb8 (Amazon AWS, eu-west-1, xenial, 16.04, amd64, hvm-ssd, 20180205). Is that a 'daily'? After 'apt update', amongst other packages, these grub packages were available to update:
grub-common:amd64 (2.02~beta2-36ubuntu3.15, 2.02~beta2-36ubuntu3.16)
grub2-common:amd64 (2.02~beta2-36ubuntu3.15, 2.02~beta2-36ubuntu3.16)
grub-pc:amd64 (2.02~beta2-36ubuntu3.15, 2.02~beta2-36ubuntu3.16)
grub-pc-bin:amd64 (2.02~beta2-36ubuntu3.15, 2.02~beta2-36ubuntu3.16)
Running 'apt dist-upgrade' resulted in me seeing the prompt regarding a changed '/boot/grub/menu.lst'.

Revision history for this message
Andy (g-launchpad-z) wrote :

In case it's relevant, I was using the new m5 instance type.

Revision history for this message
Robie Basak (racb) wrote :

I think this was fixed and then has regressed again. I filed bug 1747464 to track the regression.

Scott Moser (smoser)
no longer affects: cloud-init (Ubuntu)
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.