Ubuntu
hibagent package

agent fails if /boot/grub/menu.lst GRUB config file is not present

Bug #2013336 reported by Alberto Contreras on 2023-03-30

This bug affects 1 person

	Status	Importance	Assigned to
hibagent (Ubuntu)	Fix Released	Undecided	Unassigned
Focal	Fix Released	Undecided	Alberto Contreras
Jammy	Fix Released	Undecided	Alberto Contreras
Kinetic	Fix Released	Undecided	Alberto Contreras

Bug Description

[ Impact ]

* This bug prevents the agent to correctly set up instances
for hibernation. It disallows AWS to hibernate spot instances.

[ Test Plan ]

* Spin up an EC2 spot instance with `hibernate` as `Interruption behavior` [1].
* Install the latest hibagent: `sudo apt-get install hibagent`
* Enable hibernation: `sudo /usr/bin/enable-ec2-spot-hibernation`
* Create an AWS FIS experiment template to send a spot-instance-interruption signal [2], make it point to the created instance and launch it.
Note: This step is optional, one can wait for AWS EC2 to send the interruption signal, but it could take a lot of time.
* After some minutes, EC2 will send a signal to resume the interrupted instance.
* Verify the instance has correctly been resumed from hibernation.

[1] https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/interruption-behavior.html#specifying-spot-interruption-behavior
[2] https://catalog.us-east-1.prod.workshops.aws/workshops/5fc0039f-9f15-47f8-aff0-09dc7b1779ee/en-US/030-basic-content/078-ec2-spot/020-spot-ec2-interrup

[ Where problems could occur ]

* On affected releases, hibagent modifies config files for both GRUB and
   GRUB2. The upstream patch only modifies the GRUB config file if it is
   present. Thus, if there is a regression it would affect releases with GRUB,
   but we are going to include it in series that ship with GRUB2.

[ Other Info ]

* Bionic ships GRUB2 but the images contain the old and not needed
   /boot/grub/menu.lst . CPC has confirmed that there are no plans
   to remove that file as that would constitute a breaking change for
   an LTS. That's why this bug does not affect bionic.

[Original Description]

The agent fails to initialize the swap when /boot/grub/menu.lst is not present.
That file is related to GRUB and is not needed on systems with GRUB2 installed.

# /usr/bin/enable-ec2-spot-hibernation
# journalctl -u hibagent
...
Mar 30 10:51:10 ip-172-31-34-23 /hibagent[615]: Failed to initialize swap, reason: [Errno 2] No such file or directory: '/boot/grub/menu.lst'
Mar 30 10:51:11 ip-172-31-34-23 /hibagent[615]: Background swap initialization thread is complete.

I expected the agent to not require config files of older grub versions to be present to correctly configure the system for hibernation.

See original description

Tags:

Related branches

~aciba/ubuntu/+source/hibagent:lp1896638-path-to-swapfile-by-uuid-focal

Merged into ubuntu/+source/hibagent:ubuntu/focal-devel at revision 051f6d9b4f1fa089cbf3f15bf255e94dce33a24d

Simon Chopin (community): Approve on 2023-04-24

~aciba/ubuntu/+source/hibagent:lp1896638-path-to-swapfile-by-uuid-jammy

Merged into ubuntu/+source/hibagent:ubuntu/jammy-devel at revision 7aa580cc27d7f72594fe63db8ec10da3eb0e8b54

Simon Chopin (community): Approve on 2023-04-24

~aciba/ubuntu/+source/hibagent:lp1896638-path-to-swapfile-by-uuid-kinetic

Merged into ubuntu/+source/hibagent:ubuntu/kinetic-devel at revision 9bbca7aec6433edc5db1740ec56d6f792d5d51cc

Simon Chopin (community): Approve on 2023-04-24

Revision history for this message

Alberto Contreras (aciba) wrote on 2023-03-30:

This bug was fixed in lunar as the 1.0.1+git20230216.9ac1209f7-0ubuntu1 version made an upstream snapshot including https://github.com/aws/ec2-hibernate-linux-agent/commit/0f995e803483acbf87ddb04c62960c30e837afce

This was made to fix https://bugs.launchpad.net/ubuntu/+source/ec2-hibinit-agent/+bug/1896638 for lunar.

Changed in hibagent (Ubuntu):
status:	New → Fix Released

Alberto Contreras (aciba) on 2023-03-30

description:

updated

Alberto Contreras (aciba) on 2023-03-31

Changed in hibagent (Ubuntu Kinetic):
assignee:	nobody → Alberto Contreras (aciba)
Changed in hibagent (Ubuntu Jammy):
assignee:	nobody → Alberto Contreras (aciba)

Alberto Contreras (aciba) on 2023-03-31

Changed in hibagent (Ubuntu Focal):
assignee:	nobody → Alberto Contreras (aciba)

Alberto Contreras (aciba) on 2023-04-25

description:

updated

Alberto Contreras (aciba) on 2023-04-25

description:

updated

Alberto Contreras (aciba) on 2023-05-09

Changed in hibagent (Ubuntu Focal):
status:	New → In Progress
Changed in hibagent (Ubuntu Jammy):
status:	New → In Progress
Changed in hibagent (Ubuntu Kinetic):
status:	New → In Progress

Revision history for this message

Steve Langasek (vorlon) wrote on 2023-05-12: Please test proposed package

Hello Alberto, or anyone else affected,

Accepted hibagent into kinetic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/hibagent/1.0.1-0ubuntu2.22.10.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-kinetic to verification-done-kinetic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-kinetic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in hibagent (Ubuntu Kinetic):
status:	In Progress → Fix Committed
tags:	added: verification-needed verification-needed-kinetic
Changed in hibagent (Ubuntu Jammy):
status:	In Progress → Fix Committed
tags:	added: verification-needed-jammy

Revision history for this message

Steve Langasek (vorlon) wrote on 2023-05-12:

Hello Alberto, or anyone else affected,

Accepted hibagent into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/hibagent/1.0.1-0ubuntu2.22.04.1 in a few hours, and then in the -proposed repository.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in hibagent (Ubuntu Focal):
status:	In Progress → Fix Committed
tags:	added: verification-needed-focal

Revision history for this message

Steve Langasek (vorlon) wrote on 2023-05-12:

Hello Alberto, or anyone else affected,

Accepted hibagent into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/hibagent/1.0.1-0ubuntu1.20.04.1 in a few hours, and then in the -proposed repository.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Revision history for this message

Alberto Contreras (aciba) wrote on 2023-05-24 (last edit on 2023-05-24):

It looks like AWS EC2 has disabled the ability to request spot instances with the interruption behavior set as 'hibernate'.
I have tried to reproduce it in multiple regions and with multiple valid instance types and I consistently get the following error:

```
launchSpecTemporarilyBlacklisted Repeated errors have occurred processing the launch specification "t3.micro, ami-08d931621368a5861, Linux/UNIX, eu-west-3a while launching spot instance". It will not be retried for at least 13 minutes. Error message: The request with instanceType 't3.micro' and Linux/UNIX is not supported when instanceInterruptionBehavior is set to 'hibernate'. (Service: AmazonEC2; Status Code: 400; Error Code: InvalidParameterCombination; Proxy: null)
```

I have been able to reproduce and verify that the hibernation works and that this bug is fixed simulating the workflow on normal instance with focal, jammy and kinetic:

apt purge ec2-hibinit-agent
apt-get update
apt-get upgrade -y

cat <<EOF >/etc/apt/sources.list.d/ubuntu-$(lsb_release -cs)-proposed.list
# Enable Ubuntu proposed archive
deb http://archive.ubuntu.com/ubuntu/ $(lsb_release -cs)-proposed restricted main multiverse universe
EOF

apt-get update
apt-get install -y hibagent
apt-cache policy hibagent
systemctl is-active hibagent.target || /usr/bin/enable-ec2-spot-hibernation

# Verify no errors
systemctl status hibagent
journalctl -u hibagent

systemctl hibernate

# Start the instance and verify the hibernation resuming was okay
systemctl status hibinit-agent
journalctl --reverse

tags:

added: verification-done-focal verification-done-jammy verification-done-kinetic
removed: verification-needed verification-needed-focal verification-needed-jammy verification-needed-kinetic

Alberto Contreras (aciba) on 2023-05-24

tags:

added: verification-done

Revision history for this message

Andreas Hasenack (ahasenack) wrote on 2023-05-25:

Hi Alberto, thanks for this verification. Since you used a script, do you happen to have its output?

Revision history for this message

Alberto Contreras (aciba) wrote on 2023-05-26:

Hello Andreas. I have attached the log for this and #1896638 here: https://bugs.launchpad.net/ubuntu/+source/ec2-hibinit-agent/+bug/1896638/comments/34

Thanks.

Revision history for this message

Alberto Contreras (aciba) wrote on 2023-05-29:

I have attached the logs for the remaining series in #1896638. Thanks!

kinetic: https://bugs.launchpad.net/ubuntu/+source/ec2-hibinit-agent/+bug/1896638/+attachment/5676320/+files/verification-hibagent-lp1896638-lp2013336-kinetic.log

jammy: https://bugs.launchpad.net/ubuntu/+source/ec2-hibinit-agent/+bug/1896638/+attachment/5675852/+files/verification-hibagent-lp1896638-lp2013336-jammy.log

focal: https://bugs.launchpad.net/ubuntu/+source/ec2-hibinit-agent/+bug/1896638/+attachment/5676319/+files/verification-hibagent-lp1896638-lp2013336-focal.log

Revision history for this message

Launchpad Janitor (janitor) wrote on 2023-05-30:

This bug was fixed in the package hibagent - 1.0.1-0ubuntu1.20.04.1

---------------
hibagent (1.0.1-0ubuntu1.20.04.1) focal; urgency=medium

  * d/p/lp1896638-set-resume-device-by-partition-uuid: Set resume device
    by PARTUUID instead of by name. Thanks to Tony Nie <email address hidden>.
    (LP: #1896638)
  * d/p/lp2013336-do-not-modify-GRUB-config-on-GRUB2-systems: Do not attempt to
    modify non existent GRUB config file. Thanks to Robert Schweikert
    <email address hidden>. (LP: #2013336)

-- Alberto Contreras <email address hidden> Tue, 04 Apr 2023 18:31:39 +0200

Changed in hibagent (Ubuntu Focal):
status:	Fix Committed → Fix Released

Revision history for this message

Brian Murray (brian-murray) wrote on 2023-05-30: Update Released

#10

The verification of the Stable Release Update for hibagent has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Revision history for this message

Launchpad Janitor (janitor) wrote on 2023-05-30:

#11

This bug was fixed in the package hibagent - 1.0.1-0ubuntu2.22.04.1

---------------
hibagent (1.0.1-0ubuntu2.22.04.1) jammy; urgency=medium

-- Alberto Contreras <email address hidden> Tue, 04 Apr 2023 18:22:05 +0200

Changed in hibagent (Ubuntu Jammy):
status:	Fix Committed → Fix Released

Revision history for this message

Launchpad Janitor (janitor) wrote on 2023-05-30:

#12

This bug was fixed in the package hibagent - 1.0.1-0ubuntu2.22.10.1

---------------
hibagent (1.0.1-0ubuntu2.22.10.1) kinetic; urgency=medium

-- Alberto Contreras <email address hidden> Tue, 04 Apr 2023 18:03:13 +0200

Changed in hibagent (Ubuntu Kinetic):
status:	Fix Committed → Fix Released

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.

Ubuntuhibagent package

agent fails if /boot/grub/menu.lst GRUB config file is not present

Bug Description

Related branches

Other bug subscribers

Remote bug watches

Ubuntu
hibagent package