Merge ~mitchdz/ubuntu/+source/ec2-hibinit-agent:add-IMDSv2-jammy into ubuntu/+source/ec2-hibinit-agent:ubuntu/jammy-devel

[Mitchell Dzurick]

PPA - https://launchpad.net/~mitchdz/+archive/ubuntu/ec2-hibinit-agent-add-imdsv2-support

efb504c... by Mitchell Dzurick on 2023-05-30

changelog

[Mitchell Dzurick (mitchdz)]

This has been tested to work on Jammy

us-west-2 was used as the region for aws (this is important because AMI will change)

# To test the failure
1. Create an amazon EC2 instance with the following properties
   - AMI - ami-0c7b1f327a97d8cec (For Jammy)
   - t2.micro
   - encrypted 8GB EBS volume with default key
   - Stop - Hibernate behavior: Enable
   - Metadata accessible : Enable
   - Metadata version Info : V2 only (token required)
2. Wait for instance to say "Running"
3. ssh into your instance
4. Create a test program
```
/bin/cat <<EOM >~/allocate_mem.py
#!/usr/bin/python3
import time

# Allocate 200MB chunk of memory
size = 200 * 1024 * 1024 # 200MB
memory_chunk = bytearray(size)

print("Allocated 200MB of memory.")

# Enter indefinite loop
while True:
    time.sleep(1) # Wait for 1 second

# The script will never reach this point
EOM
```
5. Run a background process
```
python3 ~/allocate_mem.py &
```
6. Hibernate the instance
7. Wait for Instance to be in "Stopped" State
8. Start the intsance
9. ssh into instance
10. Check that the process is running
$ ps aux | grep allocate_mem | grep -v grep
$

# To test the fix
1. Create an amazon EC2 instance with the following properties
   - AMI - ami-0c7b1f327a97d8cec (For Jammy)
   - t2.micro
   - encrypted 8GB EBS volume with default key
   - Stop - Hibernate behavior: Enable
   - Metadata accessible : Enable
   - Metadata version Info : V2 only (token required)
2. Wait for instance to say "Running"
3. ssh into your instance
4. add PPA for updated package; apt update && apt upgrade
```
sudo add-apt-repository ppa:mitchdz/ec2-hibinit-agent-add-imdsv2-support -y
sudo apt update -y && sudo apt upgrade -y
```
5. Create a test program
```
/bin/cat <<EOM >~/allocate_mem.py
#!/usr/bin/python3
import time

# Allocate 200MB chunk of memory
size = 200 * 1024 * 1024 # 200MB
memory_chunk = bytearray(size)

print("Allocated 200MB of memory.")

# Enter indefinite loop
while True:
    time.sleep(1) # Wait for 1 second

# The script will never reach this point
EOM
```
6. Run a background process
```
python3 ~/allocate_mem.py &
```
7. Hibernate the instance
8. Wait for Instance to be in "Stopped" State
9. Start the instance
10. ssh into instance
11. Check that the process is running
$ ps aux | grep allocate_mem | grep -v grep
ubuntu 2532 1.1 23.0 221948 213248 pts/0 S 13:43 0:00 python3 /home/ubuntu/allocate_mem.py

[Athos Ribeiro (athos-ribeiro)]

Thanks for addressing all the requests, Mitchell. I changed one character in your changelog to make it match the other entries in the imperative form.

Uploaded.

Uploading to ubuntu (via ftp to upload.ubuntu.com):
  Uploading ec2-hibinit-agent_1.0.0-0ubuntu11.22.04.2.dsc: done.
  Uploading ec2-hibinit-agent_1.0.0-0ubuntu11.22.04.2.debian.tar.xz: done.
  Uploading ec2-hibinit-agent_1.0.0-0ubuntu11.22.04.2_source.buildinfo: done.
  Uploading ec2-hibinit-agent_1.0.0-0ubuntu11.22.04.2_source.changes: done.
Successfully uploaded packages.

[Athos Ribeiro (athos-ribeiro)]

review: Approve

[Athos Ribeiro (athos-ribeiro)]

Hi Mitchell, could you set the status of this one to merged?

diff --git a/debian/changelog b/debian/changelog
index 96bafa7..addc748 100644
--- a/debian/changelog
+++ b/debian/changelog
@@ -1,3 +1,10 @@
+ec2-hibinit-agent (1.0.0-0ubuntu11.22.04.2) jammy; urgency=medium
+
+  * d/p/lp1941785-Add-support-for-IMDSv2.patch: allows hibernation of
+    AWS EC2 instances with IMDSv2 (LP: #1941785)
+
+ -- Mitchell Dzurick <[e-mail]>  Tue, 16 May 2023 15:43:29 -0700
+
 ec2-hibinit-agent (1.0.0-0ubuntu11.22.04.1) jammy; urgency=medium
 
   * Swapon with maximum priority right before hibernation. This resolves
diff --git a/debian/patches/lp1941785-Add-support-for-IMDSv2.patch b/debian/patches/lp1941785-Add-support-for-IMDSv2.patch
new file mode 100644
index 0000000..6855eb2
--- /dev/null
+++ b/debian/patches/lp1941785-Add-support-for-IMDSv2.patch
@@ -0,0 +1,161 @@
+Description: Add support for IMDSv2
+Author: Frederick Lefebvre <[e-mail]>
+Origin: backport, https://github.com/aws/amazon-ec2-hibinit-agent/commit/9d9bca5c61fa9256289e68c88bd3747af2f62e28
+Bug-Ubuntu: https://bugs.launchpad.net/ubuntu/+source/ec2-hibinit-agent/+bug/1941785
+Last-Update: 2023-05-16
+---
+This patch header follows DEP-3: http://dep.debian.net/deps/dep3/
+--- a/agent/hibinit-agent
++++ b/agent/hibinit-agent
+@@ -18,6 +18,7 @@
+ import sys
+ import syslog
+ import math
++import requests
+ from subprocess import check_call, check_output, STDOUT
+ from threading import Thread
+ from math import ceil
+@@ -25,11 +26,6 @@
+ 
+ 
+ try:
+-    from urllib.request import urlopen, Request
+-except ImportError:
+-    from urllib2 import urlopen, Request, HTTPError
+-
+-try:
+     from ConfigParser import ConfigParser, NoSectionError, NoOptionError
+ except:
+     from configparser import ConfigParser, NoSectionError, NoOptionError
+@@ -41,7 +37,12 @@
+ log_to_syslog = True
+ log_to_stderr = True
+ SWAP_FILE = '/swap-hibinit'
+-URL = "http://169.254.169.254/latest/meta-data/hibernation/configured"
++
++DEFAULT_STATE_DIR = '/var/lib/hibinit-agent'
++HIB_ENABLED_FILE = "hibernation-enabled"
++IMDS_BASEURL = 'http://169.254.169.254'
++IMDS_API_TOKEN_PATH = 'latest/api/token'
++IMDS_SPOT_ACTION_PATH = 'latest/meta-data/hibernation/configured'
+ 
+ def log(message):
+     if log_to_syslog:
+@@ -314,6 +315,9 @@
+             get_int('swap', 'percentage-of-ram'), args.swap_ram_percentage, 100)
+         self.swap_mb = self.merge(
+             get_int('swap', 'target-size-mb'), args.swap_target_size_mb, 4000)
++        self.state_dir = get('core', 'state-dir')
++        if self.state_dir is None:
++            self.state_dir = DEFAULT_STATE_DIR
+ 
+ 
+     def merge(self, cf_value, arg_value, def_val):
+@@ -337,31 +341,55 @@
+     def __str__(self):
+         return str(self.__dict__)
+ 
+-def hibernationEnabled():
+-     """Returns a boolean indicating whether hibernation is enabled or not."""
+-     response = None
+-     try:
+-         response = urlopen(URL)
+-         data = response.read()
+-         if data.lower() in ('false', b'false'):
+-             return False
+-     except:
+-         return False
+-     finally:
+-         if response:
+-             response.close()
+-     return True
++def get_imds_token(seconds=21600):
++    """ Get a token to access instance metadata. """
++    log("Requesting new IMDSv2 token.")
++    request_header = {'X-aws-ec2-metadata-token-ttl-seconds': str(seconds)}
++    token_url = '{}/{}'.format(IMDS_BASEURL, IMDS_API_TOKEN_PATH)
++    response = requests.put(token_url, headers=request_header)
++    response.close()
++    if response.status_code != 200:
++        return None
++
++    return response.text
++
++def create_state_dir(state_dir):
++    """ Create agent run dir if it doesn't exists."""
++    if not os.path.isdir(state_dir):
++        os.makedirs(state_dir)
++
++def hibernation_enabled(state_dir):
++    """Returns a boolean indicating whether hibernation is enabled or not.
++    Hibernation can't be enabled/disabled the instance launch. If we find
++    hibernation to be enabled, we create a semephore file so that we don't
++    have to probe IMDS again. That is useful when a instance is rebooted
++    after/if the IMDS http endpoint has been disabled.
++    """
++    hib_sem_file = os.path.join(state_dir, HIB_ENABLED_FILE)
++    if os.path.isfile(hib_sem_file):
++        log("Found {!r}, configuring hibernation".format(hib_sem_file))
++        return True
++
++    imds_token = get_imds_token()
++    if imds_token is None:
++        # IMDS http endpoint is disabled
++        return False
++
++    request_header = {'X-aws-ec2-metadata-token': imds_token}
++    response = requests.get("{}/{}".format(IMDS_BASEURL, IMDS_SPOT_ACTION_PATH),
++                 headers=request_header)
++    response.close()
++    if response.status_code != 200 or response.text.lower() == "false":
++        return False
++
++    log("Hibernation Configured Flag found")
++    os.mknod(hib_sem_file)
++
++    return True
++
+ 
+ def main():
+     
+-    if not hibernationEnabled():
+-        log("Instance Launch has not enabled Hibernation Configured Flag. hibinit-agent exiting!!")
+-        exit(0)
+-    # Validate if disk space>total RAM 
+-    ram_bytes = os.sysconf('SC_PAGE_SIZE') * os.sysconf('SC_PHYS_PAGES')
+-    if get_rootfs_size()<=(math.ceil(float(ram_bytes)/(1024*1024*1024))):
+-        log("Insufficient disk space. Cannot create setup for hibernation. Please allocate a larger root device")
+-        exit(1)
+     # Parse arguments
+     parser = argparse.ArgumentParser(description="An EC2 background process that creates a setup for instance hibernation "
+                                                  "at instance launch and also registers ACPI sleep event/actions")
+@@ -388,6 +416,17 @@
+     log_to_syslog = config.log_to_syslog
+ 
+     log("Effective config: %s" % config)
++    create_state_dir(config.state_dir)
++
++    # Let's first check if we even need to run
++    if not hibernation_enabled(config.state_dir):
++        log("Instance Launch has not enabled Hibernation Configured Flag. hibinit-agent exiting!!")
++        exit(0)
++    # Validate if disk space>total RAM
++    ram_bytes = os.sysconf('SC_PAGE_SIZE') * os.sysconf('SC_PHYS_PAGES')
++    if get_rootfs_size()<=(math.ceil(float(ram_bytes)/(1024*1024*1024))):
++        log("Insufficient disk space. Cannot create setup for hibernation. Please allocate a larger root device")
++        exit(1)
+ 
+     target_swap_size = config.swap_mb * 1024 * 1024
+     swap_percentage_size = ram_bytes * config.swap_percentage // 100
+--- a/etc/hibinit-config.cfg
++++ b/etc/hibinit-config.cfg
+@@ -11,6 +11,9 @@
+ # filesystems.
+ touch-swap = False
+ 
++# Location where to create any state files
++state-dir = "/var/lib/hibinit-agent"
++
+ [swap]
+ # If there's no swap then we create it to be equal to the specified
+ # percentage of RAM or to the target size, whichever is greater
diff --git a/debian/patches/series b/debian/patches/series
index 93ae156..7412d06 100644
--- a/debian/patches/series
+++ b/debian/patches/series
@@ -8,3 +8,4 @@ detect-hibernate-cmd-by-default.patch
 0008-Always-set-resume-device-by-PARTUUID-instead-of-by-d.patch
 0010-Update-grub-configuration-when-it-needs-an-update.patch
 lp1968805-Swapon-with-maximum-priority-before-hibernation.patch
+lp1941785-Add-support-for-IMDSv2.patch