cloud-init fails when rebooting EC2 i3.metal instances

Thanks for submitting this. Is it possible to collect-logs from the initial boot of the instance before the reboot? In particular I'd like to see a cloud-init collect-logs and a tar of /var/lib/cloud before the reboot.

Ryan Harper wrote on 28/08/2019:
> Thanks for submitting this. Is it possible to collect-logs from the
> initial boot of the instance before the reboot? In particular I'd like
> to see a cloud-init collect-logs and a tar of /var/lib/cloud before the
> reboot.

Here are the first-boot logs from an instance that when rebooted hit the
problem. The post-reboot-console-log-fail.log file is the console log
collected after the reboot and that shows the cloud-init error message.
It has been collected using `aws ec2 get-console-output`.

Paride

OK, it looks like /var/lib/cloud/data/set-hostname ends up empty after the reboot, which is what causes the issue. We should handle that case gracefully, assuming that it isn't cloud-init that's causing it to be removed.

To help confirm what's going on, are you able to capture /var/lib/cloud and logs post-failure (perhaps by attaching the root disk elsewhere)?

Dan Watkins wrote on 03/09/2019:
> OK, it looks like /var/lib/cloud/data/set-hostname ends up empty after
> the reboot, which is what causes the issue. We should handle that case
> gracefully, assuming that it isn't cloud-init that's causing it to be
> removed.
>
> To help confirm what's going on, are you able to capture /var/lib/cloud
> and logs post-failure (perhaps by attaching the root disk elsewhere)?

Here are the first-boot and post-reboot logs collected from an instance
that encountered the problem.

OK, to clarify my previous comment, the issue is not that /var/lib/cloud/data/set-hostname is _missing_, the issue is specifically that it is present but empty. Unfortunately, we don't have any logging from the code path used to write set-hostname (it uses cloudinit.atomic_helper.write_json which uses atomic_helper.write_file), so we can't be 100% sure that we aren't incorrectly writing out this empty file.

So, I think this breaks down into three things:

1) We should improve the logging situation there (just filed bug 1843276)
2) We should work out why the file ends up empty in this case
3) We should gracefully treat an empty file as an absent file (probably with a WARNING?), because I think that's the best we can do once we find ourselves in that situation

(2) and (3) should _probably_ be separate bugs, but without (1) it's not 100% clear that that's the case so I haven't filed anything separate yet.

Paride, are you able to test this again using the cloud-init from the daily PPA[0]? That has some additional logging added, which should make working out (2) easier.

[0] https://launchpad.net/~cloud-init-dev/+archive/ubuntu/daily

[Expired for cloud-init because there has been no activity for 60 days.]

This didn't happen again in quite some time, not sure what fixed it. I'll reopen this bug report if I encounter the issue again.

Hi Dan Watkins,
we got a similar issue that cloud-init fails when /var/lib/cloud/data/set-hostname is present but empty during reboot. We are doing further research that why the file ends up empty in our case, meanwhile, will cloud-init handle the exception as you commented(https://bugs.launchpad.net/cloud-init/+bug/1841182/comments/6)?

Our problem details are here,
https://bugzilla.redhat.com/show_bug.cgi?id=2140893

Xiaoyi, that bugzilla bug isn't public, so we can't see the details. We'll need the resulting tarball from 'cloud-init collect-logs' (with --include-userdata if there's no sensitive userdata) in order to try to debug this issue. Also note that this bug was filed 3 years ago against a version of cloud-init that is 3 years old. Can the problem be reproduced on the current version of cloud-init?

The BZ is now public. However, there is only one suggestion that is private that might help solving the problem, so I am pasting it here:

```
Firstly, IMO, empty '/var/lib/cloud/data/set-hostname' is indeed an exception/case that cloud-init should handle properly than leave the service failed directly.

Back to this issue, hard reboot during boot or trigger sysrq b immediately when system boot up. The content of set-hostname lost after system boot up.
It seems that the content did not synced to disk when system reset.

Here is the instructions of sysrq b.

``b`` Will immediately reboot the system without syncing or unmounting
            your disks.

Below information shows that set-hostname already has content before reset, but it is 0 after boot again.
2022-11-08 01:49:17,759 - util.py[DEBUG]: Reading from /var/lib/cloud/data/set-hostname (quiet=False)
2022-11-08 01:49:17,759 - util.py[DEBUG]: Read 88 bytes from /var/lib/cloud/data/set-hostname
2022-11-08 01:49:17,759 - cc_set_hostname.py[DEBUG]: No hostname changes. Skipping set-hostname
2022-11-08 02:33:58,585 - util.py[DEBUG]: Reading from /var/lib/cloud/data/set-hostname (quiet=False)
2022-11-08 02:33:58,585 - util.py[DEBUG]: Read 0 bytes from /var/lib/cloud/data/set-hostname

Not sure if turn off buffering is helpful to mitigate this, but I did not reproduce this issue after 10+ retry times.

# diff -u /usr/lib/python3.9/site-packages/cloudinit/atomic_helper.py /usr/lib/python3.9/site-packages/cloudinit/atomic_helper.py.orig
--- /usr/lib/python3.9/site-packages/cloudinit/atomic_helper.py 2022-11-08 08:17:56.262679073 +0000
+++ /usr/lib/python3.9/site-packages/cloudinit/atomic_helper.py.orig 2022-11-08 08:17:03.722890860 +0000
@@ -25,7 +25,7 @@
     tf = None
     try:
         tf = tempfile.NamedTemporaryFile(
- dir=os.path.dirname(filename), delete=False, mode=omode, buffering=0
+ dir=os.path.dirname(filename), delete=False, mode=omode
         )
```

Tracked in Github Issues as https://github.com/canonical/cloud-init/issues/3435