cloud-init

Azure reboot with unformatted ephemeral drive won't mount reformatted volume

Bug #1825596 reported by Jason Zions
14
This bug affects 1 person
Affects Status Importance Assigned to Milestone
cloud-init
Fix Released
High
Unassigned

Bug Description

If an Azure VM is rebooted after being moved to a different host (e.g. after a deallocate operation or after a service-heal to remove a bad host from service), the ephemeral drive exposed to the VM is reset to the default state (NTFS format). The Azure data source detects this and marks cc_disk_setup and cc_mounts to be run. While cc_disk_setup reformats the volume as desired, cc_mounts determines that the appropriate mount request was already in /etc/fstab (as setup during initial provisioning). Since the normal boot process would already have mounted everything according to fstab, the cc_mounts logic is "no mount -a is required". This is not true in this scenario.

Related branches

~chad.smith/cloud-init:ubuntu/bionic
Ryan Harper: Approve
Server Team CI bot: Approve (continuous-integration)
Diff: 4639 lines (+2418/-613)
49 files modified
ChangeLog (+117/-0)
cloudinit/cmd/main.py (+5/-4)
cloudinit/config/cc_apt_configure.py (+1/-1)
cloudinit/config/cc_mounts.py (+11/-0)
cloudinit/config/cc_ubuntu_advantage.py (+116/-109)
cloudinit/config/cc_ubuntu_drivers.py (+112/-0)
cloudinit/config/tests/test_ubuntu_advantage.py (+191/-156)
cloudinit/config/tests/test_ubuntu_drivers.py (+174/-0)
cloudinit/net/eni.py (+11/-5)
cloudinit/net/network_state.py (+33/-8)
cloudinit/net/sysconfig.py (+29/-11)
cloudinit/net/tests/test_init.py (+1/-1)
cloudinit/reporting/handlers.py (+57/-60)
cloudinit/sources/DataSourceAzure.py (+179/-95)
cloudinit/sources/DataSourceCloudStack.py (+1/-1)
cloudinit/sources/DataSourceConfigDrive.py (+2/-5)
cloudinit/sources/DataSourceEc2.py (+7/-3)
cloudinit/sources/DataSourceNoCloud.py (+3/-1)
cloudinit/sources/helpers/azure.py (+42/-3)
cloudinit/util.py (+17/-13)
cloudinit/version.py (+1/-1)
config/cloud.cfg.tmpl (+3/-0)
debian/changelog (+48/-0)
debian/patches/series (+1/-0)
debian/patches/ubuntu-advantage-revert-tip.patch (+735/-0)
doc/rtd/topics/datasources/azure.rst (+35/-22)
doc/rtd/topics/datasources/nocloud.rst (+1/-1)
doc/rtd/topics/modules.rst (+1/-0)
packages/redhat/cloud-init.spec.in (+3/-1)
packages/suse/cloud-init.spec.in (+3/-1)
setup.py (+2/-1)
tests/cloud_tests/releases.yaml (+16/-0)
tests/cloud_tests/testcases/modules/apt_pipelining_disable.yaml (+1/-2)
tests/cloud_tests/testcases/modules/apt_pipelining_os.py (+3/-3)
tests/cloud_tests/testcases/modules/apt_pipelining_os.yaml (+4/-5)
tests/data/azure/non_unicode_random_string (+1/-0)
tests/unittests/test_datasource/test_azure.py (+32/-5)
tests/unittests/test_datasource/test_azure_helper.py (+7/-2)
tests/unittests/test_datasource/test_nocloud.py (+42/-0)
tests/unittests/test_distros/test_netconfig.py (+2/-0)
tests/unittests/test_ds_identify.py (+17/-0)
tests/unittests/test_handler/test_handler_mounts.py (+29/-1)
tests/unittests/test_handler/test_schema.py (+1/-0)
tests/unittests/test_net.py (+251/-18)
tests/unittests/test_reporting_hyperv.py (+49/-55)
tools/build-on-freebsd (+4/-5)
tools/ds-identify (+4/-3)
tools/read-version (+5/-2)
tox.ini (+8/-9)
~chad.smith/cloud-init:ubuntu/cosmic
Ryan Harper: Approve
Server Team CI bot: Approve (continuous-integration)
Diff: 4637 lines (+2416/-613)
49 files modified
ChangeLog (+117/-0)
cloudinit/cmd/main.py (+5/-4)
cloudinit/config/cc_apt_configure.py (+1/-1)
cloudinit/config/cc_mounts.py (+11/-0)
cloudinit/config/cc_ubuntu_advantage.py (+116/-109)
cloudinit/config/cc_ubuntu_drivers.py (+112/-0)
cloudinit/config/tests/test_ubuntu_advantage.py (+191/-156)
cloudinit/config/tests/test_ubuntu_drivers.py (+174/-0)
cloudinit/net/eni.py (+11/-5)
cloudinit/net/network_state.py (+33/-8)
cloudinit/net/sysconfig.py (+29/-11)
cloudinit/net/tests/test_init.py (+1/-1)
cloudinit/reporting/handlers.py (+57/-60)
cloudinit/sources/DataSourceAzure.py (+179/-95)
cloudinit/sources/DataSourceCloudStack.py (+1/-1)
cloudinit/sources/DataSourceConfigDrive.py (+2/-5)
cloudinit/sources/DataSourceEc2.py (+7/-3)
cloudinit/sources/DataSourceNoCloud.py (+3/-1)
cloudinit/sources/helpers/azure.py (+42/-3)
cloudinit/util.py (+17/-13)
cloudinit/version.py (+1/-1)
config/cloud.cfg.tmpl (+3/-0)
debian/changelog (+48/-0)
debian/patches/series (+1/-0)
debian/patches/ubuntu-advantage-revert-tip.patch (+733/-0)
doc/rtd/topics/datasources/azure.rst (+35/-22)
doc/rtd/topics/datasources/nocloud.rst (+1/-1)
doc/rtd/topics/modules.rst (+1/-0)
packages/redhat/cloud-init.spec.in (+3/-1)
packages/suse/cloud-init.spec.in (+3/-1)
setup.py (+2/-1)
tests/cloud_tests/releases.yaml (+16/-0)
tests/cloud_tests/testcases/modules/apt_pipelining_disable.yaml (+1/-2)
tests/cloud_tests/testcases/modules/apt_pipelining_os.py (+3/-3)
tests/cloud_tests/testcases/modules/apt_pipelining_os.yaml (+4/-5)
tests/data/azure/non_unicode_random_string (+1/-0)
tests/unittests/test_datasource/test_azure.py (+32/-5)
tests/unittests/test_datasource/test_azure_helper.py (+7/-2)
tests/unittests/test_datasource/test_nocloud.py (+42/-0)
tests/unittests/test_distros/test_netconfig.py (+2/-0)
tests/unittests/test_ds_identify.py (+17/-0)
tests/unittests/test_handler/test_handler_mounts.py (+29/-1)
tests/unittests/test_handler/test_schema.py (+1/-0)
tests/unittests/test_net.py (+251/-18)
tests/unittests/test_reporting_hyperv.py (+49/-55)
tools/build-on-freebsd (+4/-5)
tools/ds-identify (+4/-3)
tools/read-version (+5/-2)
tox.ini (+8/-9)
~chad.smith/cloud-init:ubuntu/disco
Server Team CI bot: Approve (continuous-integration)
cloud-init Commiters: Pending requested
Diff: 1142 lines (+417/-168)
25 files modified
ChangeLog (+117/-0)
cloudinit/config/cc_apt_configure.py (+1/-1)
cloudinit/config/cc_mounts.py (+11/-0)
cloudinit/net/sysconfig.py (+4/-2)
cloudinit/net/tests/test_init.py (+1/-1)
cloudinit/reporting/handlers.py (+57/-60)
cloudinit/sources/DataSourceAzure.py (+11/-6)
cloudinit/sources/DataSourceCloudStack.py (+1/-1)
cloudinit/sources/DataSourceConfigDrive.py (+2/-5)
cloudinit/sources/DataSourceEc2.py (+1/-1)
cloudinit/sources/helpers/azure.py (+11/-3)
cloudinit/util.py (+2/-13)
cloudinit/version.py (+1/-1)
debian/changelog (+27/-0)
packages/redhat/cloud-init.spec.in (+3/-1)
packages/suse/cloud-init.spec.in (+3/-1)
setup.py (+2/-1)
tests/cloud_tests/releases.yaml (+16/-0)
tests/unittests/test_datasource/test_azure.py (+10/-3)
tests/unittests/test_datasource/test_azure_helper.py (+7/-2)
tests/unittests/test_handler/test_handler_mounts.py (+29/-1)
tests/unittests/test_net.py (+42/-3)
tests/unittests/test_reporting_hyperv.py (+49/-55)
tools/build-on-freebsd (+4/-5)
tools/read-version (+5/-2)
~jasonzio/cloud-init:requestMountAll
Server Team CI bot: Approve (continuous-integration)
Ryan Harper: Approve
Diff: 87 lines (+40/-1)
2 files modified
cloudinit/config/cc_mounts.py (+11/-0)
tests/unittests/test_handler/test_handler_mounts.py (+29/-1)
Revision history for this message
Ryan Harper (raharper) wrote :

Could you supply the systemd journal from the system and a cloud-init collect-logs?

I'd like to confirm that the .mount units for the ephemeral drive were attempted, but failed due to the disk not being ext4 (yet).

Changed in cloud-init:
importance: Undecided → High
status: New → Incomplete
Revision history for this message
Jason Zions (jasonzio) wrote :

I confirmed that myself in the logs; the timestamp on the failed mnt.mount is about three seconds earlier than the timestamp on the cloud-init startup message. The error message is indeed a complaint about not being able to mount an "ntfs" formatted volume. I'll try to find a relevant repro and get you the logs you seek.

Revision history for this message
Jason Zions (jasonzio) wrote :

Per your request, the results of collect-logs

Revision history for this message
Jason Zions (jasonzio) wrote :

The contents of /var/log/messages on this CentOS system, corresponding to the VM start-after-deallocate operation covered by the previously-uploaded collect-logs bundle. In particular, see Apr 23 00:06:05 (date/times in UTC) for the attempt to mount the NTFS ephemeral volume. Meanwhile, in cloud-init.log, you'll see that cc_disk_setup doesn't run until 00:06:27.

When this VM was first created, /var/log/messages said nothing at all about /dev/disk/cloud/azure_resource-part1 or about NTFS, which is exactly as one would expect for the first-time-ever boot of the VM.

Jason Zions (jasonzio)
Changed in cloud-init:
status: Incomplete → New
Ryan Harper (raharper)
Changed in cloud-init:
status: New → In Progress
Revision history for this message
Server Team CI bot (server-team-bot) wrote :

This bug is fixed with commit acc25d8d to cloud-init on branch master.
To view that commit see the following URL:
https://git.launchpad.net/cloud-init/commit/?id=acc25d8d

Changed in cloud-init:
status: In Progress → Fix Committed
Revision history for this message
Chad Smith (chad.smith) wrote : Fixed in cloud-init version 19.1.

This bug is believed to be fixed in cloud-init in version 19.1. If this is still a problem for you, please make a comment and set the state back to New

Thank you.

Changed in cloud-init:
status: Fix Committed → Fix Released
Roufique Hossain (roufique)
Changed in cloud-init:
assignee: nobody → Roufique Hossain (roufique)
Steve Langasek (vorlon)
Changed in cloud-init:
assignee: Roufique Hossain (roufique) → nobody
Revision history for this message
James Falcon (falcojr) wrote :

Tracked in Github Issues as https://github.com/canonical/cloud-init/issues/3370

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.