AttributeError: 'DataSourceNoCloud' object has no attribute 'vendordata2_raw'

Oh, actually, this may be because of the mid-boot reboot that LXD agent installation now causes to happen.

(I can't reproduce this post-bionic, which lends credence to that hypothesis.)

I see reproduce this with the following:
lxc launch ubuntu-daily: xenial dev-x
lxc exec dev-x -- cloud-init status --wait --long # confirm no failures from status
lxc exec dev-x reboot
lxc exec dev-x -- cloud-init status --wait --long # confirm no failures from status
lxc exec dev-x -- add-apt-repository ppa:cloud-init-dev/daily
lxc exec dev-x -- apt-get update
lxc exec dev-x -- apt-get install cloud-init
lxc exec dev-x -- reboot
lxc exec dev-x -- cloud-init status --wait --long # See failure

That said, intermittent reproducer above. Second attempt didn't hit it.

I see this consistently on the two bare metal instances I have running 20.04.2 LTS (5.4.0-72-generic)

I have a PR here: https://github.com/canonical/cloud-init/pull/869

I'm probably hitting this as well.

dpkg.log:2021-04-16 09:46:40 upgrade cloud-init:all 20.4.1-0ubuntu1~18.04.1 21.1-19-gbad84ad4-0ubuntu1~18.04.1

Apr 17 06:59:03 cloud-init[580]: 2021-04-17 04:59:03,191 - util.py[WARNING]: failed stage init
Apr 17 06:59:03 cloud-init[580]: failed run of stage init
Apr 17 06:59:03 cloud-init[580]: ------------------------------------------------------------
Apr 17 06:59:03 cloud-init[580]: Traceback (most recent call last):
Apr 17 06:59:03 cloud-init[580]: File "/usr/lib/python3/dist-packages/cloudinit/cmd/main.py", line 652, in status_wrapper
Apr 17 06:59:03 cloud-init[580]: ret = functor(name, args)
Apr 17 06:59:03 cloud-init[580]: File "/usr/lib/python3/dist-packages/cloudinit/cmd/main.py", line 376, in main_init
Apr 17 06:59:03 cloud-init[580]: init.update()
Apr 17 06:59:03 cloud-init[580]: File "/usr/lib/python3/dist-packages/cloudinit/stages.py", line 371, in update
Apr 17 06:59:03 cloud-init[580]: self._store_raw_vendordata(self.datasource.get_vendordata2_raw(),
Apr 17 06:59:03 cloud-init[580]: File "/usr/lib/python3/dist-packages/cloudinit/sources/__init__.py", line 506, in get_vendordata2_raw
Apr 17 06:59:03 cloud-init[580]: return self.vendordata2_raw
Apr 17 06:59:03 cloud-init[580]: AttributeError: 'DataSourceHetzner' object has no attribute 'vendordata2_raw'
Apr 17 06:59:03 cloud-init[580]: ------------------------------------------------------------

Confirmed that removing/renaming obj.pkl appears to work around this issue without causing obvious problems. This was on Ubuntu 16.04.7 LTS, with DataSourceDigitalOcean.

A fix for this landed just now in https://github.com/canonical/cloud-init/commit/d132356cc361abef2d90d4073438f3ab759d5964

Status changed to 'Confirmed' because the bug affects multiple users.

Hello Dan, or anyone else affected,

Accepted cloud-init into groovy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cloud-init/21.1-19-gbad84ad4-0ubuntu1~20.10.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-groovy to verification-done-groovy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-groovy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Hello Dan, or anyone else affected,

Accepted cloud-init into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cloud-init/21.1-19-gbad84ad4-0ubuntu1~20.04.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Hello Dan, or anyone else affected,

Accepted cloud-init into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cloud-init/21.1-19-gbad84ad4-0ubuntu1~18.04.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Hello Dan, or anyone else affected,

Accepted cloud-init into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cloud-init/21.1-19-gbad84ad4-0ubuntu1~16.04.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

I originally filed https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/1924755 and got informed that was a duplicate of this bug. I just installed the proposed fix from https://launchpad.net/ubuntu/+source/cloud-init/21.1-19-gbad84ad4-0ubuntu1~16.04.2. I can confirm this fixes it for me on Xenial.

Thanks to all involved, regards.

Verified on 20.04.2 LTS focal.

Got this in the wild in /var/log/cloud-init.log after upgrading from 20.3-2-g371b392c-0ubuntu1~20.04.1 to 21.1-19-gbad84ad4-0ubuntu1~20.04.1:

2021-04-16 19:42:33,005 - util.py[WARNING]: failed stage init
2021-04-16 19:42:33,009 - util.py[DEBUG]: failed stage init
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/cloudinit/cmd/main.py", line 652, in stat
us_wrapper
    ret = functor(name, args)
  File "/usr/lib/python3/dist-packages/cloudinit/cmd/main.py", line 376, in main
_init
    init.update()
  File "/usr/lib/python3/dist-packages/cloudinit/stages.py", line 371, in update
    self._store_raw_vendordata(self.datasource.get_vendordata2_raw(),
  File "/usr/lib/python3/dist-packages/cloudinit/sources/__init__.py", line 506,
 in get_vendordata2_raw
    return self.vendordata2_raw
AttributeError: 'DataSourceDigitalOcean' object has no attribute 'vendordata2_ra
w'

Installed 21.1-19-gbad84ad4-0ubuntu1~20.04.2; rebooted. Confirmed no traceback in /var/log/cloud-init.log. cloud-init status --long reports:

status: done
time: Tue, 20 Apr 2021 23:32:42 +0000
detail:
DataSourceDigitalOcean

instead of an error as with the previous version.

Thanks!

Verified fixed for 20.04.2 LTS (5.4.0-72-generic)

I'm performing verification of this locally using the cloud-init integration testing framework. Specifically, I'm running test_upgrade_package[0] with the following diff applied (to trigger this bug):

@@ -104,18 +104,19 @@ def test_upgrade(session_cloud: IntegrationCloud):

 @pytest.mark.ci
 @pytest.mark.ubuntu
-def test_upgrade_package(session_cloud: IntegrationCloud):
- if get_validated_source(session_cloud) != CloudInitSource.DEB_PACKAGE:
- not_run_message = 'Test only supports upgrading to build deb'
+def test_subsequent_boot_of_upgraded_package(session_cloud: IntegrationCloud):
+ source = get_validated_source(session_cloud)
+ if not source.installs_new_version():
         if os.environ.get('TRAVIS'):
             # If this isn't running on CI, we should know
- pytest.fail(not_run_message)
+ pytest.fail(UNSUPPORTED_INSTALL_METHOD_MSG.format(source))
         else:
- pytest.skip(not_run_message)
+ pytest.skip(UNSUPPORTED_INSTALL_METHOD_MSG.format(source))
+ return # type checking doesn't understand that skip raises

     launch_kwargs = {'image_id': session_cloud.released_image_id}

     with session_cloud.launch(launch_kwargs=launch_kwargs) as instance:
- instance.install_deb()
+ instance.install_new_cloud_init(source, take_snapshot=False, clean=False)
         instance.restart()
         assert instance.execute('cloud-init status --wait --long').ok

The important changes here are that I can run it against both the release pocket and -proposed, and that it doesn't perform a clean any longer. I'll propose these changes to cloud-init upstream after validation is complete.

[0] https://github.com/canonical/cloud-init/blob/master/tests/integration_tests/test_upgrade.py#L105-L121

For groovy, I'm testing with the `ubuntu:bac1692e9ec7` image, with a serial of 20201210. First, I confirm that the test does trigger the bug on UPGRADE to the version of cloud-init in the release:

$ CLOUD_INIT_OS_IMAGE=ubuntu:bac1692e9ec7::ubuntu::groovy CLOUD_INIT_CLOUD_INIT_SOURCE=UPGRADE pytest tests/integration_tests/test_upgrade.py::test_subsequent_boot_of_upgraded_package
...
FAILED tests/integration_tests/test_upgrade.py::test_subsequent_boot_of_upgraded_package - assert False

and I then verify that the same test passes with the package in groovy-proposed:

$ CLOUD_INIT_OS_IMAGE=ubuntu:bac1692e9ec7::ubuntu::groovy CLOUD_INIT_CLOUD_INIT_SOURCE=PROPOSED pytest tests/integration_tests/test_upgrade.py::test_subsequent_boot_of_upgraded_package
...
PASSED

For focal, I'm performing the same process but with the `ubuntu:b321e3832dbb` image, serial of 20201210.

UPGRADE does fail:

$ CLOUD_INIT_OS_IMAGE=ubuntu:b321e3832dbb::ubuntu::focal CLOUD_INIT_CLOUD_INIT_SOURCE=UPGRADE pytest tests/integration_tests/test_upgrade.py::test_subsequent_boot_of_upgraded_package
...
FAILED tests/integration_tests/test_upgrade.py::test_subsequent_boot_of_upgraded_package - assert False

and PROPOSED passes:

$ CLOUD_INIT_OS_IMAGE=ubuntu:b321e3832dbb::ubuntu::focal CLOUD_INIT_CLOUD_INIT_SOURCE=PROPOSED pytest tests/integration_tests/test_upgrade.py::test_subsequent_boot_of_upgraded_package
...
PASSED

For bionic, I'm performing the same process but with the `ubuntu:c2bdb694ecc2` image, serial of 20201211.1.

UPGRADE does fail:

$ CLOUD_INIT_OS_IMAGE=ubuntu:c2bdb694ecc2::ubuntu::bionic CLOUD_INIT_CLOUD_INIT_SOURCE=UPGRADE pytest tests/integration_tests/test_upgrade.py::test_subsequent_boot_of_upgraded_package
...
FAILED tests/integration_tests/test_upgrade.py::test_subsequent_boot_of_upgraded_package - assert False

$ CLOUD_INIT_OS_IMAGE=ubuntu:c2bdb694ecc2::ubuntu::bionic CLOUD_INIT_CLOUD_INIT_SOURCE=PROPOSED pytest tests/integration_tests/test_upgrade.py::test_subsequent_boot_of_upgraded_package
...
PASSED

For xenial, I'm performing the same process but with the `ubuntu:bb8e87956495` image, serial of 20201210.

UPGRADE does fail:

$ CLOUD_INIT_OS_IMAGE=ubuntu:bb8e87956495::ubuntu::xenial CLOUD_INIT_CLOUD_INIT_SOURCE=UPGRADE pytest tests/integration_tests/test_upgrade.py::test_subsequent_boot_of_upgraded_package
...
FAILED tests/integration_tests/test_upgrade.py::test_subsequent_boot_of_upgraded_package - assert False

$ CLOUD_INIT_OS_IMAGE=ubuntu:bb8e87956495::ubuntu::xenial CLOUD_INIT_CLOUD_INIT_SOURCE=PROPOSED pytest tests/integration_tests/test_upgrade.py::test_subsequent_boot_of_upgraded_package
...
PASSED

This bug was fixed in the package cloud-init - 21.1-19-gbad84ad4-0ubuntu3

---------------
cloud-init (21.1-19-gbad84ad4-0ubuntu3) hirsute; urgency=medium

  * cherry-pick 83f6bbfb: Fix unpickle for source paths missing run_dir
    (#863) (LP: #1899299)
  * cherry-pick d132356c: fix error on upgrade caused by new vendordata2
    attributes (LP: #1922739)

 -- James Falcon <email address hidden> Mon, 19 Apr 2021 13:41:21 -0500

For hirsute, the bug does not reproduce on upgrade from the release day image. However, it can present when upgrading between releases.

To test, I launched a groovy instance with an old cloud-init (the same image as previously for groovy validation).

I performed a `do-release-upgrade -d` (-d, as upgrades to hirsute are not yet enabled) and rebooted: I saw the Traceback from this bug, as expected.

I then launched another groovy instance, performed another release upgrade but, before rebooting, installed cloud-init from hirsute-proposed. On reboot, I then did not see the Traceback for this bug.

For regression testing, I ran the suite of cloud-init integration tests against hirsute.

This bug was fixed in the package cloud-init - 21.1-19-gbad84ad4-0ubuntu3

---------------
cloud-init (21.1-19-gbad84ad4-0ubuntu3) hirsute; urgency=medium

  * cherry-pick 83f6bbfb: Fix unpickle for source paths missing run_dir
    (#863) (LP: #1899299)
  * cherry-pick d132356c: fix error on upgrade caused by new vendordata2
    attributes (LP: #1922739)

 -- James Falcon <email address hidden> Mon, 19 Apr 2021 13:41:21 -0500

The verification of the Stable Release Update for cloud-init has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

This bug was fixed in the package cloud-init - 21.1-19-gbad84ad4-0ubuntu1~20.10.2

---------------
cloud-init (21.1-19-gbad84ad4-0ubuntu1~20.10.2) groovy; urgency=medium

  * cherry-pick 83f6bbfb: Fix unpickle for source paths missing run_dir
    (#863) (LP: #1899299)
  * cherry-pick d132356c: fix error on upgrade caused by new vendordata2
    attributes (LP: #1922739)

 -- James Falcon <email address hidden> Mon, 19 Apr 2021 13:58:19 -0500

This bug was fixed in the package cloud-init - 21.1-19-gbad84ad4-0ubuntu1~20.04.2

---------------
cloud-init (21.1-19-gbad84ad4-0ubuntu1~20.04.2) focal; urgency=medium

  * cherry-pick 83f6bbfb: Fix unpickle for source paths missing run_dir
    (#863) (LP: #1899299)
  * cherry-pick d132356c: fix error on upgrade caused by new vendordata2
    attributes (LP: #1922739)

 -- James Falcon <email address hidden> Mon, 19 Apr 2021 14:00:12 -0500

This bug was fixed in the package cloud-init - 21.1-19-gbad84ad4-0ubuntu1~18.04.2

---------------
cloud-init (21.1-19-gbad84ad4-0ubuntu1~18.04.2) bionic; urgency=medium

  * cherry-pick 83f6bbfb: Fix unpickle for source paths missing run_dir
    (#863) (LP: #1899299)
  * cherry-pick d132356c: fix error on upgrade caused by new vendordata2
    attributes (LP: #1922739)

 -- James Falcon <email address hidden> Mon, 19 Apr 2021 14:02:27 -0500

This bug was fixed in the package cloud-init - 21.1-19-gbad84ad4-0ubuntu1~16.04.2

---------------
cloud-init (21.1-19-gbad84ad4-0ubuntu1~16.04.2) xenial; urgency=medium

  * cherry-pick 83f6bbfb: Fix unpickle for source paths missing run_dir
    (#863) (LP: #1899299)
  * cherry-pick d132356c: fix error on upgrade caused by new vendordata2
    attributes (LP: #1922739)

 -- James Falcon <email address hidden> Mon, 19 Apr 2021 14:06:47 -0500

This bug is believed to be fixed in cloud-init in version 21.2. If this is still a problem for you, please make a comment and set the state back to New

Thank you.

Tracked in Github Issues as https://github.com/canonical/cloud-init/issues/3864