Ubuntu
cloud-init package

do-release-upgrade from Trusty to Xenial breaks cloud-init init-local on reboot

Bug #1899299 reported by Chad Smith
10
This bug affects 1 person
Affects Status Importance Assigned to Milestone
cloud-init (Ubuntu)
Fix Released
Undecided
Unassigned
Xenial
Fix Released
Undecided
Unassigned
Bionic
Fix Released
Undecided
Unassigned
Focal
Fix Released
Undecided
Unassigned
Groovy
Fix Released
Undecided
Unassigned
Hirsute
Fix Released
Undecided
Unassigned

Bug Description

=== Begin SRU Template ===
[Impact]
The 'Paths' class has a 'run_dir' attribute that was introduced since the release of Trusty. Because of this, performing a `do-release-upgrade` on Trusty followed by a reboot will have cloud-init fail with a traceback.

The change here should fix that failure.

[Test Plan]
- Launch a Trusty instance
- Perform a `do-release-upgrade`
- Reboot

On un-patched versions, `cloud-init status` will show error, and there will be a traceback in /var/log/cloud-init.log. Patched versions should show no error or traceback.

[Where problems could occur]
Any problem here would mean we're continuing to unpickle data incorrectly, thus we'd see similar behavior to the bug we're fixing.

[Other Info]
Pull request: https://github.com/canonical/cloud-init/pull/863
Commit: https://github.com/canonical/cloud-init/commit/83f6bbfbe5b924be61a3c098f4202377d69c8947

== End SRU Template ==

== Original Description ==

After performing a do-release-upgrade from stock trusty to xenial and rebooting cloud-init traces in it's attempt to preserve instance data:

Probably obj.pkl issues across upgrade and a need to refresh Paths object after the dist upgrade.

2020-10-10 18:01:02,377 - util.py[DEBUG]: Read 298 bytes from /etc/os-release
2020-10-10 18:01:02,378 - util.py[WARNING]: failed stage init-local
2020-10-10 18:01:02,379 - util.py[DEBUG]: failed stage init-local
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/cloudinit/cmd/main.py", line 653, in status_wrapper
    ret = functor(name, args)
  File "/usr/lib/python3/dist-packages/cloudinit/cmd/main.py", line 351, in main_init
    _maybe_persist_instance_data(init)
  File "/usr/lib/python3/dist-packages/cloudinit/cmd/main.py", line 695, in _maybe_persist_instance_data
    init.datasource.persist_instance_data()
  File "/usr/lib/python3/dist-packages/cloudinit/sources/__init__.py", line 332, in persist_instance_data
    json_sensitive_file = os.path.join(self.paths.run_dir,
AttributeError: 'Paths' object has no attribute 'run_dir'

ProblemType: Bug
DistroRelease: Ubuntu 16.04
Package: cloud-init 20.3-2-g371b392c-0ubuntu1~16.04.1
ProcVersionSignature: Ubuntu 5.4.0-48.52-generic 5.4.60
Uname: Linux 5.4.0-48-generic x86_64
ApportVersion: 2.20.1-0ubuntu2.25
Architecture: amd64
CloudName: LXD
Date: Sat Oct 10 21:22:46 2020
PackageArchitecture: all
ProcEnviron:
 TERM=xterm-256color
 PATH=(custom, no user)
 LANG=C.UTF-8
SourcePackage: cloud-init
UpgradeStatus: Upgraded to xenial on 2020-10-10 (0 days ago)
cloud-init-log-warnings:
 2020-10-10 18:01:02,378 - util.py[WARNING]: failed stage init-local
 AttributeError: 'Paths' object has no attribute 'run_dir'
 2020-10-10 18:01:02,979 - util.py[WARNING]: failed stage init
 AttributeError: 'Paths' object has no attribute 'run_dir'

See original description
Revision history for this message
Chad Smith (chad.smith) wrote :
Revision history for this message
Paride Legovini (paride) wrote :

I just noticed this affects torkoal (cloud-init 20.4.1-0ubuntu1~18.04.1). On reboot we have:

[ 36.428237] cloud-init[4810]: failed run of stage modules-config
[ 36.428373] cloud-init[4810]: ------------------------------------------------------------
[ 36.428434] cloud-init[4810]: Traceback (most recent call last):
[ 36.428509] cloud-init[4810]: File "/usr/lib/python3/dist-packages/cloudinit/cmd/main.py", line 653, in status_wrapper
[ 36.428590] cloud-init[4810]: ret = functor(name, args)
[ 36.428666] cloud-init[4810]: File "/usr/lib/python3/dist-packages/cloudinit/cmd/main.py", line 494, in main_modules
[ 36.428740] cloud-init[4810]: _maybe_persist_instance_data(init)
[ 36.428809] cloud-init[4810]: File "/usr/lib/python3/dist-packages/cloudinit/cmd/main.py", line 695, in _maybe_persist_instance_data
[ 36.428873] cloud-init[4810]: init.datasource.persist_instance_data()
[ 36.428938] cloud-init[4810]: File "/usr/lib/python3/dist-packages/cloudinit/sources/__init__.py", line 333, in persist_instance_data
[ 36.429003] cloud-init[4810]: json_sensitive_file = os.path.join(self.paths.run_dir,
[ 36.429067] cloud-init[4810]: AttributeError: 'Paths' object has no attribute 'run_dir'
[ 36.429140] cloud-init[4810]: ------------------------------------------------------------
[ 38.552480] cloud-init[4899]: failed run of stage modules-final
[ 38.552630] cloud-init[4899]: ------------------------------------------------------------
[ 38.552695] cloud-init[4899]: Traceback (most recent call last):
[ 38.552746] cloud-init[4899]: File "/usr/lib/python3/dist-packages/cloudinit/cmd/main.py", line 653, in status_wrapper
[ 38.552819] cloud-init[4899]: ret = functor(name, args)
[ 38.552866] cloud-init[4899]: File "/usr/lib/python3/dist-packages/cloudinit/cmd/main.py", line 494, in main_modules
[ 38.552922] cloud-init[4899]: _maybe_persist_instance_data(init)
[ 38.553001] cloud-init[4899]: File "/usr/lib/python3/dist-packages/cloudinit/cmd/main.py", line 695, in _maybe_persist_instance_data
[ 38.553057] cloud-init[4899]: init.datasource.persist_instance_data()
[ 38.553132] cloud-init[4899]: File "/usr/lib/python3/dist-packages/cloudinit/sources/__init__.py", line 333, in persist_instance_data
[ 38.553204] cloud-init[4899]: json_sensitive_file = os.path.join(self.paths.run_dir,
[ 38.553267] cloud-init[4899]: AttributeError: 'Paths' object has no attribute 'run_dir'
[ 38.553337] cloud-init[4899]: ------------------------------------------------------------

Revision history for this message
James Falcon (falcojr) wrote :

A fix for this can be found at https://github.com/canonical/cloud-init/commit/83f6bbfbe5b924be61a3c098f4202377d69c8947

Changed in cloud-init (Ubuntu Hirsute):
status: Triaged → Fix Committed
James Falcon (falcojr)
description: updated
Revision history for this message
Brian Murray (brian-murray) wrote : Please test proposed package

Hello Chad, or anyone else affected,

Accepted cloud-init into groovy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cloud-init/21.1-19-gbad84ad4-0ubuntu1~20.10.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-groovy to verification-done-groovy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-groovy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in cloud-init (Ubuntu Groovy):
status: New → Fix Committed
tags: added: verification-needed verification-needed-groovy
Revision history for this message
Brian Murray (brian-murray) wrote :

Hello Chad, or anyone else affected,

Accepted cloud-init into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cloud-init/21.1-19-gbad84ad4-0ubuntu1~20.04.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in cloud-init (Ubuntu Focal):
status: New → Fix Committed
tags: added: verification-needed-focal
Changed in cloud-init (Ubuntu Bionic):
status: New → Fix Committed
tags: added: verification-needed-bionic
Revision history for this message
Brian Murray (brian-murray) wrote :

Hello Chad, or anyone else affected,

Accepted cloud-init into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cloud-init/21.1-19-gbad84ad4-0ubuntu1~18.04.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in cloud-init (Ubuntu Xenial):
status: New → Fix Committed
tags: added: verification-needed-xenial
Revision history for this message
Brian Murray (brian-murray) wrote :

Hello Chad, or anyone else affected,

Accepted cloud-init into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cloud-init/21.1-19-gbad84ad4-0ubuntu1~16.04.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Revision history for this message
James Falcon (falcojr) wrote :

To trigger this bug on Xenial, I did the following:

- launch a trusty instance
- Perform 'do-release-upgrade' on the trusty instance
- Reboot when complete
- Verify WARNING and Traceback present in logs
- Remove the log and reboot again
- Verify WARNING and Traceback present after second boot

To verify the bug fixed, follow the above steps, then additionally:
- Upgrade the cloud-init version to proposed
- Remove the log then reboot
- Verify no WARNING or Traceback present after boot

Revision history for this message
James Falcon (falcojr) wrote :

Note that this bug is triggered only by an upgrade from trusty. This bug is not reproduceable on Bionic, Focal, or Groovy as none of those are supported upgrade paths from Trusty. For regression testing, I ran the suite of cloud-init integration tests for Xenial, Bionic, Focal, and Groovy.

tags: added: verification-done verification-done-bionic verification-done-focal verification-done-groovy verification-done-xenial
removed: verification-needed verification-needed-bionic verification-needed-focal verification-needed-groovy verification-needed-xenial
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package cloud-init - 21.1-19-gbad84ad4-0ubuntu3

---------------
cloud-init (21.1-19-gbad84ad4-0ubuntu3) hirsute; urgency=medium

  * cherry-pick 83f6bbfb: Fix unpickle for source paths missing run_dir
    (#863) (LP: #1899299)
  * cherry-pick d132356c: fix error on upgrade caused by new vendordata2
    attributes (LP: #1922739)

 -- James Falcon <email address hidden> Mon, 19 Apr 2021 13:41:21 -0500

Changed in cloud-init (Ubuntu):
status: Fix Committed → Fix Released
Revision history for this message
James Falcon (falcojr) wrote :

For Hirsute, this bug is not reproducible as it is only triggered by an upgrade from trusty. For regression testing, I ran the suite of cloud-init integration tests against hirsute.

Łukasz Zemczak (sil2100)
tags: added: verification-done-hirsute
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package cloud-init - 21.1-19-gbad84ad4-0ubuntu3

---------------
cloud-init (21.1-19-gbad84ad4-0ubuntu3) hirsute; urgency=medium

  * cherry-pick 83f6bbfb: Fix unpickle for source paths missing run_dir
    (#863) (LP: #1899299)
  * cherry-pick d132356c: fix error on upgrade caused by new vendordata2
    attributes (LP: #1922739)

 -- James Falcon <email address hidden> Mon, 19 Apr 2021 13:41:21 -0500

Changed in cloud-init (Ubuntu Hirsute):
status: Fix Committed → Fix Released
Revision history for this message
Łukasz Zemczak (sil2100) wrote : Update Released

The verification of the Stable Release Update for cloud-init has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package cloud-init - 21.1-19-gbad84ad4-0ubuntu1~20.10.2

---------------
cloud-init (21.1-19-gbad84ad4-0ubuntu1~20.10.2) groovy; urgency=medium

  * cherry-pick 83f6bbfb: Fix unpickle for source paths missing run_dir
    (#863) (LP: #1899299)
  * cherry-pick d132356c: fix error on upgrade caused by new vendordata2
    attributes (LP: #1922739)

 -- James Falcon <email address hidden> Mon, 19 Apr 2021 13:58:19 -0500

Changed in cloud-init (Ubuntu Groovy):
status: Fix Committed → Fix Released
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package cloud-init - 21.1-19-gbad84ad4-0ubuntu1~20.04.2

---------------
cloud-init (21.1-19-gbad84ad4-0ubuntu1~20.04.2) focal; urgency=medium

  * cherry-pick 83f6bbfb: Fix unpickle for source paths missing run_dir
    (#863) (LP: #1899299)
  * cherry-pick d132356c: fix error on upgrade caused by new vendordata2
    attributes (LP: #1922739)

 -- James Falcon <email address hidden> Mon, 19 Apr 2021 14:00:12 -0500

Changed in cloud-init (Ubuntu Focal):
status: Fix Committed → Fix Released
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package cloud-init - 21.1-19-gbad84ad4-0ubuntu1~18.04.2

---------------
cloud-init (21.1-19-gbad84ad4-0ubuntu1~18.04.2) bionic; urgency=medium

  * cherry-pick 83f6bbfb: Fix unpickle for source paths missing run_dir
    (#863) (LP: #1899299)
  * cherry-pick d132356c: fix error on upgrade caused by new vendordata2
    attributes (LP: #1922739)

 -- James Falcon <email address hidden> Mon, 19 Apr 2021 14:02:27 -0500

Changed in cloud-init (Ubuntu Bionic):
status: Fix Committed → Fix Released
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package cloud-init - 21.1-19-gbad84ad4-0ubuntu1~16.04.2

---------------
cloud-init (21.1-19-gbad84ad4-0ubuntu1~16.04.2) xenial; urgency=medium

  * cherry-pick 83f6bbfb: Fix unpickle for source paths missing run_dir
    (#863) (LP: #1899299)
  * cherry-pick d132356c: fix error on upgrade caused by new vendordata2
    attributes (LP: #1922739)

 -- James Falcon <email address hidden> Mon, 19 Apr 2021 14:06:47 -0500

Changed in cloud-init (Ubuntu Xenial):
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.