IBMCloud datasource does not recognize provisioning in debug mode.

Bug #1767166 reported by Scott Moser
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
cloud-init
Fix Released
Medium
Unassigned
cloud-init (Ubuntu)
Fix Released
Low
Unassigned
Xenial
Fix Released
Medium
Unassigned
Artful
Fix Released
Medium
Unassigned
Bionic
Fix Released
Low
Unassigned
Cosmic
Fix Committed
Low
Unassigned

Bug Description

=== Begin SRU Template ===
[Impact]
Cloud-init is disabled in the provisioning state. If provisioning
artifacts are left around after debug mode, cloud-init remains disabled
and doesn't properly configure the instance.

This issue only affects images that are being tested by IBM before official
publication. Once officially published, the images will have a 'production'
tag, and bug does not reproduce.

As such, it is believed that a regular end user is not really able to produce.

[Test Case]
cat > sethostname.yaml <<EOF
#cloud-config
runcmd:
 - [sh, '-c', 'cat /proc/uptime > /run/runcmd-ran.txt']
EOF

VM_IP=`launch-softlayer --pubkey-file ~/.ssh/id_rsa.pub -u sethostname.yaml -i os:xenial | awk '/primary ip/{printf "root@%s", $3}'`

ssh root@$VM_IP -- dpkg-query --show cloud-init;
ssh root@$VM_IP -- cloud-init status --long;
ssh root@$VM_IP -- cloud-init analyze show;
ssh root@$VM_IP -- sh -c '
  mirror=http://archive.ubuntu.com/ubuntu
  echo deb $mirror $(lsb_release -sc)-proposed main | tee /etc/apt/sources.list.d/proposed.list
  apt-get update -q
  apt-get install -qy cloud-init';
ssh root@$VM_IP -- DEBUG_LEVEL=2 DI_LOG=stderr /usr/lib/cloud-init/ds-identify --force 2>&1 | grep provision
ssh root@$VM_IP -- cloud-init clean --logs --reboot;
ssh root@$VM_IP -- egrep 'provisioning|swinstall' /var/log/cloud-init.log
ssh root@$VM_IP -- grep provision /run/cloud-init/ds-identify.log

[Regression Potential]
Regression will still be limited to softlayer instances as code changes are
limited to softlayer datasource detection in ds-identify and
DataSourceIBMCloud.

[Other Info]
Upstream commit at
  https://git.launchpad.net/cloud-init/commit/?id=44a44ae18

This bug is currently fixed in bionic-proposed version
(18.2-27-g6ef92c98-0ubuntu1~18.04.1) and
cloud-init trunk, so first upload to ubuntu 'cc' will have it fixed.

=== End SRU Template ===

When IBMCloud deploys from a template, artifacts from the
provisioning stage are normally cleaned up. Cloud-init relied'
on that behavior to determine the provisioning boot from the subsequent
post-provisioning boot.

However, when testing, the provisioning stage will leave artifacts
in place (/root/provisioningConfiguration.cfg). This caused cloud-init
to permenantly believe that it was in the provisioning stage.

Related branches

Scott Moser (smoser)
Changed in cloud-init:
status: New → Confirmed
importance: Undecided → Medium
Changed in cloud-init (Ubuntu):
status: New → Confirmed
importance: Undecided → Medium
Revision history for this message
Ryan Harper (raharper) wrote :

An upstream commit landed for this bug.

To view that commit see the following URL:
https://git.launchpad.net/cloud-init/commit/?id=6ef92c98

Changed in cloud-init:
status: Confirmed → Fix Committed
Scott Moser (smoser)
Changed in cloud-init (Ubuntu Bionic):
importance: Medium → Low
status: Confirmed → Fix Committed
Changed in cloud-init (Ubuntu Xenial):
status: New → In Progress
importance: Undecided → Medium
Changed in cloud-init (Ubuntu Artful):
status: New → Confirmed
importance: Undecided → Medium
Scott Moser (smoser)
description: updated
Revision history for this message
Chris Halse Rogers (raof) wrote : Please test proposed package

Hello Scott, or anyone else affected,

Accepted cloud-init into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cloud-init/18.2-4-g05926e48-0ubuntu1~16.04.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in cloud-init (Ubuntu Xenial):
status: In Progress → Fix Committed
tags: added: verification-needed verification-needed-xenial
Changed in cloud-init (Ubuntu Artful):
status: Confirmed → Fix Committed
tags: added: verification-needed-artful
Revision history for this message
Chris Halse Rogers (raof) wrote :

Hello Scott, or anyone else affected,

Accepted cloud-init into artful-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cloud-init/18.2-4-g05926e48-0ubuntu1~17.10.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-artful to verification-done-artful. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-artful. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Revision history for this message
Chad Smith (chad.smith) wrote :
Download full text (31.9 KiB)

Validated against Xenial and Artful, while IBMCloud doesn't have any publicly available artful images, we've contrived an example upgrade from public xenial instance to artful to ensure it doesn't break.

Abridged validation logs below

=== BEGIN SRU Validation Xenial ===
root@publishing:~# cat > runcmd.yaml <<EOF
> #cloud-config
> hostname: SRU-worked
> runcmd:
> - [sh, '-c', 'cat /proc/uptime > /run/runcmd-ran.txt']
> EOF
root@publishing:~# VM_IP=`launch-softlayer --pubkey-file ~/.ssh/id_rsa.pub -u runcmd.yaml -i xenial | awk '/primary ip/{printf "root@%s", $3}'`
Destroy instance with:
  slcli virtual cancel 54555262
root@publishing:~# ssh $VM_IP -- cat /run/cloud-init/result.json
{
 "v1": {
  "datasource": "DataSourceConfigDrive [net,ver=2][source=/var/lib/cloud/seed/config_drive]",
  "errors": []
 }
}
root@publishing:~# ssh $VM_IP -- cloud-init analyze show;
-- Boot Record 01 --
The total time elapsed since completing an event is printed after the "@" character.
The time the event takes is printed after the "+" character.

Starting stage: init-local
|`->no cache found @00.01600s +00.00000s
|`->found local data from DataSourceConfigDrive @00.05300s +00.00500s
Finished stage: (init-local) 00.24200 seconds

Starting stage: init-network
|`->restored from cache with run check: DataSourceConfigDrive [net,ver=2][source=/var/lib/cloud/seed/config_drive] @01.00900s +00.00500s
|`->reading and applying user-data @01.10100s +00.00500s
|`->reading and applying vendor-data @01.10600s +00.00000s
|`->config-migrator ran successfully @01.42800s +00.00000s
|`->config-seed_random ran successfully @01.42900s +00.00500s
|`->config-bootcmd ran successfully @01.43400s +00.00100s
|`->config-write-files ran successfully @01.43500s +00.00100s
|`->config-growpart ran successfully @01.43700s +00.24800s
|`->config-resizefs ran successfully @01.68600s +00.04900s
|`->config-disk_setup ran successfully @01.73500s +00.00200s
|`->config-mounts ran successfully @01.73700s +00.22200s
|`->config-set_hostname ran successfully @01.96000s +00.01500s
|`->config-update_hostname ran successfully @01.97600s +00.00200s
|`->config-update_etc_hosts ran successfully @01.97800s +00.00200s
|`->config-ca-certs ran successfully @01.98000s +00.00100s
|`->config-rsyslog ran successfully @01.98200s +00.00100s
|`->config-users-groups ran successfully @01.98300s +00.06100s
|`->config-ssh ran successfully @02.04400s +00.58000s
Finished stage: (init-network) 01.64600 seconds

Starting stage: modules-config
|`->config-emit_upstart ran successfully @05.04800s +00.00000s
|`->config-snap_config ran successfully @05.04800s +00.00100s
|`->config-ssh-import-id ran successfully @05.05000s +00.00100s
|`->config-locale ran successfully @05.05100s +01.30800s
|`->config-set-passwords ran successfully @06.36000s +00.08500s
|`->config-grub-dpkg ran successfully @06.44500s +00.50100s
|`->config-apt-pipelining ran successfully @06.94600s +00.00400s
|`->config-apt-configure ran successfully @06.95000s +00.19600s
|`->config-ntp ran successfully @07.14700s +00.00100s
|`->config-timezone ran successfully @07.14900s +00.00100s
|`->config-disable-ec2-metadata ran successfully @07.15000s +00.00000s
|`->config-runcm...

tags: added: verification-done verification-done-artful verification-done-xenial
removed: verification-needed verification-needed-artful verification-needed-xenial
Revision history for this message
Chad Smith (chad.smith) wrote :
Download full text (196.7 KiB)

Additionally ran an integration test on proposed which covers lxd kvm and ec2 platforms with no errors:

Results available here

 https://jenkins.ubuntu.com/server/view/cloud-init/job/cloud-init-integration-proposed-x/10/consoleFull

Paste run verify details below for reference in case the job results are removed
jenkins@torkoal:/var/lib/jenkins/slaves/torkoal/workspace/cloud-init-integration-proposed-x/cloud-init$ .tox/citest/bin/python -m tests.cloud_tests verify -d ./results/ --verbose
2018-05-07 16:25:53,754 - tests.cloud_tests - DEBUG - running with args: Namespace(data_dir='/var/lib/jenkins/slaves/torkoal/workspace/cloud-init-integration-proposed-x/cloud-init/results', preserve_data=False, quiet=False, result=None, subcmd='verify', verbose=True)
2018-05-07 16:25:53,756 - tests.cloud_tests - DEBUG - found test data: {'nocloud-kvm': {'xenial': ['bugs/lp1628337', 'modules/apt_configure_sources_list', 'modules/locale', 'modules/apt_configure_primary', 'modules/lxd_bridge', 'modules/apt_configure_security', 'modules/apt_pipelining_os', 'modules/snap', 'modules/ssh_import_id', 'modules/ntp_pools', 'modules/ssh_keys_generate', 'modules/ssh_auth_key_fingerprints_enable', 'modules/apt_configure_sources_keyserver', 'modules/bootcmd', 'modules/set_hostname_fqdn', 'modules/ssh_auth_key_fingerprints_disable', 'modules/final_message', 'modules/apt_configure_sources_key', 'modules/ntp_servers', 'modules/snappy', 'modules/write_files', 'modules/set_password', 'modules/ntp', 'modules/timezone', 'modules/keys_to_console', 'modules/debug_enable', 'modules/seed_random_data', 'modules/set_password_list', 'modules/user_groups', 'modules/debug_disable', 'modules/salt_minion', 'modules/byobu', 'modules/apt_configure_proxy', 'modules/apt_configure_sources_ppa', 'modules/package_update_upgrade_install', 'modules/lxd_dir', 'modules/apt_pipelining_disable', 'modules/runcmd', 'modules/set_hostname', 'modules/apt_configure_conf', 'modules/apt_configure_disable_suites', 'modules/set_password_list_string', 'modules/set_password_expire', 'modules/ca_certs', 'main/command_output_simple']}, 'ec2': {'xenial': ['bugs/lp1628337', 'modules/apt_configure_sources_list', 'modules/locale', 'modules/apt_configure_primary', 'modules/lxd_bridge', 'modules/apt_configure_security', 'modules/apt_pipelining_os', 'modules/snap', 'modules/ssh_import_id', 'modules/ntp_pools', 'modules/ssh_keys_generate', 'modules/ssh_auth_key_fingerprints_enable', 'modules/apt_configure_sources_keyserver', 'modules/bootcmd', 'modules/set_hostname_fqdn', 'modules/ssh_auth_key_fingerprints_disable', 'modules/final_message', 'modules/apt_configure_sources_key', 'modules/ntp_servers', 'modules/snappy', 'modules/write_files', 'modules/set_password', 'modules/ntp', 'modules/timezone', 'modules/keys_to_console', 'modules/debug_enable', 'modules/seed_random_data', 'modules/set_password_list', 'modules/user_groups', 'modules/debug_disable', 'modules/salt_minion', 'modules/byobu', 'modules/apt_configure_proxy', 'modules/apt_configure_sources_ppa', 'modules/package_update_upgrade_install', 'modules/lxd_dir', 'modules/apt_pipelining_disable', 'modules/runcmd', 'modules/set_hostname', 'modules/apt_configure_conf', ...

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package cloud-init - 18.2-4-g05926e48-0ubuntu1~17.10.2

---------------
cloud-init (18.2-4-g05926e48-0ubuntu1~17.10.2) artful-proposed; urgency=medium

  * cherry-pick 11172924: IBMCloud: Disable config-drive and nocloud
    only if IBMCloud (LP: #1766401)
  * cherry-pick 6ef92c98: IBMCloud: recognize provisioning environment
    during debug (LP: #1767166)

 -- Chad Smith <email address hidden> Tue, 01 May 2018 10:36:14 -0600

Changed in cloud-init (Ubuntu Artful):
status: Fix Committed → Fix Released
Revision history for this message
Łukasz Zemczak (sil2100) wrote : Update Released

The verification of the Stable Release Update for cloud-init has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package cloud-init - 18.2-4-g05926e48-0ubuntu1~16.04.2

---------------
cloud-init (18.2-4-g05926e48-0ubuntu1~16.04.2) xenial-proposed; urgency=medium

  * cherry-pick 6ef92c98: IBMCloud: recognize provisioning environment
    during debug (LP: #1767166)
  * cherry-pick 11172924: IBMCloud: Disable config-drive and nocloud
    only if IBMCloud (LP: #1766401)

 -- Chad Smith <email address hidden> Mon, 30 Apr 2018 15:52:05 -0600

Changed in cloud-init (Ubuntu Xenial):
status: Fix Committed → Fix Released
tags: added: id-5af5b24a9d4d32fbfbbac8b0
Revision history for this message
Chad Smith (chad.smith) wrote : Fixed in cloud-init version 18.2-27-g6ef92c98-0ubuntu1~18.04.1.

This bug is believed to be fixed in cloud-init in version18.2-27-g6ef92c98-0ubuntu1~18.04.1. If this is still a problem for you, please make a comment and set the state back to New

Thank you.

Changed in cloud-init:
status: Fix Committed → Fix Released
Changed in cloud-init (Ubuntu Bionic):
status: Fix Committed → Fix Released
Revision history for this message
Scott Moser (smoser) wrote : Fixed in cloud-init version 18.3.

This bug is believed to be fixed in cloud-init in version 18.3. If this is still a problem for you, please make a comment and set the state back to New

Thank you.

Changed in cloud-init:
status: Fix Released → Fix Committed
status: Fix Committed → Fix Released
Revision history for this message
James Falcon (falcojr) wrote :
Chad Smith (chad.smith)
Changed in cloud-init (Ubuntu):
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.