fs_setup always creates new filesystem with partition 'auto'

Bug #1634678 reported by Lee Faris
16
This bug affects 3 people
Affects Status Importance Assigned to Milestone
cloud-init
Fix Released
Medium
Jonathan Ballet
cloud-init (Ubuntu)
Fix Released
Medium
Unassigned
Trusty
Confirmed
Low
Unassigned
Xenial
Fix Released
Medium
Unassigned
Yakkety
Fix Released
Medium
Unassigned

Bug Description

=== Begin SRU Template ===
[Impact]
On instance first boot cloud-init may create a filesystem when
it should have re-used an existing filesystem.

[Test Case]
The test case launches an instance, assuming that has an old version
of cloud-init inside. The user-data will not be valid for the default
configuration of disks. (Most openstack instances would have a
single 'ephemeral' disk in addition to root, with an ext4 filesystem
labelled 'ephemeral0'). We will then upgrade instance to proposed.
And create a filesystem on /dev/vdb1 that *should* match.

1. launch an instance in openstack with the following user-data.
   |#cloud-config
   |fs_setup:
   | - label: mydata
   | device: /dev/vdb
   | filesystem: ext4
   | partition: auto
   |mounts:
   | - ["/dev/vdb1", "/mnt"]

   $ cat > my-userdata.txt <<EOF
   #cloud-config
   fs_setup:
     - label: mydata
       device: /dev/vdb
       filesystem: ext4
       partition: auto
   mounts:
     - ["/dev/vdb1", "/mnt"]
   EOF

   $ openstack server create --user-data=my-userdata.txt \
      --key-name=brickies --flavor=m1.small .... my-test

2. ssh in, prepare the /dev/vdb to have a partition, and upgrade

  # run attached 'disk-setup'. This will partition the disk
  # and wipe any filesystem data off, basically making it a partitioned
  # but otherwise empty disk.
  $ sudo ./disk-setup
  umount: /mnt: not mounted
  wiping /dev/vdb
  partitioning /dev/vdb
  /dev/vdb: PTUUID="9920db9b-a1ff-4f44-834c-8fcc42bd5821" PTTYPE="gpt"
  /dev/vdb1: PARTUUID="1accde8d-8880-4a93-913a-61eee2e92535"

3. enable proposed, upgrade
4. clean out state and reboot
   sudo rm -Rf /var/lib/cloud /var/log/cloud-init*
   sudo sed -i '/comment=cloudconfig/d' /etc/fstab
   sudo reboot

5. ssh back in and look around.

   # cloud-init should have created a filesystem on /dev/vdb1
   # and mounted it at /mnt.

   $ grep /mnt /proc/mounts
   /dev/vdb1 /mnt ext4 rw,relatime,data=ordered 0 0

   # and have a filesystem 'mydata'
   $ sudo blkid /dev/vdb1
   /dev/vdb1: LABEL="mydata" UUID="79090091-800e-4348-a7a9-8c7a26ed18f7" TYPE="ext4" PARTUUID="b8ef8d3f-acb4-4bd0-ba2d-fdfb45c2e8f0"

   # put a file on there, then clean up and reboot.
   # we will expect that that this time, cloud-init will just re-use
   # the existing filesystem rather than making another.
   $ echo hi mom | sudo tee -a /mnt/my-important-data.txt

6. ssh in and expect /mnt/my-important-data.txt

   $ cat /mnt/my-important-data.txt
   hi mom

[Regression Potential]
Potentially this could re-use a partition that the user wanted reformatted.

[Other Info]
Upstream commit:
 https://git.launchpad.net/cloud-init/commit/?id=4a2b2f87ec48c2

=== End SRU Template ===

# cloud-init -v
cloud-init 0.7.5

# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 14.04.4 LTS
Release: 14.04
Codename: trusty

AMI: ami-1721ff77 - Ubuntu 14.04 20160314

fs_setup fails to detect an existing filesystem and creates a new one when using the following configuration:

fs_setup:
    label: None
    filesystem: ext4
    device: /dev/xvdf
    partition: auto

There error seems to be here - https://git.launchpad.net/cloud-init/tree/cloudinit/config/cc_disk_setup.py#n216
This line sets definition['partition'] to None.

I believe " definition['partition'] = part" should be a part of the above conditional.

Related branches

Revision history for this message
Lee Faris (leelynnef) wrote :
Revision history for this message
Jonathan Ballet (multani) wrote :

I'm facing the same problem here.

I'm trying to get this configuration into cloud-init:

    disk_setup:
      /dev/xvdb:
        overwrite: false
        layout: true
    fs_setup:
      - device: /dev/xvdb1
        filesystem: ext4
        label: test
        overwrite: false
        partition: auto

For the context, it's applied on a EBS disk on Amazon, and my hope was to be able to keep the content of the partition accross different machines.

I've tested @leelynnef 's patch and it fixes the problem for me.

Scott Moser (smoser)
Changed in cloud-init:
importance: Undecided → Medium
status: New → Confirmed
assignee: nobody → Jonathan Ballet (multani)
Scott Moser (smoser)
Changed in cloud-init:
status: Confirmed → Fix Committed
Changed in cloud-init (Ubuntu):
status: New → Confirmed
importance: Undecided → Medium
status: Confirmed → Fix Released
Changed in cloud-init (Ubuntu Trusty):
status: New → Confirmed
Changed in cloud-init (Ubuntu Xenial):
status: New → Confirmed
Changed in cloud-init (Ubuntu Yakkety):
status: New → Confirmed
Changed in cloud-init (Ubuntu Trusty):
importance: Undecided → Low
Changed in cloud-init (Ubuntu Xenial):
importance: Undecided → Medium
Changed in cloud-init (Ubuntu Yakkety):
importance: Undecided → Medium
Scott Moser (smoser)
description: updated
Revision history for this message
Brian Murray (brian-murray) wrote : Please test proposed package

Hello Lee, or anyone else affected,

Accepted cloud-init into yakkety-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cloud-init/0.7.9-90-g61eb03fe-0ubuntu1~16.10.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in cloud-init (Ubuntu Yakkety):
status: Confirmed → Fix Committed
tags: added: verification-needed
Revision history for this message
Brian Murray (brian-murray) wrote :

Hello Lee, or anyone else affected,

Accepted cloud-init into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cloud-init/0.7.9-90-g61eb03fe-0ubuntu1~16.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in cloud-init (Ubuntu Xenial):
status: Confirmed → Fix Committed
Revision history for this message
Scott Moser (smoser) wrote :
Revision history for this message
Scott Moser (smoser) wrote :

$ dpkg-query --show cloud-init
cloud-init 0.7.9-90-g61eb03fe-0ubuntu1~16.10.1

$ grep /mnt /proc/mounts
/dev/vdb1 /mnt ext4 rw,relatime,data=ordered 0 0

$ sudo blkid /dev/vdb1
/dev/vdb1: LABEL="mydata" UUID="20a0d834-2aca-450c-8d4d-a9e0a3fa74da" TYPE="ext4" PARTUUID="4d3c51ff-449e-4497-9ddc-54143b531095"

$ echo $(date -R) hi mom | sudo tee -a /mnt/my-important-data.txt
Mon, 17 Apr 2017 14:21:07 +0000 hi mom

$ sudo rm -Rf /var/lib/cloud /var/log/cloud-init*
$ sudo reboot

...
# ssh back in

$ grep cc_disk_setup.py /var/log/cloud-init.log
2017-04-17 14:22:05,078 - stages.py[DEBUG]: Running module disk_setup (<module 'cloudinit.config.cc_disk_setup' from '/usr/lib/python3/dist-packages/cloudinit/config/cc_disk_setup.py'>) with frequency once-per-instance
2017-04-17 14:22:05,080 - cc_disk_setup.py[DEBUG]: setting up filesystems: [{'label': 'mydata', 'partition': 'auto', 'device': '/dev/vdb', 'filesystem': 'ext4'}]
2017-04-17 14:22:05,166 - cc_disk_setup.py[DEBUG]: Creating new filesystem.
2017-04-17 14:22:05,166 - cc_disk_setup.py[DEBUG]: Checking /dev/vdb against default devices
2017-04-17 14:22:05,166 - cc_disk_setup.py[DEBUG]: Identifying device to create mydata filesytem on
2017-04-17 14:22:05,176 - cc_disk_setup.py[DEBUG]: Automatic device for /dev/vdb identified as /dev/vdb1
2017-04-17 14:22:05,176 - cc_disk_setup.py[DEBUG]: Found filesystem match, skipping formating.

$ cat /mnt/my-important-data.txt
Mon, 17 Apr 2017 14:21:07 +0000 hi mom

tags: added: verification-done-yakkety verification-needed-xenial
removed: verification-needed
Revision history for this message
Scott Moser (smoser) wrote :

## setup
$ git clone git://git.launchpad.net/~smoser/cloud-init/+git/sru-info
$ ./sru-info/bin/enable-proposed
$ ( sudo apt-get -q update && sudo apt-get install -qy cloud-init ) </dev/null
$ dpkg-query --show cloud-init
$ sudo ./sru-info/bugs/lp-1634678/disk-setup
$ sudo ./sru-info/bin/do-reboot clean save=orig

## ssh back in
$ grep cc_disk_setup.py /var/log/cloud-init.log
2017-04-17 14:43:48,488 - stages.py[DEBUG]: Running module disk_setup (<module 'cloudinit.config.cc_disk_setup' from '/usr/lib/python3/dist-packages/cloudinit/config/cc_disk_setup.py'>) with frequency once-per-instance
2017-04-17 14:43:48,491 - cc_disk_setup.py[DEBUG]: setting up filesystems: [{'partition': 'auto', 'label': 'mydata', 'device': '/dev/vdb', 'filesystem': 'ext4'}]
2017-04-17 14:43:48,521 - cc_disk_setup.py[DEBUG]: Creating new filesystem.
2017-04-17 14:43:48,522 - cc_disk_setup.py[DEBUG]: Checking /dev/vdb against default devices
2017-04-17 14:43:48,522 - cc_disk_setup.py[DEBUG]: Identifying device to create mydata filesytem on
2017-04-17 14:43:48,536 - cc_disk_setup.py[DEBUG]: Automatic device for /dev/vdb identified as /dev/vdb1
2017-04-17 14:43:48,536 - cc_disk_setup.py[DEBUG]: File system type 'ext4' with label 'mydata' will be created on /dev/vdb1
2017-04-17 14:43:48,542 - cc_disk_setup.py[DEBUG]: Creating file system mydata on /dev/vdb1
2017-04-17 14:43:48,542 - cc_disk_setup.py[DEBUG]: Using cmd: /sbin/mkfs.ext4 /dev/vdb1 -L mydata

$ grep /mnt /proc/mounts
/dev/vdb1 /mnt ext4 rw,relatime,data=ordered 0 0
$ sudo blkid /dev/vdb1
/dev/vdb1: LABEL="mydata" UUID="3f2d4ff8-2863-4842-b4ec-e54fc3de9a2f" TYPE="ext4" PARTUUID="9ebe6f52-30c4-4523-b771-e6857edbbd65"
$ echo $(date -R) hi mom | sudo tee -a /mnt/my-important-data.txt
Mon, 17 Apr 2017 14:45:27 +0000 hi mom
$ sudo ./sru-info/bin/do-reboot clean save=orig2

## one more time.
## we did not format this time.
$ grep cc_disk_setup.py /var/log/cloud-init.log
2017-04-17 14:46:22,927 - stages.py[DEBUG]: Running module disk_setup (<module 'cloudinit.config.cc_disk_setup' from '/usr/lib/python3/dist-packages/cloudinit/config/cc_disk_setup.py'>) with frequency once-per-instance
2017-04-17 14:46:22,929 - cc_disk_setup.py[DEBUG]: setting up filesystems: [{'filesystem': 'ext4', 'partition': 'auto', 'label': 'mydata', 'device': '/dev/vdb'}]
2017-04-17 14:46:22,948 - cc_disk_setup.py[DEBUG]: Creating new filesystem.
2017-04-17 14:46:22,949 - cc_disk_setup.py[DEBUG]: Checking /dev/vdb against default devices
2017-04-17 14:46:22,949 - cc_disk_setup.py[DEBUG]: Identifying device to create mydata filesytem on
2017-04-17 14:46:22,964 - cc_disk_setup.py[DEBUG]: Automatic device for /dev/vdb identified as /dev/vdb1
2017-04-17 14:46:22,964 - cc_disk_setup.py[DEBUG]: Found filesystem match, skipping formating.

$ cat /mnt/my-important-data.txt
Mon, 17 Apr 2017 14:45:27 +0000 hi mom

tags: added: verification-done-xenial
removed: verification-needed-xenial
Revision history for this message
Launchpad Janitor (janitor) wrote :
Download full text (3.4 KiB)

This bug was fixed in the package cloud-init - 0.7.9-90-g61eb03fe-0ubuntu1~16.10.1

---------------
cloud-init (0.7.9-90-g61eb03fe-0ubuntu1~16.10.1) yakkety; urgency=medium

  * debian/cloud-init.templates: add Bigstep to list of sources. (LP: #1676460)
  * New upstream snapshot.
    - OpenStack: add 'dvs' to the list of physical link types. (LP: #1674946)
    - Fix bug that resulted in an attempt to rename bonds or vlans.
      (LP: #1669860)
    - tests: update OpenNebula and Digital Ocean to not rely on host
      interfaces.
    - net: in netplan renderer delete known image-builtin content.
      (LP: #1675576)
    - doc: correct grammar in capabilities.rst [David Tagatac]
    - ds-identify: fix detecting of maas datasource. (LP: #1677710)
    - netplan: remove debugging prints, add debug logging [Ryan Harper]
    - ds-identify: do not write None twice to datasource_list.
    - support resizing partition and rootfs on system booted without
      initramfs. [Steve Langasek] (LP: #1677376)
    - apt_configure: run only when needed. (LP: #1675185)
    - OpenStack: identify OpenStack by product 'OpenStack Compute'.
      (LP: #1675349)
    - GCE: Search GCE in ds-identify, consider serial number in check.
      (LP: #1674861)
    - Add support for setting hashed passwords [Tore S. Lonoy] (LP: #1570325)
    - Fix filesystem creation when using "partition: auto"
      [Jonathan Ballet] (LP: #1634678)
    - ConfigDrive: support reading config drive data from /config-drive.
      (LP: #1673411)
    - ds-identify: fix detection of Bigstep datasource. (LP: #1674766)
    - test: add running of pylint [Joshua Powers]
    - ds-identify: fix bug where filename expansion was left on.
    - advertise network config v2 support (NETWORK_CONFIG_V2) in features.
    - Bigstep: fix bug when executing in python3. [root]
    - Fix unit test when running in a system deployed with cloud-init.
    - Bounce network interface for Azure when using the built-in path.
      [Brent Baude] (LP: #1674685)
    - cloudinit.net: add network config v2 parsing and rendering [Ryan Harper]
    - net: Fix incorrect call to isfile [Joshua Powers] (LP: #1674317)
    - net: add renderers for automatically selecting the renderer.
    - doc: fix config drive doc with regard to unpartitioned disks.
      (LP: #1673818)
    - test: Adding integratiron test for password as list [Joshua Powers]
    - render_network_state: switch arguments around, do not require target
    - support 'loopback' as a device type.
    - Integration Testing: improve testcase subclassing [Wesley Wiedenmeier]
    - gitignore: adding doc/rtd_html [Joshua Powers]
    - doc: add instructions for running integration tests via tox.
      [Joshua Powers]
    - test: avoid differences in 'date' output due to daylight savings.
    - Fix chef config module in omnibus install. [Jeremy Melvin] (LP: #1583837)
    - Add feature flags to cloudinit.version. [Wesley Wiedenmeier]
    - tox: add a citest environment
    - Support chpasswd/list being a list in addition to a string.
      [Sergio Lystopad] (LP: #1665694)
    - doc: Fix configuration example for cc_set_passwords module.
      [Sergio Lystopad] (LP: #1665773)
    - ...

Read more...

Changed in cloud-init (Ubuntu Yakkety):
status: Fix Committed → Fix Released
Revision history for this message
Steve Langasek (vorlon) wrote : Update Released

The verification of the Stable Release Update for cloud-init has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Revision history for this message
Launchpad Janitor (janitor) wrote :
Download full text (3.4 KiB)

This bug was fixed in the package cloud-init - 0.7.9-90-g61eb03fe-0ubuntu1~16.04.1

---------------
cloud-init (0.7.9-90-g61eb03fe-0ubuntu1~16.04.1) xenial-proposed; urgency=medium

  * debian/cloud-init.templates: add Bigstep to list of sources. (LP: #1676460)
  * New upstream snapshot.
    - OpenStack: add 'dvs' to the list of physical link types. (LP: #1674946)
    - Fix bug that resulted in an attempt to rename bonds or vlans.
      (LP: #1669860)
    - tests: update OpenNebula and Digital Ocean to not rely on host
      interfaces.
    - net: in netplan renderer delete known image-builtin content.
      (LP: #1675576)
    - doc: correct grammar in capabilities.rst [David Tagatac]
    - ds-identify: fix detecting of maas datasource. (LP: #1677710)
    - netplan: remove debugging prints, add debug logging [Ryan Harper]
    - ds-identify: do not write None twice to datasource_list.
    - support resizing partition and rootfs on system booted without
      initramfs. [Steve Langasek] (LP: #1677376)
    - apt_configure: run only when needed. (LP: #1675185)
    - OpenStack: identify OpenStack by product 'OpenStack Compute'.
      (LP: #1675349)
    - GCE: Search GCE in ds-identify, consider serial number in check.
      (LP: #1674861)
    - Add support for setting hashed passwords [Tore S. Lonoy] (LP: #1570325)
    - Fix filesystem creation when using "partition: auto"
      [Jonathan Ballet] (LP: #1634678)
    - ConfigDrive: support reading config drive data from /config-drive.
      (LP: #1673411)
    - ds-identify: fix detection of Bigstep datasource. (LP: #1674766)
    - test: add running of pylint [Joshua Powers]
    - ds-identify: fix bug where filename expansion was left on.
    - advertise network config v2 support (NETWORK_CONFIG_V2) in features.
    - Bigstep: fix bug when executing in python3. [root]
    - Fix unit test when running in a system deployed with cloud-init.
    - Bounce network interface for Azure when using the built-in path.
      [Brent Baude] (LP: #1674685)
    - cloudinit.net: add network config v2 parsing and rendering [Ryan Harper]
    - net: Fix incorrect call to isfile [Joshua Powers] (LP: #1674317)
    - net: add renderers for automatically selecting the renderer.
    - doc: fix config drive doc with regard to unpartitioned disks.
      (LP: #1673818)
    - test: Adding integratiron test for password as list [Joshua Powers]
    - render_network_state: switch arguments around, do not require target
    - support 'loopback' as a device type.
    - Integration Testing: improve testcase subclassing [Wesley Wiedenmeier]
    - gitignore: adding doc/rtd_html [Joshua Powers]
    - doc: add instructions for running integration tests via tox.
      [Joshua Powers]
    - test: avoid differences in 'date' output due to daylight savings.
    - Fix chef config module in omnibus install. [Jeremy Melvin] (LP: #1583837)
    - Add feature flags to cloudinit.version. [Wesley Wiedenmeier]
    - tox: add a citest environment
    - Support chpasswd/list being a list in addition to a string.
      [Sergio Lystopad] (LP: #1665694)
    - doc: Fix configuration example for cc_set_passwords module.
      [Sergio Lystopad] (LP: #1665773...

Read more...

Changed in cloud-init (Ubuntu Xenial):
status: Fix Committed → Fix Released
Revision history for this message
Scott Moser (smoser) wrote : Fixed in Cloud-init 17.1

This bug is believed to be fixed in cloud-init in 17.1. If this is still a problem for you, please make a comment and set the state back to New

Thank you.

Changed in cloud-init:
status: Fix Committed → Fix Released
Revision history for this message
James Falcon (falcojr) wrote :
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.