cc_growpart fails on yakkety

Bug #1619285 reported by Junien F
18
This bug affects 3 people
Affects Status Importance Assigned to Milestone
cloud-utils
Fix Released
High
Unassigned
cloud-utils (Ubuntu)
Fix Released
Undecided
Unassigned
util-linux (Ubuntu)
Fix Released
Critical
Unassigned

Bug Description

Hi,

cc_growpart fails on ppc64el, with the following messages :

Sep 1 12:06:58 ubuntu [CLOUDINIT] util.py[DEBUG]: Running command ['growpart', '--dry-run', '/dev/vda', '1'] with allowed return codes [0] (shell=False, capture=True)
Sep 1 12:06:58 ubuntu [CLOUDINIT] util.py[DEBUG]: Running command ['growpart', '/dev/vda', '1'] with allowed return codes [0] (shell=False, capture=True)
Sep 1 12:06:58 ubuntu [CLOUDINIT] util.py[WARNING]: Failed: growpart /dev/vda 1
Sep 1 12:06:58 ubuntu [CLOUDINIT] util.py[DEBUG]: Failed: growpart /dev/vda 1#012Traceback (most recent call last):#012 File "/usr/lib/python3/dist-packages/cloudinit/config/cc_growpart.py", line 109, in resize#012 util.subp(["growpart", diskdev, partnum])#012 File "/usr/lib/python3/dist-packages/cloudinit/util.py", line 1832, in subp#012 cmd=args)#012cloudinit.util.ProcessExecutionError: Unexpected error while running command.#012Command: ['growpart', '/dev/vda', '1']#012Exit code: 2#012Reason: -#012Stdout: 'FAILED: failed to resize\n'#012Stderr: "attempt to resize /dev/vda failed. sfdisk output below:\n| GPT PMBR size mismatch (4612095 != 41943039) will be corrected by w(rite).\n| Backup files:\n| PMBR (offset 0, size 512): /tmp/growpart.wr7bvu/orig.save-vda-0x00000000.bak\n| GPT Header (offset 512, size 512): /tmp/growpart.wr7bvu/orig.save-vda-0x00000200.bak\n| GPT Entries (offset 1024, size 16384): /tmp/growpart.wr7bvu/orig.save-vda-0x00000400.bak\n| \n| Disk /dev/vda: 20 GiB, 21474836480 bytes, 41943040 sectors\n| Units: sectors of 1 * 512 = 512 bytes\n| Sector size (logical/physical): 512 bytes / 512 bytes\n| I/O size (minimum/optimal): 512 bytes / 512 bytes\n| Disklabel type: gpt\n| Disk identifier: 5AF0CCE2-DDF5-4419-B991-213EE1F1B873\n| \n| Old situation:\n| \n| Device Start End Sectors Size Type\n| /dev/vda1 18432 4612062 4593631 2.2G Linux filesystem\n| /dev/vda2 2048 18431 16384 8M PowerPC PReP boot\n| \n| Partition table entries are not in disk order.\n| \n| >>> Script header accepted.\n| >>> Script header accepted.\n| >>> Script header accepted.\n| >>> Script header accepted.\n| >>> Script header accepted.\n| >>> Created a new GPT disklabel (GUID: 5AF0CCE2-DDF5-4419-B991-213EE1F1B873).\n| Created a new partition 1 of type 'Linux filesystem' and of size 20 GiB.\n| /dev/vda2: Created a new partition 2 of type 'PowerPC PReP boot' and of size 8 MiB.\n| /dev/vda3: \n| New situation:\n| \n| Device Start End Sectors Size Type\n| /dev/vda1 18432 41943006 41924575 20G Linux filesystem\n| /dev/vda2 2048 18431 16384 8M PowerPC PReP boot\n| \n| Partition table entries are not in disk order.\n| \n| The partition table has been altered.\n| Calling ioctl() to re-read partition table.\n| Re-reading the partition table failed.: Device or resource busy\n| The kernel still uses the old table. The new table will be used at the next reboot or after you run partprobe(8) or kpartx(8).\n***** WARNING: Resize failed, attempting to revert ******\n512+0 records in\n512+0 records out\n512 bytes copied, 0.000671621 s, 762 kB/s\n512+0 records in\n512+0 records out\n512 bytes copied, 0.00148717 s, 344 kB/s\n16384+0 records in\n16384+0 records out\n16384 bytes (16 kB, 16 KiB) copied, 0.0370011 s, 443 kB/s\n***** Appears to have gone OK ****\n"
Sep 1 12:06:58 ubuntu [CLOUDINIT] util.py[DEBUG]: resize_devices took 0.729 seconds

Running this command once the system is booted up appears to give the same output :

ubuntu@axino-test:~$ sudo growpart /dev/vda 1
attempt to resize /dev/vda failed. sfdisk output below:
| GPT PMBR size mismatch (4612095 != 41943039) will be corrected by w(rite).
| Backup files:
| PMBR (offset 0, size 512): /tmp/growpart.Je0SXE/orig.save-vda-0x00000000.bak
| GPT Header (offset 512, size 512): /tmp/growpart.Je0SXE/orig.save-vda-0x00000200.bak
| GPT Entries (offset 1024, size 16384): /tmp/growpart.Je0SXE/orig.save-vda-0x00000400.bak
|
| Disk /dev/vda: 20 GiB, 21474836480 bytes, 41943040 sectors
| Units: sectors of 1 * 512 = 512 bytes
| Sector size (logical/physical): 512 bytes / 512 bytes
| I/O size (minimum/optimal): 512 bytes / 512 bytes
| Disklabel type: gpt
| Disk identifier: 5AF0CCE2-DDF5-4419-B991-213EE1F1B873
|
| Old situation:
|
| Device Start End Sectors Size Type
| /dev/vda1 18432 4612062 4593631 2.2G Linux filesystem
| /dev/vda2 2048 18431 16384 8M PowerPC PReP boot
|
| Partition table entries are not in disk order.
|
| >>> Script header accepted.
| >>> Script header accepted.
| >>> Script header accepted.
| >>> Script header accepted.
| >>> Script header accepted.
| >>> Created a new GPT disklabel (GUID: 5AF0CCE2-DDF5-4419-B991-213EE1F1B873).
| Created a new partition 1 of type 'Linux filesystem' and of size 20 GiB.
| /dev/vda2: Created a new partition 2 of type 'PowerPC PReP boot' and of size 8 MiB.
| /dev/vda3:
| New situation:
|
| Device Start End Sectors Size Type
| /dev/vda1 18432 41943006 41924575 20G Linux filesystem
| /dev/vda2 2048 18431 16384 8M PowerPC PReP boot
|
| Partition table entries are not in disk order.
|
| The partition table has been altered.
| Calling ioctl() to re-read partition table.
| Re-reading the partition table failed.: Device or resource busy
| The kernel still uses the old table. The new table will be used at the next reboot or after you run partprobe(8) or kpartx(8).
FAILED: failed to resize
***** WARNING: Resize failed, attempting to revert ******
512+0 records in
512+0 records out
512 bytes copied, 0.00204201 s, 251 kB/s
512+0 records in
512+0 records out
512 bytes copied, 0.00512669 s, 99.9 kB/s
16384+0 records in
16384+0 records out
16384 bytes (16 kB, 16 KiB) copied, 0.0369471 s, 443 kB/s
***** Appears to have gone OK ****

Apparently this only affects yakkety.
ii cloud-guest-utils 0.29-0ubuntu2 all cloud guest utilities
ii cloud-init 0.7.7-22-g763f403-0ubun all Init scripts for cloud instances
ii util-linux 2.28.1-1ubuntu1 ppc64el miscellaneous system utilities

Thank you

Related bugs:
 * bug 1618525: Failure to remove kernel partition mappings in yakkety (2.28.1-1ubuntu1)

Revision history for this message
Junien F (axino) wrote :
Download full text (3.5 KiB)

This also happens with yakkety amd64 :

Sep 1 13:04:31 ubuntu [CLOUDINIT] util.py[DEBUG]: Running command ['growpart', '--dry-run', '/dev/vda', '1'] with allowed return codes [0] (shell=False, capture=True)
Sep 1 13:04:31 ubuntu [CLOUDINIT] util.py[DEBUG]: Running command ['growpart', '/dev/vda', '1'] with allowed return codes [0] (shell=False, capture=True)
Sep 1 13:04:31 ubuntu [CLOUDINIT] util.py[WARNING]: Failed: growpart /dev/vda 1
Sep 1 13:04:31 ubuntu [CLOUDINIT] util.py[DEBUG]: Failed: growpart /dev/vda 1#012Traceback (most recent call last):#012 File "/usr/lib/python3/dist-packages/cloudinit/config/cc_growpart.py", line 109, in resize#012 util.subp(["growpart", diskdev, partnum])#012 File "/usr/lib/python3/dist-packages/cloudinit/util.py", line 1832, in subp#012 cmd=args)#012cloudinit.util.ProcessExecutionError: Unexpected error while running command.#012Command: ['growpart', '/dev/vda', '1']#012Exit code: 2#012Reason: -#012Stdout: 'FAILED: failed to resize\n'#012Stderr: "attempt to resize /dev/vda failed. sfdisk output below:\n| GPT PMBR size mismatch (4612095 != 41943039) will be corrected by w(rite).\n| Backup files:\n| PMBR (offset 0, size 512): /tmp/growpart.yWKkY4/orig.save-vda-0x00000000.bak\n| GPT Header (offset 512, size 512): /tmp/growpart.yWKkY4/orig.save-vda-0x00000200.bak\n| GPT Entries (offset 1024, size 16384): /tmp/growpart.yWKkY4/orig.save-vda-0x00000400.bak\n| \n| Disk /dev/vda: 20 GiB, 21474836480 bytes, 41943040 sectors\n| Units: sectors of 1 * 512 = 512 bytes\n| Sector size (logical/physical): 512 bytes / 512 bytes\n| I/O size (minimum/optimal): 512 bytes / 512 bytes\n| Disklabel type: gpt\n| Disk identifier: 1262F37F-7675-41D2-829F-7AC253FAB611\n| \n| Old situation:\n| \n| Device Start End Sectors Size Type\n| /dev/vda1 227328 4612062 4384735 2.1G Linux filesystem\n| /dev/vda14 2048 10239 8192 4M BIOS boot\n| /dev/vda15 10240 227327 217088 106M EFI System\n| \n| Partition table entries are not in disk order.\n| \n| >>> Script header accepted.\n| >>> Script header accepted.\n| >>> Script header accepted.\n| >>> Script header accepted.\n| >>> Script header accepted.\n| >>> Created a new GPT disklabel (GUID: 1262F37F-7675-41D2-829F-7AC253FAB611).\n| Created a new partition 1 of type 'Linux filesystem' and of size 19.9 GiB.\n| /dev/vda2: Created a new partition 14 of type 'BIOS boot' and of size 4 MiB.\n| /dev/vda15: Created a new partition 15 of type 'EFI System' and of size 106 MiB.\n| /dev/vda16: \n| New situation:\n| \n| Device Start End Sectors Size Type\n| /dev/vda1 227328 41943006 41715679 19.9G Linux filesystem\n| /dev/vda14 2048 10239 8192 4M BIOS boot\n| /dev/vda15 10240 227327 217088 106M EFI System\n| \n| Partition table entries are not in disk order.\n| \n| The partition table has been altered.\n| Calling ioctl() to re-read partition table.\n| Re-reading the partition table failed.: Device or resource busy\n| The kernel still uses the old table. The new table will be used at the next reboot or after you run partprobe(8) or kpartx(8).\n***** WARNING: Resize failed, attempting to revert ******\n512+0 records in\n512...

Read more...

description: updated
summary: - cc_growpart fails on ppc64el
+ cc_growpart fails on yakkety
Revision history for this message
Matthias Klose (doko) wrote :

this breaks all autopkg tests in yakkety which need more than the two initial 2GB

Changed in cloud-init (Ubuntu):
importance: Undecided → Critical
Revision history for this message
Martin Pitt (pitti) wrote :

Confirmed in local qemu. I built an image with autopkgtest-buildvm-ubuntu-cloud (which downloads the current yakkety image and boots it and does some initialization), same error there. So not related to Scalingstack.

Changed in cloud-init (Ubuntu):
importance: Critical → Undecided
status: New → Confirmed
Revision history for this message
Martin Pitt (pitti) wrote :

Possibly/likely related bug, or even a duplicate: bug 1618525

Revision history for this message
Scott Moser (smoser) wrote :

I launched an instance that had an older util-linux (2.28-5ubuntu2).
cp /sbin/sfdisk /sbin/sfdisk.old
mkdir /tmp/bin
ln -s /sbin/sfdisk.old /tmp/bin
sudo apt-get install util-linux

Then, i can see failure with the 'test-growpart' script at [1]
 sudo ./test-growpart
but this works as expected
 sudo PATH=/tmp/bin:$PATH ./test-growpart

So something in util-linux definitely changed behavior and broke this. I suspect this is common code to the kpartx failure in bug 1618525.

[1] http://bazaar.launchpad.net/~cloud-utils-dev/cloud-utils/trunk/view/head:/test/test-growpart

Changed in util-linux (Ubuntu):
status: New → Confirmed
importance: Undecided → Critical
description: updated
Scott Moser (smoser)
Changed in cloud-utils (Ubuntu):
status: New → Confirmed
Revision history for this message
Scott Moser (smoser) wrote :

I'm going to fix this in cloud-utils (growpart) in the way that we'd addressed this issue in the past:
 scraping text output of the command ran with LANG=C.

Not the best fix in the world.

What happened that caused this was that this commit is in 2.28.1:
  https://git.kernel.org/cgit/utils/util-linux/util-linux.git/commit/?id=e0d30ef4c

and then is subsequently reverted in trunk at
  https://git.kernel.org/cgit/utils/util-linux/util-linux.git/commit/?id=78e7f78896f2

I've sent a mail to upstream development list (Subject: request for sfdisk --no-blkrrpart) asking for a flag to tell sfdisk not to call blkrrpart, and then in the future we would just use that.

Scott Moser (smoser)
Changed in cloud-utils:
status: New → Fix Committed
importance: Undecided → High
Revision history for this message
Martin Pitt (pitti) wrote :

@Scott, thanks for drilling this down! If that patch got reverted in upstream master, is there a reason to not do the same in the current yakkety package, instead of working around it in cloud-init? Or would checking the output message be more robust as the exit code is effectively broken?

Revision history for this message
Scott Moser (smoser) wrote :

Martin,
I uploaded just the workaround. As shown in that commit message, the exit values are not reliable. Long ago I'd added the workaround, so the change here was really just to extend the list of possible output messages to scrape for.

Not great, but will work.

I also added a autopackage test, so something like this will hopefully be caught next time.

Martin Pitt (pitti)
Changed in cloud-utils (Ubuntu):
status: Confirmed → Fix Committed
no longer affects: cloud-init (Ubuntu)
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package cloud-utils - 0.29-0ubuntu3

---------------
cloud-utils (0.29-0ubuntu3) yakkety; urgency=medium

  * debian/control: add Suggests for mtools to cloud-image-utils.
  * debian/tests: add dep-8 auto package test.
  * sync to trunk at revno 302
    * ubuntu-cloudimg-query: fix xenial default to be ebs-ssd
      as traditional ebs are no longer available. (LP: #1593451)
    * mount-image-callback: fix shortopt parsing (LP: #1604011)
    * growpart: improved error messages on failure.
    * mount-image-callback: add -S short flag for --system-mounts
    * growpart: ignore sfdisk failure in 2.28.1 when due to reread
      failing (LP: #1619285)

 -- Scott Moser <email address hidden> Thu, 01 Sep 2016 16:07:51 -0400

Changed in cloud-utils (Ubuntu):
status: Fix Committed → Fix Released
Martin Pitt (pitti)
Changed in util-linux (Ubuntu):
status: Confirmed → Fix Committed
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package util-linux - 2.28.2-1ubuntu1

---------------
util-linux (2.28.2-1ubuntu1) yakkety; urgency=medium

  * Merge from Debian unstable. Remaining changes:
    - Regularly trim SSDs automatically (core-1311-ssd-trimming):
      + Add debian/fstrim-all.cron: Call fstrim --all. Installed as
        /etc/cron.weekly/fstrim.
    - Drop debian/hwclock.rules and hwclock.default, recent kernels sync the
      RTC automatically.
    - Add sulogin-fallback-static-sh.patch: Add support for /bin/static-sh as
      fallback if the regular shell fails to execute. Patch ported from
      sysvinit. (see LP #505887)
    - Add sulogin-lockedpwd.patch: Make sure file systems can be fixed on
      machines with locked root accounts (as Ubuntu does by default). Don't
      require --force for sulogin.
    - Drop debian/util-linux.preinst: Unconditionally deleting files
      is wrong, and dpkg will overwrite symlinks with real files anyway.
    - Drop the Breaks: cloud-utils, Ubuntu has a different cloud-utils
      packaging and this does not affect Ubuntu > 16.04 any more.
    - Add debian/util-linux.maintscript to clean upstart jobs on upgrade. This
      needs to be kept until after 18.04 LTS.
  * The new upstream version reverts the sfdisk exit code change from 2.28.1.
    (LP: #1619285)

util-linux (2.28.2-1) unstable; urgency=medium

  * debian/gbp.conf: set upstream-vcs-tag for import-orig
  * New upstream release.
    - among other things includes fixes for HPPA testsuite failures
      (Closes: #827225)
  * Drop d/p/tests-Fix-fdisk-id-and-fdisk-mbr-nondos-mode-on-Sparc.patch
    - now include in upstream release.

 -- Martin Pitt <email address hidden> Mon, 12 Sep 2016 23:22:44 +0200

Changed in util-linux (Ubuntu):
status: Fix Committed → Fix Released
Scott Moser (smoser)
Changed in cloud-utils:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.