curtin

curtin passes wrong args to mkfs when making filesystem on advanced format disks

Bug #1597522 reported by Wesley Wiedenmeier
10
This bug affects 1 person
Affects Status Importance Assigned to Milestone
curtin
Fix Released
High
Wesley Wiedenmeier
curtin (Ubuntu)
Fix Released
Undecided
Unassigned
Xenial
Fix Released
Undecided
Unassigned

Bug Description

[Impact]

 * curtin passes wrong args to mkfs when making filesystem on advanced
   format disks (4k sector size)

   Curtin has been modified to only pass the '-s 1' sector-size flag
   to the vfat formatting tools. Curtin has been updated to specify
   the correct sector-size flags and values to other formatting
   utilities that support such settings.

[Test Case]

 * Install proposed curtin package and attempt to format a disk with
   xfs filesystem using 4K logical blocksize for the disk.

  PASS: Ubuntu successfully installs with xfs filesystem on top of a
        4k Disk

  FAIL: Ubuntu fails to install with xfs filesystem on 4k disk.

[Regression Potential]

 * Low; current users expecting xfs on 4k disks were blocked without
   this fix.

[Original Description]
During format handling, curtin detects the underlying block size of the disk and sets the filesystem block size accordingly.

In order to properly handle a bug in mkfs.vfat, curtin adds the flag '-s 1'. However, curtin adds this flag to all disk format commands, not just to commands using 'mkfs.vfat'. This can lead to unexpected behavior with some formatting tools, and can cause installation to halt with others.

For example, the mkfs.btrfs utility understands '-s' to mean sectorsize, so when curtin installs to an advanced format disk and storage config includes a btrfs formatted filesystem, curtin will create a filesystem that uses 1 byte sectors. This will greatly harm filesystem performance.

For xfs volumes, this means that installation will fail completely, as the sectorsize attribute for xfs volumes must be specified in the format '-s size=512', so '-s 1' will cause mkfs.xfs to fail.

Fortunately, this does not affect mkfs.ext* because these tools seem to silently ignore the '-s' flag, so most users likely have not been affected.

See original description

Related branches

lp:~wesley-wiedenmeier/curtin/1597522
Ryan Harper (community): Approve
Server Team CI bot: Approve (continuous-integration)
Diff: 82 lines (+25/-6)
3 files modified
curtin/block/mkfs.py (+9/-5)
examples/tests/uefi_basic.yaml (+15/-0)
tests/vmtests/test_uefi_basic.py (+1/-1)
lp:~curtin-dev/curtin/trunk
lp:~smoser/ubuntu/yakkety/curtin/pkg
lp:~raharper/ubuntu/xenial/curtin/pkg-sru-revno425
Scott Moser: Pending requested
Diff: 14513 lines (+10324/-1893)
94 files modified
Makefile (+3/-1)
curtin/__init__.py (+4/-0)
curtin/block/__init__.py (+249/-61)
curtin/block/clear_holders.py (+387/-0)
curtin/block/lvm.py (+96/-0)
curtin/block/mdadm.py (+18/-5)
curtin/block/mkfs.py (+10/-5)
curtin/commands/apply_net.py (+156/-1)
curtin/commands/apt_config.py (+668/-0)
curtin/commands/block_info.py (+75/-0)
curtin/commands/block_meta.py (+134/-263)
curtin/commands/block_wipe.py (+1/-2)
curtin/commands/clear_holders.py (+48/-0)
curtin/commands/curthooks.py (+61/-235)
curtin/commands/main.py (+4/-3)
curtin/config.py (+2/-3)
curtin/gpg.py (+74/-0)
curtin/net/__init__.py (+67/-30)
curtin/net/network_state.py (+45/-1)
curtin/util.py (+278/-81)
debian/changelog (+32/-2)
doc/conf.py (+21/-4)
doc/devel/README-vmtest.txt (+0/-152)
doc/devel/README.txt (+0/-55)
doc/devel/clear_holders_doc.txt (+85/-0)
doc/index.rst (+6/-0)
doc/topics/apt_source.rst (+164/-0)
doc/topics/config.rst (+551/-0)
doc/topics/development.rst (+68/-0)
doc/topics/integration-testing.rst (+245/-0)
doc/topics/networking.rst (+522/-0)
doc/topics/overview.rst (+7/-7)
doc/topics/reporting.rst (+3/-3)
doc/topics/storage.rst (+894/-0)
examples/apt-source.yaml (+267/-0)
examples/network-ipv6-bond-vlan.yaml (+56/-0)
examples/tests/apt_config_command.yaml (+85/-0)
examples/tests/apt_source_custom.yaml (+97/-0)
examples/tests/apt_source_modify.yaml (+92/-0)
examples/tests/apt_source_modify_arches.yaml (+102/-0)
examples/tests/apt_source_modify_disable_suite.yaml (+92/-0)
examples/tests/apt_source_preserve.yaml (+98/-0)
examples/tests/apt_source_search.yaml (+97/-0)
examples/tests/basic.yaml (+5/-1)
examples/tests/basic_network_static_ipv6.yaml (+22/-0)
examples/tests/basic_scsi.yaml (+1/-1)
examples/tests/network_alias.yaml (+125/-0)
examples/tests/network_mtu.yaml (+88/-0)
examples/tests/network_source_ipv6.yaml (+31/-0)
examples/tests/test_old_apt_features.yaml (+11/-0)
examples/tests/test_old_apt_features_ports.yaml (+10/-0)
examples/tests/uefi_basic.yaml (+15/-0)
examples/tests/vlan_network_ipv6.yaml (+92/-0)
setup.py (+2/-2)
tests/unittests/helpers.py (+41/-0)
tests/unittests/test_apt_custom_sources_list.py (+170/-0)
tests/unittests/test_apt_source.py (+1032/-0)
tests/unittests/test_block.py (+210/-0)
tests/unittests/test_block_lvm.py (+94/-0)
tests/unittests/test_block_mdadm.py (+28/-23)
tests/unittests/test_block_mkfs.py (+2/-2)
tests/unittests/test_clear_holders.py (+329/-0)
tests/unittests/test_make_dname.py (+200/-0)
tests/unittests/test_net.py (+54/-13)
tests/unittests/test_util.py (+180/-2)
tests/vmtests/__init__.py (+38/-38)
tests/vmtests/helpers.py (+129/-166)
tests/vmtests/test_apt_config_cmd.py (+55/-0)
tests/vmtests/test_apt_source.py (+238/-0)
tests/vmtests/test_basic.py (+21/-41)
tests/vmtests/test_bcache_basic.py (+5/-8)
tests/vmtests/test_bonding.py (+0/-204)
tests/vmtests/test_lvm.py (+2/-1)
tests/vmtests/test_mdadm_bcache.py (+21/-17)
tests/vmtests/test_multipath.py (+5/-13)
tests/vmtests/test_network.py (+205/-348)
tests/vmtests/test_network_alias.py (+40/-0)
tests/vmtests/test_network_bonding.py (+63/-0)
tests/vmtests/test_network_enisource.py (+91/-0)
tests/vmtests/test_network_ipv6.py (+53/-0)
tests/vmtests/test_network_ipv6_enisource.py (+26/-0)
tests/vmtests/test_network_ipv6_static.py (+42/-0)
tests/vmtests/test_network_ipv6_vlan.py (+34/-0)
tests/vmtests/test_network_mtu.py (+155/-0)
tests/vmtests/test_network_static.py (+44/-0)
tests/vmtests/test_network_vlan.py (+77/-0)
tests/vmtests/test_nvme.py (+2/-3)
tests/vmtests/test_old_apt_features.py (+89/-0)
tests/vmtests/test_raid5_bcache.py (+5/-8)
tests/vmtests/test_uefi_basic.py (+16/-18)
tools/jenkins-runner (+33/-7)
tools/launch (+9/-48)
tools/xkvm (+90/-2)
tox.ini (+30/-13)
Revision history for this message
Wesley Wiedenmeier (wesley-wiedenmeier) wrote :

Also, curtin does not specify sector size for xfs volumes and should

Wesley Wiedenmeier (wesley-wiedenmeier)
Changed in curtin:
assignee: nobody → Wesley Wiedenmeier (wesley-wiedenmeier)
status: New → In Progress
Revision history for this message
Ryan Harper (raharper) wrote :

Good catch.

Changed in curtin:
importance: Undecided → High
Wesley Wiedenmeier (wesley-wiedenmeier)
Changed in curtin:
status: In Progress → Fix Committed
Ryan Harper (raharper)
description: updated
Revision history for this message
Andy Whitcroft (apw) wrote : Please test proposed package

Hello Wesley, or anyone else affected,

Accepted curtin into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/curtin/0.1.0~bzr425-0ubuntu1~16.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

tags: added: verification-needed
Andy Whitcroft (apw)
Changed in curtin (Ubuntu):
status: New → Fix Released
Changed in curtin (Ubuntu Xenial):
status: New → Fix Committed
Revision history for this message
Wesley Wiedenmeier (wesley-wiedenmeier) wrote :

I can verify that the build in xenial-proposed resolves this bug using the example config attached.

Running an installation using curtin packed on a system with the current curtin release in xenial-updates installation fails.
Curtin release:
  root@tmp:/tmp# apt-cache policy curtin
  curtin:
    Installed: 0.1.0~bzr399-0ubuntu1~16.04.1
    Candidate: 0.1.0~bzr399-0ubuntu1~16.04.1
Failure:
  Unexpected error while running command.
  Command: ['mkfs.xfs', '-s', '1', '-f', '/dev/vdb2']
  Exit code: 1
  Reason: -
  Stdout: ''
  Stderr: 'unknown option -s 1\nUsage: mkfs.xfs\n/* blocksize */\t\t[-b log=n|size
  ...

When curtin is packed with the release in proposed, installation works

Curtin release:
  curtin:
    Installed: 0.1.0~bzr425-0ubuntu1~16.04.1
    Candidate: 0.1.0~bzr425-0ubuntu1~16.04.1

tags: added: verification-done
removed: verification-needed
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package curtin - 0.1.0~bzr425-0ubuntu1~16.04.1

---------------
curtin (0.1.0~bzr425-0ubuntu1~16.04.1) xenial-proposed; urgency=medium

  [ Scott Moser ]
  * debian/new-upstream-snapshot: add writing of debian changelog entries.

  [ Ryan Harper ]
  * New upstream snapshot.
    - unittest,tox.ini: catch and fix issue with trusty-level mock of open
    - block/mdadm: add option to ignore mdadm_assemble errors (LP: #1618429)
    - curtin/doc: overhaul curtin documentation for readthedocs.org
      (LP: #1351085)
    - curtin.util: re-add support for RunInChroot (LP: #1617375)
    - curtin/net: overhaul of eni rendering to handle mixed ipv4/ipv6 configs
    - curtin.block: refactor clear_holders logic into block.clear_holders and
      cli cmd
    - curtin.apply_net should exit non-zero upon exception. (LP: #1615780)
    - apt: fix bug in disable_suites if sources.list line is blank.
    - vmtests: disable Wily in vmtests
    - Fix the unittests for test_apt_source.
    - get CURTIN_VMTEST_PARALLEL shown correctly in jenkins-runner output
    - fix vmtest check_file_strippedline to strip lines before comparing
    - fix whitespace damage in tests/vmtests/__init__.py
    - fix dpkg-reconfigure when debconf_selections was provided.
      (LP: #1609614)
    - fix apt tests on non-intel arch
    - Add apt features to curtin. (LP: #1574113)
    - vmtest: easier use of parallel and controlling timeouts
    - mkfs.vfat: add force flag for formating whole disks (LP: #1597923)
    - block.mkfs: fix sectorsize flag (LP: #1597522)
    - block_meta: cleanup use of sys_block_path and handle cciss knames
      (LP: #1562249)
    - block.get_blockdev_sector_size: handle _lsblock multi result return
      (LP: #1598310)
    - util: add target (chroot) support to subp, add target_path helper.
    - block_meta: fallback to parted if blkid does not produce output
      (LP: #1524031)
    - commands.block_wipe: correct default wipe mode to 'superblock'
    - tox.ini: run coverage normally rather than separately
    - move uefi boot knowledge from launch and vmtest to xkvm

 -- Ryan Harper <email address hidden> Mon, 03 Oct 2016 13:43:54 -0500

Changed in curtin (Ubuntu Xenial):
status: Fix Committed → Fix Released
Revision history for this message
Martin Pitt (pitti) wrote : Update Released

The verification of the Stable Release Update for curtin has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Revision history for this message
Scott Moser (smoser) wrote : Fixed in Curtin 17.1

This bug is believed to be fixed in curtin in 17.1. If this is still a problem for you, please make a comment and set the state back to New

Thank you.

Changed in curtin:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Bug attachments

Remote bug watches

Bug watches keep track of this bug in other bug trackers.