under heavy IO encounter TOCTOU bug with bcache removal

Bug #1700564 reported by Ryan Harper
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
curtin
Fix Released
Medium
Ryan Harper
curtin (Ubuntu)
Fix Released
Medium
Unassigned

Bug Description

Here we see a check if a path exists, but when we attempt to remove it; it's gone (as the bcache device has been removed).

We need to handle the kernel async removal and not fail if something that should be removed is removed.

[ 26.387965] cloud-init[1430]: Current device storage tree:
[ 26.388370] cloud-init[1430]: vdb
[ 26.389001] cloud-init[1430]: |-- vdb1
[ 26.389441] cloud-init[1430]: `-- vdb2
[ 26.389930] cloud-init[1430]: `-- bcache0
[ 26.390170] cloud-init[1430]: vdc
[ 26.400560] cloud-init[1430]: `-- bcache0
[ 26.400881] cloud-init[1430]: shutdown running on holder type: 'bcache' syspath: '/sys/class/block/bcache0'
[ 26.401256] cloud-init[1430]: stopping bcache cacheset at: /sys/fs/bcache/661527fd-09b0-48e5-987d-1175ce570a76
[ 26.403018] cloud-init[1430]: waiting for /sys/fs/bcache/661527fd-09b0-48e5-987d-1175ce570a76 to be removed
[ 26.404351] cloud-init[1430]: /sys/fs/bcache/661527fd-09b0-48e5-987d-1175ce570a76 has been removed
[ 26.406967] cloud-init[1430]: Running command ['udevadm', 'settle'] with allowed return codes [0] (shell=False, capture=False)
[ 26.407310] cloud-init[1430]: os.path.exists on blockdevs:
[ 26.407667] cloud-init[1430]: [('/sys/class/block/bcache0', True), ('/sys/class/block/vdc/bcache', True), ('/sys/class/block/vdb/vdb2/bcache', False)]
[ 26.407996] cloud-init[1430]: stopping bcache backing device at: /sys/class/block/bcache0/bcache
[ 26.413071] cloud-init[1430]: finish: cmd-install/stage-partitioning/builtin/cmd-block-meta: FAIL: removing previous storage devices
[ 26.418728] cloud-init[1430]: finish: cmd-install/stage-partitioning/builtin/cmd-block-meta: FAIL: curtin command block-meta
[ 26.421744] cloud-init[1430]: Traceback (most recent call last):
[ 26.423874] cloud-init[1430]: File "/curtin/curtin/commands/main.py", line 215, in main
[ 26.424290] cloud-init[1430]: ret = args.func(args)
[ 26.424635] cloud-init[1430]: File "/curtin/curtin/commands/block_meta.py", line 67, in block_meta
[ 26.425253] cloud-init[1430]: meta_custom(args)
[ 26.425632] cloud-init[1430]: File "/curtin/curtin/commands/block_meta.py", line 1186, in meta_custom
[ 26.425925] cloud-init[1430]: clear_holders.clear_holders(disk_paths)
[ 26.428596] cloud-init[1430]: File "/curtin/curtin/block/clear_holders.py", line 481, in clear_holders
[ 26.428945] cloud-init[1430]: shutdown_function(dev_info['device'])
[ 26.430698] cloud-init[1430]: File "/curtin/curtin/block/clear_holders.py", line 166, in shutdown_bcache
[ 26.431769] cloud-init[1430]: '1', mode=None)
[ 26.432781] cloud-init[1430]: File "/curtin/curtin/util.py", line 435, in write_file
[ 26.433855] cloud-init[1430]: with open(filename, omode) as fp:
[ 26.435271] cloud-init[1430]: FileNotFoundError: [Errno 2] No such file or directory: '/sys/class/block/bcache0/bcache/stop'
[ 26.436969] cloud-init[1430]: [Errno 2] No such file or directory: '/sys/class/block/bcache0/bcache/stop'
[ 26.463377] cloud-init[1430]: builtin command failed
[ 26.468229] cloud-init[1430]: finish: cmd-install/stage-partitioning/builtin: FAIL: running 'curtin block-meta simple'
[ 26.468534] cloud-init[1430]: builtin took 1.020 seconds

Related branches

Scott Moser (smoser)
Changed in curtin:
status: New → Confirmed
importance: Undecided → Medium
status: Confirmed → In Progress
assignee: nobody → Ryan Harper (raharper)
Revision history for this message
Scott Moser (smoser) wrote : Fixed in Curtin 17.1

This bug is believed to be fixed in curtin in 17.1. If this is still a problem for you, please make a comment and set the state back to New

Thank you.

Changed in curtin:
status: In Progress → Fix Released
Revision history for this message
Scott Moser (smoser) wrote :

A fix for this bug is being SRU to Ubuntu 16.04 and 17.10 under bug 1743618.

Changed in curtin (Ubuntu):
status: New → Fix Released
importance: Undecided → Medium
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.