Bug #1817097 “pvmove causes file system corruption without notic...” : Bugs : lvm2 package : Ubuntu

Revision history for this message

In Red Hat Bugzilla #1669751, nkshirsa (nkshirsa-redhat-bugs) wrote on 2019-01-26:

#10

Download full text (3.5 KiB)

Description of problem:

lvm should not allow extending an LV with a PV of different sector size than existing PVs making up the LV, since the FS on the LV does not mount once LVM adds in the new PV and extends the LV.

How reproducible:
Steps to Reproduce:

** Device: sdc (using the device with default sector size of 512)

# blockdev --report /dev/sdc
RO RA SSZ BSZ StartSec Size Device
rw 8192 512 4096 0 1073741824 /dev/sdc

** LVM is created with the default sector size of 512.

# blockdev --report /dev/mapper/testvg-testlv
RO RA SSZ BSZ StartSec Size Device
rw 8192 512 4096 0 1069547520 /dev/mapper/testvg-testlv

** The filesystem will also pick up 512 sector size.

# mkfs.xfs /dev/mapper/testvg-testlv
meta-data=/dev/mapper/testvg-testlv isize=512 agcount=4, agsize=65280 blks
         = sectsz=512 attr=2, projid32bit=1
         = crc=1 finobt=0, sparse=0
data = bsize=4096 blocks=261120, imaxpct=25
         = sunit=0 swidth=0 blks
naming =version 2 bsize=4096 ascii-ci=0 ftype=1
log =internal log bsize=4096 blocks=855, version=2
         = sectsz=512 sunit=0 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0

** Now we will mount it

# xfs_info /test
meta-data=/dev/mapper/testvg-testlv isize=512 agcount=4, agsize=65280 blks
         = sectsz=512 attr=2, projid32bit=1
         = crc=1 finobt=0 spinodes=0
data = bsize=4096 blocks=261120, imaxpct=25
         = sunit=0 swidth=0 blks
naming =version 2 bsize=4096 ascii-ci=0 ftype=1
log =internal bsize=4096 blocks=855, version=2
         = sectsz=512 sunit=0 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0

** Let's extend it with a PV with a sector size of 4096:

#modprobe scsi_debug sector_size=4096 dev_size_mb=512

# fdisk -l /dev/sdd

Disk /dev/sdd: 536 MB, 536870912 bytes, 131072 sectors
Units = sectors of 1 * 4096 = 4096 bytes <==============
Sector size (logical/physical): 4096 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 262144 bytes

# blockdev --report /dev/sdd
RO RA SSZ BSZ StartSec Size Device
rw 8192 4096 4096 0 536870912 /dev/sdd

# vgextend testvg /dev/sdd
Physical volume "/dev/sdd" successfully created
Volume group "testvg" successfully extended

# lvextend -l +100%FREE /dev/mapper/testvg-testlv
Size of logical volume testvg/testlv changed from 1020.00 MiB (255 extents) to 1.49 GiB (382 extents).
Logical volume testlv successfully resized.

# umount /test

# mount /dev/mapper/testvg-testlv /test
mount: mount /dev/mapper/testvg-testlv on /test failed: Function not implemented <===========

# dmesg | grep -i dm-2

[ 477.517515] XFS (dm-2): Unmounting Filesystem
[ 486.905933] XFS (dm-2): device supports 4096 byte sectors (n...

Description of problem:

lvm should not allow extending an LV with a PV of different sector size than existing PVs making up the LV, since the FS on the LV does not mount once LVM adds in the new PV and extends the LV.

How reproducible:
Steps to Reproduce:

** Device: sdc (using the device with default sector size of 512)

# blockdev --report /dev/sdc
RO    RA   SSZ   BSZ   StartSec            Size   Device
rw  8192   512  4096          0      1073741824   /dev/sdc

** LVM is created with the default sector size of 512.

# blockdev --report /dev/mapper/testvg-testlv 
RO    RA   SSZ   BSZ   StartSec            Size   Device
rw  8192   512  4096          0      1069547520   /dev/mapper/testvg-testlv

** The filesystem will also pick up 512 sector size.

# mkfs.xfs /dev/mapper/testvg-testlv 
meta-data=/dev/mapper/testvg-testlv isize=512    agcount=4, agsize=65280 blks
         =                       sectsz=512   attr=2, projid32bit=1
         =                       crc=1        finobt=0, sparse=0
data     =                       bsize=4096   blocks=261120, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0 ftype=1
log      =internal log           bsize=4096   blocks=855, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0

** Now we will mount it

# xfs_info /test
meta-data=/dev/mapper/testvg-testlv isize=512    agcount=4, agsize=65280 blks
         =                       sectsz=512   attr=2, projid32bit=1
         =                       crc=1        finobt=0 spinodes=0
data     =                       bsize=4096   blocks=261120, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0 ftype=1
log      =internal               bsize=4096   blocks=855, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0

** Let's extend it with a PV with a sector size of 4096:

#modprobe scsi_debug sector_size=4096 dev_size_mb=512

# fdisk -l /dev/sdd
 
Disk /dev/sdd: 536 MB, 536870912 bytes, 131072 sectors
Units = sectors of 1 * 4096 = 4096 bytes <==============
Sector size (logical/physical): 4096 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 262144 bytes
 
# blockdev --report /dev/sdd
RO    RA   SSZ   BSZ   StartSec            Size   Device
rw  8192  4096  4096          0       536870912   /dev/sdd

# vgextend testvg /dev/sdd
  Physical volume "/dev/sdd" successfully created
  Volume group "testvg" successfully extended

# lvextend -l +100%FREE /dev/mapper/testvg-testlv
  Size of logical volume testvg/testlv changed from 1020.00 MiB (255 extents) to 1.49 GiB (382 extents).
  Logical volume testlv successfully resized.

# umount /test

# mount /dev/mapper/testvg-testlv /test
mount: mount /dev/mapper/testvg-testlv on /test failed: Function not implemented <===========

# dmesg | grep -i dm-2

[  477.517515] XFS (dm-2): Unmounting Filesystem
[  486.905933] XFS (dm-2): device supports 4096 byte sectors (not 512) <============

The sector size of the lv is now 4096.
# blockdev --report /dev/mapper/testvg-testlv 
RO    RA   SSZ   BSZ   StartSec            Size   Device
rw  8192  4096  4096          0      1602224128   /dev/mapper/testvg-testlv

Expected results:
LVM should fail the lvextend if sector size is different to existing PV's

Additional info:
Discussed with Zdenek during LVM meeting in Brno

Revision history for this message

In Red Hat Bugzilla #1669751, teigland (teigland-redhat-bugs) wrote on 2019-01-28:

#11

Should we just require all PVs in the VG to have the same sector size?

Revision history for this message

In Red Hat Bugzilla #1669751, zkabelac (zkabelac-redhat-bugs) wrote on 2019-01-28:

#12

Basically that's what we have agreed in meeting - since we don't know yet how to handle different sector-sized PVs.

And a short fix could be to not allow that to happen on creating time.

But still there are already users having that VGs already created - so lvm2 can't just say such VG is invalid
and disable access to it...

So I'd probably see something similar we did for 'mirrorlog' -
add lvm.conf option to disable creation - that is respected on vgcreate time

Revision history for this message

bugproxy (bugproxy) wrote on 2019-02-21: journal with debug output of pvmove operation

#1

journal with debug output of pvmove operation Edit (204.1 KiB, application/x-gz)

Default Comment by Bridge

tags:

added: architecture-s39064 bugnameltc-175696 severity-critical targetmilestone-inin1810

Revision history for this message

bugproxy (bugproxy) wrote on 2019-02-21: Detailed steps to reproduce

#2

Detailed steps to reproduce Edit (2.7 KiB, text/plain)

Default Comment by Bridge

Changed in ubuntu:
assignee:	nobody → Skipper Bug Screeners (skipper-screen-team)
affects:	ubuntu → linux (Ubuntu)

Frank Heimes (fheimes) on 2019-02-21

affects:	linux (Ubuntu) → lvm2 (Ubuntu)
Changed in ubuntu-z-systems:
assignee:	nobody → Canonical Foundations Team (canonical-foundations)
importance:	Undecided → Critical

Revision history for this message

bugproxy (bugproxy) wrote on 2019-02-21: Comment bridged from LTC Bugzilla

#3

------- Comment From <email address hidden> 2019-02-21 12:51 EDT-------
There is a minor correction needed to the setup outlined above, as the enc-loop mapper device is backed by the second loopback device, and the volume group consists of the two devices listed in [ ] :

LV
|
VG: LOOP_VG [ /dev/loop0, /dev/mapper/enc-loop ]
|
PV: /dev/loop0 --> /dev/mapper/enc-loop
( backed by /dev/loop1 )

Mind there are SCSI devices providing 4k block sizes, thus check with 'blockdev --getbsz' to actually run a scenario from 512 --> 4096 bytes block sizes.

Revision history for this message

bugproxy (bugproxy) wrote on 2019-02-22:

#4

------- Comment From <email address hidden> 2019-02-22 05:23 EDT-------
Further investigations revealed that no dm-crypt mapper device is needed at all to reproduce the behaviour but just two block devices with different physical block sizes, e.g.

# blockdev --getpbsz /dev/mapper/mpatha-part1
512
# blockdev --getpbsz /dev/dasdc1
4096

Mind to first add the SCSI device/the device with the smaller phys. block size to the volume group when running the 'vgcreate' command.
# blockdev --getpbsz /dev/mapper/TEST_VG-LV1
512

Use one SCSI disk partition (multipath devices are recommended but not required) and one DASD partition to recreate the pvmove problem. Run pvs after the move completed, and unmount, mount the fs again.
The fsck.ext4 does not detect any problems on the fs which is unexpected.

Pertaining syslog entries:
Feb 22 11:09:23 system kernel: print_req_error: I/O error, dev dasdc, sector 280770
Feb 22 11:09:23 system kernel: Buffer I/O error on dev dm-3, logical block 139265, lost sync page write
Feb 22 11:09:23 system kernel: JBD2: Error -5 detected when updating journal superblock for dm-3-8.
Feb 22 11:09:23 system kernel: Aborting journal on device dm-3-8.
Feb 22 11:09:23 system kernel: print_req_error: I/O error, dev dasdc, sector 280770
Feb 22 11:09:23 system kernel: Buffer I/O error on dev dm-3, logical block 139265, lost sync page write
Feb 22 11:09:23 system kernel: JBD2: Error -5 detected when updating journal superblock for dm-3-8.
Feb 22 11:09:23 system kernel: print_req_error: I/O error, dev dasdc, sector 2242
Feb 22 11:09:23 system kernel: Buffer I/O error on dev dm-3, logical block 1, lost sync page write
Feb 22 11:09:23 system kernel: EXT4-fs (dm-3): I/O error while writing superblock
Feb 22 11:09:23 system kernel: EXT4-fs error (device dm-3): ext4_put_super:938: Couldn't clean up the journal
Feb 22 11:09:23 system kernel: EXT4-fs (dm-3): Remounting filesystem read-only
Feb 22 11:09:23 system kernel: print_req_error: I/O error, dev dasdc, sector 2242
Feb 22 11:09:23 system kernel: Buffer I/O error on dev dm-3, logical block 1, lost sync page write
Feb 22 11:09:23 system kernel: EXT4-fs (dm-3): I/O error while writing superblock
Feb 22 11:09:32 system kernel: EXT4-fs (dm-3): bad block size 1024

The very last syslog line repeats upon 'mount /dev/mapper/TEST_VG-LV1 /mnt ' attempts, the 1024 block size is related to
# blockdev --getbsz /dev/mapper/TEST_VG-LV1
1024

After the pvmove the physical blocksize is also changed to
# blockdev --getpbsz /dev/mapper/TEST_VG-LV1
4096

------- Comment From Christian.Rund@de.ibm.com 2019-02-22 05:23 EDT-------
Further investigations revealed that no dm-crypt mapper device is needed at all to reproduce the behaviour but just two block devices with different physical block sizes, e.g.

# blockdev --getpbsz /dev/mapper/mpatha-part1
512
# blockdev --getpbsz /dev/dasdc1
4096

Mind to first add the SCSI device/the device with the smaller phys. block size to the volume group when running the 'vgcreate' command.
# blockdev --getpbsz /dev/mapper/TEST_VG-LV1
512

Use one SCSI disk partition (multipath devices are recommended but not required) and one DASD partition to recreate the pvmove problem. Run pvs after the move completed, and unmount, mount the fs again.
The fsck.ext4 does not detect any problems on the fs which is unexpected.

Pertaining syslog entries:
Feb 22 11:09:23 system kernel: print_req_error: I/O error, dev dasdc, sector 280770
Feb 22 11:09:23 system kernel: Buffer I/O error on dev dm-3, logical block 139265, lost sync page write
Feb 22 11:09:23 system kernel: JBD2: Error -5 detected when updating journal superblock for dm-3-8.
Feb 22 11:09:23 system kernel: Aborting journal on device dm-3-8.
Feb 22 11:09:23 system kernel: print_req_error: I/O error, dev dasdc, sector 280770
Feb 22 11:09:23 system kernel: Buffer I/O error on dev dm-3, logical block 139265, lost sync page write
Feb 22 11:09:23 system kernel: JBD2: Error -5 detected when updating journal superblock for dm-3-8.
Feb 22 11:09:23 system kernel: print_req_error: I/O error, dev dasdc, sector 2242
Feb 22 11:09:23 system kernel: Buffer I/O error on dev dm-3, logical block 1, lost sync page write
Feb 22 11:09:23 system kernel: EXT4-fs (dm-3): I/O error while writing superblock
Feb 22 11:09:23 system kernel: EXT4-fs error (device dm-3): ext4_put_super:938: Couldn't clean up the journal
Feb 22 11:09:23 system kernel: EXT4-fs (dm-3): Remounting filesystem read-only
Feb 22 11:09:23 system kernel: print_req_error: I/O error, dev dasdc, sector 2242
Feb 22 11:09:23 system kernel: Buffer I/O error on dev dm-3, logical block 1, lost sync page write
Feb 22 11:09:23 system kernel: EXT4-fs (dm-3): I/O error while writing superblock
Feb 22 11:09:32 system kernel: EXT4-fs (dm-3): bad block size 1024

The very last syslog line repeats upon 'mount /dev/mapper/TEST_VG-LV1 /mnt ' attempts, the 1024 block size is related to
# blockdev --getbsz /dev/mapper/TEST_VG-LV1
1024

After the pvmove the physical blocksize is also changed to
# blockdev --getpbsz /dev/mapper/TEST_VG-LV1
4096

Revision history for this message

In Red Hat Bugzilla #1669751, teigland (teigland-redhat-bugs) wrote on 2019-02-25:

#13

Another report of this problem
https://www.redhat.com/archives/linux-lvm/2019-February/msg00018.html

Revision history for this message

In Red Hat Bugzilla #1669751, nsoffer (nsoffer-redhat-bugs) wrote on 2019-02-25:

#14

Interesting, I asked about this here few weeks ago:
https://www.redhat.com/archives/linux-lvm/2019-February/msg00002.html

Based on the info in this bug, it looks like RHV should care about the
only the logical block size when extending or creating a VG.

David, Zdenek, what do you think?

Revision history for this message

In Red Hat Bugzilla #1669751, teigland (teigland-redhat-bugs) wrote on 2019-03-05:

#15

Here's an initial, lightly-tested solution to the VG-consistency part. It does not address the issue of checking that a given LV is used with a consistent sector size. Perhaps if a user overrides the VG consistency check, it should be their responsibility to ensure LVs are consistent.

https://sourceware.org/git/?p=lvm2.git;a=commit;h=dd6ff9e3a75801fc5c6166aa0983fa8df098e91a

vgcreate/vgextend: check for inconsistent logical block sizes

When creating or extending a VG, check if the PVs have
inconsisent logical block sizes (value from BLKSSZGET ioctl).
If so, return an error. The error can be changed to a warning,
allowing the command to proceed with the mixed values, by
setting lvm.conf allow_mixed_logical_block_sizes=1.

Revision history for this message

Dimitri John Ledkov (xnox) wrote on 2019-03-11:

#5

I see that this bug was created with Ubuntu 18.10 (judging by the tags).
I am trying to reproduce the issue on Ubuntu 19.04 (current development release).

I am failing to produce a mixed blocksize cryptsetup device:
$ sudo cryptsetup luksFormat --type luks2 --sector-size 4096 /dev/loop1

is failing for me with:
"Device size is not aligned to the requested sector size."

And on this machine, I do not have access to native 4k and non-4k drives at the same time. Let me get a better machine to debug this further.

Revision history for this message

bugproxy (bugproxy) wrote on 2019-03-11:

#6

------- Comment From <email address hidden> 2019-03-11 08:22 EDT-------
The message "Device size is not aligned to the requested sector size." is because the size of your loopback device is not a multiple of 4096 bytes. With --sector-size 4096, it's size must be a multiple of 4096 bytes, otherwise you would a half sector at the end of the device.

Besides the setup with cryptsetup and 4K sector size, you can also try to setup your loopback devices with different sector sizes (and omit dm-crypt/cryptsetup totally).
By default loopback devices use a physical block size of 512, however, you can create them with -b 4096 to get a loopback device with 4K physical block size. With that you should also be able to reproduce this.

BTW: Please also see the thread about this topic on the LVM Mailing list that I have started in parallel, and especially the following post from David Teigland:
https://www.redhat.com/archives/linux-lvm/2019-March/msg00018.html

There is also already a draft patch from David Teigland for this in a private branch of the LVM2 git repository: https://sourceware.org/git/?p=lvm2.git;a=commit;h=dd6ff9e3a75801fc5c6166aa0983fa8df098e91a
I hope that this fix will make it into the master branch at some point in time.

Revision history for this message

Dimitri John Ledkov (xnox) wrote on 2019-03-11:

#7

Ok, reproduced this on x86_64 with raw files which are in multiples of 4k.

This is not an architecture specific issue.

Revision history for this message

Dimitri John Ledkov (xnox) wrote on 2019-03-11:

#8

This is a well-known upstream issue/bug.
This is not s390x, Ubuntu 18.10, any other Ubuntu release specific.
There is no dataloss -> one can execute pvmove operation in reverse (or i guess onto any 512 sector size PV) to mount the filesystems again.

Thus this is not critical at all.

Also, I am failing to understand what is the expectation for Canonical to do, w.r.t. this bug report?

If you want support, as a workaround one can force using 4k sizes, with vgcreate and ext4, then moving volumes to/from 512/4k physical volumes appears to work seamlessly:

$ sudo vgcreate --physicalextentsize 4k newtestvg /dev/...
$ sudo mkfs.ext4 -b 4096 /dev/mapper/...

For a more general solution, do create stand-alone new VGs/LVs/FSs, and migrate data over using higther level tools - e.g. dump/restore, rsync, etc.

But note, that launchpad should not be used for support requests. Please use your UA account (salesforce) for support request for your production systems.

This is discussed upstream, where they are trying to introduce a soft check to prevent from moving data across. https://bugzilla.redhat.com/show_bug.cgi?id=1669751 But it's not a real solution, just a weak safety check. As one can still force create ext4fs of either 512 or 4k, and move the volume to the "wrong" size. As ideally it would be user friendly if moving to/from mixed sector sizes would just work(tm) but that's unlikely to happen upstream, thus is wont-fix downstream too.

Was there anything in particular that you were expecting for us to change?

We could change the cloud-images (if they don't already), installers (i.e. d-i / subiquity) or the utils (i.e. vgcreate, mkfs.ext4) to default to 4k minimum sector sizes. But at the moment, these utils try to guess the sector sizes based on heuristics at creation time, and obviously get is "wrong" if the underlying device is swapped away from under their feet post creation time. Thus this is expected.

References:
The upstream bug report is https://bugzilla.redhat.com/show_bug.cgi?id=1669751
The upstream overridable weak safety-net check is https://sourceware.org/git/?p=lvm2.git;a=commitdiff;h=dd6ff9e3a75801fc5c6166aa0983fa8df098e91a
And that will make it into ubuntu eventually, when released in a stable lvm2 release and integration into ubuntu.

Please remove severity critical
Please remove target ubuntu 18.10
Please provide explanation as to why this issue was filed