dev test in ubuntu_stress_smoke_tests throws usercopy warning s390x Ubuntu on LPAR

Bug #1800609 reported by Po-Hsu Lin
20
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Stress-ng
Fix Released
High
Colin Ian King
ubuntu-kernel-tests
Fix Released
Undecided
Unassigned
stress-ng (Ubuntu)
Fix Released
Undecided
Unassigned
Bionic
Fix Released
Undecided
Unassigned
Cosmic
Fix Released
Undecided
Unassigned

Bug Description

SRU Request [Bionic, Cosmic]

== Justification ==

When running stress-ng with the dev stressor on s390x the FIBMAP ioctl is being incorrectly invoked in a raw block device. FIBMAP is intended for files
and not block devices. This ioctl needs to be removed from the stress test.

== Fix ==

Upstream commit http://kernel.ubuntu.com/git/cking/stress-ng.git/commit/?id=53e5a98cd58d3b146cf8ffe52540722ff410a5ad

== Testing ==

Run:

sudo stress-ng --dev 1

Without the fix the kernel will throw warning errors. With the fix the test will run correctly without kernel issues.

== Regression Potential ==

Minimal. This affects just the dev stressor in stress-ng and removes functionality that was broken.

-----

The dev test on a s390x node (Ubuntu on LPAR) will hang, kernel bug could be found in the syslog

[54084.082277] ------------[ cut here ]------------
[54084.082279] kernel BUG at mm/usercopy.c:100!
[54084.082294] illegal operation: 0001 ilc:1 [#256] SMP
[54084.082297] Modules linked in: vhost_net vhost tap cuse dccp_ipv4 dccp algif_rng salsa20_generic camellia_generic cast6_generic cast_common serpent_generic twofish_generic twofish_common lrw algif_skcipher tgr192 wp512 rmd320 rmd256 rmd160 rmd128 md4 algif_hash af_alg aufs ip6t_MASQUERADE ip6table_nat ip6table_filter ip6_tables xt_comment xt_CHECKSUM iptable_mangle ipt_MASQUERADE iptable_nat xt_tcpudp bridge iptable_filter bpfilter openvswitch nsh nf_conntrack_ipv6 nf_nat_ipv6 nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_defrag_ipv6 nf_conncount nf_nat nf_conntrack 8021q garp mrp stp llc binfmt_misc zfs(PO) zunicode(PO) zavl(PO) icp(PO) dm_service_time zcommon(PO) znvpair(PO) dm_multipath scsi_dh_rdac scsi_dh_emc spl(O) scsi_dh_alua ghash_s390 prng aes_s390 des_s390 des_generic sha512_s390 sha256_s390
[54084.082354] sha1_s390 sha_common genwqe_card crc_itu_t chsc_sch eadm_sch ctcm fsm vfio_ccw vfio_mdev mdev vfio_iommu_type1 vfio sch_fq_codel nfsd ib_iser rdma_cm auth_rpcgss nfs_acl iw_cm lockd grace ib_cm iscsi_tcp libiscsi_tcp sunrpc libiscsi scsi_transport_iscsi ip_tables x_tables btrfs zstd_compress zlib_deflate raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 linear mlx4_en mlx4_ib qeth_l2 ib_core ptp pps_core crc32_vx_s390 mlx4_core devlink zfcp scsi_transport_fc qeth qdio ccwgroup dasd_eckd_mod dasd_mod
[54084.082418] CPU: 3 PID: 217520 Comm: stress-ng-dev Tainted: P D O 4.18.0-11-generic #12-Ubuntu
[54084.082420] Hardware name: IBM 2964 N63 400 (LPAR)
[54084.082422] Krnl PSW : 00000000ed7ed85f 00000000816ddc5c (usercopy_abort+0xb2/0xb8)
[54084.082433] R:0 T:1 IO:1 EX:1 Key:0 M:1 W:0 P:0 AS:3 CC:0 PM:0 RI:0 EA:3
[54084.082436] Krnl GPRS: 0000000000000320 0000000000000006 0000000000000056 0000000000000007
[54084.082438] 0000000000000007 000000007fe00000 0000000000a96d3a 0000000000000000
[54084.082441] 000003ff00000000 0000000000000006 0000000000000006 00000002acb4ee00
[54084.082443] 0000000000a96d3a 0000000000914430 0000000000386606 000000038f087a78
[54084.082449] Krnl Code: 00000000003865fa: c02000390dc4 larl %r2,aa8182
                          0000000000386600: c0e5fff2b418 brasl %r14,1dce30
                         #0000000000386606: a7f40001 brc 15,386608
                         >000000000038660a: 0707 bcr 0,%r7
                          000000000038660c: 0707 bcr 0,%r7
                          000000000038660e: 0707 bcr 0,%r7
                          0000000000386610: c00400000000 brcl 0,386610
                          0000000000386616: ebcff0780024 stmg %r12,%r15,120(%r15)
[54084.082484] Call Trace:
[54084.082487] ([<0000000000386606>] usercopy_abort+0xae/0xb8)
[54084.082489] [<0000000000386522>] __check_object_size+0x19a/0x1d0
[54084.082492] [<00000000005c69d8>] sg_scsi_ioctl+0x120/0x3d8
[54084.082495] [<00000000005c75b6>] scsi_cmd_ioctl+0x3ce/0x3d8
[54084.082497] [<00000000005c87d0>] bsg_ioctl+0x118/0x250
[54084.082499] [<00000000003a3c9c>] do_vfs_ioctl+0xac/0x650
[54084.082502] [<00000000003a42cc>] ksys_ioctl+0x8c/0xb8
[54084.082504] [<00000000003a432a>] sys_ioctl+0x32/0x40
[54084.082507] [<00000000008f0dc0>] system_call+0xdc/0x2c8
[54084.082508] Last Breaking-Event-Address:
[54084.082510] [<0000000000386606>] usercopy_abort+0xae/0xb8
[54084.082512]
[54084.082513] ---[ end trace 5009dd6293a23031 ]---

ProblemType: Bug
DistroRelease: Ubuntu 18.10
Package: linux-image-4.18.0-11-generic 4.18.0-11.12
ProcVersionSignature: Ubuntu 4.18.0-11.12-generic 4.18.12
Uname: Linux 4.18.0-11-generic s390x
NonfreeKernelModules: zfs zunicode zavl icp zcommon znvpair
AlsaDevices: Error: command ['ls', '-l', '/dev/snd/'] failed with exit code 2: ls: cannot access '/dev/snd/': No such file or directory
AplayDevices: Error: [Errno 2] No such file or directory: 'aplay': 'aplay'
ApportVersion: 2.20.10-0ubuntu13
Architecture: s390x
ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord': 'arecord'
CRDA: Error: command ['iw', 'reg', 'get'] failed with exit code 1: nl80211 not found.
Date: Tue Oct 30 09:05:17 2018
HibernationDevice: RESUME=UUID=46fcf640-03e2-4fd5-b7d6-321d6a11ff8d
IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig': 'iwconfig'
Lsusb: Error: command ['lsusb'] failed with exit code 1:
PciMultimedia:

ProcFB: Error: [Errno 2] No such file or directory: '/proc/fb'
ProcKernelCmdLine: root=UUID=7ca41340-b34a-4b16-87e9-3ceb5b51767d crashkernel=196M BOOT_IMAGE=0
RelatedPackageVersions:
 linux-restricted-modules-4.18.0-11-generic N/A
 linux-backports-modules-4.18.0-11-generic N/A
 linux-firmware 1.175
RfKill: Error: [Errno 2] No such file or directory: 'rfkill': 'rfkill'
SourcePackage: linux
UpgradeStatus: No upgrade log present (probably fresh install)

Revision history for this message
Po-Hsu Lin (cypressyew) wrote :
Revision history for this message
Ubuntu Kernel Bot (ubuntu-kernel-bot) wrote : Status changed to Confirmed

This change was made by a bot.

Changed in linux (Ubuntu):
status: New → Confirmed
Revision history for this message
Colin Ian King (colin-king) wrote : Re: dev test in ubuntu_stress_smoke_tests will hang on s390x Ubuntu on LPAR

Occurs when performing the FIBMAP ioctl on /dev/bsg/rport-1:0-0

Revision history for this message
Cristian Aravena Romero (caravena) wrote :

Did this issue start happening after an update/upgrade? Was there a
prior kernel version where you were not having this particular problem?

Would it be possible for you to test the latest upstream kernel? Refer
to https://wiki.ubuntu.com/KernelMainlineBuilds . Please test the latest
v4.19 kernel[0].

If this bug is fixed in the mainline kernel, please add the following
tag 'kernel-fixed-upstream'.

If the mainline kernel does not fix this bug, please add the tag:
'kernel-bug-exists-upstream'.

Once testing of the upstream kernel is complete, please mark this bug as
"Confirmed".

Thanks in advance.

[0] http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.19

Changed in linux (Ubuntu):
status: Confirmed → Incomplete
Changed in linux (Ubuntu):
assignee: nobody → Colin Ian King (colin-king)
importance: Undecided → Medium
status: Incomplete → In Progress
Revision history for this message
Colin Ian King (colin-king) wrote :

Fix committed to stress-ng:

http://kernel.ubuntu.com/git/cking/stress-ng.git/commit/?id=53e5a98cd58d3b146cf8ffe52540722ff410a5ad

FIBMAP on a block device is plainly wrong, it should only be applied to a file. The FIBMAP ioctl to a scsi block device is the same as the SCSI_IOCTL_SEND_COMMAND ioctl that requires a scsi_ioctl_command struct. As it stands, we are passing an int with stack garbage as this struct which is passing garbage to the ioctl. The deprecated SCSI ioctl creates a scsi_request from the garbage and this causes the kernel to correctly warn with a usercopy_abort that the structure is bogos. Fix this by not calling FIBMAP on a raw SCSI device.

Changed in stress-ng:
status: New → Fix Committed
importance: Undecided → High
assignee: nobody → Colin Ian King (colin-king)
Changed in linux (Ubuntu):
status: In Progress → Invalid
no longer affects: linux (Ubuntu)
summary: - dev test in ubuntu_stress_smoke_tests will hang on s390x Ubuntu on LPAR
+ dev test in ubuntu_stress_smoke_tests throws usercopy warning s390x
+ Ubuntu on LPAR
Revision history for this message
Po-Hsu Lin (cypressyew) wrote :

Tested on the same node, the system can pass with the dev test now.
Thanks!

Changed in ubuntu-kernel-tests:
status: New → Fix Released
Revision history for this message
Colin Ian King (colin-king) wrote :

This requires SRU'ing.

Changed in stress-ng:
status: Fix Committed → Fix Released
description: updated
description: updated
Revision history for this message
Timo Aaltonen (tjaalton) wrote : Please test proposed package

Hello Po-Hsu, or anyone else affected,

Accepted stress-ng into cosmic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/stress-ng/0.09.42-1ubuntu1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-cosmic to verification-done-cosmic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-cosmic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in stress-ng (Ubuntu):
status: New → Fix Released
Changed in stress-ng (Ubuntu Cosmic):
status: New → Fix Committed
tags: added: verification-needed verification-needed-cosmic
Revision history for this message
Timo Aaltonen (tjaalton) wrote :

Hello Po-Hsu, or anyone else affected,

Accepted stress-ng into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/stress-ng/0.09.25-1ubuntu1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in stress-ng (Ubuntu Bionic):
status: New → Fix Committed
tags: added: verification-needed-bionic
Revision history for this message
Colin Ian King (colin-king) wrote :

verified 0.09.42-1ubuntu1 is fixed for Cosmic

tags: added: verification-done-cosmic
removed: verification-needed-cosmic
Revision history for this message
Colin Ian King (colin-king) wrote :

verified 0.09.25-1ubuntu1 is fixed for Bionic

tags: added: verification-done verification-done-bionic
removed: verification-needed verification-needed-bionic
Revision history for this message
Łukasz Zemczak (sil2100) wrote : Update Released

The verification of the Stable Release Update for stress-ng has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package stress-ng - 0.09.25-1ubuntu1

---------------
stress-ng (0.09.25-1ubuntu1) bionic; urgency=medium

  * stress-dev: remove FIBMAP ioctl on device (LP: #1800609)
    - fixes error of incorrect ioctl on block device
  * stress-revio: ensure fiemap extent struct is zero'd (LP: #1798566)
    - fixes fiemap ioctl failures because of garbage in struct

 -- Colin King <email address hidden> Wed, 14 Nov 2018 18:10:15 +0000

Changed in stress-ng (Ubuntu Bionic):
status: Fix Committed → Fix Released
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package stress-ng - 0.09.42-1ubuntu1

---------------
stress-ng (0.09.42-1ubuntu1) cosmic; urgency=medium

  * stress-affinity: fix set affinity setting across all
    stressors (LP: #1803322)
    - fixes --seq 0 affinity setting stopping all CPUs from being used
  * stress-seccomp: add sanity check to see if SECCOMP_SET_MODE_FILTER
    works (LP: #1801856)
    - fixes false failure errors if filtering is not enabled
  * stress-dev: remove FIBMAP ioctl on device (LP: #1800609)
    - fixes error of incorrect ioctl on block device
  * stress-revio: ensure fiemap extent struct is zero'd (LP: #1798566)
    - fixes fiemap ioctl failures because of garbage in struct

 -- Colin King <email address hidden> Wed, 14 Nov 2018 17:28:22 +0000

Changed in stress-ng (Ubuntu Cosmic):
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.