ubuntu-kernel-tests

ubuntu_boot test report kernel tainted on s390x (flag 4097)

Bug #1892124 reported by Po-Hsu Lin
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
ubuntu-kernel-tests
Fix Released
Undecided
Po-Hsu Lin

Bug Description

START ---- ---- timestamp=1597806884 localtime=Aug 18 23:14:44
 Persistent state client._record_indent now set to 1
        START ubuntu_boot.kernel_tainted ubuntu_boot.kernel_tainted timestamp=1597806884 localtime=Aug 18 23:14:44
 Persistent state client._record_indent now set to 2
 Persistent state client.unexpected_reboot now set to ('ubuntu_boot.kernel_tainted', 'ubuntu_boot.kernel_tainted')
 Checking kernel tainted flags in /proc/sys/kernel/tainted
 ERROR: kernel tainted flag != 0: 4097

I think this is because of the zfs module.
As we don't re-deploy these manual provisioned system every time. Loading this out-of-tree module taints the kernel

see bug 1876990

A possible solution is to unload the module after the corresponding tests.

See original description
Revision history for this message
Po-Hsu Lin (cypressyew) wrote :

ubuntu@s2lp3:~$ ps aux | grep zed
root 3317 0.0 0.0 6688 2772 ? Ss 23:16 0:00 /usr/sbin/zed -F
ubuntu 5411 0.0 0.0 7132 764 pts/0 S+ 23:32 0:00 grep --color=auto zed
ubuntu@s2lp3:~$ lsmod | grep zfs
zfs 3158016 3
zunicode 335872 1 zfs
zcommon 73728 1 zfs
znvpair 110592 2 zfs,zcommon
spl 126976 3 zfs,zcommon,znvpair
zavl 16384 1 zfs

Po-Hsu Lin (cypressyew)
description: updated
Po-Hsu Lin (cypressyew)
tags: added: groovy sru-20200831
Kelsey Steele (kelsey-steele)
tags: added: 5.4 focal s390x sru-20210315
Revision history for this message
Marcelo Cerri (mhcerri) wrote :

Also seen in bionic linux-fips 4.15.0-1057.65 for sru-20210315

tags: added: fips
Kelsey Steele (kelsey-steele)
summary: - ubuntu_boot test report kernel tainted on s390x
+ ubuntu_boot test report kernel tainted on s390x (flag 4097)
tags: added: 4.15 bionic
Revision history for this message
Marcelo Cerri (mhcerri) wrote :

Also seen in xenial linux-fips 4.4.0-1060.66 for sru-20210315

tags: added: 4.4
Kelsey Steele (kelsey-steele)
tags: added: sru-20210412
Kelsey Steele (kelsey-steele)
tags: added: sru-20210621
Revision history for this message
Po-Hsu Lin (cypressyew) wrote :

One interesting finding is that kernel06 is not failing with this issue. Command `dpkg -l | grep zfs` returns nothing.

On other s390x instances,
The zfs module can be removed by stopping the zed service first:
ubuntu@kernel04:~$ lsmod | grep zfs
zfs 3936256 6
zunicode 335872 1 zfs
zlua 192512 1 zfs
zcommon 114688 1 zfs
znvpair 110592 2 zfs,zcommon
zavl 16384 1 zfs
icp 299008 1 zfs
spl 159744 5 zfs,icp,znvpair,zcommon,zavl
ubuntu@kernel04:~$ dpkg -l |grep zfs
ii libzfs2linux 0.8.3-1ubuntu12.10 s390x OpenZFS filesystem library for Linux
ii zfs-zed 0.8.3-1ubuntu12.10 s390x OpenZFS Event Daemon
ii zfsutils-linux 0.8.3-1ubuntu12.10 s390x command-line tools to manage OpenZFS filesystems
ubuntu@kernel04:~$ sudo systemctl stop zed
ubuntu@kernel04:~$ lsmod | grep zfs
zfs 3936256 0
zunicode 335872 1 zfs
zlua 192512 1 zfs
zcommon 114688 1 zfs
znvpair 110592 2 zfs,zcommon
zavl 16384 1 zfs
icp 299008 1 zfs
spl 159744 5 zfs,icp,znvpair,zcommon,zavl
ubuntu@kernel04:~$
ubuntu@kernel04:~$ sudo modprobe -r zfs

Dig deeper, it shows that the zed (zfs-zed) comes with the zfsutils-linux package which was installed by zfs related tests in our test suite. We should add a remove code to remove that package in the end of the test.

Po-Hsu Lin (cypressyew)
Changed in ubuntu-kernel-tests:
status: New → In Progress
assignee: nobody → Po-Hsu Lin (cypressyew)
Revision history for this message
Po-Hsu Lin (cypressyew) wrote :

As we don't run zfs related tests on s390x anymore, simply remove zfsutils-linux will do the trick. But it's still more reliable to do a cleanup for these tests to guarantee we won't tread on this again in the future.

Revision history for this message
Po-Hsu Lin (cypressyew) wrote :

Test case improved:
https://git.launchpad.net/~canonical-kernel-team/+git/autotest-client-tests/commit/?id=4c9ac393e457971106b8d55b5437ee58210f2f0b

And all of the zfs module on s390x instance have been removed manually.

Changed in ubuntu-kernel-tests:
status: In Progress → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.