backport dkms fixes to build modules correctly for hwe-5.19+ kernels with custom compiler

Bug #1991664 reported by Dimitri John Ledkov
12
This bug affects 1 person
Affects Status Importance Assigned to Milestone
backport-iwlwifi-dkms (Ubuntu)
Fix Released
Undecided
Unassigned
Jammy
New
Undecided
Unassigned
dkms (Ubuntu)
Fix Released
Undecided
Unassigned
Jammy
Fix Released
Undecided
Dimitri John Ledkov
linux (Ubuntu)
Fix Released
Undecided
Unassigned
Jammy
Incomplete
Undecided
Unassigned
v4l2loopback (Ubuntu)
Fix Released
Undecided
Unassigned
Jammy
Fix Released
Undecided
Unassigned
zfs-linux (Ubuntu)
Fix Released
Undecided
Unassigned
Jammy
Fix Released
Undecided
Unassigned

Bug Description

tl; dr

this has proven to be a bit futile, and instead we went ahead with changing the default kernel makefiles to default to the correct toolchain and we have reverted hwe-5.19 kernel to use stock gcc on jammy for now.

So the things described here are not strictly even needed in jammy for now.

[ Impact ]

 * hwe-5.19 kernel has to be compiled with gcc-12+ for the new security kernel config options that have been enabled in that kernel (init zero structs).

 * dkms needs to be patched to attempt to use the same compiler as was used to build the kernel, specifically use gcc for v5.15 but gcc-12 for v5.19+ in jammy

 * dkms autopkgtests also need to be improved to ensure there are no false negatives when kernel has module under test already built-in at the same version.

 * because linux-headers packages in Ubuntu do not depend on a toolchain, and it is very hard to correctly depend on a native or cross-toolchain, add gcc-12 dependency in the dkms package.

[ Test Plan ]

 * Retrigger dkms module test for a built-in package of the same version as already built into the kernel, for example zfs-linux dkms. It should pass.

 * Attempt to build dkms modules against v5.15 kernel and v5.19 kernel from proposed. dkms module should be built with gcc and gcc-12 respectively.

[ Where problems could occur ]

 * New gcc-12 dependency on the dkms package may result in two compilers being installed in the Ubuntu Desktop Live image

 * New gcc-12 dependency may prevent automatic upgrade/installation of the dkms package update via unattended upgrades.

[ Other Info ]

 * It is questionable for dkms to depend on any c-compiler, given that a module for a given kernel can really be built with the same toolchain and the same compiler features.

 * In debian, linux kernel packages ship and extra configuration file in the kernel headers, dkms uses said configuration file to use a matching compiler, and linux-headers packages depend on the toolchain used to build a given kernel. Maybe in the future Ubuntu could consider using the same packaging.

 * Some DKMS modules need further fixes to preserve and not loose the compiler setting

Changed in dkms (Ubuntu):
status: New → Fix Released
Changed in dkms (Ubuntu Jammy):
status: New → Triaged
assignee: nobody → Dimitri John Ledkov (xnox)
Changed in dkms (Ubuntu):
status: Fix Released → Triaged
Revision history for this message
Ubuntu Kernel Bot (ubuntu-kernel-bot) wrote : Missing required logs.

This bug is missing log files that will aid in diagnosing the problem. While running an Ubuntu kernel (not a mainline or third-party kernel) please enter the following command in a terminal window:

apport-collect 1991664

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

Changed in linux (Ubuntu):
status: New → Incomplete
Changed in linux (Ubuntu Jammy):
status: New → Incomplete
description: updated
summary: backport dkms fixes to build modules correctly for hwe-5.19+ kernels
+ with custom compiler
Changed in v4l2loopback (Ubuntu):
status: New → In Progress
Changed in backport-iwlwifi-dkms (Ubuntu):
status: New → In Progress
Changed in zfs-linux (Ubuntu):
status: New → In Progress
Changed in dkms (Ubuntu):
status: Triaged → In Progress
Revision history for this message
Timo Aaltonen (tjaalton) wrote : Please test proposed package

Hello Dimitri, or anyone else affected,

Accepted dkms into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/dkms/2.8.7-2ubuntu2.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in dkms (Ubuntu Jammy):
status: Triaged → Fix Committed
tags: added: verification-needed verification-needed-jammy
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package backport-iwlwifi-dkms - 9904-0ubuntu3

---------------
backport-iwlwifi-dkms (9904-0ubuntu3) kinetic; urgency=medium

  * Preserve CC setting in the make wrapper script, to ensure module can
    be built correctly with the kernel's custom compiler. Fixes FTBFS in
    hwe-5.19 build. LP: #1991664

 -- Dimitri John Ledkov <email address hidden> Thu, 06 Oct 2022 12:52:22 +0100

Changed in backport-iwlwifi-dkms (Ubuntu):
status: In Progress → Fix Released
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package zfs-linux - 2.1.5-1ubuntu6

---------------
zfs-linux (2.1.5-1ubuntu6) kinetic; urgency=medium

  * Use detected CC compiler, if KERNEL_CC is undefined. Fixes
    cross-building using non-standard compiler, i.e. $(arch)-gcc-12. LP:
    #1991664

 -- Dimitri John Ledkov <email address hidden> Tue, 04 Oct 2022 14:34:16 +0100

Changed in zfs-linux (Ubuntu):
status: In Progress → Fix Released
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package v4l2loopback - 0.12.7-2ubuntu2

---------------
v4l2loopback (0.12.7-2ubuntu2) kinetic; urgency=medium

  * Preserve CC setting when invoking sub-make, to ensure module can be
    built correctly with the kernel's custom compiler. Fixes FTBFS in
    hwe-5.19 build. LP: #1991664

 -- Dimitri John Ledkov <email address hidden> Thu, 06 Oct 2022 15:01:47 +0100

Changed in v4l2loopback (Ubuntu):
status: In Progress → Fix Released
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package dkms - 3.0.6-2ubuntu1

---------------
dkms (3.0.6-2ubuntu1) kinetic; urgency=medium

  * Export exact CC compiler earlier, before pre_build script is run. Some
    dkms modules already perform compilation in pre_build script, and thus
    need correct CC variable earlier. LP: #1991664

 -- Dimitri John Ledkov <email address hidden> Tue, 04 Oct 2022 15:22:46 +0100

Changed in dkms (Ubuntu):
status: In Progress → Fix Released
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package linux - 5.19.0-21.21

---------------
linux (5.19.0-21.21) kinetic; urgency=medium

  * kinetic/linux: 5.19.0-21.21 -proposed tracker (LP: #1992639)

  * cannot change mount namespace (LP: #1991691)
    - SAUCE: apparmor: Fix getaatr mediation causing snap failures

  * Kernel regresses openjdk on riscv64 (LP: #1992484)
    - SAUCE: Revert "riscv: mmap with PROT_WRITE but no PROT_READ is invalid"

 -- Andrea Righi <email address hidden> Wed, 12 Oct 2022 19:53:36 +0200

Changed in linux (Ubuntu):
status: Incomplete → Fix Released
Revision history for this message
Dimitri John Ledkov (xnox) wrote :
Download full text (5.2 KiB)

# Setup
apt install software-properties-common ubuntu-dev-tools
add-apt-repository ppa:canonical-kernel-team/ubuntu/ppa
apt install linux-generic linux-generic-wip dkms=2.8.7-2ubuntu2
pull-lp-debs v4l2loopback kinetic

This installs v5.15 and preliminary v5.19 kernels in a jammy chroot, together with dkms from jammy release pocket.
It also fetches a sample dkms module from kinetic, that supports both kernels.

# apt install ./v4l2loopback-dkms_0.12.7-2ubuntu2_all.deb
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Note, selecting 'v4l2loopback-dkms' instead of './v4l2loopback-dkms_0.12.7-2ubuntu2_all.deb'
Suggested packages:
  v4l2loopback-utils
The following NEW packages will be installed:
  v4l2loopback-dkms
0 upgraded, 1 newly installed, 0 to remove and 2 not upgraded.
Need to get 0 B/29.3 kB of archives.
After this operation, 113 kB of additional disk space will be used.
Get:1 /root/v4l2loopback-dkms_0.12.7-2ubuntu2_all.deb v4l2loopback-dkms all 0.12.7-2ubuntu2 [29.3 kB]
Selecting previously unselected package v4l2loopback-dkms.
(Reading database ... 93941 files and directories currently installed.)
Preparing to unpack .../v4l2loopback-dkms_0.12.7-2ubuntu2_all.deb ...
Unpacking v4l2loopback-dkms (0.12.7-2ubuntu2) ...
Setting up v4l2loopback-dkms (0.12.7-2ubuntu2) ...
Loading new v4l2loopback-0.12.7 DKMS files...
It is likely that 5.15.0-52-generic belongs to a chroot's host
Building for 5.15.0-54-generic and 5.19.0-18-generic
Building initial module for 5.15.0-54-generic
Done.

v4l2loopback.ko:
Running module version sanity check.
 - Original module
 - Installation
   - Installing to /lib/modules/5.15.0-54-generic/updates/dkms/

depmod...
Building initial module for 5.19.0-18-generic
Error! Bad return status for module build on kernel: 5.19.0-18-generic (x86_64)
Consult /var/lib/dkms/v4l2loopback/0.12.7/build/make.log for more information.
dpkg: error processing package v4l2loopback-dkms (--configure):
 installed v4l2loopback-dkms package post-installation script subprocess returned error exit
status 10
Errors were encountered while processing:
 v4l2loopback-dkms
E: Sub-process /usr/local/libexec/mk-sbuild/dpkg returned an error code (1)

# dkms status
v4l2loopback/0.12.7, 5.15.0-54-generic, x86_64: installed

With current dkms, the module is built correctly for v5.15 kernel bug fails for v5.19 kernel.

In the build log we can see that gcc-11 is used to build the module instead of gcc-12.

warning: the compiler differs from the one used to build the kernel
  The kernel was built by: x86_64-linux-gnu-gcc-12 (Ubuntu 12.1.0-2ubuntu1~22.04) 12.1.0
  You are using: cc (Ubuntu 11.3.0-1ubuntu1~22.04) 11.3.0

Upgrade dkms package to the proposed version

# apt install dkms
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following additional packages will be installed:
  cpp-12 gcc-12 libasan8 libgcc-12-dev libtsan2
Suggested packages:
  gcc-12-locales menu gcc-12-multilib gcc-12-doc
The following NEW packages will be installed:
  cpp-12 gcc-12 libasan8 libgcc-12-dev libtsan2
The following packages will be upgraded:
  dkms
...

Setting...

Read more...

tags: added: verification-done verification-done-jammy
removed: verification-needed verification-needed-jammy
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package dkms - 2.8.7-2ubuntu2.1

---------------
dkms (2.8.7-2ubuntu2.1) jammy; urgency=medium

  * Cherry-pick patches from kinetic/stable to address building dkms
    modules correctly for HWE kernels (LP: #1991664):
    + Fix dkms-autopkgtest when a given dkms package is built into the
    kernel already of the same version. (i.e. zfs-linux on Ubuntu).
    + Use exact compiler for dkms as used to build the kernel, when
    possible.
    + Depend on gcc-12 to build modules for HWE kernels correctly.

 -- Dimitri John Ledkov <email address hidden> Tue, 04 Oct 2022 13:25:35 +0100

Changed in dkms (Ubuntu Jammy):
status: Fix Committed → Fix Released
Revision history for this message
Łukasz Zemczak (sil2100) wrote : Update Released

The verification of the Stable Release Update for dkms has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Revision history for this message
Ubuntu Kernel Bot (ubuntu-kernel-bot) wrote :

This bug is awaiting verification that the linux-hwe-5.19/5.19.0-24.25~22.04.1 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-jammy' to 'verification-done-jammy'. If the problem still exists, change the tag 'verification-needed-jammy' to 'verification-failed-jammy'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

tags: added: kernel-spammed-jammy-linux-hwe-5.19 verification-needed-jammy
removed: verification-done verification-done-jammy
Revision history for this message
You-Sheng Yang (vicamo) wrote (last edit ): Re: [Bug 1991664] Re: backport dkms fixes to build modules correctly for hwe-5.19+ kernels with custom compiler

Attach debdiff for ipu6-drivers/ivsc-driver for Lunar.

WIP. Please skip this for now.

Revision history for this message
Łukasz Zemczak (sil2100) wrote : Please test proposed package

Hello Dimitri, or anyone else affected,

Accepted v4l2loopback into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/v4l2loopback/0.12.7-2ubuntu2~22.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in v4l2loopback (Ubuntu Jammy):
status: New → Fix Committed
tags: added: verification-needed
Revision history for this message
Dimitri John Ledkov (xnox) wrote :

tl; dr

this has proven to be a bit futile, and instead we went ahead with changing the default kernel makefiles to default to the correct toolchain and we have reverted hwe-5.19 kernel to use stock gcc on jammy for now.

So the things described here are not strictly even needed in jammy for now.

tags: added: verification-done verification-done-jammy
removed: verification-needed verification-needed-jammy
description: updated
Revision history for this message
Dimitri John Ledkov (xnox) wrote :

v4l2loopback tested 0.12.7-2ubuntu2~22.04.1 to build on ga & hwe kernels in jammy.

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package zfs-linux - 2.1.5-1ubuntu6~22.04.1

---------------
zfs-linux (2.1.5-1ubuntu6~22.04.1) jammy; urgency=medium

  * No change, rebuild with the jammy toolchain (LP: #2003174).

 -- Andrea Righi <email address hidden> Wed, 18 Jan 2023 08:18:30 +0000

Changed in zfs-linux (Ubuntu Jammy):
status: New → Fix Released
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package v4l2loopback - 0.12.7-2ubuntu2~22.04.1

---------------
v4l2loopback (0.12.7-2ubuntu2~22.04.1) jammy; urgency=medium

  * Backport dkms support for HWE kernels. LP: #1999670

v4l2loopback (0.12.7-2ubuntu2) kinetic; urgency=medium

  * Preserve CC setting when invoking sub-make, to ensure module can be
    built correctly with the kernel's custom compiler. Fixes FTBFS in
    hwe-5.19 build. LP: #1991664

v4l2loopback (0.12.7-2ubuntu1) kinetic; urgency=medium

  * Merge from Debian, remaining changes:
    - drop module-assistant autopkgtest
    - add Ubuntu patch track active-readers
    - add Ubuntu patch implement support for V4L2_EVENT_PRI_CLIENT_USAGE

v4l2loopback (0.12.7-2) unstable; urgency=medium

  * Fix DKMS build on linux>=5.18

v4l2loopback (0.12.7-1) unstable; urgency=medium

  * New upstream version 0.12.7

  [ Andreas Beckmann ]
  * Switch to dh-sequence-dkms.
  * Switch to Testsuite: autopkgtest-pkg-dkms.

  [ Debian Janitor ]
  * Set upstream metadata fields: Bug-Database, Bug-Submit, Repository,
    Repository-Browse.
  * Remove constraints unnecessary since buster

  [ IOhannes m zmölnig ]
  * Drop patches applied upstream
  * Drop unnecessary B-Ds
  * Bump dh-compat to 13
  * Bump standards version to 4.6.1

v4l2loopback (0.12.5-1ubuntu7) kinetic; urgency=medium

  * Drop obsolte REMAKE_INITRD option from dkms.conf.
  * Update module init & exit calls to use kernel provided macros, such
    that they gain correct IBT attributes. LP: #1980484

v4l2loopback (0.12.5-1ubuntu6) kinetic; urgency=low

  * Check CONFIG_VIDEO_DEV against kernel version >= 5.18. (LP: #1970983)

 -- Dimitri John Ledkov <email address hidden> Mon, 16 Jan 2023 11:22:08 +0000

Changed in v4l2loopback (Ubuntu Jammy):
status: Fix Committed → Fix Released
You-Sheng Yang (vicamo)
no longer affects: ivsc-driver (Ubuntu Jammy)
Revision history for this message
You-Sheng Yang (vicamo) wrote :

Removed ipu6-drivers, ivsc-driver from the affected list for they're no longer affected.

no longer affects: ivsc-driver (Ubuntu)
no longer affects: ipu6-drivers (Ubuntu Jammy)
no longer affects: ipu6-drivers (Ubuntu)
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.