fix build issue for v5.19 kernels (jammy-hwe, kinetic, lunar-riscv)
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
linux (Ubuntu) |
Invalid
|
Medium
|
Roxana Nicolescu | ||
Focal |
Invalid
|
Undecided
|
Roxana Nicolescu | ||
Jammy |
Invalid
|
Undecided
|
Roxana Nicolescu | ||
Kinetic |
Invalid
|
Undecided
|
Roxana Nicolescu | ||
Lunar |
Invalid
|
Undecided
|
Roxana Nicolescu | ||
Mantic |
Invalid
|
Medium
|
Roxana Nicolescu | ||
lttng-modules (Ubuntu) |
Won't Fix
|
Medium
|
Roxana Nicolescu | ||
Jammy |
Won't Fix
|
Undecided
|
Unassigned | ||
Kinetic |
Won't Fix
|
Undecided
|
Unassigned | ||
Lunar |
Won't Fix
|
Medium
|
Roxana Nicolescu |
Bug Description
SRU Justification
[ Impact ]
Upstream stable added a change in the format of jbd2 in 5.15.87 that was cherry-picked to kinetic:linux.
This is incompatible with the current changes in the lttng-module kinetic.
We previously encountered this issue for focal https:/
and we proactively released a new version for kinetic.
The problem is the fix does not really work as expected.
As shown in the logs below,
The build error is triggered because some function declaration does not match the kernel's.
In `include/
Build logs:
35:35 DEBUG| [stdout] In file included from /var/lib/
860 02:35:35 DEBUG| [stdout] from /var/lib/
861 02:35:35 DEBUG| [stdout] from /var/lib/
862 02:35:35 DEBUG| [stdout] /var/lib/
863 02:35:35 DEBUG| [stdout] 133 | void trace_#
864 02:35:35 DEBUG| [stdout] | ^~~~~~
865 02:35:35 DEBUG| [stdout] /var/lib/
866 02:35:35 DEBUG| [stdout] 45 | LTTNG_TRACEPOIN
867 02:35:35 DEBUG| [stdout] | ^~~~~~~
868 02:35:35 DEBUG| [stdout] /var/lib/
869 02:35:35 DEBUG| [stdout] 87 | LTTNG_TRACEPOIN
870 02:35:35 DEBUG| [stdout] | ^~~~~~~
871 02:35:35 DEBUG| [stdout] /var/lib/
872 02:35:35 DEBUG| [stdout] 180 | LTTNG_TRACEPOIN
873 02:35:35 DEBUG| [stdout] | ^~~~~~~
874 02:35:35 DEBUG| [stdout] In file included from ./include/
875 02:35:35 DEBUG| [stdout] from /var/lib/
876 02:35:35 DEBUG| [stdout] ./include/
877 02:35:35 DEBUG| [stdout] 245 | static inline void trace_##name(proto) \
878 02:35:35 DEBUG| [stdout] | ^~~~~~
879 02:35:35 DEBUG| [stdout] ./include/
880 02:35:35 DEBUG| [stdout] 419 | __DECLARE_
881 02:35:35 DEBUG| [stdout] | ^~~~~~~~~~~~~~~
882 02:35:35 DEBUG| [stdout] ./include/
883 02:35:35 DEBUG| [stdout] 553 | DECLARE_TRACE(name, PARAMS(proto), PARAMS(args))
884 02:35:35 DEBUG| [stdout] | ^~~~~~~~~~~~~
885 02:35:35 DEBUG| [stdout] ./include/
886 02:35:35 DEBUG| [stdout] 248 | TRACE_EVENT(
887 02:35:35 DEBUG| [stdout] | ^~~~~~~~~~~
888 02:35:35 DEBUG| [stdout] /var/lib/
889 02:35:35 DEBUG| [stdout] 133 | void trace_#
890 02:35:35 DEBUG| [stdout] | ^~~~~~
891 02:35:35 DEBUG| [stdout] /var/lib/
892 02:35:35 DEBUG| [stdout] 45 | LTTNG_TRACEPOIN
893 02:35:35 DEBUG| [stdout] | ^~~~~~~
894 02:35:35 DEBUG| [stdout] /var/lib/
895 02:35:35 DEBUG| [stdout] 87 | LTTNG_TRACEPOIN
896 02:35:35 DEBUG| [stdout] | ^~~~~~~
897 02:35:35 DEBUG| [stdout] /var/lib/
898 02:35:35 DEBUG| [stdout] 200 | LTTNG_TRACEPOIN
899 02:35:35 DEBUG| [stdout] | ^~~~~~~
900 02:35:35 DEBUG| [stdout] ./include/
901 02:35:35 DEBUG| [stdout] 245 | static inline void trace_##name(proto) \
902 02:35:35 DEBUG| [stdout] | ^~~~~~
903 02:35:35 DEBUG| [stdout] ./include/
904 02:35:35 DEBUG| [stdout] 419 | __DECLARE_
905 02:35:35 DEBUG| [stdout] | ^~~~~~~~~~~~~~~
906 02:35:35 DEBUG| [stdout] ./include/
907 02:35:35 DEBUG| [stdout] 553 | DECLARE_TRACE(name, PARAMS(proto), PARAMS(args))
908 02:35:35 DEBUG| [stdout] | ^~~~~~~~~~~~~
909 02:35:35 DEBUG| [stdout] ./include/
910 02:35:35 DEBUG| [stdout] 296 | TRACE_EVENT(
911 02:35:35 DEBUG| [stdout] | ^~~~~~~~~~~
912 02:35:35 DEBUG| [stdout] make[3]: *** [scripts/
913 02:35:35 DEBUG| [stdout] make[2]: *** [scripts/
914 02:35:35 DEBUG| [stdout] make[1]: *** [scripts/
915 02:35:35 DEBUG| [stdout] make: *** [Makefile:1850: /var/lib/
916 02:35:35 DEBUG| [stdout] make: Leaving directory '/usr/src/
917 02:35:35 ERROR| Exception escaping from test:
918 Traceback (most recent call last):
919 File "/home/
920 _cherry_
921 File "/home/
922 return func(*p_args, **p_dargs)
923 File "/home/
924 raise error.TestError
[ Fix ]
Initially, we tried adding a new interval [5.19.0 - 5.19.19]. But because 5.19.0 is not maintained upstream and there are no stable fixes from there, the actual upstream version has remained unchanged since last year. It is currently 5.19.17. Therefore we can use this version to make a distinction between kernels that have this commit and the ones that do not have it and it won't be compatible with older versions.
We expose the abi version of the kernel, but it alone will not solve
this issue because we deliver many derivatives, each with its own
version. Hence a base main abi version needs to be exposed in the headers.
Therefore this requires 2 steps:
1. Expose UTS_UBUNTU_
jammy:linux-
will be used. In case there is no inheritance, the actual kernel abi
will be used. This is needed for jammy, kinetic and lunar but it may be needed in the future for similar use cases, so it will be pushed to all releases.
2. Use this variable in the lttng-module to construct LTTNG_UBUNTU_
LTTNG_UBUNTU_
The lttng-module fix will be done only for lunar, kinetic, and jammy.
Attached there is a patch with the proposed solution, debdiffs for kinetic, lunar and jammy will be attached separately.
This requires a new release for kinetic and jammy because jammy 5.19 backports are affected as well. Also lunar, because some riscv kernels are still on 5.19 as well.
[ Where problems would occur ]
We may notice new failures in ubuntu_
[ Test Plan ]
Since we hit backward issues in the past, multiple kernel versions were used to make sure this is the proper fix.
General test instructions:
1. Install kinetic (disired version)
2. sudo apt install lttng-modules-dkms
3. Depending on the version it may work or not
4. Additionally lttng-modules regression test should be triggered to confirm everything works file.
Steps to run rt:
# Install packages for autotest
sudo apt-get install git python2 gdb python-yaml -y || sudo apt-get install git python gdb python-yaml -y
# Enable deb-src
sudo sed -i 's/^# deb-src/deb-src/' /etc/apt/
sudo apt-get update
git clone --depth=1 https:/
git clone --depth=1 https:/
rm -fr autotest/
ln -sf ~/autotest-
AUTOTEST_
My testing:
A. I built a kinetic kernel where I expose UTS_UBUNTU_
Results:
ubuntu@
ii lttng-modules-dkms 2.13.8-
RT results:
09:32:05 INFO | Found 325223 dd and 390615 context switches
09:32:05 INFO | PASSED (simple system call tracing with babeltrace)
09:32:05 INFO |
09:32:05 INFO | Summary: 7 passed, 0 failed
B. The break commit was introduced in 5.19.0.42. Versions between 42 and 45(hopefully this will land in the next release) will break.
C. Tested with a kernel version that does not have the break commit: 5.19.0-41 to see if it's backwards compatible.
ubuntu@kinetic:~$ uname -a
Linux kinetic 5.19.0-41-generic #42-Ubuntu SMP PREEMPT_DYNAMIC Mon Apr 17 19:17:03 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
ubuntu@
ii lttng-modules-dkms 2.13.8-
D. Tested with jammy-aws-5.19 that has the commit with the fix:
ubuntu@
Linux jammmy-lttng 5.19.0-1027-aws #28~22.04.1 SMP Mon Jun 5 08:27:20 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
~
Build is fine
ubuntu@
ii lttng-modules-dkms 2.13.8-
E. Tested with linux-aws-
ubuntu@
ii lttng-modules-dkms 2.13.8-
ubuntu@
Linux jammmy-lttng 5.19.0-1024-aws #25~22.04.1-Ubuntu SMP Tue Apr 18 23:41:58 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
description: | updated |
tags: | added: sru-20230417 |
Changed in lttng-modules (Ubuntu): | |
assignee: | nobody → Roxana Nicolescu (roxanan) |
importance: | Undecided → Medium |
Changed in lttng-modules (Ubuntu Lunar): | |
status: | New → Fix Released |
status: | Fix Released → New |
description: | updated |
summary: |
- fix build issue for kinetic + fix build issue for v5.19 kernels (jammy-hwe, kinetic, lunar-riscv) |
Changed in lttng-modules (Ubuntu Lunar): | |
status: | New → In Progress |
Changed in lttng-modules (Ubuntu Kinetic): | |
status: | New → In Progress |
Changed in lttng-modules (Ubuntu Jammy): | |
status: | New → In Progress |
description: | updated |
description: | updated |
Changed in lttng-modules (Ubuntu Lunar): | |
status: | Incomplete → In Progress |
Changed in lttng-modules (Ubuntu): | |
status: | Incomplete → In Progress |
description: | updated |
Changed in linux (Ubuntu Jammy): | |
status: | Incomplete → Confirmed |
Changed in linux (Ubuntu Kinetic): | |
status: | Incomplete → Confirmed |
Changed in linux (Ubuntu Lunar): | |
status: | Incomplete → Confirmed |
no longer affects: | lttng-modules (Ubuntu Focal) |
no longer affects: | lttng-modules (Ubuntu Mantic) |
Changed in linux (Ubuntu Focal): | |
assignee: | nobody → Roxana Nicolescu (roxanan) |
status: | New → In Progress |
Changed in linux (Ubuntu Jammy): | |
assignee: | nobody → Roxana Nicolescu (roxanan) |
status: | Confirmed → In Progress |
Changed in linux (Ubuntu Kinetic): | |
assignee: | nobody → Roxana Nicolescu (roxanan) |
status: | Confirmed → In Progress |
Changed in linux (Ubuntu Lunar): | |
assignee: | nobody → Roxana Nicolescu (roxanan) |
status: | Confirmed → In Progress |
Changed in linux (Ubuntu Mantic): | |
assignee: | nobody → Roxana Nicolescu (roxanan) |
status: | Confirmed → In Progress |
Changed in lttng-modules (Ubuntu): | |
status: | In Progress → Won't Fix |
Changed in lttng-modules (Ubuntu Jammy): | |
status: | Fix Committed → Won't Fix |
Changed in lttng-modules (Ubuntu Kinetic): | |
status: | Confirmed → Won't Fix |
Changed in lttng-modules (Ubuntu Lunar): | |
status: | Fix Committed → Won't Fix |
Changed in linux (Ubuntu Focal): | |
status: | In Progress → Invalid |
Changed in linux (Ubuntu Jammy): | |
status: | In Progress → Invalid |
Changed in linux (Ubuntu Kinetic): | |
status: | In Progress → Invalid |
Changed in linux (Ubuntu Lunar): | |
status: | In Progress → Invalid |
Changed in linux (Ubuntu Mantic): | |
status: | In Progress → Invalid |
Kinetic fix is attached.
Tested on a VM with the latest kinetic version : 5.19.0-42-generic kinetic: ~/lttng- modules$ dpkg -l | grep lttng-modules 1~ubuntu22. 10.1 all Linux Trace Toolkit (LTTng) kernel modules (DKMS)
ubuntu@
ii lttng-modules-dkms 2.13.8-