Ubuntu
dpdk package

DPDK dkms packages build for wrong kernel

Bug #1687492 reported by Patrick MacArthur on 2017-05-01

This bug affects 3 people

	Status	Importance	Assigned to
dpdk (Debian)	Fix Released	Unknown	debbugs #843864
dpdk (Ubuntu)	Fix Released	Undecided	Unassigned
Yakkety	Fix Released	Medium	Christian Ehrhardt 

Bug Description

[Impact]

* DPDK DKMS modules can be unusable depending on the kernel that was
running when triggering DKMS rebuilds last time

* Fix is a backport of the fix already in zesty for a while, see links in
comment # #1

* Essentially the fix passes the right kerneldir to the DKMS build

[Test Case]

* Set up a yakkety system with kernels (VM or BM).

* Install DPDK (this DKMS build for krernel "A" will work)

* Now install another kernel "B", the DKMS rebuild will be for "A" but
installed in "B". On reboot into "B" the module will fail

* With the fix applied on installing "B" the build will be for "B" and
install in "B" and thereby modules usable after reboot.

[Regression Potential]

* The changes are "only" affecting the DKMS configuration of DPDK
modules. Those DKMS modules are only used by a subset of DKMS setups
therefore the scope of potentially regressing systems should be small

* Also the fix is in zesty for quite a while now and worked fine there.

* Finally given the code change it should be fixing a case for those who
are broken at the moment, but not affect anyone else.

* Thereby overall I'd rate the risk low and definitely worth to fix the
case for those affected (Currently only usable with workaround to
re-trigger DKMS once in the new kernel).

[Other Info]

* n/a

---

The DPDK build system uses the RTE_KERNELDIR make variable to determine which kernel version to build for. DKMS provides the kernel version to build for in the $kernelver and $kernel_source_dir variables. Because you use a custom MAKE command, this must be used in the MAKE variable of dkms.conf to generate the correct RTE_KERNELDIR for the build, by doing something like this:

--- /usr/src/dpdk-rte-kni-16.07.2/dkms.conf 2017-05-01 18:23:13.242496251 -0400
+++ /usr/src/dpdk-rte-kni-16.07.2/dkms.conf.new 2017-05-01 18:23:20.638620058 -0400
@@ -1,7 +1,7 @@
PACKAGE_NAME="dpdk-rte-kni"
PACKAGE_VERSION="16.07.2"
BUILT_MODULE_NAME[0]="rte_kni"
-MAKE="source /usr/share/dpdk/dpdk-sdk-env.sh; make MODULE_CFLAGS='-I/usr/include/dpdk -include /usr/include/dpdk/rte_config.h -I$source_tree/dpdk-rte-kni-16.07.2/ethtool/ixgbe -I$source_tree/dpdk-rte-kni-16.07.2/ethtool/igb'"
+MAKE="source /usr/share/dpdk/dpdk-sdk-env.sh; make MODULE_CFLAGS='-I/usr/include/dpdk -include /usr/include/dpdk/rte_config.h -I$source_tree/dpdk-rte-kni-16.07.2/ethtool/ixgbe -I$source_tree/dpdk-rte-kni-16.07.2/ethtool/igb' RTE_KERNELDIR=$kernel_source_dir"
CLEAN="source /usr/share/dpdk/dpdk-sdk-env.sh; make clean"
DEST_MODULE_LOCATION[0]="/updates/dkms"
AUTOINSTALL="YES"

Because this is not done, whenever the kernel package is upgraded and DKMS triggers a rebuild, it will trigger a rebuild for the *running* kernel which gets installed into the *newly installed* kernel directory, which is wrong and causes the module to fail to load when the newly installed kernel is booted.

See original description

Tags:

Revision history for this message

Christian Ehrhardt  (paelzer) wrote on 2017-05-02:

Hi Patrick,
thanks for your report.
I sounded familiar to me and so it is - we fixed that a while ago in [1].

Also we reworked some of the related code later in [2] and [3] which made my memory a bit mixed there.
So back then I didn't consider code too far back to be affected but your report shows it is.

I checked and I see that Zesty already released 16.11 with the fix.
So there and later things are good.
Back in Xenial we had no dkms modules yet so good as well, only Yakkety in the middle will need an SRU for this.

Flagging bug tasks accordingly.

[1]: https://gerrit.fd.io/r/gitweb?p=deb_dpdk.git;a=commit;h=1dda674f3ed515ca67c4a833fbbdb8d01b0db93a
[2]: https://gerrit.fd.io/r/gitweb?p=deb_dpdk.git;a=blobdiff;f=debian/rules;h=03d6e443cc824b075f9132b17a95d9db3dbfdc08;hp=2b7db7573353296b314faa3cfb2e8b64077ab6dd;hb=bc603db838495a6149b866fbdf2413670a0c96f2;hpb=84adab31e2d11153f8f83eafc4cc54af5deb7806
[3]: https://gerrit.fd.io/r/gitweb?p=deb_dpdk.git;a=blobdiff;f=debian/rules;h=9e272aa649aef4c120a1fb65a9f7c8522400b5be;hp=03d6e443cc824b075f9132b17a95d9db3dbfdc08;hb=41d44cf3d64183ffff0d83a28ccb67b03c56414f;hpb=bc603db838495a6149b866fbdf2413670a0c96f2

Changed in dpdk (Ubuntu):
status:	New → Fix Released
Changed in dpdk (Ubuntu Yakkety):
status:	New → Triaged
importance:	Undecided → Medium

Revision history for this message

Christian Ehrhardt  (paelzer) wrote on 2017-05-02:

I added an SRU template for latter processing, but I'd be really thankful if you could verify the build in the ppa at [1] if you agree that it fixes the issue for your. Always better to not self-test.

It is building now, and should be available in approximately 45 minutes.

Once you can confirm that this is working for you and I had the time to QA check the upload a bit more I can push to Yakkety as SRU.

[1]: https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/2744

description:	updated
Changed in dpdk (Ubuntu Yakkety):
status:	Triaged → In Progress
assignee:	nobody → ChristianEhrhardt (paelzer)

Bug Watch Updater (bug-watch-updater) on 2017-05-02

Changed in dpdk (Debian):
status:	Unknown → Fix Released

Revision history for this message

Patrick MacArthur (patrickmacarthur) wrote on 2017-05-02:

I tested the updated packages in a VM and it appears to have fixed the issue. Thanks!

Revision history for this message

Christian Ehrhardt  (paelzer) wrote on 2017-05-03:

Thanks for the check, it also passed my basic tests - might run them again and more once in proposed then, but given the nature of the change it is unlikely to affect other things than DKMS builds of DPDK modules.

Revision history for this message

Christian Ehrhardt  (paelzer) wrote on 2017-05-03:

Ok, SRU template ready, PPA verified by reporter, Tests on PPA good, releasing to the Yakkety SRU queue now.
Next this bug should get an update once accepted (or not) to finally verify in the release via -proposed.

Changed in dpdk (Ubuntu Yakkety):
status:	In Progress → Fix Committed

Revision history for this message

Łukasz Zemczak (sil2100) wrote on 2017-05-09: Please test proposed package

Hello Patrick, or anyone else affected,

Accepted dpdk into yakkety-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/dpdk/16.07.2-0ubuntu0.16.10.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

tags:

added: verification-needed

Revision history for this message

Patrick MacArthur (patrickmacarthur) wrote on 2017-05-10:

dpdk-rte-kni-dkms-16.07.2-0ubuntu0.16.10.2 appears to fix the issue.

tags:

added: verification-done
removed: verification-needed

Revision history for this message

Brian Murray (brian-murray) wrote on 2017-05-18: Update Released

The verification of the Stable Release Update for dpdk has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Revision history for this message

Launchpad Janitor (janitor) wrote on 2017-05-18:

This bug was fixed in the package dpdk - 16.07.2-0ubuntu0.16.10.2

---------------
dpdk (16.07.2-0ubuntu0.16.10.2) yakkety; urgency=medium

  [ Luca Boccassi ]
  * Fix DKMS build to use the requested kernel version rather than the running
    one. (LP: #1687492)

-- Christian Ehrhardt <email address hidden> Tue, 02 May 2017 09:01:21 +0200

Changed in dpdk (Ubuntu Yakkety):
status:	Fix Committed → Fix Released

Revision history for this message

Pranav Sharma (sudopluto-deactivatedaccount) wrote on 2017-05-25:

#10

hi, i am having this issue in xenial with bbswitch

Revision history for this message

Christian Ehrhardt  (paelzer) wrote on 2017-05-29:

#11

Hi Pranav, the dpdk in Xenial doesn't have a DKMS module yet - so it can't build for the wrong kernel :-)

I assume you use the Ubuntu Cloud Archive or a ppa to trigger this.
Assuming you have UCA-Newton to hit this it will need some extra time to be picked up there as well.
I see it is already in newton-staging [2], so it should appear for you as well soon.

[1]: https://wiki.ubuntu.com/OpenStack/CloudArchive
[2]: https://launchpad.net/~ubuntu-cloud-archive/+archive/ubuntu/newton-staging/+packages

Revision history for this message

Pranav Sharma (sudopluto-deactivatedaccount) wrote on 2017-06-06:

#12

Sorry for not seeing this sooner. I have a xenial machine with the latest stable HWE kernel. Whenever there is a kernel upgrade, DKMS builds for the old kernel version. After each update I have to manually run "sudo dpkg-reconfigure bbswitch-dkms" Maybe I will take a look at how you fixed it in this package to fix in bbswitch-dkms.

Revision history for this message

Pranav Sharma (sudopluto-deactivatedaccount) wrote on 2017-06-06:

#13

I would appreciate it if someone could show me what all was changed. Was it just those three commits?

Revision history for this message

Christian Ehrhardt  (paelzer) wrote on 2017-06-07:

#14

Hi Pranav,
sometimes there are upstream commits about kernel compatibility - those have to be backported.
I can't promise the one we fixed here is the one for your particular case - in general we have one about every ~2-4 months.
This change here in particular was different as it was not a usual patch by DPDK for the modules, but instead a change in how we call our build.

That said via Launchpad you can easily take a look at the effective change which is http://launchpadlibrarian.net/318966241/dpdk_16.07.2-0ubuntu0.16.10.1_16.07.2-0ubuntu0.16.10.2.diff.gz

Hope that helps you.