systemtap fails to discover installed debug modules

eu-readelf is provided by elfutils and is required to run the attached script.

added a kernel task for this. Will investigate and go from there.

~JFo

You might also want to take a look at how we fixed this in Debian.

On 11/02/2010 02:04 AM, Peter Petrakis wrote:
> # stap -l 'kernel.function("acpi_*")' | sort
>
> will succeed
>
> and this will fail.
>
> # stap -l 'module("ohci1394").function("*")' | sort
>
> Now there's no shortage of blogs and wikis on how to work
> around this but no one has seemingly ever investigated the root
> cause. If one were to run this through strace you would find
> the following.
>
> open("/usr/lib/debug/.build-id/3b/6eb5a0f22ba2bc92c3c3f1fcb14fe7f31f3807.debug", O_RDONLY) = -1 ENOENT (No such file or directory)
> open("/lib/modules/2.6.32-25-generic/kernel/drivers/ieee1394/ohci1394.ko.debug", O_RDONLY) = -1 ENOENT (No such file or directory)
> open("/lib/modules/2.6.32-25-generic/kernel/drivers/ieee1394/.debug/ohci1394.ko.debug", O_RDONLY) = -1 ENOENT (No such file or directory)
> open("/usr/lib/debug/lib/modules/2.6.32-25-generic/kernel/drivers/ieee1394/ohci1394.ko.debug", O_RDONLY) = -1 ENOENT (No such file or directory)
> open("/lib/modules/2.6.32-25-generic/kernel/drivers/ieee1394/build/ohci1394.ko.debug", O_RDONLY) = -1 ENOENT (No such file or directory)
>
> Note that the path is correct, it's the name of the KO that stap is expecting that is wrong,
> Kernel debug symbols provided by Ubuntu are unstripped yet maintain the .ko extension,
> systemtap (actually libelf/elfutils) is expecting the filename to be module.ko.debug.
>
> After discussing this on #systemtap on Freenode, the following script was proposed to
> generate a symlink tree, by build id, in /usr/lib/debug, with the proper extension. This
> completely solves the issue, and is also of benefit to things like gdb and oprofile.
>
> #!/bin/sh
>
> for file in `find /usr/lib/debug -name '*.ko' -print`
> do
> buildid=`eu-readelf -n $file| grep Build.ID: | awk '{print $3}'`
> dir=`echo $buildid | cut -c1-2`
> fn=`echo $buildid | cut -c3-`
> mkdir -p /usr/lib/debug/.build-id/$dir
> ln -s $file /usr/lib/debug/.build-id/$dir/$fn
> ln -s $file /usr/lib/debug/.build-id/$dir/${fn}.debug
> done
>
> If we could integrate this into the ddeb postinstall script, the problem would
> be solved.
>
>

--
Ritesh Raj Sarraf | http://people.debian.org/~rrs
Debian - The Universal Operating System

Ritesh, can you provide a pointer to the details?

Hello Frank,

http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=555549

This bug report should have all the information.

Ritesh

On 11/03/2010 01:10 AM, Frank Ch. Eigler wrote:
> Ritesh, can you provide a pointer to the details?
>
>

--
Ritesh Raj Sarraf
RESEARCHUT - http://www.researchut.com
"Necessity is the mother of invention."

Fixed at the linux-linaro packages by commit:
http://git.linaro.org/gitweb?p=ubuntu/linux-linaro-oneiric.git;a=commitdiff;h=4a0c288fa1eddb0b9183f5169c88cef8db99ff54;hp=8035803c27c612dde7607c33dfd8d96f3c53cdb1

Thank you for triaging this. Do I understand it correctly that you are going to fix this in the kernel package and I don't need to change anything in systemtap packaging?

This patch is taken from the Linaro packaging patch, I'm building the .debs now for testing.

You can download a quantal amd64 build here:
http://people.canonical.com/~arges/lp669641/

Please test this with this package and see if systemtap is working properly.
--chris

The attachment "0001-UBUNTU-PACKAGING-add-.gnu_debuglink-sections-to-.ko-.patch" of this bug report has been identified as being a patch. The ubuntu-reviewers team has been subscribed to the bug report so that they can review the patch. In the event that this is in fact not a patch you can resolve this situation by removing the tag 'patch' from the bug report and editing the attachment so that it is not flagged as a patch. Additionally, if you are member of the ubuntu-reviewers team please also unsubscribe the team from this bug report.

[This is an automated message performed by a Launchpad user owned by Brian Murray. Please contact him regarding any issues with the action taken in this bug report.]

The linaro patch alone doesn't work. objcopy fails. I'll upload the exact build error.

@Chris, this doesn't work.

BTW, you don't have to bounce this back to me to verify, this is
a one line test, either the probed module will return a list of
entry points, or it won't. Also, please provide the main "headers"
package next time, the archive is marching on and deleted the
last build.

root@ubuntu:~# uname -a
Linux ubuntu 3.4.0-3-generic #7~lp669641v201205242144 SMP Thu May 24 21:45:43 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux

root@ubuntu:~# stap -l 'module("psmouse").function("*")'
root@ubuntu:~#

There's no debug links section either.

root@ubuntu:~# objdump -h /lib/modules/3.4.0-3-generic/kernel/drivers/input/mouse/psmouse.ko
/lib/modules/3.4.0-3-generic/kernel/drivers/input/mouse/psmouse.ko: file format elf64-x86-64

Sections:
Idx Name Size VMA LMA File off Algn
  0 .note.gnu.build-id 00000024 0000000000000000 0000000000000000 00000040 2**2
                  CONTENTS, ALLOC, LOAD, READONLY, DATA
  1 .text 0000ca84 0000000000000000 0000000000000000 00000070 2**4
                  CONTENTS, ALLOC, LOAD, RELOC, READONLY, CODE
  2 .init.text 000000cb 0000000000000000 0000000000000000 0000caf4 2**0
                  CONTENTS, ALLOC, LOAD, RELOC, READONLY, CODE
  3 .exit.text 0000001e 0000000000000000 0000000000000000 0000cbbf 2**0
                  CONTENTS, ALLOC, LOAD, RELOC, READONLY, CODE
  4 .rodata 000005f0 0000000000000000 0000000000000000 0000cbe0 2**5
                  CONTENTS, ALLOC, LOAD, RELOC, READONLY, DATA
  5 .rodata.str1.1 00000522 0000000000000000 0000000000000000 0000d1d0 2**0
                  CONTENTS, ALLOC, LOAD, READONLY, DATA
  6 .rodata.str1.8 00000f3d 0000000000000000 0000000000000000 0000d6f8 2**3
                  CONTENTS, ALLOC, LOAD, READONLY, DATA
  7 .parainstructions 0000007c 0000000000000000 0000000000000000 0000e638 2**3
                  CONTENTS, ALLOC, LOAD, RELOC, READONLY, DATA
  8 .modinfo 00000330 0000000000000000 0000000000000000 0000e6b4 2**0
                  CONTENTS, ALLOC, LOAD, READONLY, DATA
  9 __param 000000c0 0000000000000000 0000000000000000 0000e9e8 2**3
                  CONTENTS, ALLOC, LOAD, RELOC, READONLY, DATA
 10 __mcount_loc 000004d0 0000000000000000 0000000000000000 0000eaa8 2**3
                  CONTENTS, ALLOC, LOAD, RELOC, READONLY, DATA
 11 .init.rodata 00001998 0000000000000000 0000000000000000 0000ef80 2**5
                  CONTENTS, ALLOC, LOAD, RELOC, READONLY, DATA
 12 __bug_table 0000000c 0000000000000000 0000000000000000 00010918 2**0
                  CONTENTS, ALLOC, LOAD, RELOC, READONLY, DATA
 13 .smp_locks 00000010 0000000000000000 0000000000000000 00010924 2**2
                  CONTENTS, ALLOC, LOAD, RELOC, READONLY, DATA
 14 __versions 00001400 0000000000000000 0000000000000000 00010940 2**5
                  CONTENTS, ALLOC, LOAD, READONLY, DATA
 15 .data 00000e90 0000000000000000 0000000000000000 00011d40 2**5
                  CONTENTS, ALLOC, LOAD, RELOC, DATA
 16 .gnu.linkonce.this_module 00000258 0000000000000000 0000000000000000 00012be0 2**5
                  CONTENTS, ALLO...

OK, so I think you have a build issue, as I can add the run the objcopy
line alone and verify the heading was added.

root@ubuntu:/usr/lib/debug# lsb_release -r
Release: 12.04
root@ubuntu:/usr/lib/debug# uname -a
Linux ubuntu 3.4.0-3-generic #7~lp669641v201205242144 SMP Thu May 24 21:45:43 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux

root@ubuntu:~# objcopy \
> --add-gnu-debuglink=/usr/lib/debug/lib/modules/3.4.0-3-generic/kernel/drivers/scsi/libiscsi.ko /lib/modules/3.4.0-3-generic/kernel/drivers/scsi/libiscsi.ko

root@ubuntu:~# objdump -h /lib/modules/3.4.0-3-generic/kernel/drivers/scsi/libiscsi.ko | grep -A2 debuglink
 21 .gnu_debuglink 00000010 0000000000000000 0000000000000000 00009aae 2**0
                  CONTENTS, READONLY

but systemtap still fumbles over it, no entries found.

root@ubuntu:~# stap -l 'module("libiscsi").function("*")'

But strace confirms that the debug file was opened

stat("/lib/modules/3.4.0-3-generic/kernel/drivers/scsi/libiscsi.ko", {st_mode=S_IFREG|0644, st_size=87192, ...}) = 0
open("/lib/modules/3.4.0-3-generic/kernel/drivers/scsi/libiscsi.ko", O_RDONLY) = 4
fstat(4, {st_mode=S_IFREG|0644, st_size=87192, ...}) = 0
close(4) = 0
open("/lib/modules/3.4.0-3-generic/kernel/drivers/scsi/.debug/libiscsi.ko", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/lib/debug/lib/modules/3.4.0-3-generic/kernel/drivers/scsi/libiscsi.ko", O_RDONLY) = 4
fstat(4, {st_mode=S_IFREG|0644, st_size=689768, ...}) = 0
fcntl(4, F_GETFL) = 0x8000 (flags O_RDONLY|O_LARGEFILE)
fstat(4, {st_mode=S_IFREG|0644, st_size=689768, ...}) = 0
mmap(NULL, 689768, PROT_READ|PROT_WRITE, MAP_PRIVATE, 4, 0) = 0x7faaa4333000
munmap(0x7faaa4333000, 689768) = 0
close(4) = 0
close(4) = -1 EBADF (Bad file descriptor)

So what went wrong? When I go back and apply the script that
creates the shadow links to the debug files the result is the same, no
output, regardless of the module I choose to test.

I think your build is broken.

Got a chance to retest against an officially built kernel and dbgsym, the technique works, so
now all that's left is to independently verify the entire patch on a new build and call
it done.

root@ubuntu:~# uname -a
Linux ubuntu 3.2.0-23-generic #36-Ubuntu SMP Tue Apr 10 20:39:51 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux

cat > /etc/apt/sources.list.d/ddebs.list << EOF
deb http://ddebs.ubuntu.com/ precise main restricted universe multiverse
EOF

apt-key adv --keyserver keyserver.ubuntu.com --recv-keys ECDCAD72428D7C01
apt-get update
apt-get install linux-image-$(uname -r)-dbgsym

serio='/lib/modules/3.2.0-23-generic/kernel/drivers/input/serio/serio_raw.ko'
dbg='/usr/lib/debug/lib/modules/3.2.0-23-generic/kernel/drivers/input/serio/serio_raw.ko'

objcopy --add-gnu-debuglink=$dbg $serio
objdump -h $serio | grep -A2 debuglink

root@ubuntu:~# stap -l 'module("serio_raw").function("*")'
module("serio_raw").function("INIT_LIST_HEAD@/build/buildd/linux-3.2.0/include/linux/list.h:24")
module("serio_raw").function("__list_add@/build/buildd/linux-3.2.0/include/linux/list.h:37")
module("serio_raw").function("__list_del@/build/buildd/linux-3.2.0/include/linux/list.h:86")
module("serio_raw").function("__list_del_entry@/build/buildd/linux-3.2.0/include/linux/list.h:99")
...

any other module I try to probe comes up empty.

Building against precise is interesting, at first we thought (ayan and myself) that
we were modifying the same object twice which results in an error. To remedy
that, sort -u was added to the processing loop. After another precise build
with that precaution we get this failure.

  MKDIR /home/ppetraki/ubuntu-precise/debian/linux-image-3.2.0-26-generic-dbgsym/usr/lib/debug/lib/firmware/yam
  INSTALL debian/linux-image-3.2.0-26-generic-dbgsym/usr/lib/debug/lib/firmware/yam/1200.bin
  INSTALL debian/linux-image-3.2.0-26-generic-dbgsym/usr/lib/debug/lib/firmware/yam/9600.bin
  DEPMOD 3.2.0-26-generic
make[1]: Leaving directory `/home/ppetraki/ubuntu-precise'
# Add .gnu_debuglink sections to each stripped .ko
# pointing to unstripped verson
find /home/ppetraki/ubuntu-precise/debian/linux-image-3.2.0-26-generic -name '*.ko' | sed 's|/home/ppetraki/ubuntu-precise/debian/linux-image-3.2.0-26-generic||'| sort -u | while read module ; do \
  objcopy \
   --add-gnu-debuglink=/home/ppetraki/ubuntu-precise/debian/linux-image-3.2.0-26-generic-dbgsym/usr/lib/debug/$module \
   /home/ppetraki/ubuntu-precise/debian/linux-image-3.2.0-26-generic/$module; \
 done
objcopy:/home/ppetraki/ubuntu-precise/debian/linux-image-3.2.0-26-generic//lib/modules/3.2.0-26-generic/initrd/sttHRG8l: cannot fill debug link section `/home/ppetraki/ubuntu-precise/debian/linux-image-3.2.0-26-generic-dbgsym/usr/lib/debug//lib/modules/3.2.0-26-generic/initrd/vesafb.ko': No such file or directory
make: *** [install-generic] Error 1

There is no "initrd" dir in the debug symbols tree

on an installed system...
root@ubuntu:~# find /usr/lib/debug -name vesafb.ko
/usr/lib/debug/lib/modules/3.2.0-23-generic/kernel/drivers/video/vesafb.ko

back to the build.

./debian/build/build-generic/drivers/video/vesafb.ko
./debian/linux-image-3.2.0-26-generic-dbgsym/usr/lib/debug/lib/modules/3.2.0-26-generic/kernel/drivers/video/vesafb.ko
./debian/linux-image-3.2.0-26-generic/lib/modules/3.2.0-26-generic/kernel/drivers/video/vesafb.ko
./debian/linux-image-3.2.0-26-generic/lib/modules/3.2.0-26-generic/initrd/vesafb.ko

ppetraki@tangerine:~/ubuntu-precise$ cmp ./debian/linux-image-3.2.0-26-generic/lib/modules/3.2.0-26-generic/kernel/drivers/video/vesafb.ko ./debian/linux-image-3.2.0-26-generic/lib/modules/3.2.0-26-generic/initrd/vesafb.ko
ppetraki@tangerine:~/ubuntu-precise$ echo $?
0

Perhaps we should attempt this sections update earlier in the build? Or we may have to get smarter
about evaluating our destination arg for --add-gnu-debuglink=${...}.

The root cause of the build failure under precise was that we were trying to assign
an entry to debuglink that didn't exist. When the build was iterating over
kos, it simply expects that the path in /lib/modules will mirror that found
in /usr/lib/debug, this is not always true. There are modules copied to
places like /lib/modules/3.2.0-25-generic/initrd/, of which there exists no
analog under the current debug tree structure, nor should there as it's a
copy of a perfectly probable module already found in kernel/

This updated v2 patches checks that the destination debug ko exists before
attempt to update the gnu_debuglink section. It's also noteworthy that
objcopy isn't just recording a pathname into that space, it's reading the unstripped
ko.

I've tested this patch extensively on precise, including developer builds where
I delete the build stamp, change the source code for a single module and incrementally
rebuild the entire package. The dbgsyms remain consistent.

I've developed a test suite to verify this new part of the build but am not
sure how to integrate it. Attached.

You can download a precise amd64 build here:

Please test this with this package and see if systemtap is working properly.

http://people.canonical.com/~ppetraki/bug-669641/

I hesitate to upload the ddeb as it's comparable to an iso in size. It's
probably faster to rebuild the kernel than download the ddeb.

git://kernel.ubuntu.com/ubuntu/ubuntu-precise.git

apply patch and build https://wiki.ubuntu.com/Kernel/Dev/QuickBuildLocal

schroot -c precise-amd64 fakeroot debian/rules binary-generic skipdbg=false

Ok. I can verify this also works on Quantal. Formatting the SRU.

Here is the submitted patch.

This bug was fixed in the package linux - 3.5.0-2.2

---------------
linux (3.5.0-2.2) quantal-proposed; urgency=low

  [ Andy Whitcroft ]

  * rebase to v3.5-rc4

  [ Arend van Spriel ]

  * SAUCE: (drop after 3.5) brcmsmac: fix NULL pointer crash in
    brcms_c_regd_init()
    - LP: #950320

  [ Bryan Wu ]

  * [Config] Sync CONFIG_CGROUP_MEM_RES_CTLR_SWAP for ARM

  [ Chris J Arges ]

  * PACKAGING: add .gnu_debuglink sections to .ko files
    - LP: #669641

  [ Leann Ogasawara ]

  * d-i: Add hid-generic to input-modules
    - LP: #1017879

  [ Ming Lei ]

  * SAUCE: Revert "mmc: omap_hsmmc: Enable Auto CMD12"
    - LP: #1017717, #225

  [ Paolo Pisati ]

  * SAUCE: Revert "Fix OMAP EHCI suspend/resume failure (i693)"
    - LP: #1017718
  * [Config] Disable generic USB_EHCI_HCD_PLATFORM on omap3

  [ Seth Forshee ]

  * SAUCE: (drop after 3.5) brcm80211: smac: don't set up tx power limits
    during initialization
    - LP: #950320
  * SAUCE: (drop after 3.5) brcm80211: smac: always set channel specified
    by mac80211
    - LP: #950320
  * SAUCE: (drop after 3.5) brcm80211: smac: remove unused code for 40MHz
    channels
    - LP: #950320
  * SAUCE: (drop after 3.5) brcm80211: smac: clean up channel.c
    - LP: #950320
  * SAUCE: (drop after 3.5) brcm80211: smac: inform mac80211 of the X2
    regulatory domain
    - LP: #950320
  * SAUCE: (drop after 3.5) brcm80211: smac: enable/disable radio on
    regulatory updates
    - LP: #950320
  * SAUCE: (drop after 3.5) brcm80211: smac: use mac80211 channel data for
    tx power limits
    - LP: #950320
  * SAUCE: (drop after 3.5) brcm80211: smac: don't validate channels
    against internal regulatory data
    - LP: #950320
  * SAUCE: (drop after 3.5) brcm80211: smac: use current regulatory domain
    when checking whether OFDM is allowed
    - LP: #950320

  [ Tim Gardner ]

  * [Config] Enable CONFIG_CGROUPS for highbank
    - LP: #1014692
  * [Config] FB_OMAP*=y and PANEL_TFP410=y

  [ Upstream Kernel Changes ]

  * rebase to v3.5-rc4
 -- Leann Ogasawara <email address hidden> Tue, 26 Jun 2012 06:21:05 -0700

This bug is awaiting verification that the kernel for Precise in -proposed solves the problem (3.2.0-27.43). Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-precise' to 'verification-done-precise'.

If verification is not done by one week from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

= summary =
 * looks good

# apt-get install -t precise-proposed linux-image-generic
# apt-get install -t precise-proposed linux-headers-3.2.0-27-generic
# apt-get install -t precise-proposed linux-image-3.2.0-27-generic-dbgsym
# apt-get install systemtap

# reboot

# zgrep -B2 669641 /usr/share/doc/linux-image-$(uname -r)/changelog.Debian.gz

  * PACKAGING: add .gnu_debuglink sections to .ko files
    - LP: #669641

# stap -l 'module("serio_raw").function("*")'
module("serio_raw").function("INIT_LIST_HEAD@/build/buildd/linux-3.2.0/include/linux/list.h:24")
module("serio_raw").function("__list_add@/build/buildd/linux-3.2.0/include/linux/list.h:37")
module("serio_raw").function("__list_del@/build/buildd/linux-3.2.0/include/linux/list.h:86")
module("serio_raw").function("__list_del_entry@/build/buildd/linux-3.2.0/include/linux/list.h:99")
module("serio_raw").function("arch_local_irq_enable@/build/buildd/linux-3.2.0/arch/x86/include/asm/paravirt.h:881")
...

# stap -l 'kernel.function("acpi_*")' | sort
kernel.function("acpi_ac_add@/build/buildd/linux-3.2.0/drivers/acpi/ac.c:263")
kernel.function("acpi_ac_add_fs@/build/buildd/linux-3.2.0/drivers/acpi/ac.c:196")
kernel.function("acpi_ac_get_state@/build/buildd/linux-3.2.0/drivers/acpi/ac.c:108")
kernel.function("acpi_ac_notify@/build/buildd/linux-3.2.0/drivers/acpi/ac.c:236")
kernel.function("acpi_ac_open_fs@/build/buildd/linux-3.2.0/drivers/acpi/ac.c:191")
...

Great, thanks for testing. Tagging as verified as per comment #24.

This bug was fixed in the package linux - 3.2.0-27.43

---------------
linux (3.2.0-27.43) precise-proposed; urgency=low

  [ Andy Whitcroft ]

  * No change upload to fix .ddeb generation in the PPA.

  [ Luis Henriques ]

  * Release Tracking Bug
    - LP: #1020016

linux (3.2.0-27.42) precise-proposed; urgency=low

  [Luis Henriques]

  * Release Tracking Bug
    - LP: #1020016

  [ Chris J Arges ]

  * PACKAGING: add .gnu_debuglink sections to .ko files
    - LP: #669641

  [ Ike Panhc ]

  * [Config] Updateconfigs
    - LP: #1008345

  [ Luis Henriques ]

  * SAUCE: (upstreamed) [media] ene_ir: Fix driver initialisation
    - LP: #1014800
  * SAUCE: ocfs2: Fix NULL pointer dereferrence in
    __ocfs2_change_file_space
    - LP: #1006012

  [ Rob Herring ]

  * SAUCE: net: calxedaxgmac: enable rx cut-thru mode
    - LP: #1008345
  * SAUCE: EDAC: Add support for the highbank platform memory
    - LP: #1008345
  * SAUCE: EDAC: add support for highbank platform L2 cache ecc
    - LP: #1008345

  [ Seth Forshee ]

  * (pre-stable): bcma: add ext PA workaround for BCM4331 and BCM43431
    - LP: #925577

  [ Takashi Iwai ]

  * SAUCE: ALSA: hda - Fix power-map regression for HP dv6 & co
    - LP: #1013183

  [ Tim Gardner ]

  * [Config] Enable CONFIG_CGROUPS for highbank
    - LP: #1014692

  [ Upstream Kernel Changes ]

  * Revert "net: maintain namespace isolation between vlan and real device"
    - LP: #1013723
  * x86/amd: Re-enable CPU topology extensions in case BIOS has disabled it
    - LP: #1009087
  * hwmon: (k10temp) Add support for AMD Trinity CPUs
    - LP: #1009086
  * hwmon: (fam15h_power) Increase output resolution
    - LP: #1009086
  * Input: wacom - use BTN_TOOL_FINGER to indicate touch device type
    - LP: #1009435
  * Input: wacom - use switch statement for wacom_tpc_irq()
    - LP: #1009435
  * Input: wacom - isolate input registration
    - LP: #1009435
  * Input: wacom - wireless monitor framework
    - LP: #1009435
  * Input: wacom - create inputs when wireless connect
    - LP: #1009435
  * Input: wacom - wireless battery status
    - LP: #1009435
  * Input: wacom - check for allocation failure in probe()
    - LP: #1009435
  * Input: wacom - add basic Intuos5 support
    - LP: #1009435
  * Input: wacom - add Intuos5 Touch Ring/ExpressKey support
    - LP: #1009435
  * Input: wacom - add Intuos5 Touch Ring LED support
    - LP: #1009435
  * Input: wacom - add Intuos5 multitouch sensor support
    - LP: #1009435
  * iommu/amd: Add workaround for event log erratum
    - LP: #1013723
  * MIPS: BCM63XX: Add missing include for bcm63xx_gpio.h
    - LP: #1013723
  * cifs: Include backup intent search flags during searches {try #2)
    - LP: #1013723
  * sunrpc: fix loss of task->tk_status after rpc_delay call in
    xprt_alloc_slot
    - LP: #1013723
  * exofs: Fix CRASH on very early IO errors.
    - LP: #1013723
  * cifs: fix oops while traversing open file list (try #4)
    - LP: #1013723
  * Fix dm-multipath starvation when scsi host is busy
    - LP: #1013723
  * ixp4xx: fix compilation by adding gpiolib support
    - LP: #1013723
  * drm/i915: properly handle interlaced bit for sdvo dtd conversion
    - LP: #1013723
...

The verification of this Stable Release Update has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regresssions.