Ubuntu 16.04 - Full EEH Recovery Support for NVMe devices

------- Comment From <email address hidden> 2016-07-18 10:24 EDT-------
*** Bug 143100 has been marked as a duplicate of this bug. ***

https://lists.ubuntu.com/archives/kernel-team/2016-July/079171.html

Test kernel at http://people.canonical.com/~rtg/eeh-lp1602724/

------- Comment From <email address hidden> 2016-08-08 15:05 EDT-------
(In reply to comment #9)
> I Have Tested this with , Test Kernel available at
> http://people.canonical.com/~rtg/eeh-lp1602724/ . on Ubuntu 16.04.1
>
> Test Kernel :
> root@everest-lp13-leaf:~# uname -a
> Linux everest-lp13-leaf 4.4.0-32-generic #51 SMP Tue Jul 19 21:41:04 UTC
> 2016 ppc64le ppc64le ppc64le GNU/Linux
>
>
>
> Nvme (Leaf) is getting recovered till 5 times on triggering the EEH, But
> "hitting a kernel crash" after on 6th time trigger of EEH.
>

This is most likely fixed by

http://lists.infradead.org/pipermail/linux-nvme/2016-August/005670.html
("[PATCH v2] nvme: Suspend all queues before deletion")

Which is not upstream yet. Once it gets accepted, it should be pushed to Ubuntu via another bugzilla. When that happens, we'll need a new test kernel for this one.

------- Comment From <email address hidden> 2016-08-08 15:09 EDT-------
(In reply to comment #13)
> (In reply to comment #9)
> > I Have Tested this with , Test Kernel available at
> > http://people.canonical.com/~rtg/eeh-lp1602724/ . on Ubuntu 16.04.1
> >
> > Test Kernel :
> > root@everest-lp13-leaf:~# uname -a
> > Linux everest-lp13-leaf 4.4.0-32-generic #51 SMP Tue Jul 19 21:41:04 UTC
> > 2016 ppc64le ppc64le ppc64le GNU/Linux
> >
> >
> >
> > Nvme (Leaf) is getting recovered till 5 times on triggering the EEH, But
> > "hitting a kernel crash" after on 6th time trigger of EEH.
> >
>
> This is most likely fixed by
>
> http://lists.infradead.org/pipermail/linux-nvme/2016-August/005670.html
> ("[PATCH v2] nvme: Suspend all queues before deletion")
>
> Which is not upstream yet. Once it gets accepted, it should be pushed to
> Ubuntu via another bugzilla. When that happens, we'll need a new test
> kernel for this one.

Canonical,

For a little more context, we have identified an issue in the test kernel you provided. After a sequence of 6 EEHs, DD will attempt to remove the adapter, which ends up hitting a BUG_ON.

We think the above patch is a fix, but it's still not confirmed. Can you provide a kernel with that patch also applied for verification? It's still not upstream yet, but it has already been ack-ed by the driver maintainer, Keith Busch.

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-xenial' to 'verification-done-xenial'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

Test kernel at http://people.canonical.com/~rtg/eeh-lp1602724/ with upstream commit c21377f8366c95440d533edbe47d070f662c62ef ('nvme: Suspend all queues before deletion') applied.

------- Comment From <email address hidden> 2016-08-22 13:11 EDT-------
(In reply to comment #16)
> Test kernel at http://people.canonical.com/~rtg/eeh-lp1602724/ with upstream
> commit c21377f8366c95440d533edbe47d070f662c62ef ('nvme: Suspend all queues
> before deletion') applied.

This test kernel is not ok, it stalls the wq:

[ 540.097661] INFO: rcu_sched detected stalls on CPUs/tasks:
[ 540.103320] 1-...: (1 GPs behind) idle=d35/140000000000000/0 softirq=2385/2386 fqs=65
[ 540.103411] (detected by 11, t=5472 jiffies, g=1335, c=1334, q=793)
[ 540.103492] Task dump for CPU 1:
[ 540.103539] kworker/u32:1 D 0000000000000000 0 101 0 0x00000800
[ 540.103656] Call Trace:
[ 540.103692] [c00000017bc539c0] [c00000017bc53a00] 0xc00000017bc53a00 (unreliable)
[ 540.103805] [c00000017bc53a00] [d000000001614480] nvme_suspend_queue+0x30/0x150 [nvme]
[ 540.103914] [c00000017bc53a30] [d000000001616850] nvme_dev_disable+0x110/0x440 [nvme]
[ 540.104022] [c00000017bc53b10] [d000000001617e60] nvme_reset_work+0xe0/0x1120 [nvme]
[ 540.104132] [c00000017bc53c50] [c0000000000dd630] process_one_work+0x1e0/0x5a0
[ 540.104239] [c00000017bc53ce0] [c0000000000ddb84] worker_thread+0x194/0x680
[ 540.104331] [c00000017bc53d80] [c0000000000e6680] kthread+0x110/0x130
[ 540.104424] [c00000017bc53e30] [c000000000009538] ret_from_kernel_thread+0x5c/0xa4
[ 604.094501] INFO: rcu_sched detected stalls on CPUs/tasks:
[ 604.094699] 1-...: (1 GPs behind) idle=d35/140000000000000/0 softirq=2385/2386 fqs=82
[ 604.094700] (detected by 5, t=21472 jiffies, g=1335, c=1334, q=1283)
[ 604.094705] Task dump for CPU 1:

Can you provide the backported patch for verification?

git://kernel.ubuntu.com/rtg/ubuntu-xenial.git eeh-lp1602724
commit fcb182b0c9d79b9dae70f2c9104997a0ec63f080 ('nvme: Suspend all queues before deletion')

The 6 patches mentioned in the bug description have already been merged but not yet packaged in a released version, though they _are_ included in this test kernel.

Part of the fix was in the current update. Marking the verification done to proceed with the current cycle. This bug needs to be reset to fix-committed for tracking the remaining patch after it gets closed by automatic processes.

This bug was fixed in the package linux - 4.4.0-36.55

---------------
linux (4.4.0-36.55) xenial; urgency=low

  [ Stefan Bader ]

  * Release Tracking Bug
    - LP: #1612305

  * I2C touchpad does not work on AMD platform (LP: #1612006)
    - SAUCE: pinctrl/amd: Remove the default de-bounce time

  * CVE-2016-5696
    - tcp: make challenge acks less predictable

linux (4.4.0-35.54) xenial; urgency=low

  [ Stefan Bader ]

  * Release Tracking Bug
    - LP: #1611215

  * [i915_bpo] Sync with v4.7 (LP: #1609742)
    - SAUCE: i915_bpo: Sync with v4.7

  * s390/cio: fix reset of channel measurement block (LP: #1609415)
    - s390/cio: allow to reset channel measurement block

  * in Ubuntu16.10: Hit on Call traces and system goes down when transactional
    memory tests are running in 32TB Brazos system (LP: #1606786)
    - powerpc/tm: Avoid SLB faults in treclaim/trecheckpoint when RI=0
    - powerpc/tm: Fix stack pointer corruption in __tm_recheckpoint()

  * Power Menu does not display after press the Power Button (LP: #1609204)
    - intel-vbtn: new driver for Intel Virtual Button
    - [config] enable CONFIG_INTEL_VBTN=m

  * OptiPlex 7450 AIO hangs when rebooting (LP: #1608762)
    - x86/reboot: Add Dell Optiplex 7450 AIO reboot quirk

  * virtualbox+usb 3.0 breaks boot, -28 kernel works (LP: #1604058)
    - SAUCE: xhci: Fix soft lockup in xhci_pci_probe path when XHCI_STATE_HALTED

  * linux-kernel: Freeing IRQ from IRQ context (LP: #1597908)
    - block: defer timeouts to a workqueue

  * Tunnel offload indications not stripped from encapsulated packets, causing
    performance overhead (LP: #1602755)
    - tunnels: Remove encapsulation offloads on decap.

  * lm-sensors is throwing "ERROR: Can't get value of subfeature temp1_input:
    I/O error" for be2net driver (LP: #1607387)
    - be2net: perform temperature query in adapter regardless of its interface
      state

  * Dell dock MAC Address pass through doesn't work in Ubuntu (LP: #1579984)
    - r8152: Add support for setting pass through MAC address on RTL8153-AD

  * vmxnet3 LRO IPv6 performance issues (stalling TCP) (LP: #1605494)
    - Driver: Vmxnet3: set CHECKSUM_UNNECESSARY for IPv6 packets

  * ISST-LTE:pVM:monklp5:Ubuntu16.04.1:system crashed at
    lpfc_sli4_scmd_to_wqidx_distr (LP: #1597974)
    - SAUCE: lpfc: fix oops in lpfc_sli4_scmd_to_wqidx_distr() from
      lpfc_send_taskmgmt()

  * Backport cxlflash shutdown patch to Xenial SRU (LP: #1605405)
    - SAUCE: cxlflash: Verify problem state area is mapped before notifying
      shutdown

  * Xenial update to v4.4.16 stable release (LP: #1607404)
    - mac80211: fix fast_tx header alignment
    - mac80211: mesh: flush mesh paths unconditionally
    - mac80211_hwsim: Add missing check for HWSIM_ATTR_SIGNAL
    - mac80211: Fix mesh estab_plinks counting in STA removal case
    - EDAC, sb_edac: Fix rank lookup on Broadwell
    - IB/cm: Fix a recently introduced locking bug
    - IB/mlx4: Properly initialize GRH TClass and FlowLabel in AHs
    - powerpc/pseries: Fix IBM_ARCH_VEC_NRCORES_OFFSET since POWER8NVL was added
    - powerpc/tm: Always reclaim in start_thread() for exec() class syscalls
    - usb: dwc2: fix reg...

------- Comment From <email address hidden> 2016-09-12 22:46 EDT-------
(In reply to comment #20)
> Part of the fix was in the current update. Marking the verification done to
> proceed with the current cycle. This bug needs to be reset to fix-committed
> for tracking the remaining patch after it gets closed by automatic processes.

Hi smb,

The final patch, which was not included in the current release depends on a fixup I pushed upstream. I retested this, and it allowed it to successfully disable the device after 6 EEHs;

The fixup is not in Linus tree yet, but it should be included in the next -rc. it's queued in Jens Axboe's linux-block tree:

82469c59d222 ("nvme: Don't suspend admin queue that wasn't created")

<https://git.kernel.org/cgit/linux/kernel/git/axboe/linux-block.git/commit/?h=for-linus&id=82469c59d222f839ded5cd282172258e026f9112>

This needs to go on top of

c21377f8366c ("nvme: Suspend all queues before deletion")

Which is already in the Ubuntu's kernel -next branch.

Thanks,

https://lists.ubuntu.com/archives/kernel-team/2016-September/079944.html

This bug was fixed in the package linux - 4.4.0-38.57

---------------
linux (4.4.0-38.57) xenial; urgency=low

  [ Tim Gardner ]

  * Release Tracking Bug
    - LP: #1620658

  * CIFS client: access problems after updating to kernel 4.4.0-29-generic
    (LP: #1612135)
    - Revert "UBUNTU: SAUCE: (namespace) Bypass sget() capability check for nfs"
    - fs: Call d_automount with the filesystems creds

  * apt-key add fails in overlayfs (LP: #1618572)
    - SAUCE: overlayfs: fix regression in whiteout detection

linux (4.4.0-37.56) xenial; urgency=low

  [ Tim Gardner ]

  * Release Tracking Bug
    - LP: #1618040

  * [Feature] Instruction decoder support for new SKX instructions- AVX512
    (LP: #1591655)
    - x86/insn: perf tools: Fix vcvtph2ps instruction decoding
    - x86/insn: Add AVX-512 support to the instruction decoder
    - perf tools: Add AVX-512 support to the instruction decoder used by Intel PT
    - perf tools: Add AVX-512 instructions to the new instructions test

  * [Ubuntu 16.04] FCoE Lun not visible in OS with inbox driver - Issue with
    ioremap() call on 32bit kernel (LP: #1608652)
    - lpfc: Correct issue with ioremap() call on 32bit kernel

  * [Feature] turbostat support for Skylake-SP server (LP: #1591802)
    - tools/power turbostat: decode more CPUID fields
    - tools/power turbostat: CPUID(0x16) leaf shows base, max, and bus frequency
    - tools/power turbostat: decode HWP registers
    - tools/power turbostat: Decode MSR_MISC_PWR_MGMT
    - tools/power turbostat: allow sub-sec intervals
    - tools/power turbostat: Intel Xeon x200: fix erroneous bclk value
    - tools/power turbostat: Intel Xeon x200: fix turbo-ratio decoding
    - tools/power turbostat: re-name "%Busy" field to "Busy%"
    - tools/power turbostat: add --out option for saving output in a file
    - tools/power turbostat: fix compiler warnings
    - tools/power turbostat: make fewer systems calls
    - tools/power turbostat: show IRQs per CPU
    - tools/power turbostat: show GFXMHz
    - tools/power turbostat: show GFX%rc6
    - tools/power turbostat: detect and work around syscall jitter
    - tools/power turbostat: indicate SMX and SGX support
    - tools/power turbostat: call __cpuid() instead of __get_cpuid()
    - tools/power turbostat: correct output for MSR_NHM_SNB_PKG_CST_CFG_CTL dump
    - tools/power turbostat: bugfix: TDP MSRs print bits fixing
    - tools/power turbostat: SGX state should print only if --debug
    - tools/power turbostat: print IRTL MSRs
    - tools/power turbostat: initial BXT support
    - tools/power turbostat: decode BXT TSC frequency via CPUID
    - tools/power turbostat: initial SKX support

  * [BYT] display hotplug doesn't work on console (LP: #1616894)
    - drm/i915/vlv: Make intel_crt_reset() per-encoder
    - drm/i915/vlv: Reset the ADPA in vlv_display_power_well_init()
    - drm/i915/vlv: Disable HPD in valleyview_crt_detect_hotplug()
    - drm/i915: Enable polling when we don't have hpd

  * [Feature]intel_idle enabling on Broxton-P (LP: #1520446)
    - intel_idle: add BXT support

  * [Feature] EDAC: Update driver for SKX-SP (LP: #1591815)
    - [Config] CONFIG_EDAC_SKX=m
    - EDAC, skx_edac: Ad...

------- Comment From <email address hidden> 2016-09-22 18:01 EDT-------
Thanks Canonical,

Ran 6 EEHs in a row and got recovered each time. The last one removed the device correctly.

Closing this.

------- Comment (attachment only) From <email address hidden> 2016-09-22 18:00 EDT-------

hello,

who can give me the test script of 6 EEHs ?

Thanks a lot.