hibmc-drm Causes Unreadable Display for Huawei amd64 Servers

This bug is missing log files that will aid in diagnosing the problem. While running an Ubuntu kernel (not a mainline or third-party kernel) please enter the following command in a terminal window:

apport-collect 1762940

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

Hey Mao, Is this visible after install?

In other words can you install 16.04 on the failing system, then add the 4.15 kernel and boot into it?

Once you do that, does the issue appear?

Just to summarize where we are:

16.04, 4.13: No problem
18.04, 4.15: Problem with frame buffer
16.04, 4.15: ???

The screen also get blur after ubuntu 18.04 beta2 was installed by legacy bios on Huawei server (VGA card is SM750 which was embedded in Hi1710 iMBC chips). The ubuntu-destktop was installed by "apt-get install ubuntu-desktop and init 5 to switch to x.

For Xenial ubuntu-deskto on Huawei server testing (UEFI), all kernel 4.4 (default) and 4.13 (hwe) and 4.15 (hwe-edge) works well (no screen blur).

Today, Huawei side did x-window testing 16.04.04 in legacy BIOS model, all 4.4,4.13(hwe),4.15(hwe-edge which as as 18.04), 4.17rc1 works fine (no screen blur).

Below image uses the Subiquity and it can be installed successfully. But after install ubuntu-desktop, the login screen would become screen blur after reboot or init 5, but it can be start by startx after init 3 in the terminal.
http://releases.ubuntu.com/18.04/ubuntu-18.04-live-server-amd64.iso.

For ISO of classic Debian installer (d-i) which was download from below, the setup (install) would still become bur and can't continue to finish install by manual which was tested by iMBC(KVM) and virtual CD-ROM.
http://cdimage.ubuntu.com/releases/18.04/release/ubuntu-18.04-server-amd64.iso

Blur screen for GDM login was caused by wayland, it can be fix by force to use Xorg in /etc/gdm3/customer.conf to uncoment WaylandEnable=false

For ISO of classic Debian installer (d-i), it would become screen blur after kernel load and in the first screen of "select a langue"

This bug seems related with 4.15.0-20, the test result on UEFI model for different combination of 16.04, 18.04 and their kernel are below:
18.04(d-i not subiquity) with 4.13 kerel: no screen blur
18.04(d-i not subiquity) with 4.15.22 kerenl: no screen blur
16.04 with 4.15.20 kernel screen blur
18.04(d-i) with 4.15.20 kernel screen blur
Notes: 4.13 kernel come from 16.04.4 ISO
       4.15.0-22 kernel come from upgrading installed 18.04
       screen blur means blur screen picture for language select page
       Test with customized ISO which was replaced by vmlinux and initrd (new kernel)

In #18, 4.13 should replaced with 4.4.0-116 which used by 16.04.4 ISO.

The daily build is still use 4.15.0-20 kernel. It was expecting of update and fix this blur problem. http://cdimage.ubuntu.com/ubuntu-server/bionic/daily/20180625/bionic-server-amd64.iso

This looks to me as if Ubuntu (the kernel, Wayland, and/or the X server, depending on when the problem occurs) is delivering a video mode that the display device can't handle. (With VGA, the display should deliver its capabilities to the computer via EDID, but this doesn't always work correctly.) Thus, the first question is what that "display device" is. The screen shots look like a remote KVM of some sort. Is that the BMC's built-in screen-sharing tool, a network-accessible KVM hardware device (like a Raritan or Avocent), or something else? If possible, I recommend testing in person with a plain monitor plugged directly into the computer, at least as a diagnostic procedure. If that works, then it's likely that the issue is either a matter of a restrictive set of video modes in the remote-access tool, miscommunication between the remote-access tool and the kernel/Wayland/X, or both. Also, if a physical monitor is plugged into the server while it's being accessed remotely, it's conceivable that the kernel/Wayland/X is taking its cues about video mode from the physical monitor rather than from the remote-access tool, thus creating a mismatch. If the problem is caused by the remote access tool, or by an interaction between that and the kernel/Wayland/X, then updating the remote access tool's firmware may fix the problem.

Note that the video mode used can vary depending on the kernel version, X vs. Wayland, EFI vs. BIOS boot mode, and other factors. Such variables likely explain why earlier kernels worked but newer ones don't.

As a workaround, it might be possibly to forcibly reconfigure the system to use a video mode that will work with the display device. Unfortunately, there are a lot of different ways to do this, depending on the subsystem involved. For instance:

* Kernel parameters can set the kernel's video mode. Unfortunately, there are
  a lot of possibilities even here, including, but not limited to:
  * nomodeset
  * video=vga16fb:off
  * vga16fb.modeset=0
  * video=efifb:width:640,height:480
  * Try Googling "kernel video mode setting" or something similar for
    additional possibilities.
* X configuration
  * If you can get X working, even temporarily, you can use the GUI
    setup tools to change the resolution.
  * The xrandr tool can be used to probe and set video options; see
    https://www.x.org/wiki/Projects/XRandR/
  * I've not kept up with text-mode (manual) X configuration, but
    you may be able to find some documentation on how to get it
    to change video mode. Start here:
    https://help.ubuntu.com/community/VideoDriverHowto#Xorg.conf.d
* Wayland configuration
  * I'm even less of an expert on this, so I suggest you try
    Googling "wayland set resolution" or something similar.

Finally, note that Ubuntu server certification doesn't cover display devices, so this is NOT a certification blocker.

This bug seems related with the kernel or 4.15.0-20 of d-i ISO. Please refer to #18 for test results on different kernel.

The daily build is still use 4.15.0-20 kernel.
http://cdimage.ubuntu.com/ubuntu-server/bionic/daily/current/bionic-server-amd64.iso

Anyone can confirm that the 18.04.1 d-i install iso would update the kernel.

I have tested with 18.04.1 on Huawei FusionServer 2288 again. The screen blure is same. The kernel of this iso have been update to 4.15.0-29-generic.

I just tested with /boot/vmlinuz-4.15.0-29-generic which from my laptop ( to replace install/vmlinuz with this file), it would be screen blur. If I use 4.15.0-32-generic, blur would disappear.

I test 16.04.5 ga-kernel would not be screen blur but the hwe-kernel(4.15.0-29) which is same as 18.04.1 would become screen blur.

The 16.04.5 hwe (blur) syslog are enclosed.

The 16.04.5 ga-kernel wouldn't be screen blur and their syslog are enclosed.

The 18.04.1 ga-kernel (4.15.0-29) would be screen blur. The syslog are enclosed too.

replace 18.04.1 with 4.15.0-29 kernel from my laptop (16.04) and it would be screen blur too. The syslog are enclosed here. If I replacing it with 4.15.0-32-generic from my laptop, it would not be screen bure on language select screen, but the system seems hanged (died) in case I can't switch to tty2 or press "enter" key to continue.

In the screen blur kernel syslog, it show below error which 16.04.5 GA kernel don't have this error:
...
Aug 21 10:25:50 kernel: [ 0.589408] ACPI Exception: Could not find/resolve named package element: LNKA (20170831/dspkginit-381)
(repeated 942 lines)
...

ACPI Exception have been reported here and it seems not related to screen blur

https://bugzilla.kernel.org/show_bug.cgi?id=198167

Tested on 17.10 server d-i install, it would be screen blur too during installation, the kernel is 4.13.0.

Tested with 18.04.1 Desktop, it wouldn't screen blur. The kernel log are enclosed.

In case 18.04.1 server and destop iso use same kernel of v4.15.0-29 and the GUI works fine on destop, so the kernel should not be blamed.

In 18.04.1 server installation, if I kill bterm process of below:
/usr/bin/bterm -f /lib/unifont.bgf -l C.UTF-i /lib/debian-installer/menu

Some blur picture would be gone. Please see attach screen shot.

Huawei hisilicon team thought it might be caused by hibmc_drm which are upstream and included in v4.10 kernel for their ARM64 server. It is right. After disable this module by:
 modprobe.blacklist=hibmc_drm
In the grub whiling booting iso. The install screen blur was disappeared.

Hisilicon team said this driver (hibmc_drm) are only for arm64. For X86 architecture, it need to use default driver for this SM750.

It seems we incorrectly include this model in X86.

Hisicon team have plans to fix in upstream source code which to limit this modules configure options only to arm64 to prevent someone compile (configure) it for x86.

This issue also occurs on arm64. Attached is a screenshot from a HiSilicon D05 running 18.04.

Why would hibmc_drm be arm64-specific? It is the same PCI device (19e5:1711), so shouldn't it use the same driver regardless of architecture?

As observed on x86, disabling wayland also avoids the problem on ARM.

For #37 screen shot and #39. We found this wayland issues for 18.04 and Huawei x86 server, but it seems no for 18.04.1. In my test, apt-get install ubuntu-desktop and reboot works fine.

For #37 "Why would hibmc_drm be arm64-specific?", I guess it might because vendors (sm750 vga) have already provided driver x86.

For #37-#39

This screen blur issues during install are only happened on Huawei X86 server for 18.04 and 18.04.1 d-i iso. As I know, 18.04 and even 18.04.1 ARM64 iso works fine with this new hibmc-drm module for installation on Hisilicon D05 board.

This is definitely reproducible on a D05 (arm64) with 18.04 running wayland, and the symptoms look identical. I re-tested with the 18.04.1 ISO and confirmed.

Dann,

For the wanland issues, I understood it as comments in #15, that is:

      Blur screen for GDM login was caused by wayland, it can be fix by force to use Xorg in
      /etc/gdm3/customer.conf to uncoment WaylandEnable=false

Is this right for your wanland issues on D05?

In 18.04, we found 2 issues on Huawei X86 server. One is screen blur during d-i install on UEFI model in Language selection screen, but it works for legacy bios model. The other is after start GUI (x-window or Ubuntu desktop). After we sucessfully installed 18.04 in legacy bios model on X86 Huawei server, I tried to start X-windows via apt-get install ubuntu-dekstop and reboot, we would get this blur screen again. This seems caused by this wanyland and can be fixed by this "WaylandEnable=false".

As I know, this wayland issues have been happened for D05 but I don't think D05 have above issues
during install.

@Zhanglei

Could you try these 2 patches?

https://patchwork.freedesktop.org/patch/246810/
https://patchwork.freedesktop.org/patch/246811/

I think these 2 patches will fix the issue that dual GPU install on UEFI boot mode.

@Mao: correct, I do not observe screen blur in d-i on D05.

btw, note also bug 1780076

@AaronMa
Do you have docs for how to compile a new kernel. I can found below doc link
https://help.ubuntu.com/community/Kernel/Compile
But it seems a bit old.

The other difficult is how to get compile config from d-i iso kernel. I found if I use a kernel from running/installed system to replace d-i iso kernel, it would hang (died) after booting in the first language selection screen.

Hi:
Please refer to this wiki:
https://wiki.ubuntu.com/Kernel/BuildYourOwnKernel

Usually you just need to do the following first:

1, git clone xxx;
2, apply patches;

With 18.04 UEFI on x86, does passing the kernel parameter "video=efifb:off" make a difference?

summary for workaround via kernel parameter to blocker loading:

  modprobe.blacklist=hibmc_drm

**prevent load during install
To add "modprobe.blacklist=hibmc_drm" kernel parameter in grub when boot from d-i iso (press ‘e’ key and then press F10 or ctrl+x ( those key was noticed on the screen). It would prevent this module to load and let you install as usual.

**prevent load for installed system
Although, we didn't find any issues for X-window during our testing. If you want to prevent loading on a finished installed system. You can add similar setting to /etc/default/grub for line as:
GRUB_CMDLINE_LINUX_DEFAULT="modprobe.blacklist=hibmc_drm"
You need to update grub by "sudo grub-update" and reboot to make it be efectly.

@Aaron Ma
I want to test your patch. But seems it would patch with error. I guess, the new 18.04 source code have been change to:
        list_add_tail(&vgadev->list, &vga_list);
instead of:
- list_add(&vgadev->list, &vga_list);
+ list_add_tail(&vgadev->list, &vga_list);
Does it mean this patch have been already merged in our new source. In last month, I testing with 18.04.2 iso (d-i) and this screen blur is still existing. Does it mean this patch did not solve our cases.

For #44 Aaron's suggested patch, it seems patch have been included after 4.15.0-46. The testing 18.04.2 iso with 4.15.0-48 kernel seems the system would be hang on language selection after select install server of grub screen. So it seems this hibmc_drm would caused another this hang issues too.

This hibmc_drm driver can only work for ARM64, as I know, chip vendors don't have any intend/interesting to make this drm/kernel driver to work for x86. For x86, it must use the existing driver which was based Xorg.

For #44, in the following testing, if disable hibmc_drm loading, then it works fine and would not be hang.

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-disco' to 'verification-done-disco'. If the problem still exists, change the tag 'verification-needed-disco' to 'verification-failed-disco'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-bionic' to 'verification-done-bionic'. If the problem still exists, change the tag 'verification-needed-bionic' to 'verification-failed-bionic'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

I installed 4.15.0-56-generic to a bionic VM, with the -modules and
-modules-extra packages. I went into /var/lib/modules/4.15.0-56/ and did a search for
hibmc_drm. There was no mention apart from a header Makefile. The module was not
built and not present on amd64.

I did the same for 5.0.0-22-generic in a disco VM, and the module was not
built and was not present.

4.15.0-56-generic has been installed to a amd64 based Huawei server which was
having the issues explained in this bug, and since the module was not present,
the screen was not "blurry" when a desktop environment was installed and viewed
from the remote access tool used by Huawei.

To ensure that ARM64 still has the kernel module built, I also downloaded
linux-modules-extra for 4.15.0-56-generic for ARM64. The hibmc_drm module is
still present in /var/lib/modules/4.15.0-56/kernel/drivers/gpu/drm/hisilicon/hibmc
This means that Huawei's arm64 based servers will not be affected by the change.

The fixes solve the problems of the video mode not being right for amd64 based
platforms, and does not cause any regressions for ARM64. I am happy to mark this
as verified.

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-xenial' to 'verification-done-xenial'. If the problem still exists, change the tag 'verification-needed-xenial' to 'verification-failed-xenial'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

I installed linux-hwe 4.15.0-57 #63~16.04.1-Ubuntu on xenial, and ensured the -modules and -modules-extra package was installed.

I went into /var/lib/modules/4.15.0-56/ and did a search for hibmc_drm. There was no mention apart from a header Makefile. The module was not built and not present on amd64.

Tests have been successful on bionic, so I am happy to mark this as fixed for xenial.

This bug was fixed in the package linux - 5.2.0-10.11

---------------
linux (5.2.0-10.11) eoan; urgency=medium

  * eoan/linux: 5.2.0-10.11 -proposed tracker (LP: #1838113)

  * Packaging resync (LP: #1786013)
    - [Packaging] resync git-ubuntu-log

  * Eoan update: v5.2.4 upstream stable release (LP: #1838428)
    - bnx2x: Prevent load reordering in tx completion processing
    - caif-hsi: fix possible deadlock in cfhsi_exit_module()
    - hv_netvsc: Fix extra rcu_read_unlock in netvsc_recv_callback()
    - igmp: fix memory leak in igmpv3_del_delrec()
    - ipv4: don't set IPv6 only flags to IPv4 addresses
    - ipv6: rt6_check should return NULL if 'from' is NULL
    - ipv6: Unlink sibling route in case of failure
    - net: bcmgenet: use promisc for unsupported filters
    - net: dsa: mv88e6xxx: wait after reset deactivation
    - net: make skb_dst_force return true when dst is refcounted
    - net: neigh: fix multiple neigh timer scheduling
    - net: openvswitch: fix csum updates for MPLS actions
    - net: phy: sfp: hwmon: Fix scaling of RX power
    - net_sched: unset TCQ_F_CAN_BYPASS when adding filters
    - net: stmmac: Re-work the queue selection for TSO packets
    - net/tls: make sure offload also gets the keys wiped
    - nfc: fix potential illegal memory access
    - r8169: fix issue with confused RX unit after PHY power-down on RTL8411b
    - rxrpc: Fix send on a connected, but unbound socket
    - sctp: fix error handling on stream scheduler initialization
    - sctp: not bind the socket in sctp_connect
    - sky2: Disable MSI on ASUS P6T
    - tcp: be more careful in tcp_fragment()
    - tcp: fix tcp_set_congestion_control() use from bpf hook
    - tcp: Reset bytes_acked and bytes_received when disconnecting
    - vrf: make sure skb->data contains ip header to make routing
    - net/mlx5e: IPoIB, Add error path in mlx5_rdma_setup_rn
    - net: bridge: mcast: fix stale nsrcs pointer in igmp3/mld2 report handling
    - net: bridge: mcast: fix stale ipv6 hdr pointer when handling v6 query
    - net: bridge: don't cache ether dest pointer on input
    - net: bridge: stp: don't cache eth dest pointer before skb pull
    - macsec: fix use-after-free of skb during RX
    - macsec: fix checksumming after decryption
    - netrom: fix a memory leak in nr_rx_frame()
    - netrom: hold sock when setting skb->destructor
    - selftests: txring_overwrite: fix incorrect test of mmap() return value
    - net/tls: fix poll ignoring partially copied records
    - net/tls: reject offload of TLS 1.3
    - net/mlx5e: Fix port tunnel GRE entropy control
    - net/mlx5e: Rx, Fix checksum calculation for new hardware
    - net/mlx5e: Fix return value from timeout recover function
    - net/mlx5e: Fix error flow in tx reporter diagnose
    - bnxt_en: Fix VNIC accounting when enabling aRFS on 57500 chips.
    - mlxsw: spectrum_dcb: Configure DSCP map as the last rule is removed
    - net/mlx5: E-Switch, Fix default encap mode
    - mlxsw: spectrum: Do not process learned records with a dummy FID
    - dma-buf: balance refcount inbalance
    - dma-buf: Discard old fence_excl on retrying get_fences_rcu for realloc
    - Revert "gpio/spi: Fix spi-gpio...

This bug was fixed in the package linux - 5.0.0-25.26

---------------
linux (5.0.0-25.26) disco; urgency=medium

  * CVE-2019-1125
    - x86/cpufeatures: Carve out CQM features retrieval
    - x86/cpufeatures: Combine word 11 and 12 into a new scattered features word
    - x86/speculation: Prepare entry code for Spectre v1 swapgs mitigations
    - x86/speculation: Enable Spectre v1 swapgs mitigations
    - x86/entry/64: Use JMP instead of JMPQ
    - x86/speculation/swapgs: Exclude ATOMs from speculation through SWAPGS

 -- Kleber Sacilotto de Souza <email address hidden> Thu, 01 Aug 2019 12:04:35 +0200

This bug was fixed in the package linux - 4.15.0-58.64

---------------
linux (4.15.0-58.64) bionic; urgency=medium

  * unable to handle kernel NULL pointer dereference at 000000000000002c (IP:
    iget5_locked+0x9e/0x1f0) (LP: #1838982)
    - Revert "ovl: set I_CREATING on inode being created"
    - Revert "new primitive: discard_new_inode()"

linux (4.15.0-57.63) bionic; urgency=medium

  * CVE-2019-1125
    - x86/cpufeatures: Carve out CQM features retrieval
    - x86/cpufeatures: Combine word 11 and 12 into a new scattered features word
    - x86/speculation: Prepare entry code for Spectre v1 swapgs mitigations
    - x86/speculation: Enable Spectre v1 swapgs mitigations
    - x86/entry/64: Use JMP instead of JMPQ
    - x86/speculation/swapgs: Exclude ATOMs from speculation through SWAPGS

  * Packaging resync (LP: #1786013)
    - update dkms package versions

linux (4.15.0-56.62) bionic; urgency=medium

  * bionic/linux: 4.15.0-56.62 -proposed tracker (LP: #1837626)

  * Packaging resync (LP: #1786013)
    - [Packaging] resync git-ubuntu-log
    - [Packaging] update helper scripts

  * CVE-2019-2101
    - media: uvcvideo: Fix 'type' check leading to overflow

  * hibmc-drm Causes Unreadable Display for Huawei amd64 Servers (LP: #1762940)
    - [Config] Set CONFIG_DRM_HISI_HIBMC to arm64 only
    - SAUCE: Make CONFIG_DRM_HISI_HIBMC depend on ARM64

  * Bionic: support for Solarflare X2542 network adapter (sfc driver)
    (LP: #1836635)
    - sfc: make mem_bar a function rather than a constant
    - sfc: support VI strides other than 8k
    - sfc: add Medford2 (SFC9250) PCI Device IDs
    - sfc: improve PTP error reporting
    - sfc: update EF10 register definitions
    - sfc: populate the timer reload field
    - sfc: update MCDI protocol headers
    - sfc: support variable number of MAC stats
    - sfc: expose FEC stats on Medford2
    - sfc: expose CTPIO stats on NICs that support them
    - sfc: basic MCDI mapping of 25/50/100G link speeds
    - sfc: support the ethtool ksettings API properly so that 25/50/100G works
    - sfc: add bits for 25/50/100G supported/advertised speeds
    - sfc: remove tx and MCDI handling from NAPI budget consideration
    - sfc: handle TX timestamps in the normal data path
    - sfc: add function to determine which TX timestamping method to use
    - sfc: use main datapath for HW timestamps if available
    - sfc: only enable TX timestamping if the adapter is licensed for it
    - sfc: MAC TX timestamp handling on the 8000 series
    - sfc: on 8000 series use TX queues for TX timestamps
    - sfc: only advertise TX timestamping if we have the license for it
    - sfc: simplify RX datapath timestamping
    - sfc: support separate PTP and general timestamping
    - sfc: support second + quarter ns time format for receive datapath
    - sfc: support Medford2 frequency adjustment format
    - sfc: add suffix to large constant in ptp
    - sfc: mark some unexported symbols as static
    - sfc: update MCDI protocol headers
    - sfc: support FEC configuration through ethtool
    - sfc: remove ctpio_dmabuf_start from stats
    - sfc: stop the TX queue before pushing new buffers

  * [18.04 FEAT] zKVM: Add hardwar...

I have submitted the patch upstream to dri-devel and CC'd the hibmc maintainers.

We will see where the discussion goes.

https://lists.freedesktop.org/archives/dri-devel/2019-August/231284.html
https://lists.freedesktop.org/archives/dri-devel/2019-August/231285.html

The patch has been mainlined as of Linux 5.4-rc1 with:

commit feeb07d0ca5ad0d6edb5bb728e33e2e86118529d
Author: Matthew Ruffell <email address hidden>
Date: Thu Aug 15 16:26:40 2019 +1200
Subject: drm/hisilicon/hibmc: Make CONFIG_DRM_HISI_HIBMC depend on ARM64

https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=feeb07d0ca5ad0d6edb5bb728e33e2e86118529d