Merge ~williamhsu/plainbox-provider-checkbox/+git/plainbox-provide-checkbox:bug/lp-1884235/exchanged-the-sequence-of-graphics-log-parsing into plainbox-provider-checkbox:master

[William Hsu]

Exchanged the sequence of graphics log parsing

# Description
SWE team found the information of graphics driver is recorded on the user folder instead of root folder if Nvidia modeset=1. Therefore, I created the patch to check the sequence of graphics log parsing. Details can be found on lp:1884235. Thanks!

# Changes
1. Checking the Nvidia mode settings before log parsing.
  - If the modeset=1 (/etc/modprobe.d/oem-nvidia-drm.conf), the Nvidia graphics driver info is logged on the user folder instead of root folder
2. Seeking the log file
  - Sometimes, the name of xorg log isn't named "Xorg.0.log". E.g.
    whsu@thinkpad-x:/var/log$ ll X*
    -rw-r--r-- 1 root root 8.9K Jun 18 10:15 Xorg.1.log
    -rw-r--r-- 1 root root 8.9K Jun 17 19:01 Xorg.1.log.old
    Suggesting to seek the xorg log before log parsing

# Patch verification result:
  ## Scenario 1: Validation on the discrete graphics card (Nvidia driver without modeset=1) ##
    u@u-ThinkStationP520:/usr/lib/plainbox-provider-checkbox/bin$ ./graphics_driver
    ------------- VIDEO DRIVER INFORMATION -------------
    Video Driver: nvidia
    Driver Version: 440.64

    ------------- HYBRID GRAPHICS CHECK ----------------
    Graphics Chipset: NVIDIA (10de:1c31)
    Loaded DDX Drivers: nouveau, modesetting, fbdev, vesa
    Hybrid Graphics: no
    u@u-ThinkStationP520$ sudo cat /sys/module/nvidia_drm/parameters/modeset
    N

  ## Scenario 2: Validation on the discrete graphics card (Nvidia driver with modeset=1) ##
    u@u-ThinkStation-P720R4:/usr/lib/plainbox-provider-checkbox/bin$ ./graphics_driver
    ------------- VIDEO DRIVER INFORMATION -------------
    Video Driver: nvidia
    Driver Version: 440.64

    ------------- HYBRID GRAPHICS CHECK ----------------
    Graphics Chipset: NVIDIA (10de:1e84)
    Loaded DDX Drivers: nouveau, modesetting, fbdev, vesa
    Hybrid Graphics: no
    u@u-ThinkStation-P720R4$ sudo cat /sys/module/nvidia_drm/parameters/modeset
    Y

  ## Scenario 3: Validation on the UMA config ##
    u@u-ThinkStation-P340-SFF:/usr/lib/plainbox-provider-checkbox/bin$ ./graphics_driver
    ------------- VIDEO DRIVER INFORMATION -------------
    Video Driver: modesetting
    Driver Version: 1.20.8

    ------------- HYBRID GRAPHICS CHECK ----------------
    Graphics Chipset: Intel (8086:9be6)
    Loaded DDX Drivers: modesetting, fbdev, vesa
    Hybrid Graphics: no

[Pierre Equoy (pieq)]

A few comments inline.

review: Needs Fixing

[William Hsu (williamhsu)]

Thanks for Pierre's reminder, the scenario 1 should be as follows. Sorry for the typo.
## Scenario 1: Validation on the discrete graphics card (Nvidia driver "without" modeset=1) ##

[Jonathan Cave (jocave)]

Comments below

review: Needs Fixing

[William Hsu (williamhsu)]

# The validation for the 2nd patch.
-----------------------------------------------------------
## Scenario 1: Validation on the discrete graphics card (Nvidia driver without modeset=1) ##
u@u-ThinkStationP520c:/usr/lib/plainbox-provider-checkbox/bin$ sudo ./graphics_driver
------------- VIDEO DRIVER INFORMATION -------------
Video Driver: nvidia
Driver Version: 440.64

------------- HYBRID GRAPHICS CHECK ----------------
Graphics Chipset: NVIDIA (10de:1eb1)
Loaded DDX Drivers: nouveau, modesetting, fbdev, vesa
Hybrid Graphics: no

u@u-ThinkStationP520c:/usr/lib/plainbox-provider-checkbox/bin$ sudo cat /sys/module/nvidia_drm/parameters/modeset
N

## Scenario 2: Validation on the discrete graphics card (Nvidia driver with modeset=1) ##
u@u-ThinkStationP520c:/usr/lib/plainbox-provider-checkbox/bin$ sudo ./graphics_driver
------------- VIDEO DRIVER INFORMATION -------------
Video Driver: nvidia
Driver Version: 440.64

------------- HYBRID GRAPHICS CHECK ----------------
Graphics Chipset: NVIDIA (10de:1eb1)
Loaded DDX Drivers: nouveau, modesetting, fbdev, vesa
Hybrid Graphics: no

u@u-ThinkStationP520c:/usr/lib/plainbox-provider-checkbox/bin$ sudo cat /sys/module/nvidia_drm/parameters/modeset
Y

[Jonathan Cave (jocave)]

I suggest you squash the commits - please make sure the commit message has a short summary line and then the longer description of changes

Suggest use of resource manager for the file read (see below)

review: Needs Fixing

[William Hsu (williamhsu)]

> I suggest you squash the commits - please make sure the commit message has a
> short summary line and then the longer description of changes
>
> Suggest use of resource manager for the file read (see below)

Thank you, Jonathan!
I used resource manager to read file and then squashed (rebase) the commits.
Have a great day! :)

[Jonathan Cave (jocave)]

Thanks for changes - unfortunately the script was renamed by another merge while you were working on this.

This script bin/graphics_driver has been renamed to bin/graphics_driver.py

I think it should be possible to resolve this doing a rebase:

$ git checkout master
$ git pull
$ git checkout bug/lp-188...
$ git rebase master
$ git push -f <remote> bug/lp-188...

[William Hsu (williamhsu)]

@Jonathan, thank you for all your help and information. I have done the rebase.

[Jonathan Cave (jocave)]

+1, thanks

review: Approve

[William Hsu (williamhsu)]

@Jonathan, cool! My pleasure! Thank you!

@ Pierre, @Hugh, and @Nara, may I have your help? I was wondering if I can have your input regarding the commit. It would be appreciated if I can have your review and endorsement. Thank you! :)

[hugh chao (hugh712)]

I think the topic should not just specific in "nvidia" due to AMD driver also has this issue[0]

[0] https://bugs.launchpad.net/stella/+bug/1886000

review: Needs Information

[William Hsu (williamhsu)]

@Hugh, thanks for the input. May I borrow your expertise? The following file can help us to know if the modeset is enabled with Nvidia card.
- "/sys/module/nvidia_drm/parameters/modeset"
Therefore, I was wondering if any file/configure can help me to identify the modeset value with AMD graphic card. Thanks!

[William Hsu (williamhsu)]

@ Hugh, according to the information from HWE and SWE, the Xorg logs don't have the necessary AMD driver information. So, we may need to use the other methods to retrieve the AMD driver info. Thanks! :)

@ Everyone, the validations for the patch (Used the Xorg owner to judge the Xorg log location)

### Scenario 1 - Intel Graphic Card (Built-in) ###
(venv_ldtp) whsu@thinkpad-x:~/ws/plainbox-provider-checkbox/bin$ ./graphics_driver.py
------------- VIDEO DRIVER INFORMATION -------------
Video Driver: modesetting
Driver Version: 1.20.5
------------- HYBRID GRAPHICS CHECK ----------------
Graphics Chipset: Intel (8086:0a16)
Loaded DDX Drivers: modesetting, fbdev, vesa
Hybrid Graphics: no

### Scenario 2 - Nvidia Graphic Card, NV modeset = Y ###
u@u-ThinkStation-xxx:/usr/lib/plainbox-provider-checkbox/bin$ sudo cat /sys/module/nvidia_drm/parameters/modeset
Y
u@u-ThinkStation-xxx:/usr/lib/plainbox-provider-checkbox/bin$ ./graphics_driver
------------- VIDEO DRIVER INFORMATION -------------
Video Driver: nvidia
Driver Version: 440.64
------------- HYBRID GRAPHICS CHECK ----------------
Graphics Chipset: NVIDIA (10de:1c31)
Loaded DDX Drivers: nouveau, modesetting, fbdev, vesa
Hybrid Graphics: no

### Scenario 3 - Nvidia Graphic Card, NV modeset = N ###
u@u-ThinkStationP520c:/usr/lib/plainbox-provider-checkbox/bin$ sudo cat /sys/module/nvidia_drm/parameters/modeset
N
u@u-ThinkStationP520c:/usr/lib/plainbox-provider-checkbox/bin$ ./graphics_driver
------------- VIDEO DRIVER INFORMATION -------------
Video Driver: nvidia
Driver Version: 440.64
------------- HYBRID GRAPHICS CHECK ----------------
Graphics Chipset: NVIDIA (10de:1eb1)
Loaded DDX Drivers: nouveau, modesetting, fbdev, vesa
Hybrid Graphics: no

### Scenario 4 - AMD graphic card (A known issue, no AMD driver info on the Xorg log) ###
u@u-ThinkStation-xxx:/usr/lib/plainbox-provider-checkbox/bin$ ./graphics_driver
------------- VIDEO DRIVER INFORMATION -------------
ERROR: No video driver loaded! Possibly in failsafe mode!
------------- HYBRID GRAPHICS CHECK ----------------
Graphics Chipset: AMD (1002:6981)
Loaded DDX Drivers: ati, radeon, modesetting, fbdev, vesa
Hybrid Graphics: no

[hugh chao (hugh712)]

so do we have any idea to get AMD info?
I will suggest to fix AMD one in this MP.

[jeremyszu (os369510)]

I leave some suggestions in below lines.

BTW, currently the Ubuntu supports x11 and wayland. If this python script is a generic test case for "graphics_driver" then better to confirm the current session whether executing under x11 before starting the verification (might leave wayland as unsupported test environment).

[William Hsu (williamhsu)]

> so do we have any idea to get AMD info?
> I will suggest to fix AMD one in this MP.

@ Hugh, we would like to use the other patches/cases to fix AMD one because parsing the Xorg log (Current implementation) is not a good way to get AMD graphics driver info. Thanks! :)

[William Hsu (williamhsu)]

> so do we have any idea to get AMD info?
> I will suggest to fix AMD one in this MP.

@Hugh, my bad. Thanks to Jeremy's reminder, the Xorg log can help us to find out the AMD driver info. I also ran the script on a AMD test config, it does work. Test result is as follow

u@u-INVALID:/usr/lib/plainbox-provider-checkbox/bin$ lspci | grep -i --color 'vga\|3d\|2d'
61:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Lexa XT [Radeon PRO WX 3200] (rev 10)
u@u-INVALID:/usr/lib/plainbox-provider-checkbox/bin$ ./graphics_driver
------------- VIDEO DRIVER INFORMATION -------------
Video Driver: amdgpu
Driver Version: 19.1.0

------------- HYBRID GRAPHICS CHECK ----------------
Graphics Chipset: AMD (1002:6981)
Loaded DDX Drivers: amdgpu, ati, modesetting, fbdev, vesa
Hybrid Graphics: no

The reason why it failed in the previous test, it's because of the blacklist-amdgpu.conf. It blocked the graphics tests. So, please skip the comment - https://code.launchpad.net/~williamhsu/plainbox-provider-checkbox/+git/plainbox-provide-checkbox/+merge/386796/comments/1020120.

In short, I'm going to implement the Jeremy's idea. Afterwards, we can get together to see if the MR meets the requirements. Thanks!

[William Hsu (williamhsu)]

@Hugh and Jeremy,
I merged Jeremy's idea into the latest patch. Would it be possible to have your time to review it? Thanks!

# Test result:
## Scenario 1 (NV modeset enabled) ##

u@u:~/WorkSPace/plainbox-provide-checkbox/bin$ ./graphics_driver.py
------------- VIDEO DRIVER INFORMATION -------------
Video Driver: nvidia
Driver Version: 440.64
------------- HYBRID GRAPHICS CHECK ----------------
Graphics Chipset: NVIDIA (10de:1bb1)
Loaded DDX Drivers: nouveau, modesetting, fbdev, vesa
Hybrid Graphics: no

u@u:~/WorkSPace/plainbox-provide-checkbox/bin$ sudo cat /sys/module/nvidia_drm/parameters/modeset
Y

u@u:~/WorkSPace/plainbox-provide-checkbox/bin$ cat /etc/buildstamp
Jenkins Sun, 28 Jun 2020 22:47:15 +0000
pc-sutton-bachman-focal-amd64-X00-20200628-134

## Scenario 2 (NV modeset disabled) ##
u@u-ThinkStation-P920:~/plainbox-provide-checkbox/bin$ ./graphics_driver.py
------------- VIDEO DRIVER INFORMATION -------------
Video Driver: nvidia
Driver Version: 418.56
------------- HYBRID GRAPHICS CHECK ----------------
Graphics Chipset: NVIDIA (10de:1e30)
Loaded DDX Drivers: nouveau, modesetting, fbdev, vesa
Hybrid Graphics: no

u@u-ThinkStation-P920:~/plainbox-provide-checkbox/bin$ sudo cat /sys/module/nvidia_drm/parameters/modeset
N

u@u-ThinkStation-P920:~/plainbox-provide-checkbox/bin$ cat /etc/buildstamp
Jenkins Thu, 23 May 2019 09:08:29 +0000
sutton-mccarthy-bionic-amd64-20190523-122

[hugh chao (hugh712)]

review: Needs Fixing

[William Hsu (williamhsu)]

@Hugh and Jeremy, the latest change has been verified. May I have a few minutes of your time to do a double-check? Thank you! :)

####################### Validate the change on stock Ubuntu #######################
whsu@thinkpad-x:~/Workspace/plainbox-provider-checkbox/bin$ ./graphics_driver.py
------------- VIDEO DRIVER INFORMATION -------------
Video Driver: modesetting
Driver Version: 1.20.5

------------- HYBRID GRAPHICS CHECK ----------------
Graphics Chipset: Intel (8086:0a16)
Loaded DDX Drivers: modesetting, fbdev, vesa
Hybrid Graphics: no

################# Validate the change on OEM build with NV modeset=Y #################
u@u-ThinkStationP520c:~/Workspace/plainbox-provider-checkbox/bin$ ./graphics_driver.py
['gdm']
------------- VIDEO DRIVER INFORMATION -------------
Video Driver: nvidia
Driver Version: 440.64

------------- HYBRID GRAPHICS CHECK ----------------
Graphics Chipset: NVIDIA (10de:1cb6)
Loaded DDX Drivers: nouveau, modesetting, fbdev, vesa
Hybrid Graphics: no
u@u-ThinkStationP520c:~/Workspace$ sudo cat /sys/module/nvidia_drm/parameters/modeset
Y

################# Validate the change on OEM build with NV modeset=N #################

u@u-ThinkStationP520c:~/Workspace/plainbox-provider-checkbox/bin$ ./graphics_driver.py
['root', 'root']
------------- VIDEO DRIVER INFORMATION -------------
Video Driver: nvidia
Driver Version: 440.64

------------- HYBRID GRAPHICS CHECK ----------------
Graphics Chipset: NVIDIA (10de:1cb6)
Loaded DDX Drivers: nouveau, modesetting, fbdev, vesa
Hybrid Graphics: no
u@u-ThinkStationP520c:~/Workspace$ sudo cat /sys/module/nvidia_drm/parameters/modeset
N

[hugh chao (hugh712)]

it's in a good shape now,
so I'm ok with this commit,
good job~!

review: Approve

[jeremyszu (os369510)]

LGTM, if you did verified it on Sutton/Stella.

review: Approve

[William Hsu (williamhsu)]

> LGTM, if you did verified it on Sutton/Stella.

Yes, I have verified the patch on a UMA SKU and Sutton platform. The results are as follows.

## NV modeset enabled ##
u@u-ThinkStationP520c:~/workspace$ ./graphics_driver.py
['gdm', 'u']
------------- VIDEO DRIVER INFORMATION -------------
Video Driver: nvidia
Driver Version: 440.64

------------- HYBRID GRAPHICS CHECK ----------------
Graphics Chipset: NVIDIA (10de:1cb6)
Loaded DDX Drivers: nouveau, modesetting, fbdev, vesa
Hybrid Graphics: no

## NV modeset disabled (UMA) ##
whsu@thinkpad-x:~/Workspace/plainbox-provider-checkbox/bin$ ./graphics_driver.py
------------- VIDEO DRIVER INFORMATION -------------
Video Driver: modesetting
Driver Version: 1.20.5

------------- HYBRID GRAPHICS CHECK ----------------
Graphics Chipset: Intel (8086:0a16)
Loaded DDX Drivers: modesetting, fbdev, vesa
Hybrid Graphics: no

[Pierre Equoy (pieq)]

It's getting there :) 加油！

review: Needs Fixing

[William Hsu (williamhsu)]

Cool! Thank you, Pierre! I'm going to work on it:)

[William Hsu (williamhsu)]

@Pierre, the latest change has been verified and pushed into the MR/branch. Results are as follows. Would it be possible to check it again? Thank you! :)

## NV modeset enabled ##
u@u-ThinkStationP520c:~/workspace$ ./graphics_driver.py
['gdm', 'u']
------------- VIDEO DRIVER INFORMATION -------------
Video Driver: nvidia
Driver Version: 440.64

------------- HYBRID GRAPHICS CHECK ----------------
Graphics Chipset: NVIDIA (10de:1cb6)
Loaded DDX Drivers: nouveau, modesetting, fbdev, vesa
Hybrid Graphics: no

## NV modeset disabled (UMA) ##
whsu@thinkpad-x:~/Workspace/plainbox-provider-checkbox/bin$ ./graphics_driver.py
------------- VIDEO DRIVER INFORMATION -------------
Video Driver: modesetting
Driver Version: 1.20.5

------------- HYBRID GRAPHICS CHECK ----------------
Graphics Chipset: Intel (8086:0a16)
Loaded DDX Drivers: modesetting, fbdev, vesa
Hybrid Graphics: no

[Pierre Equoy (pieq)]

One comment inline.

Also, why do you have this output:

    ['gdm', 'u']

when running the script with NV modeset enabled?

This looks like a Python side-effect and should be avoided. (sorry I didn't catch that earlier)

review: Needs Fixing

[William Hsu (williamhsu)]

> One comment inline.
>
> Also, why do you have this output:
>
> ['gdm', 'u']
> when running the script with NV modeset enabled?
>
> This looks like a Python side-effect and should be avoided. (sorry I didn't
> catch that earlier)
@Pierre, oh! It's just a debugging message in my local branch. Before I pushed the patch onto the launchpad, it has been removed. Sorry for the confusion!

[William Hsu (williamhsu)]

@Pierre, I fine-turned the MR. The latest change has been verified. Results are as follows. May I have a few minutes of your time to do a double-check. Thank you! :)

### NV modeset enabled ###
u@u-ThinkStationP520c:~/workspace$ date
Thu 13 Aug 2020 11:09:21 AM EDT
u@u-ThinkStationP520c:~/Desktop$ ./graphics_driver.py
------------- VIDEO DRIVER INFORMATION -------------
Video Driver: nvidia
Driver Version: 440.64

------------- HYBRID GRAPHICS CHECK ----------------
Graphics Chipset: NVIDIA (10de:1cb6)
Loaded DDX Drivers: nouveau, modesetting, fbdev, vesa
Hybrid Graphics: no

### NV modeset disabled (UMA) ###
whsu@thinkpad-x:~/workspace/plainbox-provider-checkbox/bin$ date
Thu Aug 13 23:09:38 CST 2020
whsu@thinkpad-x:~/workspace/plainbox-provider-checkbox/bin$ ./graphics_driver.py
------------- VIDEO DRIVER INFORMATION -------------
Video Driver: modesetting
Driver Version: 1.20.5

------------- HYBRID GRAPHICS CHECK ----------------
Graphics Chipset: Intel (8086:0a16)
Loaded DDX Drivers: modesetting, fbdev, vesa
Hybrid Graphics: no

[Pierre Equoy (pieq)]

+1

Thanks for all the effort!

review: Approve

diff --git a/bin/graphics_driver.py b/bin/graphics_driver.py
index f9bdff3..52ca157 100755
--- a/bin/graphics_driver.py
+++ b/bin/graphics_driver.py
@@ -37,6 +37,7 @@
 import re
 import sys
 import os
+import glob
 
 from subprocess import Popen, PIPE, check_output, CalledProcessError
 
@@ -362,10 +363,30 @@ def hybrid_graphics_check(xlog):
 
 
 def main():
-    if os.path.isfile("/var/log/Xorg.0.log"):
-        xlog = XorgLog("/var/log/Xorg.0.log")
+    usr_xorg_dir = os.path.expanduser("~/.local/share/xorg/")
+    root_xorg_dir = "/var/log/"
+    xlog = None
+    xorg_owner = []
+    tgt_dir = ""
+
+    # Output the Xorg owner
+    xorg_owner = check_output("ps -o user= -p $(pidof Xorg)",
+                              shell=True,
+                              universal_newlines=True).split()
+
+    # Check the Xorg owner and then judge the Xorg log location
+    if "root" in xorg_owner:
+        tgt_dir = root_xorg_dir
+    elif xorg_owner:
+        tgt_dir = usr_xorg_dir
     else:
-        xlog = XorgLog(os.path.expanduser("~/.local/share/xorg/Xorg.0.log"))
+        print("ERROR: No Xorg process found!", file=sys.stderr)
+
+    if tgt_dir:
+        xorg_file_paths = list(glob.iglob(tgt_dir + 'Xorg.*.log'))
+        target_file = xorg_file_paths[0]
+        xlog = XorgLog(target_file)
+
     results = []
 
     results.append(get_driver_info(xlog))