'utf-8' codec can't decode byte 0xc0

Bug #1979036 reported by Bin Li
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Checkbox Support Library
Fix Released
High
Bin Li
OEM Priority Project
Fix Released
High
Bin Li

Bug Description

In our OEM project, we found lots of ThinkPad's battery could not be parsed correctly.

For the 'udev_resource.py' will call 'udevadm info --export-db | checkbox-support-parse udevadm'

The test failed with error:

$ checkbox-cli run com.canonical.certification::udev_json
===========================[ Running Selected Jobs ]============================
--------------[ Running job 1 / 1. Estimated time left: unknown ]---------------
------------------[ Attaches json dumps of udev_resource.py ]-------------------
ID: com.canonical.certification::udev_json
Category: com.canonical.plainbox::info
... 8< -------------------------------------------------------------------------
Failed to decode input stream: 'utf-8' codec can't decode byte 0xc0 in position 39452: invalid start byte
------------------------------------------------------------------------- >8 ---
Outcome: job failed

[Steps to reproduce]
checkbox-cli run com.canonical.certification::udev_json

Revision history for this message
Bin Li (binli) wrote :

Another testcase is failed.

The test com.canonical.certification::ethernet/detect failed with the error list blow:
-------------[ Detect if at least one ethernet device is detected ]-------------
ID: com.canonical.certification::ethernet/detect
Category: com.canonical.plainbox::ethernet
... 8< -------------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/nest-m_b8ini3.9a94080dfa9d0226560c697ac550c8dc840cf7385cc17eb075a488d5c7c11d21/network_device_info.py", line 393, in <module>
    udev = UdevDevices(args.category)
  File "/tmp/nest-m_b8ini3.9a94080dfa9d0226560c697ac550c8dc840cf7385cc17eb075a488d5c7c11d21/network_device_info.py", line 338, in __init__
    self._collect_devices()
  File "/tmp/nest-m_b8ini3.9a94080dfa9d0226560c697ac550c8dc840cf7385cc17eb075a488d5c7c11d21/network_device_info.py", line 346, in _collect_devices
    output = check_output(cmd).decode(sys.stdout.encoding)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xc0 in position 38854: invalid start byte

Revision history for this message
Bin Li (binli) wrote :

E: DEVPATH=/devices/LNXSYSTM:00/LNXSYBUS:00/PNP0A08:00/device:3b/PNP0C09:00/PNP0C0A:00/power_supply/BAT0
E: SUBSYSTEM=power_supply
E: POWER_SUPPLY_NAME=BAT0
E: POWER_SUPPLY_TYPE=Battery
E: POWER_SUPPLY_STATUS=Full
E: POWER_SUPPLY_PRESENT=1
E: POWER_SUPPLY_TECHNOLOGY=Li-poly
E: POWER_SUPPLY_CYCLE_COUNT=2
E: POWER_SUPPLY_VOLTAGE_MIN_DESIGN=11580000
E: POWER_SUPPLY_VOLTAGE_NOW=13278000
E: POWER_SUPPLY_POWER_NOW=0
E: POWER_SUPPLY_ENERGY_FULL_DESIGN=39300000
E: POWER_SUPPLY_ENERGY_FULL=39300000
E: POWER_SUPPLY_ENERGY_NOW=39300000
E: POWER_SUPPLY_CAPACITY=100
E: POWER_SUPPLY_CAPACITY_LEVEL=Full
E: POWER_SUPPLY_MODEL_NAME=LNV-5B10W51860Àà
E: POWER_SUPPLY_MANUFACTURER=LGES
E: POWER_SUPPLY_SERIAL_NUMBER= 496

The POWER_SUPPLY_MODEL_NAME=LNV-5B10W51860Àà has some non-utf8 character.

Revision history for this message
Bin Li (binli) wrote :

The E0 and C0 are à and À in extended ASCII.

POWER_SUPPLY_MODEL_NAME=LNV-5B10W51860Àà

$ hexdump /sys/class/power_supply/BAT0/model_name
0000000 4e4c 2d56 4235 3031 3557 3831 3036 e0c0
0000010 000a
0000011

tags: added: oem-priority originate-from-1976462 sutton
tags: added: originate-from-1976471
tags: added: originate-from-1979034
Revision history for this message
Bin Li (binli) wrote :

I found a method to fix this issue, I could use 'replace' or 'ignore' errors, from the result I prefer the 'ignore'.

--- a/checkbox_support/parsers/__init__.py
+++ b/checkbox_support/parsers/__init__.py
@@ -71,7 +71,7 @@ def main():
         raise SystemExit()
     parser = AVAILABLE_PARSERS[args.parser_name]
     stdin = sys.stdin
- with io.TextIOWrapper(sys.stdin.buffer, encoding='utf-8') as stdin:
+ with io.TextIOWrapper(sys.stdin.buffer, encoding='utf-8', errors='ignore') as stdin:
         try:
             text = stdin.read()
             print(run_parsing(parser, text))

Using 'replace'
$ checkbox-cli run com.canonical.certification::udev_json | grep LNV
        "product": "LNV-5B10W51860\ufffd\ufffd",
        "product_slug": "LNV-5B10W51860__",
Finalizing session that hasn't been submitted anywhere: checkbox-run-2022-06-17T08.03.06

Using 'ignore'.
$ checkbox-cli run com.canonical.certification::udev_json | grep LNV
        "product": "LNV-5B10W51860",
        "product_slug": "LNV-5B10W51860",
Finalizing session that hasn't been submitted anywhere: checkbox-run-2022-06-17T08.03.32

Bin Li (binli)
Changed in oem-priority:
assignee: nobody → Bin Li (binli)
importance: Undecided → High
status: New → In Progress
Revision history for this message
Bin Li (binli) wrote :

The test failed with error list below:
-----[ Identify what service is managing each physical network interface ]------
ID: com.canonical.certification::net_if_management
Category: com.canonical.plainbox::uncategorised
... 8< -------------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/nest-q6kekfja.85214daceec20cd8651bf53a569971cc3a614b158baeaeabe09973416fbf3505/net_if_management.py", line 199, in <module>
    main()
  File "/tmp/nest-q6kekfja.85214daceec20cd8651bf53a569971cc3a614b158baeaeabe09973416fbf3505/net_if_management.py", line 188, in main
    results = identify_managers(has_netplan=is_netplan_available(),
  File "/tmp/nest-q6kekfja.85214daceec20cd8651bf53a569971cc3a614b158baeaeabe09973416fbf3505/net_if_management.py", line 97, in identify_managers
    wired = get_network_interfaces('NETWORK')
  File "/tmp/nest-q6kekfja.85214daceec20cd8651bf53a569971cc3a614b158baeaeabe09973416fbf3505/net_if_management.py", line 39, in get_network_interfaces
    output = sp.check_output(cmd, shell=True).decode(sys.stdout.encoding)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xc0 in position 38854: invalid start byte

Revision history for this message
Bin Li (binli) wrote :

The test failed with error list below:

ID: com.canonical.certification::touchpad/palm-rejection-firmware-labeling_ELAN0678_00_04F3_3195_Touchpad
Category: com.canonical.plainbox::touchpad
... 8< -------------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/nest-9roc0600.478c4952642bd256030a1147ed323c16a2d1f00251873c42ea6472976890f74e/touchpad_confidence_bit.py", line 54, in <module>
    main()
  File "/tmp/nest-9roc0600.478c4952642bd256030a1147ed323c16a2d1f00251873c42ea6472976890f74e/touchpad_confidence_bit.py", line 33, in main
    path = TouchpadDevices().devices[product_slug]
  File "/tmp/nest-9roc0600.478c4952642bd256030a1147ed323c16a2d1f00251873c42ea6472976890f74e/touchpad_confidence_bit.py", line 12, in __init__
    self._collect_devices()
  File "/tmp/nest-9roc0600.478c4952642bd256030a1147ed323c16a2d1f00251873c42ea6472976890f74e/touchpad_confidence_bit.py", line 17, in _collect_devices
    output = check_output(cmd).decode(sys.stdout.encoding)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xc0 in position 38854: invalid start byte
------------------------------------------------------------------------- >8 ---

Bin Li (binli)
tags: added: originate-from-1962833
tags: added: originate-from-1969884
tags: added: originate-from-1979035
Bin Li (binli)
tags: added: originate-from-1976479
Changed in checkbox-support:
milestone: none → 0.56.0
status: New → Fix Committed
importance: Undecided → Medium
importance: Medium → High
assignee: nobody → Bin Li (binli)
Pierre Equoy (pieq)
tags: added: ce-qa-concern
tags: removed: ce-qa-concern
Changed in checkbox-support:
milestone: 0.56.0 → 0.55.0
Bin Li (binli)
Changed in oem-priority:
status: In Progress → Fix Released
Changed in checkbox-support:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.