Traceback when finishing a test run on s390x

Bug #1543162 reported by Jeff Lane 
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Next Generation Checkbox (CLI)
Fix Released
Critical
Sylvain Pineau
PlainBox (Toolkit)
Fix Released
Critical
Sylvain Pineau

Bug Description

Running the suite from the dev-ppa on an s390x guest in z/KVM. The testing had some issues I'll dive into later, but for now, I have not been able to submit the results.

I added the SID to the config file and at the end, checkbox-ng tries to submit the results and that results in the traceback below:

hwe@s1lp9g003:~$ canonical-certification-server
=========================[ Resume Incomplete Session ]==========================
There are 2 incomplete sessions that might be resumed
Do you want to resume session 'pbox-8kv5ebmk'?
  r => resume this session
  n => next session
  c => create new session
[rnc]: r
=============================[ Session Statistics ]=============================
This session is about 93.33% complete
Estimated duration cannot be determined for automated jobs.
Estimated duration cannot be determined for manual jobs.
Size of the desired job list: 60
Size of the effective execution plan: 60
===========================[ Running Selected Jobs ]============================
--------------[ Running job 1 / 1. Estimated time left: 0:00:00 ]---------------
------------------------------[ info/kvm_output ]-------------------------------
ID: 2013.com.canonical.certification::info/kvm_output
Category: 2013.com.canonical.plainbox::info
(Command output hidden)
Outcome: job failed
==================================[ Results ]===================================
WARNING plainbox.exporter: Incorrect stream encoding. Got ANSI_X3.4-1968, expected utf-8. some characters won't be printed
  : Collect information about installed software packages
  : Attaches information about disk block devices and their mount points
  : requirements
  : Collect information about dpkg version
  : Collect information about installed system (lsb-release)
  : Collect information about the CPU
  : Attach detailed sysfs property output from udev
  : Attach hardware database (udev)
  : Attach a copy of /sys/class/dmi/id/*
  : Collect information about hardware devices (DMI)
  : Collect information about the running kernel
  : miscellanea/submission-resources
  : Attach a copy of /proc/cpuinfo
  : Attach a copy of /var/log/dmesg
  : Attach a copy of of the output of dmidecode
  : efi_attachment
  : Attach a list of PCI devices
  : lshw_attachment
  : lsusb_attachment
  : meminfo_attachment
  : Attach the contents of modprobe configuration files
  : Attach the contents of /etc/modules
  : Attach sysctl configuration files.
  : Attach a list of currently running kernel modules
  : info/disk_partitions
  : Retrieve the computer's make and model
  : Gets the version of MAAS used to deploy the SUT
  : EFI-mode boot test
  : CPU sanity check
  : DMI ID sanity check (servers)
  : miscellanea/fwts_test
  : miscellanea/fwts_results.log
  : miscellanea/ipmi_test
  : Gather BMC Identification info
  : cpu/clocktest
  : cpu/scaling_test
  : cpu/scaling_test-log-attach
  : cpu/maxfreq_test
  : cpu/maxfreq_test-log-attach
  : cpu/offlining_test
  : cpu/topology
  : disk/detect
  : Collect information about hardware devices (udev)
  : ethernet/detect
  : Enumerate available system executables
  : ethernet/info_automated
  : memory/info
  : memory/check
  : optical/detect
  : rtc
  : power-management/rtc
  : stress/cpu_stress_test
  : stress/cpu_stress_ng_test
  : usb/detect
  : usb/storage-preinserted
  : Collect information about supported types of USB
  : usb3/storage-preinserted
  : virtualization
  : Verify KVM VM boots
  : info/kvm_output

View results (xml): file:///home/hwe/.local/share/plainbox/submission.xml
View results (html): file:///home/hwe/.local/share/plainbox/submission.html
View results (xlsx): file:///home/hwe/.local/share/plainbox/submission.xlsx
View results (json): file:///home/hwe/.local/share/plainbox/submission.json
Submitting results to https://certification.canonical.com/submissions/submit/ for secure_id a00D000000HrCJx
CRITICAL plainbox.crashes: Executable 'checkbox-launcher' invoked with Namespace(color=None, command=<checkbox_ng.commands.launcher.LauncherCommand object at 0x3ffb4fb56a0>, debug_console=False, debug_interrupt=False, dont_suppress_output=False, dry_run=False, exclude_pattern_list=[], include_pattern_list=[], launcher='/usr/bin/canonical-certification-server', log_level=None, non_interactive=False, pdb=False, secure_id=None, test_plan=None, trace=[], url=None, whitelist=[]) has crashed
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/plainbox/impl/clitools.py", line 560, in dispatch_and_catch_exceptions
    return self.dispatch_command(ns)
  File "/usr/lib/python3/dist-packages/plainbox/impl/clitools.py", line 556, in dispatch_command
    return ns.command.invoked(ns)
  File "/usr/lib/python3/dist-packages/checkbox_ng/commands/launcher.py", line 90, in invoked
    self.provider_loader, lambda: self.config, ns, launcher
  File "/usr/lib/python3/dist-packages/checkbox_ng/commands/newcli.py", line 150, in run
    return self.do_normal_sequence()
  File "/usr/lib/python3/dist-packages/checkbox_ng/commands/newcli.py", line 212, in do_normal_sequence
    self.export_and_send_results()
  File "/usr/lib/python3/dist-packages/checkbox_ng/commands/newcli.py", line 413, in export_and_send_results
    self.submit_certification_results()
  File "/usr/lib/python3/dist-packages/checkbox_ng/commands/newcli.py", line 482, in submit_certification_results
    result = transport.send(stream, self.config)
  File "/usr/lib/python3/dist-packages/checkbox_ng/certification.py", line 161, in send
    self.url, files=form_payload, headers=headers, proxies=proxies)
  File "/usr/lib/python3/dist-packages/requests/api.py", line 107, in post
    return request('post', url, data=data, json=json, **kwargs)
  File "/usr/lib/python3/dist-packages/requests/api.py", line 53, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 454, in request
    prep = self.prepare_request(req)
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 388, in prepare_request
    hooks=merge_hooks(request.hooks, self.hooks),
  File "/usr/lib/python3/dist-packages/requests/models.py", line 296, in prepare
    self.prepare_body(data, files, json)
  File "/usr/lib/python3/dist-packages/requests/models.py", line 447, in prepare_body
    (body, content_type) = self._encode_files(files, data)
  File "/usr/lib/python3/dist-packages/requests/models.py", line 150, in _encode_files
    fdata = fp.read()
  File "/usr/lib/python3.5/encodings/ascii.py", line 26, in decode
    return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 13029: ordinal not in range(128)
Traceback (most recent call last):
  File "/usr/bin/checkbox-launcher", line 9, in <module>
    load_entry_point('checkbox-ng==0.24.dev0', 'console_scripts', 'checkbox-launcher')()
  File "/usr/lib/python3/dist-packages/checkbox_ng/main.py", line 54, in launcher
    raise SystemExit(CheckboxLauncherTool().main(argv))
  File "/usr/lib/python3/dist-packages/plainbox/impl/clitools.py", line 324, in main
    return self.dispatch_and_catch_exceptions(ns)
  File "/usr/lib/python3/dist-packages/plainbox/impl/clitools.py", line 560, in dispatch_and_catch_exceptions
    return self.dispatch_command(ns)
  File "/usr/lib/python3/dist-packages/plainbox/impl/clitools.py", line 556, in dispatch_command
    return ns.command.invoked(ns)
  File "/usr/lib/python3/dist-packages/checkbox_ng/commands/launcher.py", line 90, in invoked
    self.provider_loader, lambda: self.config, ns, launcher
  File "/usr/lib/python3/dist-packages/checkbox_ng/commands/newcli.py", line 150, in run
    return self.do_normal_sequence()
  File "/usr/lib/python3/dist-packages/checkbox_ng/commands/newcli.py", line 212, in do_normal_sequence
    self.export_and_send_results()
  File "/usr/lib/python3/dist-packages/checkbox_ng/commands/newcli.py", line 413, in export_and_send_results
    self.submit_certification_results()
  File "/usr/lib/python3/dist-packages/checkbox_ng/commands/newcli.py", line 482, in submit_certification_results
    result = transport.send(stream, self.config)
  File "/usr/lib/python3/dist-packages/checkbox_ng/certification.py", line 161, in send
    self.url, files=form_payload, headers=headers, proxies=proxies)
  File "/usr/lib/python3/dist-packages/requests/api.py", line 107, in post
    return request('post', url, data=data, json=json, **kwargs)
  File "/usr/lib/python3/dist-packages/requests/api.py", line 53, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 454, in request
    prep = self.prepare_request(req)
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 388, in prepare_request
    hooks=merge_hooks(request.hooks, self.hooks),
  File "/usr/lib/python3/dist-packages/requests/models.py", line 296, in prepare
    self.prepare_body(data, files, json)
  File "/usr/lib/python3/dist-packages/requests/models.py", line 447, in prepare_body
    (body, content_type) = self._encode_files(files, data)
  File "/usr/lib/python3/dist-packages/requests/models.py", line 150, in _encode_files
    fdata = fp.read()
  File "/usr/lib/python3.5/encodings/ascii.py", line 26, in decode
    return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 13029: ordinal not in range(128)

Related branches

Revision history for this message
Jeff Lane  (bladernr) wrote :

Using canonical-certification-submit succeeds without complaint:
hwe@s1lp9g003:~$ canonical-certification-submit --secure_id a00D000000HrCJx .local/share/plainbox/submission.xml
Successfully sent, submission status at https://certification.canonical.com/submissions/status/19130

However, I'm still waiting for that to result into a submission attached to the hardware. It's still in the pending status after several minutes.

tags: added: hwcert-server
Revision history for this message
Jeff Lane  (bladernr) wrote :

Also, here's a tarball of the results files. The HTML one looks fine to me (rendered locally in Chrome).

And package versions:
hwe@s1lp9g003:~$ dpkg -l |grep checkbox
ii checkbox-ng 0.24+bzr4205+pkg36~ubuntu16.04.1 all PlainBox based test runner
ii plainbox-provider-checkbox 0.26+bzr4205+pkg33~ubuntu16.04.1 s390x CheckBox provider for PlainBox
ii python3-checkbox-ng 0.24+bzr4205+pkg36~ubuntu16.04.1 all PlainBox based test runner (Python 3 library)
ii python3-checkbox-support 0.24+bzr4205+pkg26~ubuntu16.04.1 all collection of Python modules used by PlainBox providers
hwe@s1lp9g003:~$ dpkg -l |grep plainbox
ii plainbox-insecure-policy 0.26+bzr4205+pkg40~ubuntu16.04.1 all policykit policy required to use plainbox (insecure version)
ii plainbox-provider-certification-server 0.21+201602041003~ubuntu16.04.1 all Server Certification provider for Plainbox
ii plainbox-provider-checkbox 0.26+bzr4205+pkg33~ubuntu16.04.1 s390x CheckBox provider for PlainBox
ii plainbox-provider-resource-generic 0.24+bzr4205+pkg25~ubuntu16.04.1 s390x CheckBox generic resource jobs provider
ii python3-plainbox 0.26+bzr4205+pkg40~ubuntu16.04.1 all toolkit for software and hardware testing (python3 module)

Revision history for this message
Jeff Lane  (bladernr) wrote :

https://certification.canonical.com/hardware/201012-6906/submission/108428/

I am able to successfully submit that failing result from the SUT using canonical-certification-submit, and C3 properly parses and logs it and attaches it to the correct system.

So it seems that the XML is valid. I am not sure exactly why checkbox flaked on this, but I have a suspicion it has to do with that weird ANSI error:

WARNING plainbox.exporter: Incorrect stream encoding. Got ANSI_X3.4-1968, expected utf-8. some characters won't be printed

Revision history for this message
Jeff Lane  (bladernr) wrote :

The C3 parsing error in Comment #1 was due to swift errors after an upgrade, not related to this issue.

Changed in checkbox:
importance: Undecided → Critical
status: New → Confirmed
affects: checkbox → checkbox-ng
Changed in checkbox-ng:
milestone: none → 0.24
Revision history for this message
Zygmunt Krynicki (zyga) wrote :

Can we identify which part of the submission causes this.

Revision history for this message
Jeff Lane  (bladernr) wrote :

I'm curious about that myself, because I was able to submit manually and it worked without a problem.

Revision history for this message
Sylvain Pineau (sylvain-pineau) wrote :

We would need the session folder ~/.cache/plainbox to identify the guilty job but I noticed a subtle difference between canonical-certification-submit and the newcli.py module of checkbox-ng:

c-c-s is opening the submission xml file as follows:

            with open(self.ns.submission, "r", encoding='utf-8') as subm_file:
                result = transport.send(subm_file)

nut newcli code is:

        with open(self.submission_file) as stream:
            try:
                # Send the data, reading from the fallback file
                result = transport.send(stream, self.config)

I'm pretty sure that if we forced encoding like c-c-s to be utf-8 we wouldn’t have the error.

Changed in checkbox-ng:
assignee: nobody → Sylvain Pineau (sylvain-pineau)
status: Confirmed → In Progress
Revision history for this message
Jeff Lane  (bladernr) wrote :

Per request from Sylvain, here's the plainbox cache from, the failing system

Changed in checkbox-ng:
status: In Progress → Fix Committed
Revision history for this message
Jeff Lane  (bladernr) wrote :
Download full text (3.3 KiB)

This still seems to be a problem :(

hwe@s1lp9g003:~$ apt-cache policy checkbox-ng
checkbox-ng:
  Installed: 0.24+bzr4275+pkg36~ubuntu16.04.1
  Candidate: 0.24+bzr4275+pkg36~ubuntu16.04.1

Undeclared exception UnicodeDecodeError raised from export_to_transport

                        Application Malfunction Detected

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/guacamole/core.py", line 248, in eat
    return self._dispatch()
  File "/usr/lib/python3/dist-packages/guacamole/core.py", line 298, in _dispatch
    result = ingredient.dispatch(self.context)
  File "/usr/lib/python3/dist-packages/guacamole/ingredients/cmdtree.py", line 152, in dispatch
    return self._dispatch(context, 0)
  File "/usr/lib/python3/dist-packages/guacamole/ingredients/cmdtree.py", line 165, in _dispatch
    retval = command.invoked(context)
  File "/usr/bin/checkbox-cli", line 213, in invoked
    self._export_results()
  File "/usr/bin/checkbox-cli", line 749, in _export_results
    exporter_id, transport)
  File "/usr/lib/python3/dist-packages/plainbox/impl/decorators.py", line 104, in wrapper
    raise exc
  File "/usr/lib/python3/dist-packages/plainbox/impl/decorators.py", line 98, in wrapper
    return func(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/plainbox/impl/session/assistant.py", line 1395, in export_to_transport
    result = transport.send(exported_stream)
  File "/usr/lib/python3/dist-packages/plainbox/impl/transport.py", line 349, in send
    copyfileobj(data, translating_stream)
  File "/usr/lib/python3.5/shutil.py", line 76, in copyfileobj
    fdst.write(buf)
  File "/usr/lib/python3/dist-packages/plainbox/impl/exporter/__init__.py", line 391, in write
    return self.dest_stream.write(data.decode(self.encoding))
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 8: ordinal not in range(128)

  Please report a bug including the information from the paragraph above. To
  report the bug visit https://bugs.launchpad.net/checkbox-ng/+filebug

  We are sorry for the inconvenience! ...

Read more...

Changed in checkbox-ng:
status: Fix Committed → Triaged
status: Triaged → Confirmed
Changed in checkbox-ng:
status: Confirmed → In Progress
Changed in checkbox-ng:
status: In Progress → Fix Committed
Changed in plainbox:
status: New → Fix Committed
importance: Undecided → Critical
assignee: nobody → Sylvain Pineau (sylvain-pineau)
milestone: none → 0.26
Revision history for this message
Jeff Lane  (bladernr) wrote :

Thanks, I believe this is resolved now...

Pierre Equoy (pieq)
Changed in checkbox-ng:
status: Fix Committed → Fix Released
Changed in plainbox:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.