Checkbox Provider - Canonical Certification Client

Merge ~stanley31/plainbox-provider-certification-client:client-cert-manual2 into plainbox-provider-certification-client:master

  1. Git
  2. lp:~stanley31/plainbox-provider-certification-client
  3. client-cert-manual2
  4. Merge into master
Proposed by StanleyHuang
Status: Rejected
Rejected by: StanleyHuang
Proposed branch: ~stanley31/plainbox-provider-certification-client:client-cert-manual2
Merge into: plainbox-provider-certification-client:master
Diff against target: 63 lines (+56/-0)
1 file modified
units/client-cert-manual.pxu (+56/-0)
Reviewer Review Type Date Requested Status
Jonathan Cave (community) Needs Fixing
Betty Lin (community) Approve
Review via email: mp+395586@code.launchpad.net

Commit message

Create a fully manual test plan for doing manual testing while kernel update

Description of the change

We need a new test plan to cover the manual part while doing testing for kernel update

To post a comment you must log in.
Revision history for this message
StanleyHuang (stanley31) wrote :
Download full text (16.1 KiB)

run checkbox list-bootstrapped to show all jobs in this test plan as following:

com.canonical.certification::miscellanea/device_check
com.canonical.certification::cpuinfo
com.canonical.certification::system_info_json
com.canonical.certification::modprobe_json
com.canonical.certification::executable
com.canonical.certification::lsblk_attachment
com.canonical.certification::dmi_present
com.canonical.certification::dmi_attachment
com.canonical.certification::sysfs_attachment
com.canonical.certification::dmi
com.canonical.certification::efi
com.canonical.certification::cdimage
com.canonical.certification::lspci_standard_config_json
com.canonical.certification::kernel_cmdline_attachment
com.canonical.certification::environment
com.canonical.certification::dkms_info_json
com.canonical.certification::meminfo
com.canonical.certification::udev_json
com.canonical.certification::uname
com.canonical.certification::dpkg
com.canonical.certification::lsb
com.canonical.certification::module
com.canonical.certification::package
com.canonical.certification::requirements
com.canonical.certification::raw_devices_dmi_json
com.canonical.certification::udev_attachment
com.canonical.certification::snap
com.canonical.certification::miscellanea/submission-resources
com.canonical.certification::info/systemd-analyze
com.canonical.certification::device
com.canonical.certification::audio/alsa_record_playback_automated
com.canonical.certification::info/detect_recovery
com.canonical.certification::recovery_info_attachment
com.canonical.certification::acpi_sleep_attachment
com.canonical.certification::codecs_attachment
com.canonical.certification::cpuinfo_attachment
com.canonical.certification::dkms_info_attachment
com.canonical.certification::dmesg_attachment
com.canonical.certification::dmidecode_attachment
com.canonical.certification::efi_attachment
com.canonical.certification::info/buildstamp
com.canonical.certification::info/disk_partitions
com.canonical.certification::block_device
com.canonical.certification::info/hdparm_nvme0n1.txt
com.canonical.certification::info/touchpad_driver
com.canonical.certification::installer_debug.gz
com.canonical.certification::lsmod_attachment
com.canonical.certification::lspci_attachment
com.canonical.certification::lspci_standard_config_attachment
com.canonical.certification::lstopo_verbose_attachment
com.canonical.certification::lstopo_visual_attachment
com.canonical.certification::lsusb_attachment
com.canonical.certification::meminfo_attachment
com.canonical.certification::modinfo_attachment
com.canonical.certification::modprobe_attachment
com.canonical.certification::modules_attachment
com.canonical.certification::sysctl_attachment
com.canonical.certification::audio/speaker-headphone-plug-detection
com.canonical.certification::audio/microphone-plug-detection
com.canonical.certification::audio/list_devices
com.canonical.certification::audio/playback_headphones
com.canonical.certification::audio/alsa_record_playback_external
com.canonical.certification::audio/playback_auto
com.canonical.certification::audio/alsa_record_playback_internal
com.canonical.certification::audio/channels
com.canonical.certification::audio/external-linein
com....

Revision history for this message
Betty Lin (bettyl) wrote :

Since this test plan is manual, the auto test plan - misc-client-cert-full should not in it.

review: Needs Fixing
~stanley31/plainbox-provider-certification-client:client-cert-manual2 updated
6cbed3b... by StanleyHuang

remove automatic test case

remote automatic test case

Revision history for this message
StanleyHuang (stanley31) wrote :
Download full text (19.5 KiB)

Remove the automatic test plan and run checkbox list-bootstrapped to show all jobs in this test plan as following:

u@u-Dell-G15-5510:~$ checkbox-cli list-bootstrapped com.canonical.certification::client-cert-manual
com.canonical.certification::miscellanea/device_check
com.canonical.certification::executable
com.canonical.certification::lsblk_attachment
com.canonical.certification::meminfo
com.canonical.certification::requirements
com.canonical.certification::modprobe_json
com.canonical.certification::udev_json
com.canonical.certification::cdimage
com.canonical.certification::package
com.canonical.certification::dmi_present
com.canonical.certification::dmi
com.canonical.certification::dmi_attachment
com.canonical.certification::lsb
com.canonical.certification::efi
com.canonical.certification::dkms_info_json
com.canonical.certification::uname
com.canonical.certification::cpuinfo
com.canonical.certification::system_info_json
com.canonical.certification::kernel_cmdline_attachment
com.canonical.certification::udev_attachment
com.canonical.certification::dpkg
com.canonical.certification::environment
com.canonical.certification::raw_devices_dmi_json
com.canonical.certification::module
com.canonical.certification::sysfs_attachment
com.canonical.certification::snap
com.canonical.certification::lspci_standard_config_json
com.canonical.certification::miscellanea/submission-resources
com.canonical.certification::info/systemd-analyze
com.canonical.certification::device
com.canonical.certification::audio/alsa_record_playback_automated
com.canonical.certification::info/detect_recovery
com.canonical.certification::recovery_info_attachment
com.canonical.certification::acpi_sleep_attachment
com.canonical.certification::codecs_attachment
com.canonical.certification::cpuinfo_attachment
com.canonical.certification::dkms_info_attachment
com.canonical.certification::dmesg_attachment
com.canonical.certification::dmidecode_attachment
com.canonical.certification::efi_attachment
com.canonical.certification::info/buildstamp
com.canonical.certification::info/disk_partitions
com.canonical.certification::block_device
com.canonical.certification::info/hdparm_nvme0n1.txt
com.canonical.certification::info/touchpad_driver
com.canonical.certification::installer_debug.gz
com.canonical.certification::lsmod_attachment
com.canonical.certification::lspci_attachment
com.canonical.certification::lspci_standard_config_attachment
com.canonical.certification::lstopo_verbose_attachment
com.canonical.certification::lstopo_visual_attachment
com.canonical.certification::lsusb_attachment
com.canonical.certification::meminfo_attachment
com.canonical.certification::modinfo_attachment
com.canonical.certification::modprobe_attachment
com.canonical.certification::modules_attachment
com.canonical.certification::sysctl_attachment
com.canonical.certification::audio/speaker-headphone-plug-detection
com.canonical.certification::audio/microphone-plug-detection
com.canonical.certification::audio/list_devices
com.canonical.certification::audio/playback_headphones
com.canonical.certification::audio/alsa_record_playback_external
com.canonical.certification::audio/playback_auto
com.canonical.certification::...

Revision history for this message
Betty Lin (bettyl) wrote :

LGTM +1

review: Approve
Revision history for this message
Jonathan Cave (jocave) wrote :

When you say this is need for testing "kernel update" do you mean as part of SRU?

review: Needs Information
Revision history for this message
StanleyHuang (stanley31) wrote :

> When you say this is need for testing "kernel update" do you mean as part of
> SRU?

Actually, this is for 5.6 kernel retirement testing.
We would like to increase the test coverage to make sure the system is good enough after update kernel from 5.6 to 5.8 or 5.10.

Revision history for this message
Jonathan Cave (jocave) wrote :

I would like it to be clearer what this test plan is for and who uses it. I think this needs to be reflected both in the ID of the test plan and with a more verbose description.

I made an initial attempt to document some naming conventions in this file: https://git.launchpad.net/plainbox-provider-certification-client/tree/README.rst

It's not clear to me where this fits it to Certification programmes

review: Needs Fixing
Revision history for this message
Maciej Kisielewski (kissiel) wrote :

Looks like this MR is awaiting another patch. I'm marking it as "Work in progress".

Revision history for this message
OEM Taipei Bot (oem-taipei-bot) wrote :

[BOT]
$ cat plainbox-provider-certification-client-0.46.0-1-6cbed3b-in-linux-container-focal-summary.log
bootstrap-client-cert-desktop-20-04-automated FAIL non-zero exit status 1
https://oem-share.canonical.com/share/lyoncore/artifacts/plainbox-provider-certification-client-0.46.0-1-6cbed3b-in-linux-container-focal

Revision history for this message
OEM Taipei Bot (oem-taipei-bot) wrote :

Execute `curl -X POST http://10.102.135.31/api/v1/teams/self-contained/pipelines/plainbox-provider-certification-client/resources/merge-proposal-395586/check/webhook?webhook_token=merge-proposal-395586` within TW VPN to restart the test.
[autopkgtest]
$ cat plainbox-provider-certification-client-0.46.0-1-6cbed3b-in-linux-container-focal-summary.log
bootstrap-client-cert-desktop-20-04-automated FAIL non-zero exit status 1
https://oem-share.canonical.com/partners/lyoncore/share/artifacts/plainbox-provider-certification-client-0.46.0-1-6cbed3b-in-linux-container-focal

Revision history for this message
StanleyHuang (stanley31) wrote :

We don't need this MR anymore.

Unmerged commits

6cbed3b... by StanleyHuang

remove automatic test case

remote automatic test case

26db0b4... by StanleyHuang

Add client-cert-manual test plan

Add the client-cert-manual test plan for Kernel update testing

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1diff --git a/units/client-cert-manual.pxu b/units/client-cert-manual.pxu
2new file mode 100644
3index 0000000..6dfc372
4--- /dev/null
5+++ b/units/client-cert-manual.pxu
6@@ -0,0 +1,56 @@
7+id: client-cert-manual
8+_name: Fully Manual Client Certification Tests
9+unit: test plan
10+include:
11+ com.canonical.certification::audio/alsa_record_playback_automated
12+ com.canonical.certification::recovery_info_attachment
13+nested_part:
14+ com.canonical.certification::submission-cert-full
15+ com.canonical.certification::info-attachment-cert-full
16+ com.canonical.certification::audio-cert-manual
17+ com.canonical.certification::bluetooth-cert-manual
18+ com.canonical.certification::camera-cert-manual
19+ com.canonical.certification::thunderbolt-cert-manual
20+ com.canonical.certification::monitor-integrated-gpu-cert-manual
21+ com.canonical.certification::graphics-integrated-gpu-cert-manual
22+ com.canonical.certification::graphics-discrete-gpu-cert-manual
23+ com.canonical.certification::monitor-discrete-gpu-cert-manual
24+ com.canonical.certification::input-cert-manual
25+ com.canonical.certification::disk-cert-manual
26+ com.canonical.certification::fingerprint-manual
27+ com.canonical.certification::keys-cert-manual
28+ com.canonical.certification::led-cert-manual
29+ com.canonical.certification::mediacard-cert-manual
30+ com.canonical.certification::mobilebroadband-cert-manual
31+ com.canonical.certification::networking-cert-manual
32+ com.canonical.certification::optical-cert-manual
33+ com.canonical.certification::touchpad-cert-manual
34+ com.canonical.certification::touchscreen-cert-manual
35+ com.canonical.certification::usb-cert-manual
36+ com.canonical.certification::usb3-cert-manual
37+ # we have this test plan only for usb-c
38+ usb-c-cert-full
39+ # Skip wireless-cert-manual due to it's same as automated
40+ com.canonical.certification::suspend-tp
41+ com.canonical.certification::after-suspend-graphics-discrete-gpu-cert-manual
42+ com.canonical.certification::after-suspend-manual-monitor-discrete-gpu-cert-manual
43+ com.canonical.certification::after-suspend-graphics-integrated-gpu-cert-manual
44+ com.canonical.certification::after-suspend-manual-monitor-integrated-gpu-cert-manual
45+ com.canonical.certification::after-suspend-audio-cert-full
46+ com.canonical.certification::after-suspend-camera-cert-full
47+ com.canonical.certification::after-suspend-input-cert-full
48+ com.canonical.certification::after-suspend-keys-cert-blockers
49+ com.canonical.certification::after-suspend-mediacard-cert-full
50+ com.canonical.certification::after-suspend-optical-cert-blockers
51+ com.canonical.certification::after-suspend-touchpad-cert-blockers
52+ com.canonical.certification::after-suspend-touchscreen-cert-manual
53+ com.canonical.certification::after-suspend-usb-c-cert-full
54+ com.canonical.certification::after-suspend-usb3-cert-full
55+ com.canonical.certification::after-suspend-usb-cert-full
56+ # Skip after-suspend-wireless-cert-full has automated tests only
57+ com.canonical.certification::after-suspend-thunderbolt-cert-manual
58+ com.canonical.certification::power-management-reboot-poweroff-cert
59+ com.canonical.certification::info-attachment-cert-full
60+bootstrap_include:
61+ com.canonical.certification::device
62+ com.canonical.certification::graphics_card
63\ No newline at end of file

Subscribers

People subscribed via source and target branches