Merge ~sylvain-pineau/checkbox-limerick/+git/checkbox-provider-limerick:components into ~checkbox-dev/checkbox-limerick/+git/checkbox-limerick-classic:master

[Sylvain Pineau]

Add tests for components + test plans cleanup

Unmerged commits

79301dc... by Sylvain Pineau on 2020-09-30

exclude disk/storage_device_mtd.*

This Flash memory is not really meant for write operations:

"The only case where the mtdblock device is useful is to mount read-only
file systems such as Squashfs on top of reliable flash memory such as NOR."

Src: https://www.oreilly.com/library/view/mastering-embedded-linux/9781787283282/64271306-bd52-47d8-8118-6b618630d307.xhtml

23db591... by Sylvain Pineau on 2020-09-30

Add DTB components tests

58fcb31... by Sylvain Pineau on 2020-09-30

Add the limerick_sku resource job

810d9f6... by Sylvain Pineau on 2020-09-30

test plan cleanup

ba28088... by PMR <pmr@pmr-lander> on 2020-09-16

Merge #390817 from ~bettyl/checkbox-limerick/+git/checkbox-provider-limerick:add_after_suspend_testplan

5848f51... by Betty Lin on 2020-09-16

add after suspend tests

1. add after suspend tests
2. exclude some monitor tests and power management lid related tests
3. Remove kernel-snap-automated test which not need test in desktop
image

e5f4fd9... by PMR <pmr@pmr-lander> on 2020-09-15

Merge #390722 from ~zongminl/checkbox-limerick/+git/checkbox-provider-limerick:test-plan-cleanup

Clean up the test plan:
1. Remove limerick-stress from Full test plan.
2. Exclude cpu/scaling_test since it's not supported on ARM.

f87985a... by Vic Liu on 2020-09-15

1. Remove limerick-stress from Full test plan.
2. Exclude cpu/scaling_test since it's not supported on ARM.

260a92f... by PMR <pmr@pmr-lander> on 2020-09-14

Merge #390654 from ~bettyl/checkbox-limerick/+git/checkbox-provider-limerick:modify_testplan

4d0696b... by Betty Lin on 2020-09-14

modify test plan

1. Exclude some cpu tests which can't run on ARM device
2. Exlcude hibernate stress test
3. Exlcude dvi related tests from monitor

diff --git a/bin/component_test.py b/bin/component_test.py
new file mode 100755
index 0000000..4f4c4ba
--- /dev/null
+++ b/bin/component_test.py
@@ -0,0 +1,55 @@
+#!/usr/bin/env python3
+#
+# This file is part of Checkbox.
+#
+# Copyright 2020 Canonical Ltd.
+#
+# Authors:
+#    Sylvain Pineau <[e-mail]>
+#
+# Checkbox is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 3,
+# as published by the Free Software Foundation.
+#
+# Checkbox is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with Checkbox.  If not, see <http://www.gnu.org/licenses/>.
+
+import codecs
+import glob
+import os
+import re
+import subprocess
+import sys
+
+p = glob.glob('/usr/lib/firmware/*/device-tree/{}'.format(
+    codecs.decode("kvyvak", "rot-13")))[0]
+dtb = os.path.join(p, '{}.dtb'.format(sys.argv[1]))
+dt_lookup_list = []
+exit_status = 0
+
+dt_compatible_patterns = subprocess.check_output(
+    'find /proc/device-tree/ -name compatible -exec '
+    'bash -c "cat {} && echo" \\; | sort -u',
+    shell=True, universal_newlines=True).splitlines()
+for line in dt_compatible_patterns:
+    line = line.replace("\0", ", ")
+    dt_lookup_list.append(re.split(', ', line)[:-1])
+
+compatible_patterns = subprocess.check_output(
+    'fdtdump {} 2>/dev/null | grep -oP "(?<= compatible = ).*"'
+    ' | sort -u'.format(dtb),
+    shell=True, universal_newlines=True).splitlines()
+for l in compatible_patterns:
+    item = re.split(', ', l.replace('"', '').replace(';', ''))
+    if item in dt_lookup_list:
+        print('Found in /proc/device-tree/:', l)
+    else:
+        print('ERROR: Not found:', l, file=sys.stderr)
+        exit_status = 1
+
+raise SystemExit(exit_status)
diff --git a/manage.py b/manage.py
new file mode 100755
index 0000000..da21a67
--- /dev/null
+++ b/manage.py
@@ -0,0 +1,16 @@
+#!/usr/bin/env python3
+# Copyright 2020 Canonical Ltd.
+# All rights reserved.
+
+"""Management script for the limerick provider."""
+
+from plainbox.provider_manager import setup
+from plainbox.provider_manager import N_
+
+setup(
+    name='checkbox-provider-limerick',
+    namespace='com.canonical.qa.limerick',
+    version="0.1",
+    description=N_("Checkbox Provider for limerick devices"),
+    gettext_domain='checkbox-provider-limerick',
+)
diff --git a/units/components/category.pxu b/units/components/category.pxu
new file mode 100644
index 0000000..95a11eb
--- /dev/null
+++ b/units/components/category.pxu
@@ -0,0 +1,9 @@
+# Copyright 2020 Canonical Ltd.
+# All rights reserved.
+#
+# Written by:
+#   Sylvain Pineau <[e-mail]>
+
+unit: category
+id: dtb_components
+_name: Device Tree Components
diff --git a/units/components/jobs.pxu b/units/components/jobs.pxu
new file mode 100644
index 0000000..524bcd6
--- /dev/null
+++ b/units/components/jobs.pxu
@@ -0,0 +1,10 @@
+unit: template
+template-resource: limerick_sku
+id: dtb/components_{name}
+_summary: Check all DTB compatible properties are loaded in /proc/device-tree
+command: component_test.py {name}
+plugin: shell
+category_id: dtb_components
+estimated_duration: 1m
+imports: from com.canonical.certification import package
+requires: package.name == 'device-tree-compiler'
diff --git a/units/components/test-plan.pxu b/units/components/test-plan.pxu
new file mode 100644
index 0000000..cefd8a5
--- /dev/null
+++ b/units/components/test-plan.pxu
@@ -0,0 +1,6 @@
+id: dtb-automated
+unit: test plan
+_name: Automated DTB tests
+_description: Automated DTB tests
+include:
+    dtb/components_.*
diff --git a/units/device/jobs.pxu b/units/device/jobs.pxu
new file mode 100644
index 0000000..3a9fde5
--- /dev/null
+++ b/units/device/jobs.pxu
@@ -0,0 +1,34 @@
+id: connect-plugs
+estimated_duration: 60
+plugin: resource
+user: root
+command:
+    checkbox-support-snap_connect \
+        hardware-observe:core:hardware-observe \
+        mount-observe:core:mount-observe \
+        network-bind:core:network-bind \
+        system-observe:core:system-observe \
+        # bluez:bluez:service \
+        # bluetooth-control:core:bluetooth-control \
+        # network-manager:network-manager:service \
+        home:core:home
+_summary: Connect all checkbox plugs
+_description: Connect all checkbox plugs
+
+id: limerick_sku
+plugin: resource
+_summary: Detect which Limerick SKU the tool is running on
+command:
+ perl -ne 's/ /-/g;s/(.*?)([A-Z]?)\W+$/\L$1\U$2/; print "name: $_\n"' /proc/device-tree/model
+estimated_duration: 1s
+
+id: device-connections-tp
+unit: test plan
+_name: Test plan that connects all checkbox plugs
+_description:
+ This test plan is necessary to assure the proper ordering of resource jobs
+ being run
+estimated_duration: 60
+bootstrap_include:
+ connect-plugs
+include:
diff --git a/units/test-plan-classic.pxu b/units/test-plan-classic.pxu
new file mode 100644
index 0000000..b333acf
--- /dev/null
+++ b/units/test-plan-classic.pxu
@@ -0,0 +1,184 @@
+# Copyright 2020 Canonical Ltd.
+# All rights reserved.
+#
+# Test plans and (optionally) jobs unique to the limerick device.
+#
+
+id: limerick-classic
+unit: test plan
+_name: Full manual + automated tests for limerick classic image
+_description:
+ Combined manual and automated test plans for limerick classic image.
+include:
+nested_part:
+    limerick-manual-classic
+    limerick-automated-classic
+
+id: limerick-manual-classic
+unit: test plan
+_name: Manual only QA tests for limerick classic image
+_description:
+ QA test plan for the limerick classic image. This test plan contains
+ all of the tests that require manual control of device hardware
+ or some other user input to complete.
+estimated_duration: 3600
+include:
+mandatory_include:
+    com.canonical.certification::connections
+    com.canonical.certification::dmi_attachment
+    com.canonical.certification::dpkg
+    com.canonical.certification::info/buildstamp
+    com.canonical.certification::info/systemd-analyze
+    com.canonical.certification::interface
+    com.canonical.certification::lsb
+    com.canonical.certification::lspci_attachment
+    com.canonical.certification::lsusb_attachment
+    com.canonical.certification::meminfo
+    com.canonical.certification::package
+    com.canonical.certification::parts_meta_info_attachment
+    com.canonical.certification::serial_assertion
+    com.canonical.certification::snap
+    com.canonical.certification::sysfs_attachment
+    com.canonical.certification::system_info_json
+    com.canonical.certification::udev_attachment
+    com.canonical.certification::udev_json
+    com.canonical.certification::uname
+    com.canonical.certification::net_if_management_attachment
+bootstrap_include:
+    com.canonical.certification::cpuinfo
+    com.canonical.certification::device
+    com.canonical.certification::dmi
+    com.canonical.certification::model_assertion
+    com.canonical.certification::net_if_management
+    limerick_sku
+nested_part:
+    dtb-automated
+    com.canonical.certification::ethernet-manual
+    com.canonical.certification::monitor-manual
+    com.canonical.certification::power-manual
+    com.canonical.certification::serial-manual
+    # com.canonical.certification::snappy-snap-manual # covered by automated tests
+    com.canonical.certification::usb-manual
+    com.canonical.certification::usb3-manual
+    com.canonical.certification::watchdog-manual
+    # com.canonical.certification::wireless-manual # covered by automated tests
+    com.canonical.certification::graphics-integrated-gpu-cert-manual
+    com.canonical.certification::camera-cert-manual
+    com.canonical.certification::after-suspend-monitor-manual
+    com.canonical.certification::after-suspend-usb-manual
+    com.canonical.certification::after-suspend-usb3-manual
+    com.canonical.certification::after-suspend-graphics-integrated-gpu-cert-manual
+    com.canonical.certification::after-suspend-camera-cert-full
+exclude:
+    com.canonical.certification::monitor/dvi
+    com.canonical.certification::monitor/dvi-to-vga
+    com.canonical.certification::monitor/hdmi-to-vga
+    com.canonical.certification::after-suspend-monitor/dvi
+    com.canonical.certification::after-suspend-monitor/dvi-to-vga
+    com.canonical.certification::after-suspend-monitor/hdmi-to-vga
+    com.canonical.certification::power-management/lid
+    com.canonical.certification::power-management/lid_close
+    com.canonical.certification::power-management/lid_open
+
+id: limerick-automated-classic
+unit: test plan
+_name: Automated only QA tests for limerick classic image
+_description:
+ QA test plan for the limerick classic image. This test plan contains
+ all of the automated tests used to validate the limerick device.
+include:
+mandatory_include:
+    com.canonical.certification::connections
+    com.canonical.certification::dmi_attachment
+    com.canonical.certification::dpkg
+    com.canonical.certification::info/buildstamp
+    com.canonical.certification::info/systemd-analyze
+    com.canonical.certification::interface
+    com.canonical.certification::lsb
+    com.canonical.certification::lspci_attachment
+    com.canonical.certification::lsusb_attachment
+    com.canonical.certification::meminfo
+    com.canonical.certification::package
+    com.canonical.certification::parts_meta_info_attachment
+    com.canonical.certification::serial_assertion
+    com.canonical.certification::snap
+    com.canonical.certification::sysfs_attachment
+    com.canonical.certification::system_info_json
+    com.canonical.certification::udev_attachment
+    com.canonical.certification::udev_json
+    com.canonical.certification::uname
+    com.canonical.certification::net_if_management_attachment
+bootstrap_include:
+    com.canonical.certification::cpuinfo
+    com.canonical.certification::device
+    com.canonical.certification::dmi
+    com.canonical.certification::model_assertion
+    com.canonical.certification::net_if_management
+    limerick_sku
+nested_part:
+    dtb-automated
+    com.canonical.certification::self-automated
+    com.canonical.certification::graphics-integrated-gpu-cert-automated
+    com.canonical.certification::usb-automated
+    com.canonical.certification::cpu-automated
+    com.canonical.certification::disk-automated
+    com.canonical.certification::ethernet-automated
+    com.canonical.certification::i2c-automated
+    com.canonical.certification::memory-automated
+    com.canonical.certification::networking-automated
+    com.canonical.certification::camera-cert-automated
+    com.canonical.certification::snappy-snap-automated
+    com.canonical.certification::watchdog-automated
+    com.canonical.certification::suspend-tp
+    com.canonical.certification::after-suspend-ethernet-automated
+    com.canonical.certification::after-suspend-networking-automated
+exclude:
+    com.canonical.certification::cpu/maxfreq_test
+    com.canonical.certification::cpu/maxfreq_test-log-attach
+    com.canonical.certification::cpu/offlining_test
+    com.canonical.certification::cpu/topology
+    com.canonical.certification::cpu/cstates
+    com.canonical.certification::cpu/cstates_results.log
+    com.canonical.certification::cpu/scaling_test
+    com.canonical.certification::cpu/scaling_test-log-attach
+    com.canonical.certification::disk/storage_device_mtd.*
+
+id: limerick-stress-classic
+unit: test plan
+_name: Stress only QA tests for limerick classic image
+_description:
+ QA test plan for the limerick classic image. This test plan contains
+ all of the stree tests used to validate the limerick device.
+include:
+mandatory_include:
+    com.canonical.certification::connections
+    com.canonical.certification::dmi_attachment
+    com.canonical.certification::dpkg
+    com.canonical.certification::info/buildstamp
+    com.canonical.certification::info/systemd-analyze
+    com.canonical.certification::interface
+    com.canonical.certification::lsb
+    com.canonical.certification::lspci_attachment
+    com.canonical.certification::lsusb_attachment
+    com.canonical.certification::meminfo
+    com.canonical.certification::package
+    com.canonical.certification::parts_meta_info_attachment
+    com.canonical.certification::serial_assertion
+    com.canonical.certification::snap
+    com.canonical.certification::sysfs_attachment
+    com.canonical.certification::system_info_json
+    com.canonical.certification::udev_attachment
+    com.canonical.certification::udev_json
+    com.canonical.certification::uname
+    com.canonical.certification::net_if_management_attachment
+bootstrap_include:
+    com.canonical.certification::cpuinfo
+    com.canonical.certification::device
+    com.canonical.certification::dmi
+    com.canonical.certification::model_assertion
+    com.canonical.certification::net_if_management
+    limerick_sku
+nested_part:
+    com.canonical.certification::stress-full
+exclude:
+    com.canonical.certification::stress-tests/hibernate.*
diff --git a/units/test-plan.pxu b/units/test-plan.pxu
new file mode 100644
index 0000000..8d2b8ae
--- /dev/null
+++ b/units/test-plan.pxu
@@ -0,0 +1,167 @@
+# Copyright 2020 Canonical Ltd.
+# All rights reserved.
+#
+# Test plans and (optionally) jobs unique to the limerick device.
+#
+
+id: limerick
+unit: test plan
+_name: Full manual + automated tests for limerick
+_description:
+ Combined manual and automated test plans for limerick devices.
+include:
+nested_part:
+    limerick-manual
+    limerick-automated
+
+id: limerick-manual
+unit: test plan
+_name: Manual only QA tests for limerick
+_description:
+ QA test plan for the limerick hardware. This test plan contains
+ all of the tests that require manual control of device hardware
+ or some other user input to complete.
+estimated_duration: 3600
+include:
+mandatory_include:
+    com.canonical.certification::connections
+    com.canonical.certification::dmi_attachment
+    com.canonical.certification::dpkg
+    com.canonical.certification::info/buildstamp
+    com.canonical.certification::info/systemd-analyze
+    com.canonical.certification::interface
+    com.canonical.certification::lsb
+    com.canonical.certification::lspci_attachment
+    com.canonical.certification::lsusb_attachment
+    com.canonical.certification::meminfo
+    com.canonical.certification::package
+    com.canonical.certification::parts_meta_info_attachment
+    com.canonical.certification::serial_assertion
+    com.canonical.certification::snap
+    com.canonical.certification::sysfs_attachment
+    com.canonical.certification::system_info_json
+    com.canonical.certification::udev_attachment
+    com.canonical.certification::udev_json
+    com.canonical.certification::uname
+    com.canonical.certification::net_if_management_attachment
+bootstrap_include:
+    com.canonical.certification::cpuinfo
+    com.canonical.certification::device
+    com.canonical.certification::dmi
+    com.canonical.certification::model_assertion
+    com.canonical.certification::net_if_management
+    limerick_sku
+nested_part:
+    device-connections-tp
+    com.canonical.certification::ethernet-manual
+    com.canonical.certification::monitor-manual
+    com.canonical.certification::power-manual
+    com.canonical.certification::serial-manual
+    # com.canonical.certification::snappy-snap-manual # covered by automated tests
+    com.canonical.certification::usb-manual
+    com.canonical.certification::usb3-manual
+    com.canonical.certification::watchdog-manual
+    # com.canonical.certification::wireless-manual # covered by automated tests
+exclude:
+    com.canonical.certification::monitor/dvi
+    com.canonical.certification::monitor/dvi-to-vga
+
+
+id: limerick-automated
+unit: test plan
+_name: Automated only QA tests for limerick
+_description:
+ QA test plan for the limerick hardware. This test plan contains
+ all of the automated tests used to validate the limerick device.
+include:
+mandatory_include:
+    com.canonical.certification::connections
+    com.canonical.certification::dmi_attachment
+    com.canonical.certification::dpkg
+    com.canonical.certification::info/buildstamp
+    com.canonical.certification::info/systemd-analyze
+    com.canonical.certification::interface
+    com.canonical.certification::lsb
+    com.canonical.certification::lspci_attachment
+    com.canonical.certification::lsusb_attachment
+    com.canonical.certification::meminfo
+    com.canonical.certification::package
+    com.canonical.certification::parts_meta_info_attachment
+    com.canonical.certification::serial_assertion
+    com.canonical.certification::snap
+    com.canonical.certification::sysfs_attachment
+    com.canonical.certification::system_info_json
+    com.canonical.certification::udev_attachment
+    com.canonical.certification::udev_json
+    com.canonical.certification::uname
+    com.canonical.certification::net_if_management_attachment
+bootstrap_include:
+    com.canonical.certification::cpuinfo
+    com.canonical.certification::device
+    com.canonical.certification::dmi
+    com.canonical.certification::model_assertion
+    com.canonical.certification::net_if_management
+    limerick_sku
+nested_part:
+    device-connections-tp
+    com.canonical.certification::self-automated
+    com.canonical.certification::usb-automated
+    com.canonical.certification::cpu-automated
+    com.canonical.certification::disk-automated
+    com.canonical.certification::ethernet-automated
+    com.canonical.certification::i2c-automated
+    com.canonical.certification::kernel-snap-automated
+    com.canonical.certification::memory-automated
+    com.canonical.certification::networking-automated
+    com.canonical.certification::snappy-snap-automated
+    com.canonical.certification::watchdog-automated
+exclude:
+    com.canonical.certification::cpu/maxfreq_test
+    com.canonical.certification::cpu/maxfreq_test-log-attach
+    com.canonical.certification::cpu/offlining_test
+    com.canonical.certification::cpu/topology
+    com.canonical.certification::cpu/cstates
+    com.canonical.certification::cpu/cstates_results.log
+    com.canonical.certification::cpu/scaling_test
+    com.canonical.certification::cpu/scaling_test-log-attach
+    com.canonical.certification::disk/storage_device_mtd.*
+
+id: limerick-stress
+unit: test plan
+_name: Stress only QA tests for limerick
+_description:
+ QA test plan for the limerick hardware. This test plan contains
+ all of the stree tests used to validate the limerick device.
+include:
+mandatory_include:
+    com.canonical.certification::connections
+    com.canonical.certification::dmi_attachment
+    com.canonical.certification::dpkg
+    com.canonical.certification::info/buildstamp
+    com.canonical.certification::info/systemd-analyze
+    com.canonical.certification::interface
+    com.canonical.certification::lsb
+    com.canonical.certification::lspci_attachment
+    com.canonical.certification::lsusb_attachment
+    com.canonical.certification::meminfo
+    com.canonical.certification::package
+    com.canonical.certification::parts_meta_info_attachment
+    com.canonical.certification::serial_assertion
+    com.canonical.certification::snap
+    com.canonical.certification::sysfs_attachment
+    com.canonical.certification::system_info_json
+    com.canonical.certification::udev_attachment
+    com.canonical.certification::udev_json
+    com.canonical.certification::uname
+    com.canonical.certification::net_if_management_attachment
+bootstrap_include:
+    com.canonical.certification::cpuinfo
+    com.canonical.certification::device
+    com.canonical.certification::dmi
+    com.canonical.certification::model_assertion
+    com.canonical.certification::net_if_management
+    limerick_sku
+nested_part:
+    com.canonical.certification::stress-full
+exclude:
+    com.canonical.certification::stress-tests/hibernate.*