Merge ~jocave/plainbox-provider-checkbox:iot-image-checks into plainbox-provider-checkbox:master

[Jonathan Cave]

I wrote these tests to go in to the test plan I had been working on whose goal was to certify Ubuntu Core IoT images. I'm not sure how relevant these are going to be now, but since I put the time in I may as well propose them.

I test them on an amd64 machine running generic UC18.

[Sheila Miguez (codersquid)]

I have a very minor comment about a summary. Otherwise things seem ok.

[Jonathan Cave (jocave)]

Good spot, thanks. Fixed and squashed

[Sheila Miguez (codersquid)]

lgtm

review: Approve

[Chris Wayne (cwayne)]

Obviously this is still relevant for the core images, so very worth writing, thanks. Given the ODM program pricing structure has core as 50% of the cost of classic, I think we can still assume the majority will be core.

diff --git a/bin/ubuntucore_image_checks.py b/bin/ubuntucore_image_checks.py
new file mode 100755
index 0000000..214fc42
--- /dev/null
+++ b/bin/ubuntucore_image_checks.py
@@ -0,0 +1,116 @@
+#!/usr/bin/env python3
+# Copyright 2020 Canonical Ltd.
+# All rights reserved.
+#
+# Written by:
+#   Jonathan Cave <[e-mail]>
+
+import io
+import sys
+
+from checkbox_support.snap_utils.snapd import Snapd
+
+
+class SnapInfo():
+
+    def __init__(self):
+        self.kernel = None
+        self.gadget = None
+        data = Snapd().list()
+        for snap in data:
+            if snap['type'] == 'kernel':
+                self.kernel = snap
+            if snap['type'] == 'gadget':
+                self.gadget = snap
+
+    def test_kernel_publisher(self):
+        if not self.kernel:
+            raise SystemExit('ERROR: failed to get kernel snap info')
+        if self.kernel['publisher']['id'] != 'canonical':
+            raise SystemExit('ERROR: kernel snap publisher must be canonical')
+        if self.kernel['publisher']['validation'] != 'verified':
+            raise SystemExit('ERROR: kernel snap publisher must be verified')
+        print('PASS')
+
+    def test_kernel_tracking(self):
+        if not self.kernel:
+            raise SystemExit('ERROR: failed to get kernel snap info')
+        if not self.kernel.get('tracking-channel'):
+            raise SystemExit('ERROR: kernel must be tracking a store channel')
+        if 'stable' not in self.kernel['tracking-channel']:
+            raise SystemExit('ERROR: kernel must be on a stable channel')
+        print('PASS')
+
+    def test_gadget_publisher(self):
+        if not self.gadget:
+            raise SystemExit('ERROR: failed to get gadget snap info')
+        if self.gadget['publisher']['id'] != 'canonical':
+            raise SystemExit('ERROR: gadget snap publisher must be canonical')
+        if self.gadget['publisher']['validation'] != 'verified':
+            raise SystemExit('ERROR: gadget snap publisher must be verified')
+        print('PASS')
+
+    def test_gadget_tracking(self):
+        if not self.gadget:
+            raise SystemExit('ERROR: failed to get gadget snap info')
+        if not self.gadget.get('tracking-channel'):
+            raise SystemExit('ERROR: gadget must be tracking a store channel')
+        if 'stable' not in self.gadget['tracking-channel']:
+            raise SystemExit('ERROR: gadget must be on a stable channel')
+        print('PASS')
+
+
+class ModelInfo():
+
+    def __init__(self):
+        self.authority = None
+        self.brand = None
+        for line in io.StringIO(Snapd().get_assertions('model').text):
+            if ':' in line:
+                entry = line.split(':', maxsplit=1)
+                if entry[0] == 'authority-id':
+                    self.authority = entry[1].strip()
+                if entry[0] == 'brand-id':
+                    self.brand = entry[1].strip()
+
+    def test_model_authority(self):
+        if not self.authority:
+            raise SystemExit('ERROR: failed to get model authority info')
+        if self.authority != 'canonical':
+            raise SystemExit('ERROR: model authority must be canonical')
+        print('PASS')
+
+    def test_model_brand(self):
+        if not self.brand:
+            raise SystemExit('ERROR: failed to get model brand info')
+        if self.brand != 'canonical':
+            raise SystemExit('ERROR: model brand must be canonical')
+        print('PASS')
+
+
+def main():
+    if len(sys.argv) != 2:
+        raise SystemExit('USAGE: ubuntucore_image_checks.py [action]')
+    action = sys.argv[1]
+
+    snapinfo = SnapInfo()
+    modelinfo = ModelInfo()
+
+    if action == 'kernel-publisher':
+        snapinfo.test_kernel_publisher()
+    elif action == 'kernel-tracking':
+        snapinfo.test_kernel_tracking()
+    elif action == 'gadget-publisher':
+        snapinfo.test_gadget_publisher()
+    elif action == 'gadget-tracking':
+        snapinfo.test_gadget_tracking()
+    elif action == 'model-authority':
+        modelinfo.test_model_authority()
+    elif action == 'model-brand':
+        modelinfo.test_model_authority()
+    else:
+        raise SystemExit('ERROR: unrecognised action')
+
+
+if __name__ == '__main__':
+    main()
diff --git a/units/image/category.pxu b/units/image/category.pxu
new file mode 100644
index 0000000..919d6cf
--- /dev/null
+++ b/units/image/category.pxu
@@ -0,0 +1,3 @@
+unit: category
+id: image
+_name: Image verification tests
\ No newline at end of file
diff --git a/units/image/jobs.pxu b/units/image/jobs.pxu
new file mode 100644
index 0000000..efe8d4e
--- /dev/null
+++ b/units/image/jobs.pxu
@@ -0,0 +1,72 @@
+
+id: image/kernel-publisher-canonical
+category_id: image
+summary: Check that the kernel snap publisher is Canonical
+description:
+  During certification testing IoT devices must be running a kernel supplied
+  by Canonical
+plugin: shell
+command:
+  ubuntucore_image_checks.py kernel-publisher
+estimated_duration: 2.0
+flags: preserve-locale
+
+id: image/kernel-tracking-stable
+category_id: image
+summary: Check that the kernel snap is tracking stable channel
+description:
+  During certification testing IoT devices must be running a kernel that is on
+  a stable channel
+plugin: shell
+command:
+  ubuntucore_image_checks.py kernel-tracking
+estimated_duration: 2.0
+flags: preserve-locale
+
+id: image/gadget-publisher-canonical
+category_id: image
+summary: Check that the gadget snap publisher is Canonical
+description:
+  During certification testing IoT devices must be running a kernel supplied
+  by Canonical
+plugin: shell
+command:
+  ubuntucore_image_checks.py gadget-publisher
+estimated_duration: 2.0
+flags: preserve-locale
+
+id: image/gadget-tracking-stable
+category_id: image
+summary: Check that the gadget snap is tracking stable channel
+description:
+  During certification testing IoT devices must be running a gadget that is on
+  a stable channel
+plugin: shell
+command:
+  ubuntucore_image_checks.py gadget-tracking
+estimated_duration: 2.0
+flags: preserve-locale
+
+id: image/model-authority-canonical
+category_id: image
+summary: Check that model authority-id is canonical
+description:
+  The authority-id declares on whose authority this assertion is made. This
+  must be Canonical for the certification of IoT devices.
+plugin: shell
+command:
+  ubuntucore_image_checks.py model-authority
+estimated_duration: 2.0
+flags: preserve-locale
+
+id: image/model-brand-canonical
+category_id: image
+summary: Check the model brand-id is canoncial
+description:
+  For the certification of IoT devices canonical provided generic images must
+  be used. Hence, the brand-id must be canonical.
+plugin: shell
+command:
+  ubuntucore_image_checks.py model-brand
+estimated_duration: 2.0
+flags: preserve-locale
\ No newline at end of file
diff --git a/units/image/test-plan.pxu b/units/image/test-plan.pxu
new file mode 100644
index 0000000..48af204
--- /dev/null
+++ b/units/image/test-plan.pxu
@@ -0,0 +1,20 @@
+
+id: iot-cert-image-full
+unit: test plan
+name: Sanity checks for images being certified on IoT devices
+include:
+nested_part:
+    iot-cert-image-manual
+    iot-cert-image-automated
+
+id: iot-cert-image-automated
+unit: test plan
+name: Automated Sanity checks for images being certified on IoT devices
+include:
+  image/.*
+
+id: iot-cert-image-manual
+unit: test plan
+name: Manual Sanity checks for images being certified on IoT devices
+include:
+