Merge ~jocave/plainbox-provider-checkbox:dtb-mismatch-test into plainbox-provider-checkbox:master

[Jonathan Cave]

Create a new test to run on UC18 that checks that the DTB files supplied in the kernel snap are correctly extracted to the boot directory and everything is in sync.

I've tested that it correctly fails on current RPi devices running UC18, but I'm unsure whether there are other devices that use u-boot where this might fail.

[Maciej Kisielewski (kissiel)]

Looks good modulo some nitpicking below.
One cool thing that could be done in this script is to introduce a partially filled print that always does file=sys.stderr, flush=True. (functools.partial goodies)

review: Approve

[Jonathan Cave (jocave)]

Tidied up as per comments, squashed, and re-pushed

[Maciej Kisielewski (kissiel)]

+1

review: Approve

diff --git a/bin/booted_dtb_tests.py b/bin/booted_dtb_tests.py
new file mode 100755
index 0000000..9ebe2e4
--- /dev/null
+++ b/bin/booted_dtb_tests.py
@@ -0,0 +1,72 @@
+#!/usr/bin/env python3
+#
+# This file is part of Checkbox.
+#
+# Copyright 2021 Canonical Ltd.
+# Written by:
+#   Jonathan Cave <[e-mail]>
+#
+# Checkbox is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 3,
+# as published by the Free Software Foundation.
+#
+# Checkbox is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with Checkbox.  If not, see <http://www.gnu.org/licenses/>.
+
+import filecmp
+import sys
+from os import walk
+from os.path import join, relpath
+
+from checkbox_support.snap_utils.system import get_kernel_snap
+
+
+if len(sys.argv) != 2:
+    raise SystemExit('ERROR: please specify the bootloader')
+
+dtb_dir = '/var/lib/snapd/hostfs/boot/{}'.format(sys.argv[1])
+print('Bootloader DTB location: {}'.format(dtb_dir))
+
+kernel = get_kernel_snap()
+if kernel is None:
+    raise SystemExit('ERROR: failed to get kernel snap')
+snap_dtbs = '/snap/{}/current/dtbs'.format(kernel)
+print('Kernel snap DTB location: {}'.format(
+    snap_dtbs), end='\n\n', flush=True)
+
+snap_files = []
+for (dirpath, dirs, files) in walk(snap_dtbs):
+    if dirpath == snap_dtbs:
+        snap_files.extend(files)
+    else:
+        snap_files.extend([join(relpath(dirpath, snap_dtbs), f)
+                           for f in files])
+
+match, mismatch, errors = filecmp.cmpfiles(
+    snap_dtbs, dtb_dir, snap_files, shallow=True)
+
+if match:
+    print('{} matching DTB files found'.format(
+        len(match)), end='\n\n', flush=True)
+
+if mismatch:
+    print('FAIL: DTB files shipped in kernel snap with mismatched versions'
+          ' in bootloader dir:', file=sys.stderr)
+    for f in mismatch:
+        print(f, file=sys.stderr)
+    print('', file=sys.stderr, flush=True)
+
+if errors:
+    print('FAIL: DTB files shipped in kernel snap which could not be'
+          ' found in bootloader dir:', file=sys.stderr)
+    for f in errors:
+        print(f, file=sys.stderr)
+    print('', file=sys.stderr, flush=True)
+
+if mismatch or errors:
+    raise SystemExit('Comparison failed')
diff --git a/units/kernel-snap/jobs.pxu b/units/kernel-snap/jobs.pxu
index 950d1a6..b03d6a8 100644
--- a/units/kernel-snap/jobs.pxu
+++ b/units/kernel-snap/jobs.pxu
@@ -33,3 +33,22 @@ imports:
   from com.canonical.certification import ubuntu_core_features
 requires:
   ubuntu_core_features.force_kernel_extraction == 'True'
+
+unit: template
+template-resource: bootloader
+template-filter: bootloader.name == 'uboot'
+id: kernel-snap/booted-dtbs-match-current-{name}
+requires:
+  lsb.description == 'Ubuntu Core 18'
+  snap.name == 'snapd' and int(snap.revision) >= 11841
+category_id: kernel-snap
+_summary: The dtbs in the bootloader match those in current kernel snap
+_description:
+  Since snapd version 2.50 DTB files can be distributed in the kernel snap and
+  the extracted in to the boot partition. This checks that if the DTBs are in
+  sync.
+plugin: shell
+user: root
+estimated_duration: 2.0
+command:
+  booted_dtb_tests.py {name}
diff --git a/units/kernel-snap/test-plan.pxu b/units/kernel-snap/test-plan.pxu
index ffaa5d1..585bd01 100644
--- a/units/kernel-snap/test-plan.pxu
+++ b/units/kernel-snap/test-plan.pxu
@@ -14,6 +14,7 @@ _name: Automated Kernel Snap tests
 _description: Automated Kernel Snap tests for Ubuntu Core devices
 include:
     kernel-snap/booted-kernel-matches-current-.*
+    kernel-snap/booted-dtbs-match-current-.*
 bootstrap_include:
     bootloader