Merge ~kristinchuang/plainbox-provider-checkbox:model-grade-check-revamp into plainbox-provider-checkbox:master

[Kristin Chuang]

Modify the existing Ubuntu Core model grade check from simply 'checking if the grade != dangerous' into 'checking if the actual grade set is as expected':

===== Changes =====
* Rename the existing `image/model-grade-not-dangerous` job into `image/model-grade`

* Updated the description of `image/model-grade` job

* Modified the logic of model checking function in `ubuntucore_image_checks.py`:
- always fail if model grade is missing
- always fail if model grade is 'dangerous'
- if a MODEL_GRADE exists in configuration variables, make sure the model grade matches MODEL_GRADE
- if no MODEL_GRADE was given, make sure the model grade is 'secured'

===== Trial run with the change =====
[OK] model grade missing - case expected to fail
[OK] image is classic - case expected to be skipped
-
[OK] model grade is dangerous, MODEL_GRADE set - case expected to fail
[OK] model grade is dangerous, MODEL_GRADE not set - case expected to fail
-
[OK] MODEL_GRADE set, matches actual model grade - case expected to pass
[OK] MODEL_GRADE set, does not match actual model grade - case expected to fail
-
[OK] MODEL_GRADE not set, matches actual model grade - case expected to pass
[OK] MODEL_GRADE not set, does not match actual model grade - case expected to fail

Model assertion grade documentation:
https://ubuntu.com/core/docs/reference/assertions/model

[Jonathan Cave (jocave)]

Could you just modify the description to use the accurate grade "secured" rather than "secure"? Otherwise looks good, thanks

review: Approve

[Kristin Chuang (kristinchuang)]

Good catch, thanks Jonathan!
Re-pushed with typo corrected.

[Jonathan Cave (jocave)]

Thanks, still approved :)

review: Approve

[Devices Certification Bot (ce-certification-qa)]

The merge was fine but running tests failed.

[bionic] [09:06:22] starting container
[focal] [09:06:22] starting container
[xenial] [09:06:23] starting container
Device project added to bionic-testing
Device project added to focal-testing
Device project added to xenial-testing
[xenial] [09:06:35] provisioning container
[bionic] [09:06:44] provisioning container
[focal] [09:06:44] provisioning container
[xenial] [09:06:54] Starting tests...
[xenial] Found a test script: ./requirements/container-tests-provider-checkbox
[bionic] [09:07:04] Starting tests...
[bionic] Found a test script: ./requirements/container-tests-provider-checkbox
[focal] [09:07:18] Starting tests...
[focal] Found a test script: ./requirements/container-tests-provider-checkbox
[xenial] [09:08:47] container-tests-provider-checkbox: FAIL
[xenial] output: https://paste.ubuntu.com/p/qpRgXnxCBR/
[xenial] [09:08:50] Fixing file permissions in source directory
[xenial] [09:08:50] Destroying container
[bionic] [09:09:07] container-tests-provider-checkbox: FAIL
[bionic] output: https://paste.ubuntu.com/p/W5RkWKQNFh/
[bionic] [09:09:09] Fixing file permissions in source directory
[bionic] [09:09:09] Destroying container
[focal] [09:09:21] container-tests-provider-checkbox: FAIL
[focal] output: https://paste.ubuntu.com/p/XdQHHvd4jr/
[focal] [09:09:23] Fixing file permissions in source directory
[focal] [09:09:24] Destroying container

review: Needs Fixing

[Jonathan Cave (jocave)]

test_flake8_/root/project/bin/ubuntucore_image_checks.py (plainbox.provider_manager.Flake8Tests) ... /root/project/bin/ubuntucore_image_checks.py:101:80: E501 line too long (80 > 79 characters)

[Kristin Chuang (kristinchuang)]

re-pushed with flake8 error fixed

diff --git a/bin/ubuntucore_image_checks.py b/bin/ubuntucore_image_checks.py
index e381e61..ff14784 100755
--- a/bin/ubuntucore_image_checks.py
+++ b/bin/ubuntucore_image_checks.py
@@ -6,6 +6,7 @@
 #   Jonathan Cave <[e-mail]>
 
 import io
+import os
 import sys
 
 from checkbox_support.snap_utils.snapd import Snapd
@@ -91,10 +92,14 @@ class ModelInfo():
         print('PASS')
 
     def test_model_grade(self):
+        MODEL_GRADE = os.environ.get('MODEL_GRADE', 'secured')
         if not self.grade:
             raise SystemExit('ERROR: failed to get model grade info')
         if self.grade == 'dangerous':
             raise SystemExit('ERROR: model grade must not be dangerous')
+        if self.grade != MODEL_GRADE:
+            raise SystemExit('ERROR: model grade is "{}",'.format(self.grade) +
+                             ' but "{}" is expected'.format(MODEL_GRADE))
         print('PASS')
 
 
diff --git a/units/image/jobs.pxu b/units/image/jobs.pxu
index d185550..d5aa7d4 100644
--- a/units/image/jobs.pxu
+++ b/units/image/jobs.pxu
@@ -71,16 +71,18 @@ command:
 estimated_duration: 2.0
 flags: preserve-locale
 
-id: image/model-grade-not-dangerous
+id: image/model-grade
 category_id: image
-_summary: Check that the model grade is not missing or set to dangerous
+_summary: Check that the model grade is correctly set
 requires:
  lsb.distributor_id == "Ubuntu Core" and int(lsb.release) >= 20
 _description:
   Images with the 'dangerous' grade (the lowest of all available grades)
   results in certain security measures to be relaxed.
   Images that require strict security related implementations must
-  have the model grade set to a grade higher than 'dangerous'.
+  have the model grade set to non-dangerous grades - either the highest
+  grade of 'secured', or a grade passed to checkbox for checking via the
+  MODEL_GRADE configuration variable.
 plugin: shell
 command:
   ubuntucore_image_checks.py model-grade