Merge ~chad.smith/cloud-init:tests/cii-summary-fallback-to-traceback into cloud-init:master

[Chad Smith]

tests: integration test failure summary to use traceback if empty error

When integration tests verification fails, the object returned
contains has 'error' and 'traceback' keys. Each key can contain empty
strings. If the simplified 'error' message is empty, fallback and use
the more verbose full 'traceback' text in the failure summary.

[Chad Smith (chad.smith)]

Will hold on committing this until cosmic freeze is over.

Basically it changes the following output

    test failures:
    * tests.cloud_tests.testcases.modules.apt_configure_sources_key.TestAptconfigureSourcesKey.test_instance_data_json_kvm

----- CHANGED TO-----

    test failures:
    * tests.cloud_tests.testcases.modules.apt_configure_sources_key.TestAptconfigureSourcesKey.test_instance_data_json_kvm
Traceback (most recent call last):
  File "/var/lib/jenkins/slaves/torkoal/workspace/cloud-init-integration-nocloud-kvm-c/cloud-init/tests/cloud_tests/testcases/base.py", line 265, in test_instance_data_json_kvm
    self.assertEqual('config-disk (/dev/vda)', v1_data['subplatform'])
AssertionError: 'config-disk (/dev/vda)' != 'config-disk (/dev/vdb)'
- config-disk (/dev/vda)
? ^
+ config-disk (/dev/vdb)
?

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:f9d86009ec65a42cb8c61aae02f79f3cd8b5363a
https://jenkins.ubuntu.com/server/job/cloud-init-ci/391/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Unit & Style Tests
    SUCCESS: Ubuntu LTS: Build
    SUCCESS: Ubuntu LTS: Integration
    IN_PROGRESS: Declarative: Post Actions

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/391/rebuild

review: Approve (continuous-integration)

[Scott Moser (smoser)]

@Chad. the first commit is in.
So rebase on mater, and push for c-i.

can you give an example of the end result of this chagne?

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:f14a6d926eb8fc5ee39831ebc63cf034dc723ea0
https://jenkins.ubuntu.com/server/job/cloud-init-ci/507/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Unit & Style Tests
    SUCCESS: Ubuntu LTS: Build
    SUCCESS: Ubuntu LTS: Integration
    IN_PROGRESS: Declarative: Post Actions

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/507/rebuild

review: Approve (continuous-integration)

[Dan Watkins (oddbloke)]

review: Approve

diff --git a/tests/cloud_tests/verify.py b/tests/cloud_tests/verify.py
index 9911ecf..7018f4d 100644
--- a/tests/cloud_tests/verify.py
+++ b/tests/cloud_tests/verify.py
@@ -61,12 +61,17 @@ def format_test_failures(test_result):
     if not test_result['failures']:
         return ''
     failure_hdr = '    test failures:'
-    failure_fmt = '    * {module}.{class}.{function}\n          {error}'
+    failure_fmt = '    * {module}.{class}.{function}\n          '
     output = []
     for failure in test_result['failures']:
         if not output:
             output = [failure_hdr]
-        output.append(failure_fmt.format(**failure))
+        msg = failure_fmt.format(**failure)
+        if failure.get('error'):
+            msg += failure['error']
+        else:
+            msg += failure.get('traceback', '')
+        output.append(msg)
     return '\n'.join(output)