Merge lp:~canonical-platform-qa/ubuntu-system-tests/add-adt-run-debugging into lp:ubuntu-system-tests

[Allan LeSage]

Add -ddd debugging to adt-run calls.

This doesn't change the behavior of the tests, only adds a lot of debugging output. If we insist it can be made into a command-line option but we'll have to choose a letter other than 'd' as rhuddie is squatting that one ;) .

[platform-qa-bot (platform-qa-bot)]

FAILED: Continuous integration, rev:369
No commit message was specified in the merge proposal. Click on the following link and set the commit message (if you want a jenkins rebuild you need to trigger it yourself):
https://code.launchpad.net/~canonical-platform-qa/ubuntu-system-tests/add-adt-run-debugging/+merge/291956/+edit-commit-message

https://platform-qa-jenkins.ubuntu.com/job/ubuntu-system-tests-ci/65/
Executed test runs:
    SUCCESS: https://platform-qa-jenkins.ubuntu.com/job/build-wily-amd64-package/101
        deb: https://platform-qa-jenkins.ubuntu.com/job/build-wily-amd64-package/101/artifact/work/output/*zip*/output.zip
    SUCCESS: https://platform-qa-jenkins.ubuntu.com/job/build-wily-i386-package/101
        deb: https://platform-qa-jenkins.ubuntu.com/job/build-wily-i386-package/101/artifact/work/output/*zip*/output.zip
    None: https://platform-qa-jenkins.ubuntu.com/job/generic-update-mp/95/console

Click here to trigger a rebuild:
https://platform-qa-jenkins.ubuntu.com/job/ubuntu-system-tests-ci/65/rebuild

review: Needs Fixing (continuous-integration)

[platform-qa-bot (platform-qa-bot)]

PASSED: Continuous integration, rev:369
https://platform-qa-jenkins.ubuntu.com/job/ubuntu-system-tests-ci/66/
Executed test runs:
    SUCCESS: https://platform-qa-jenkins.ubuntu.com/job/build-wily-amd64-package/102
        deb: https://platform-qa-jenkins.ubuntu.com/job/build-wily-amd64-package/102/artifact/work/output/*zip*/output.zip
    SUCCESS: https://platform-qa-jenkins.ubuntu.com/job/build-wily-i386-package/102
        deb: https://platform-qa-jenkins.ubuntu.com/job/build-wily-i386-package/102/artifact/work/output/*zip*/output.zip
    None: https://platform-qa-jenkins.ubuntu.com/job/generic-update-mp/96/console

Click here to trigger a rebuild:
https://platform-qa-jenkins.ubuntu.com/job/ubuntu-system-tests-ci/66/rebuild

review: Approve (continuous-integration)

[PS Jenkins bot (ps-jenkins)]

FAILED: Continuous integration, rev:369
http://s-jenkins.ubuntu-ci:8080/job/ubuntu-sanity-tests-ci/1499/
Executed test runs:
    FAILURE: http://s-jenkins.ubuntu-ci:8080/job/ubuntu-sanity-tests-wily-amd64-ci/1226/console
    SUCCESS: http://s-jenkins.ubuntu-ci:8080/job/ubuntu-sanity-tests-wily-armhf-ci/1228
    FAILURE: http://s-jenkins.ubuntu-ci:8080/job/ubuntu-sanity-tests-wily-i386-ci/1225/console

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/ubuntu-sanity-tests-ci/1499/rebuild

review: Needs Fixing (continuous-integration)

[Richard Huddie (rhuddie)]

Agree that a command line option would be best to enable this. I'd suggest using -v or --verbose for that. I think -v is free :)

review: Needs Fixing

[lp:~canonical-platform-qa/ubuntu-system-tests/add-adt-run-debugging]

370. By Allan LeSage on 2016-04-15

Add -v for verbose logging.

[platform-qa-bot (platform-qa-bot)]

FAILED: Continuous integration, rev:370
https://platform-qa-jenkins.ubuntu.com/job/ubuntu-system-tests-ci/69/
Executed test runs:
    FAILURE: https://platform-qa-jenkins.ubuntu.com/job/build-wily-amd64-package/107/console
    FAILURE: https://platform-qa-jenkins.ubuntu.com/job/build-wily-i386-package/107/console
    None: https://platform-qa-jenkins.ubuntu.com/job/generic-update-mp/99/console

Click here to trigger a rebuild:
https://platform-qa-jenkins.ubuntu.com/job/ubuntu-system-tests-ci/69/rebuild

review: Needs Fixing (continuous-integration)

[lp:~canonical-platform-qa/ubuntu-system-tests/add-adt-run-debugging]

371. By Allan LeSage on 2016-04-16

Fix tests.

[platform-qa-bot (platform-qa-bot)]

PASSED: Continuous integration, rev:371
https://platform-qa-jenkins.ubuntu.com/job/ubuntu-system-tests-ci/70/
Executed test runs:
    SUCCESS: https://platform-qa-jenkins.ubuntu.com/job/build-wily-amd64-package/108
        deb: https://platform-qa-jenkins.ubuntu.com/job/build-wily-amd64-package/108/artifact/work/output/*zip*/output.zip
    SUCCESS: https://platform-qa-jenkins.ubuntu.com/job/build-wily-i386-package/108
        deb: https://platform-qa-jenkins.ubuntu.com/job/build-wily-i386-package/108/artifact/work/output/*zip*/output.zip
    None: https://platform-qa-jenkins.ubuntu.com/job/generic-update-mp/100/console

Click here to trigger a rebuild:
https://platform-qa-jenkins.ubuntu.com/job/ubuntu-system-tests-ci/70/rebuild

review: Approve (continuous-integration)

[PS Jenkins bot (ps-jenkins)]

FAILED: Continuous integration, rev:370
http://s-jenkins.ubuntu-ci:8080/job/ubuntu-sanity-tests-ci/1502/
Executed test runs:
    FAILURE: http://s-jenkins.ubuntu-ci:8080/job/ubuntu-sanity-tests-wily-amd64-ci/1229/console
    FAILURE: http://s-jenkins.ubuntu-ci:8080/job/ubuntu-sanity-tests-wily-armhf-ci/1231/console
    FAILURE: http://s-jenkins.ubuntu-ci:8080/job/ubuntu-sanity-tests-wily-i386-ci/1228/console

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/ubuntu-sanity-tests-ci/1502/rebuild

review: Needs Fixing (continuous-integration)

[PS Jenkins bot (ps-jenkins)]

PASSED: Continuous integration, rev:371
http://s-jenkins.ubuntu-ci:8080/job/ubuntu-sanity-tests-ci/1503/
Executed test runs:
    SUCCESS: http://s-jenkins.ubuntu-ci:8080/job/ubuntu-sanity-tests-wily-amd64-ci/1230
    SUCCESS: http://s-jenkins.ubuntu-ci:8080/job/ubuntu-sanity-tests-wily-armhf-ci/1232
    SUCCESS: http://s-jenkins.ubuntu-ci:8080/job/ubuntu-sanity-tests-wily-i386-ci/1229

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/ubuntu-sanity-tests-ci/1503/rebuild

review: Approve (continuous-integration)

[lp:~canonical-platform-qa/ubuntu-system-tests/add-adt-run-debugging]

372. By Allan LeSage on 2016-04-17

Merge trunk.

[platform-qa-bot (platform-qa-bot)]

PASSED: Continuous integration, rev:372
https://platform-qa-jenkins.ubuntu.com/job/ubuntu-system-tests-ci/71/
Executed test runs:
    SUCCESS: https://platform-qa-jenkins.ubuntu.com/job/build-wily-amd64-package/109
        deb: https://platform-qa-jenkins.ubuntu.com/job/build-wily-amd64-package/109/artifact/work/output/*zip*/output.zip
    SUCCESS: https://platform-qa-jenkins.ubuntu.com/job/build-wily-i386-package/109
        deb: https://platform-qa-jenkins.ubuntu.com/job/build-wily-i386-package/109/artifact/work/output/*zip*/output.zip
    None: https://platform-qa-jenkins.ubuntu.com/job/generic-update-mp/101/console

Click here to trigger a rebuild:
https://platform-qa-jenkins.ubuntu.com/job/ubuntu-system-tests-ci/71/rebuild

review: Approve (continuous-integration)

[PS Jenkins bot (ps-jenkins)]

PASSED: Continuous integration, rev:372
http://s-jenkins.ubuntu-ci:8080/job/ubuntu-sanity-tests-ci/1504/
Executed test runs:
    SUCCESS: http://s-jenkins.ubuntu-ci:8080/job/ubuntu-sanity-tests-wily-amd64-ci/1231
    SUCCESS: http://s-jenkins.ubuntu-ci:8080/job/ubuntu-sanity-tests-wily-armhf-ci/1233
    SUCCESS: http://s-jenkins.ubuntu-ci:8080/job/ubuntu-sanity-tests-wily-i386-ci/1230

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/ubuntu-sanity-tests-ci/1504/rebuild

review: Approve (continuous-integration)

[Allan LeSage (allanlesage)]

Review me please!

[Richard Huddie (rhuddie)]

Looks good and works fine. Just a couple of minor points below, thanks.

review: Needs Fixing

[lp:~canonical-platform-qa/ubuntu-system-tests/add-adt-run-debugging]

373. By Allan LeSage on 2016-04-18

Add a verbosity command-line test, conform to no-defaulting param style.

[platform-qa-bot (platform-qa-bot)]

PASSED: Continuous integration, rev:373
https://platform-qa-jenkins.ubuntu.com/job/ubuntu-system-tests-ci/75/
Executed test runs:
    SUCCESS: https://platform-qa-jenkins.ubuntu.com/job/build-wily-amd64-package/114
        deb: https://platform-qa-jenkins.ubuntu.com/job/build-wily-amd64-package/114/artifact/work/output/*zip*/output.zip
    SUCCESS: https://platform-qa-jenkins.ubuntu.com/job/build-wily-i386-package/114
        deb: https://platform-qa-jenkins.ubuntu.com/job/build-wily-i386-package/114/artifact/work/output/*zip*/output.zip
    None: https://platform-qa-jenkins.ubuntu.com/job/generic-update-mp/105/console

Click here to trigger a rebuild:
https://platform-qa-jenkins.ubuntu.com/job/ubuntu-system-tests-ci/75/rebuild

review: Approve (continuous-integration)

[PS Jenkins bot (ps-jenkins)]

PASSED: Continuous integration, rev:373
http://s-jenkins.ubuntu-ci:8080/job/ubuntu-sanity-tests-ci/1507/
Executed test runs:
    SUCCESS: http://s-jenkins.ubuntu-ci:8080/job/ubuntu-sanity-tests-wily-amd64-ci/1234
    SUCCESS: http://s-jenkins.ubuntu-ci:8080/job/ubuntu-sanity-tests-wily-armhf-ci/1236
    SUCCESS: http://s-jenkins.ubuntu-ci:8080/job/ubuntu-sanity-tests-wily-i386-ci/1233

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/ubuntu-sanity-tests-ci/1507/rebuild

review: Approve (continuous-integration)

[Richard Huddie (rhuddie)]

Thanks for update, all looks good so approving.

review: Approve

=== modified file 'ubuntu_system_tests/command_line.py'
--- ubuntu_system_tests/command_line.py	2016-04-14 09:48:49 +0000
+++ ubuntu_system_tests/command_line.py	2016-04-18 15:40:01 +0000
@@ -61,7 +61,7 @@
     return run_system_tests(args.config, args.suites,
                             args.update_apt, args.network_ready,
                             args.touch_visualization, args.install_deps,
-                            args.silent, args.cache_deps)
+                            args.silent, args.cache_deps, args.verbose)
 
 
 def _parse_arguments(argv=None):
@@ -94,12 +94,16 @@
              'downloaded for every test run.')
     parser.add_argument(
         'suites', nargs='*', help='The list of test suites to run.')
+    parser.add_argument(
+        '--verbose', '-v', action='store_true',
+        help='Provide verbose logging.', default=False)
     return parser.parse_args(args=argv)
 
 
 def run_system_tests(conf_name, tests_to_run, update_apt=False,
                      network_ready=False, touch_visualization=False,
-                     install_deps=False, silent=False, cache_deps=False):
+                     install_deps=False, silent=False, cache_deps=False,
+                     verbose=False):
     """Run system tests in a device connected via USB and report the results.
     :param conf_name: The name of the configuration file that will be used
         for the system tests.
@@ -125,7 +129,7 @@
 
     enable_autopilot_introspection(config_stack)
     return_code = run_autopkg_test(config_stack, update_apt, network_ready,
-                                   install_deps, cache_deps)
+                                   install_deps, cache_deps, verbose)
 
     if return_code <= 6:
         # Return codes higher than 6 mean that there was a problem with adt or
@@ -222,7 +226,7 @@
 
 
 def run_autopkg_test(config_stack, update_apt, network_ready, install_deps,
-                     cache_deps):
+                     cache_deps, verbose):
     """ Run the autopkgtest in the revice for the tests selected
     :param config_stack: The config objects
     :param update_apt: True if the packages hasve to be updated
@@ -232,6 +236,7 @@
     permanently in the device instead of install those in the tmp dir
     :param cache_deps: If True then cache the downloaded dependencies so that
                        they don't have to be downloaded for every test run.
+    :param verbose: True to provide verbose logging.
     :return: the autopkgtest execution exit code
     """
     extra_depends = _get_extra_dependencies(
@@ -245,7 +250,8 @@
             network_ready=network_ready,
             add_ppa=False,
             install_deps=False,
-            cache_deps=True)
+            cache_deps=True,
+            verbose=verbose)
         subprocess.call(adt_run_cmd)
         clean_output_directory(config_stack)
         logger.info('apt-cacher-ng installed on device')
@@ -257,7 +263,8 @@
             update_apt=update_apt,
             network_ready=network_ready,
             add_ppa=add_ppa,
-            install_deps=install_deps)
+            install_deps=install_deps,
+            verbose=verbose)
         subprocess.call(adt_run_cmd)
 
         clean_output_directory(config_stack)
@@ -273,7 +280,8 @@
         update_apt=update_apt,
         network_ready=network_ready,
         add_ppa=add_ppa,
-        install_deps=install_deps)
+        install_deps=install_deps,
+        verbose=verbose)
     return subprocess.call(adt_run_cmd)
 
 
@@ -370,11 +378,13 @@
             src=keyfile, dest=ssh.get_device_keyfile_path(keyfile))])
 
 
-def _build_basic_adt_run_test_command(config_stack, extra_depends):
+def _build_basic_adt_run_test_command(config_stack, extra_depends, verbose):
     """ Build and return the first part of the adt-run """
     output_dir = config_stack.get('output_dir')
 
     run_cmd = ['adt-run']
+    if verbose:
+        run_cmd.append('-ddd')
     if extra_depends:
         run_cmd.append('--override-control={}'.format(
             _create_temp_control_file(extra_depends))
@@ -403,10 +413,11 @@
 def _build_adt_run_test_command(config_stack, extra_depends=None,
                                 update_apt=False, network_ready=False,
                                 add_ppa=True, install_deps=False,
-                                cache_deps=False):
+                                cache_deps=False, verbose=False):
 
     device_password = config_stack.get('device_password')
-    run_cmd = _build_basic_adt_run_test_command(config_stack, extra_depends)
+    run_cmd = _build_basic_adt_run_test_command(
+        config_stack, extra_depends, verbose)
 
     ssh_cmd = (
         ['ssh', '-s', 'adb', '--', '-p', device_password] +

=== modified file 'ubuntu_system_tests/selftests/test_command_line.py'
--- ubuntu_system_tests/selftests/test_command_line.py	2016-04-08 09:17:43 +0000
+++ ubuntu_system_tests/selftests/test_command_line.py	2016-04-18 15:40:01 +0000
@@ -351,6 +351,8 @@
     DONT_USE_SILENT = False
     DO_USE_CACHE_DEPS = True
     DONT_USE_CACHE_DEPS = False
+    DO_USE_VERBOSE_LOGGING = True
+    DONT_USE_VERBOSE_LOGGING = False
 
     scenarios = [
         ('no arguments', {
@@ -363,7 +365,9 @@
                 VISUALIZATION_OFF,
                 DONT_INSTALL_DEPS,
                 DONT_USE_SILENT,
-                DONT_USE_CACHE_DEPS)
+                DONT_USE_CACHE_DEPS,
+                DONT_USE_VERBOSE_LOGGING,
+            )
         }),
         ('alternative config', {
             'command_line_arguments': ['--config=alternative.conf'],
@@ -375,7 +379,9 @@
                 VISUALIZATION_OFF,
                 DONT_INSTALL_DEPS,
                 DONT_USE_SILENT,
-                DONT_USE_CACHE_DEPS)
+                DONT_USE_CACHE_DEPS,
+                DONT_USE_VERBOSE_LOGGING,
+            )
         }),
         ('with apt-get update', {
             'command_line_arguments': ['--update-apt'],
@@ -387,7 +393,9 @@
                 VISUALIZATION_OFF,
                 DONT_INSTALL_DEPS,
                 DONT_USE_SILENT,
-                DONT_USE_CACHE_DEPS)
+                DONT_USE_CACHE_DEPS,
+                DONT_USE_VERBOSE_LOGGING,
+            )
         }),
         ('one test suite', {
             'command_line_arguments': ['test_suite'],
@@ -399,7 +407,9 @@
                 VISUALIZATION_OFF,
                 DONT_INSTALL_DEPS,
                 DONT_USE_SILENT,
-                DONT_USE_CACHE_DEPS)
+                DONT_USE_CACHE_DEPS,
+                DONT_USE_VERBOSE_LOGGING,
+            )
         }),
         ('multiple test suites', {
             'command_line_arguments': [
@@ -411,7 +421,9 @@
                 DONT_APT_UPDATE, NETWORK_NOT_ALREADY_SETUP, VISUALIZATION_OFF,
                 DONT_INSTALL_DEPS,
                 DONT_USE_SILENT,
-                DONT_USE_CACHE_DEPS)
+                DONT_USE_CACHE_DEPS,
+                DONT_USE_VERBOSE_LOGGING,
+            )
         }),
         ('with network already setup', {
             'command_line_arguments': ['--network-ready'],
@@ -421,7 +433,9 @@
                 VISUALIZATION_OFF,
                 DONT_INSTALL_DEPS,
                 DONT_USE_SILENT,
-                DONT_USE_CACHE_DEPS)
+                DONT_USE_CACHE_DEPS,
+                DONT_USE_VERBOSE_LOGGING,
+            )
         }),
         ('with touch visualisation enabled', {
             'command_line_arguments': ['--touch-visualization'],
@@ -431,7 +445,9 @@
                 VISUALIZATION_ON,
                 DONT_INSTALL_DEPS,
                 DONT_USE_SILENT,
-                DONT_USE_CACHE_DEPS)
+                DONT_USE_CACHE_DEPS,
+                DONT_USE_VERBOSE_LOGGING,
+            )
         }),
         ('with install dependencies enabled', {
             'command_line_arguments': ['--install-deps'],
@@ -441,7 +457,9 @@
                 VISUALIZATION_OFF,
                 DO_INSTALL_DEPS,
                 DONT_USE_SILENT,
-                DONT_USE_CACHE_DEPS)
+                DONT_USE_CACHE_DEPS,
+                DONT_USE_VERBOSE_LOGGING,
+            )
         }),
         ('with silent mode specified', {
             'command_line_arguments': ['--silent'],
@@ -451,7 +469,9 @@
                 VISUALIZATION_OFF,
                 DONT_INSTALL_DEPS,
                 DO_USE_SILENT,
-                DONT_USE_CACHE_DEPS)
+                DONT_USE_CACHE_DEPS,
+                DONT_USE_VERBOSE_LOGGING,
+            )
         }),
         ('with cache deps specified', {
             'command_line_arguments': ['--cache-deps'],
@@ -461,7 +481,21 @@
                 VISUALIZATION_OFF,
                 DONT_INSTALL_DEPS,
                 DONT_USE_SILENT,
-                DO_USE_CACHE_DEPS)
+                DO_USE_CACHE_DEPS,
+                DONT_USE_VERBOSE_LOGGING,
+            )
+        }),
+        ('with verbose logging specified', {
+            'command_line_arguments': ['--verbose'],
+            'parsed_arguments': (
+                DEFAULT_CONFIG_FILE, [], DONT_APT_UPDATE,
+                NETWORK_NOT_ALREADY_SETUP,
+                VISUALIZATION_OFF,
+                DONT_INSTALL_DEPS,
+                DONT_USE_SILENT,
+                DONT_USE_CACHE_DEPS,
+                DO_USE_VERBOSE_LOGGING,
+            )
         }),
     ]