Merge lp:~veebers/autopilot/fix_for_config_init into lp:autopilot

[Christopher Lee]

Move config parsing to happen before test import.

Move config parsing to happen before test import. Ensures config details available within decorators.

[PS Jenkins bot (ps-jenkins)]

PASSED: Continuous integration, rev:545
http://jenkins.qa.ubuntu.com/job/autopilot-ci/1017/
Executed test runs:
    SUCCESS: http://jenkins.qa.ubuntu.com/job/autopilot-vivid-amd64-ci/80
        deb: http://jenkins.qa.ubuntu.com/job/autopilot-vivid-amd64-ci/80/artifact/work/output/*zip*/output.zip
    SUCCESS: http://jenkins.qa.ubuntu.com/job/autopilot-vivid-armhf-ci/80
        deb: http://jenkins.qa.ubuntu.com/job/autopilot-vivid-armhf-ci/80/artifact/work/output/*zip*/output.zip
    SUCCESS: http://jenkins.qa.ubuntu.com/job/autopilot-vivid-i386-ci/80
        deb: http://jenkins.qa.ubuntu.com/job/autopilot-vivid-i386-ci/80/artifact/work/output/*zip*/output.zip
    UNSTABLE: http://jenkins.qa.ubuntu.com/job/generic-deb-autopilot-vivid-touch/1657
    SUCCESS: http://jenkins.qa.ubuntu.com/job/generic-mediumtests-vivid-autopilot/118
    UNSTABLE: http://jenkins.qa.ubuntu.com/job/generic-deb-autopilot-runner-vivid-mako/1470
    SUCCESS: http://jenkins.qa.ubuntu.com/job/generic-mediumtests-builder-vivid-armhf/1655
        deb: http://jenkins.qa.ubuntu.com/job/generic-mediumtests-builder-vivid-armhf/1655/artifact/work/output/*zip*/output.zip
    SUCCESS: http://s-jenkins.ubuntu-ci:8080/job/touch-flash-device/18573
    SUCCESS: http://jenkins.qa.ubuntu.com/job/autopilot-testrunner-otto-vivid-autopilot/114
    SUCCESS: http://jenkins.qa.ubuntu.com/job/generic-mediumtests-builder-vivid-amd64/802
        deb: http://jenkins.qa.ubuntu.com/job/generic-mediumtests-builder-vivid-amd64/802/artifact/work/output/*zip*/output.zip

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/autopilot-ci/1017/rebuild

review: Approve (continuous-integration)

[Leo Arias (elopio)]

This looks good. Can we write a test for it?
I think that using a mock and checking the order of the calls would be good.
Even better if we add a skip decorator in a test that uses a config value.

[lp:~veebers/autopilot/fix_for_config_init]

546. By Christopher Lee on 2015-03-10

Failing test for config available in decorator

547. By Christopher Lee on 2015-03-10

test is passing (no functional code change, demos that config now available)

[Christopher Lee (veebers)]

@Leo, I've added a first pass at a functional test for this as requested.

I'm not son keen on doing a test using mocks etc. to check the order of the calls. The calls are static and we can see them in the source file (e.g. they aren't conditionally run etc.) as well as the whole suite of tests that cover the execution of this part of the code which should give us coverage already.

[PS Jenkins bot (ps-jenkins)]

FAILED: Continuous integration, rev:547
http://jenkins.qa.ubuntu.com/job/autopilot-ci/1018/
Executed test runs:
    SUCCESS: http://jenkins.qa.ubuntu.com/job/autopilot-vivid-amd64-ci/81
        deb: http://jenkins.qa.ubuntu.com/job/autopilot-vivid-amd64-ci/81/artifact/work/output/*zip*/output.zip
    SUCCESS: http://jenkins.qa.ubuntu.com/job/autopilot-vivid-armhf-ci/81
        deb: http://jenkins.qa.ubuntu.com/job/autopilot-vivid-armhf-ci/81/artifact/work/output/*zip*/output.zip
    SUCCESS: http://jenkins.qa.ubuntu.com/job/autopilot-vivid-i386-ci/81
        deb: http://jenkins.qa.ubuntu.com/job/autopilot-vivid-i386-ci/81/artifact/work/output/*zip*/output.zip
    FAILURE: http://jenkins.qa.ubuntu.com/job/generic-mediumtests-vivid-autopilot/119/console
    FAILURE: http://jenkins.qa.ubuntu.com/job/autopilot-testrunner-otto-vivid-autopilot/115/console
    SUCCESS: http://jenkins.qa.ubuntu.com/job/generic-mediumtests-builder-vivid-amd64/827
        deb: http://jenkins.qa.ubuntu.com/job/generic-mediumtests-builder-vivid-amd64/827/artifact/work/output/*zip*/output.zip

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/autopilot-ci/1018/rebuild

review: Needs Fixing (continuous-integration)

[Leo Arias (elopio)]

Without the fix the test fails. With it the test passes.
I actually prefer this to be tested in the functional suite, so +2.

review: Approve

[PS Jenkins bot (ps-jenkins)]

FAILED: Continuous integration, rev:547
http://jenkins.qa.ubuntu.com/job/autopilot-ci/1019/
Executed test runs:
    SUCCESS: http://jenkins.qa.ubuntu.com/job/autopilot-vivid-amd64-ci/82
        deb: http://jenkins.qa.ubuntu.com/job/autopilot-vivid-amd64-ci/82/artifact/work/output/*zip*/output.zip
    SUCCESS: http://jenkins.qa.ubuntu.com/job/autopilot-vivid-armhf-ci/82
        deb: http://jenkins.qa.ubuntu.com/job/autopilot-vivid-armhf-ci/82/artifact/work/output/*zip*/output.zip
    SUCCESS: http://jenkins.qa.ubuntu.com/job/autopilot-vivid-i386-ci/82
        deb: http://jenkins.qa.ubuntu.com/job/autopilot-vivid-i386-ci/82/artifact/work/output/*zip*/output.zip
    FAILURE: http://jenkins.qa.ubuntu.com/job/generic-mediumtests-vivid-autopilot/120/console
    FAILURE: http://jenkins.qa.ubuntu.com/job/autopilot-testrunner-otto-vivid-autopilot/116/console
    SUCCESS: http://jenkins.qa.ubuntu.com/job/generic-mediumtests-builder-vivid-amd64/828
        deb: http://jenkins.qa.ubuntu.com/job/generic-mediumtests-builder-vivid-amd64/828/artifact/work/output/*zip*/output.zip

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/autopilot-ci/1019/rebuild

review: Needs Fixing (continuous-integration)

[PS Jenkins bot (ps-jenkins)]

FAILED: Continuous integration, rev:547
http://jenkins.qa.ubuntu.com/job/autopilot-ci/1020/
Executed test runs:
    SUCCESS: http://jenkins.qa.ubuntu.com/job/autopilot-vivid-amd64-ci/83
        deb: http://jenkins.qa.ubuntu.com/job/autopilot-vivid-amd64-ci/83/artifact/work/output/*zip*/output.zip
    SUCCESS: http://jenkins.qa.ubuntu.com/job/autopilot-vivid-armhf-ci/83
        deb: http://jenkins.qa.ubuntu.com/job/autopilot-vivid-armhf-ci/83/artifact/work/output/*zip*/output.zip
    SUCCESS: http://jenkins.qa.ubuntu.com/job/autopilot-vivid-i386-ci/83
        deb: http://jenkins.qa.ubuntu.com/job/autopilot-vivid-i386-ci/83/artifact/work/output/*zip*/output.zip
    FAILURE: http://jenkins.qa.ubuntu.com/job/generic-mediumtests-vivid-autopilot/121/console
    FAILURE: http://jenkins.qa.ubuntu.com/job/autopilot-testrunner-otto-vivid-autopilot/117/console
    SUCCESS: http://jenkins.qa.ubuntu.com/job/generic-mediumtests-builder-vivid-amd64/837
        deb: http://jenkins.qa.ubuntu.com/job/generic-mediumtests-builder-vivid-amd64/837/artifact/work/output/*zip*/output.zip

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/autopilot-ci/1020/rebuild

review: Needs Fixing (continuous-integration)

[Richard Huddie (rhuddie)]

This all looks good, very interesting how the new test case is done. Test passes for me.

review: Approve

[Leo Arias (elopio)]

Asking ci again about the tests. They are stuck.

[PS Jenkins bot (ps-jenkins)]

PASSED: Continuous integration, rev:547
http://jenkins.qa.ubuntu.com/job/autopilot-ci/1021/
Executed test runs:
    SUCCESS: http://jenkins.qa.ubuntu.com/job/autopilot-vivid-amd64-ci/84
        deb: http://jenkins.qa.ubuntu.com/job/autopilot-vivid-amd64-ci/84/artifact/work/output/*zip*/output.zip
    SUCCESS: http://jenkins.qa.ubuntu.com/job/autopilot-vivid-armhf-ci/84
        deb: http://jenkins.qa.ubuntu.com/job/autopilot-vivid-armhf-ci/84/artifact/work/output/*zip*/output.zip
    SUCCESS: http://jenkins.qa.ubuntu.com/job/autopilot-vivid-i386-ci/84
        deb: http://jenkins.qa.ubuntu.com/job/autopilot-vivid-i386-ci/84/artifact/work/output/*zip*/output.zip
    UNSTABLE: http://jenkins.qa.ubuntu.com/job/generic-deb-autopilot-vivid-touch/1734
    SUCCESS: http://jenkins.qa.ubuntu.com/job/generic-mediumtests-vivid-autopilot/123
    UNSTABLE: http://jenkins.qa.ubuntu.com/job/generic-deb-autopilot-runner-vivid-mako/1540
    SUCCESS: http://jenkins.qa.ubuntu.com/job/generic-mediumtests-builder-vivid-armhf/1732
        deb: http://jenkins.qa.ubuntu.com/job/generic-mediumtests-builder-vivid-armhf/1732/artifact/work/output/*zip*/output.zip
    SUCCESS: http://s-jenkins.ubuntu-ci:8080/job/touch-flash-device/18772
    SUCCESS: http://jenkins.qa.ubuntu.com/job/autopilot-testrunner-otto-vivid-autopilot/121
    SUCCESS: http://jenkins.qa.ubuntu.com/job/generic-mediumtests-builder-vivid-amd64/843
        deb: http://jenkins.qa.ubuntu.com/job/generic-mediumtests-builder-vivid-amd64/843/artifact/work/output/*zip*/output.zip

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/autopilot-ci/1021/rebuild

review: Approve (continuous-integration)

[PS Jenkins bot (ps-jenkins)]

PASSED: Continuous integration, rev:547
http://jenkins.qa.ubuntu.com/job/autopilot-ci/1022/
Executed test runs:
    SUCCESS: http://jenkins.qa.ubuntu.com/job/autopilot-vivid-amd64-ci/85
        deb: http://jenkins.qa.ubuntu.com/job/autopilot-vivid-amd64-ci/85/artifact/work/output/*zip*/output.zip
    SUCCESS: http://jenkins.qa.ubuntu.com/job/autopilot-vivid-armhf-ci/85
        deb: http://jenkins.qa.ubuntu.com/job/autopilot-vivid-armhf-ci/85/artifact/work/output/*zip*/output.zip
    SUCCESS: http://jenkins.qa.ubuntu.com/job/autopilot-vivid-i386-ci/85
        deb: http://jenkins.qa.ubuntu.com/job/autopilot-vivid-i386-ci/85/artifact/work/output/*zip*/output.zip
    UNSTABLE: http://jenkins.qa.ubuntu.com/job/generic-deb-autopilot-vivid-touch/1736
    SUCCESS: http://jenkins.qa.ubuntu.com/job/generic-mediumtests-vivid-autopilot/124
    UNSTABLE: http://jenkins.qa.ubuntu.com/job/generic-deb-autopilot-runner-vivid-mako/1542
    SUCCESS: http://jenkins.qa.ubuntu.com/job/generic-mediumtests-builder-vivid-armhf/1734
        deb: http://jenkins.qa.ubuntu.com/job/generic-mediumtests-builder-vivid-armhf/1734/artifact/work/output/*zip*/output.zip
    SUCCESS: http://s-jenkins.ubuntu-ci:8080/job/touch-flash-device/18774
    SUCCESS: http://jenkins.qa.ubuntu.com/job/autopilot-testrunner-otto-vivid-autopilot/122
    SUCCESS: http://jenkins.qa.ubuntu.com/job/generic-mediumtests-builder-vivid-amd64/844
        deb: http://jenkins.qa.ubuntu.com/job/generic-mediumtests-builder-vivid-amd64/844/artifact/work/output/*zip*/output.zip

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/autopilot-ci/1022/rebuild

review: Approve (continuous-integration)

[PS Jenkins bot (ps-jenkins)]

PASSED: Continuous integration, rev:547
http://jenkins.qa.ubuntu.com/job/autopilot-ci/1023/
Executed test runs:
    SUCCESS: http://jenkins.qa.ubuntu.com/job/autopilot-vivid-amd64-ci/86
        deb: http://jenkins.qa.ubuntu.com/job/autopilot-vivid-amd64-ci/86/artifact/work/output/*zip*/output.zip
    SUCCESS: http://jenkins.qa.ubuntu.com/job/autopilot-vivid-armhf-ci/86
        deb: http://jenkins.qa.ubuntu.com/job/autopilot-vivid-armhf-ci/86/artifact/work/output/*zip*/output.zip
    SUCCESS: http://jenkins.qa.ubuntu.com/job/autopilot-vivid-i386-ci/86
        deb: http://jenkins.qa.ubuntu.com/job/autopilot-vivid-i386-ci/86/artifact/work/output/*zip*/output.zip
    UNSTABLE: http://jenkins.qa.ubuntu.com/job/generic-deb-autopilot-vivid-touch/1737
    SUCCESS: http://jenkins.qa.ubuntu.com/job/generic-mediumtests-vivid-autopilot/125
    UNSTABLE: http://jenkins.qa.ubuntu.com/job/generic-deb-autopilot-runner-vivid-mako/1543
    SUCCESS: http://jenkins.qa.ubuntu.com/job/generic-mediumtests-builder-vivid-armhf/1735
        deb: http://jenkins.qa.ubuntu.com/job/generic-mediumtests-builder-vivid-armhf/1735/artifact/work/output/*zip*/output.zip
    SUCCESS: http://s-jenkins.ubuntu-ci:8080/job/touch-flash-device/18776
    SUCCESS: http://jenkins.qa.ubuntu.com/job/autopilot-testrunner-otto-vivid-autopilot/123
    SUCCESS: http://jenkins.qa.ubuntu.com/job/generic-mediumtests-builder-vivid-amd64/845
        deb: http://jenkins.qa.ubuntu.com/job/generic-mediumtests-builder-vivid-amd64/845/artifact/work/output/*zip*/output.zip

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/autopilot-ci/1023/rebuild

review: Approve (continuous-integration)

[Christopher Lee (veebers)]

Top approving as it has the required 2x approves and now that CI is resolved they are passing.

=== modified file 'autopilot/run.py'
--- autopilot/run.py	2014-09-05 04:55:50 +0000
+++ autopilot/run.py	2015-03-10 00:23:59 +0000
@@ -662,16 +662,6 @@
 
     def run_tests(self):
         """Run tests, using input from `args`."""
-        test_suite, error_encountered = load_test_suite_from_name(
-            self.args.suite
-        )
-
-        if not test_suite.countTestCases():
-            raise RuntimeError('Did not find any tests')
-
-        if self.args.random_order:
-            shuffle(test_suite._tests)
-            print("Running tests in random order")
 
         _configure_debug_profile(self.args)
         _configure_timeout_profile(self.args)
@@ -688,6 +678,17 @@
         if self.args.verbose:
             autopilot.globals.set_log_verbose(True)
 
+        test_suite, error_encountered = load_test_suite_from_name(
+            self.args.suite
+        )
+
+        if not test_suite.countTestCases():
+            raise RuntimeError('Did not find any tests')
+
+        if self.args.random_order:
+            shuffle(test_suite._tests)
+            print("Running tests in random order")
+
         result = construct_test_result(self.args)
         result.startTestRun()
         try:

=== modified file 'autopilot/tests/functional/test_autopilot_functional.py'
--- autopilot/tests/functional/test_autopilot_functional.py	2014-07-24 00:24:41 +0000
+++ autopilot/tests/functional/test_autopilot_functional.py	2015-03-10 00:23:59 +0000
@@ -94,6 +94,31 @@
 
 class FunctionalTestMain(AutopilotFunctionalTestsBase):
 
+    def test_config_available_in_decorator(self):
+        """Any commandline config values must be available for decorators."""
+        unique_config_value = self.getUniqueString()
+        self.create_test_file(
+            'test_config_skip.py', dedent("""\
+            from testtools import skipIf
+            import autopilot
+            from autopilot.testcase import AutopilotTestCase
+
+            class ConfigTest(AutopilotTestCase):
+                @skipIf(
+                    autopilot.get_test_configuration().get('skipme', None)
+                    == '{unique_config_value}',
+                    'Skipping Test')
+                def test_config(self):
+                    self.fail('Should not run.')
+            """.format(unique_config_value=unique_config_value))
+        )
+
+        config_string = 'skipme={}'.format(unique_config_value)
+        code, output, error = self.run_autopilot(
+            ['run', 'tests', '--config', config_string])
+
+        self.assertThat(code, Equals(0))
+
     def test_can_list_empty_test_dir(self):
         """Autopilot list must report 0 tests found with an empty test
         module."""