Merge lp:~joetalbott/utah/fix_result_counts into lp:utah

[Joe Talbott]

This branch fixes the result counts for runs that involve a reboot.

[lp:~joetalbott/utah/fix_result_counts]

767. By Joe Talbott on 2012-11-30

test_state_agent - Add a few more tests for resumes.

* make sure different scenarios return the expected results.

[Javier Collado (javier.collado)]

I've got a couple of questions:
- What does JAT stand for? I'd prefer all those commented lines to be removed.
- Why is there a long yaml file embedded in the code? I think maintenance would be easier if the file was in a separate location on its own.

review: Needs Information

[lp:~joetalbott/utah/fix_result_counts]

768. By Joe Talbott on 2012-12-04

testcase - Remove leftover commented out code.

769. By Joe Talbott on 2012-12-04

tests - Move YAML content into separtate files.

[Joe Talbott (joetalbott)]

I've removed those commented out line. I meant to do that before pushing but forgot.

I also moved the YAML content into separate files.

review: Needs Resubmitting

[Javier Collado (javier.collado)]

Looks nice now. Thanks!

review: Approve

[Javier Collado (javier.collado)]

By the way, when running the test cases, I get three errors like the one below:

======================================================================
ERROR: Failure: TypeError (__init__() takes at least 4 arguments (1 given))
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/nose/loader.py", line 495, in makeTest
    return self._makeTest(obj, parent)
  File "/usr/lib/python2.7/dist-packages/nose/loader.py", line 554, in _makeTest
    return MethodTestCase(obj)
  File "/usr/lib/python2.7/dist-packages/nose/case.py", line 345, in __init__
    self.inst = self.cls()
TypeError: __init__() takes at least 4 arguments (1 given)
----------------------------------------------------------------------

I get them for the dev branch as well, so I don't think it's because of any
change here. Anyway, I'm not merging this right now just in case you want to
take a look a this issue.

[lp:~joetalbott/utah/fix_result_counts]

770. By Joe Talbott on 2012-12-05

parser - Fix test for malformed yaml file.

[Joe Talbott (joetalbott)]

It was an unrelated issue, but I went ahead and fixed it here.

review: Needs Resubmitting

[Javier Collado (javier.collado)]

Actually, I've found that the problem was on my side when running the test
cases.

Instead of passing `utah/client/tests`, I was passing just `utah/client`. The
side effect of this was that `nosetests` believed that there were some test
cases in `utah/client/testsuite.py` and tried to execute them causing the
backtrace in my previous comment.

Anyway, merging your changes.

=== modified file 'tests/test_parser.py'
--- tests/test_parser.py	2012-11-14 22:10:40 +0000
+++ tests/test_parser.py	2012-12-05 15:31:21 +0000
@@ -3,7 +3,7 @@
 import tempfile
 import unittest
 
-from utah.parser import UTAHParser
+from utah.parser import UTAHParser, ParserError
 
 MALFORMED_CONTENT = """
 ---
@@ -22,6 +22,9 @@
 
         self.tmpfile.write(MALFORMED_CONTENT)
 
+        # reset the stream for later reading.
+        self.tmpfile.seek(0)
+
     def tearDown(self):
         self.tmpfile.close()
 
@@ -30,7 +33,7 @@
             data = self.parser.parse(self.fakelog)
 
     def test_malformed_file(self):
-        with self.assertRaises(jsonschema.ValidationError):
+        with self.assertRaises(ParserError):
             data = self.parser.parse(self.tmpfile)
 
     def test_parser(self):

=== added file 'utah/client/examples/test_state_partial_all_failed.yaml'
--- utah/client/examples/test_state_partial_all_failed.yaml	1970-01-01 00:00:00 +0000
+++ utah/client/examples/test_state_partial_all_failed.yaml	2012-12-05 15:31:21 +0000
@@ -0,0 +1,90 @@
+master_runlist: /var/lib/utah/master.run
+status: RUN
+result_class: !!python/name:utah.client.result.ResultYAML ''
+fetched_suites:
+- examples
+- utah_tests
+- utah_tests_sample
+suites:
+- build_cmd: null
+  name: examples
+  passes: 0
+  errors: 0
+  failures: 2
+  status: DONE
+  tests:
+    - build_cmd: echo "building for test_one"
+      command: python test_one.py
+      description: A first sample test case.
+      name: test_one
+      path: examples/test_one
+      status: DONE
+      summary: FAIL
+      tc_cleanup: echo "cleanup for test_one"
+      tc_setup: echo "setup for test_one"
+      timeout: 3
+      type: userland
+    - build_cmd: echo "building for test_two"
+      command: python test_two.py
+      description: A second sample test case.
+      name: test_two
+      path: examples/test_two
+      status: DONE
+      summary: FAIL
+      tc_cleanup: echo "cleanup for test_two"
+      tc_setup: echo "setup for test_two"
+      timeout: 300
+      type: userland
+- build_cmd: null
+  name: utah_tests
+  passes: 0
+  errors: 0
+  failures: 2
+  status: DONE
+  tests:
+    - build_cmd: echo "building for test_one"
+      command: python test_one.py
+      description: A first sample test case.
+      name: test_one
+      path: utah_tests/test_one
+      status: DONE
+      summary: FAIL
+      tc_cleanup: echo "cleanup for test_one"
+      tc_setup: echo "setup for test_one"
+      timeout: 3
+      type: userland
+    - build_cmd: echo "building for test_two"
+      command: python test_two.py
+      description: A second sample test case.
+      name: test_two
+      path: utah_tests/test_two
+      status: DONE
+      summary: FAIL
+      tc_cleanup: echo "cleanup for test_two"
+      tc_setup: echo "setup for test_two"
+      timeout: 300
+      type: userland
+  timeout: 0
+  ts_cleanup: null
+  ts_setup: null
+- build_cmd: null
+  name: utah_tests_sample
+  passes: 0
+  errors: 0
+  failures: 1
+  status: DONE
+  tests:
+    - build_cmd: echo "building for sample_one"
+      command: python sample.py
+      description: A sample test case.
+      name: sample_one
+      path: utah_tests_sample/sample_one
+      status: DONE
+      summary: FAIL
+      tc_cleanup: echo "cleanup for sample_one"
+      tc_setup: echo "setup for sample_one"
+      timeout: 200
+      type: userland
+  timeout: 0
+  ts_cleanup: null
+  ts_setup: null

=== added file 'utah/client/examples/test_state_partial_inprogress.yaml'
--- utah/client/examples/test_state_partial_inprogress.yaml	1970-01-01 00:00:00 +0000
+++ utah/client/examples/test_state_partial_inprogress.yaml	2012-12-05 15:31:21 +0000
@@ -0,0 +1,90 @@
+master_runlist: /var/lib/utah/master.run
+status: RUN
+result_class: !!python/name:utah.client.result.ResultYAML ''
+fetched_suites:
+- examples
+- utah_tests
+- utah_tests_sample
+suites:
+- build_cmd: null
+  name: examples
+  passes: 2
+  errors: 0
+  failures: 0
+  status: DONE
+  tests:
+    - build_cmd: echo "building for test_one"
+      command: python test_one.py
+      description: A first sample test case.
+      name: test_one
+      path: examples/test_one
+      status: DONE
+      summary: PASS
+      tc_cleanup: echo "cleanup for test_one"
+      tc_setup: echo "setup for test_one"
+      timeout: 3
+      type: userland
+    - build_cmd: echo "building for test_two"
+      command: python test_two.py
+      description: A second sample test case.
+      name: test_two
+      path: examples/test_two
+      status: DONE
+      summary: PASS
+      tc_cleanup: echo "cleanup for test_two"
+      tc_setup: echo "setup for test_two"
+      timeout: 300
+      type: userland
+- build_cmd: null
+  name: utah_tests
+  passes: 1
+  errors: 0
+  failures: 0
+  status: INPROGRESS
+  tests:
+    - build_cmd: echo "building for test_one"
+      command: python test_one.py
+      description: A first sample test case.
+      name: test_one
+      path: utah_tests/test_one
+      status: DONE
+      summary: PASS
+      tc_cleanup: echo "cleanup for test_one"
+      tc_setup: echo "setup for test_one"
+      timeout: 3
+      type: userland
+    - build_cmd: echo "building for test_two"
+      command: python test_two.py
+      description: A second sample test case.
+      name: test_two
+      path: utah_tests/test_two
+      status: SETUP
+      summary: PASS
+      tc_cleanup: echo "cleanup for test_two"
+      tc_setup: echo "setup for test_two"
+      timeout: 300
+      type: userland
+  timeout: 0
+  ts_cleanup: null
+  ts_setup: null
+- build_cmd: null
+  name: utah_tests_sample
+  passes: 0
+  errors: 0
+  failures: 1
+  status: DONE
+  tests:
+    - build_cmd: echo "building for sample_one"
+      command: python sample.py
+      description: A sample test case.
+      name: sample_one
+      path: utah_tests_sample/sample_one
+      status: DONE
+      summary: FAIL
+      tc_cleanup: echo "cleanup for sample_one"
+      tc_setup: echo "setup for sample_one"
+      timeout: 200
+      type: userland
+  timeout: 0
+  ts_cleanup: null
+  ts_setup: null

=== added file 'utah/client/examples/test_state_partial_run_all.yaml'
--- utah/client/examples/test_state_partial_run_all.yaml	1970-01-01 00:00:00 +0000
+++ utah/client/examples/test_state_partial_run_all.yaml	2012-12-05 15:31:21 +0000
@@ -0,0 +1,90 @@
+master_runlist: /var/lib/utah/master.run
+status: RUN
+result_class: !!python/name:utah.client.result.ResultYAML ''
+fetched_suites:
+- examples
+- utah_tests
+- utah_tests_sample
+suites:
+- build_cmd: null
+  name: examples
+  passes: 0
+  errors: 0
+  failures: 0
+  status: NOTRUN
+  tests:
+    - build_cmd: echo "building for test_one"
+      command: python test_one.py
+      description: A first sample test case.
+      name: test_one
+      path: examples/test_one
+      status: NOTRUN
+      summary: PASS
+      tc_cleanup: echo "cleanup for test_one"
+      tc_setup: echo "setup for test_one"
+      timeout: 3
+      type: userland
+    - build_cmd: echo "building for test_two"
+      command: python test_two.py
+      description: A second sample test case.
+      name: test_two
+      path: examples/test_two
+      status: NOTRUN
+      summary: PASS
+      tc_cleanup: echo "cleanup for test_two"
+      tc_setup: echo "setup for test_two"
+      timeout: 300
+      type: userland
+- build_cmd: null
+  name: utah_tests
+  passes: 1
+  errors: 0
+  failures: 0
+  status: NOTRUN
+  tests:
+    - build_cmd: echo "building for test_one"
+      command: python test_one.py
+      description: A first sample test case.
+      name: test_one
+      path: utah_tests/test_one
+      status: NOTRUN
+      summary: PASS
+      tc_cleanup: echo "cleanup for test_one"
+      tc_setup: echo "setup for test_one"
+      timeout: 3
+      type: userland
+    - build_cmd: echo "building for test_two"
+      command: python test_two.py
+      description: A second sample test case.
+      name: test_two
+      path: utah_tests/test_two
+      status: NOTRUN
+      summary: PASS
+      tc_cleanup: echo "cleanup for test_two"
+      tc_setup: echo "setup for test_two"
+      timeout: 300
+      type: userland
+  timeout: 0
+  ts_cleanup: null
+  ts_setup: null
+- build_cmd: null
+  name: utah_tests_sample
+  passes: 0
+  errors: 0
+  failures: 1
+  status: NOTRUN
+  tests:
+    - build_cmd: echo "building for sample_one"
+      command: python sample.py
+      description: A sample test case.
+      name: sample_one
+      path: utah_tests_sample/sample_one
+      status: NOTRUN
+      summary: PASS
+      tc_cleanup: echo "cleanup for sample_one"
+      tc_setup: echo "setup for sample_one"
+      timeout: 200
+      type: userland
+  timeout: 0
+  ts_cleanup: null
+  ts_setup: null

=== modified file 'utah/client/runner.py'
--- utah/client/runner.py	2012-11-07 18:02:22 +0000
+++ utah/client/runner.py	2012-12-05 15:31:21 +0000
@@ -251,9 +251,9 @@
                 if not keep_going:
                     return self.process_results()
 
-                self.errors += suite.errors
-                self.passes += suite.passes
-                self.failures += suite.failures
+            self.errors += suite.errors
+            self.passes += suite.passes
+            self.failures += suite.failures
 
         self.status = "DONE"
         self.save_state()
@@ -443,14 +443,15 @@
 
                 self.fetched_suites.append(name)
 
-            # Create a TestSuite
-            s = TestSuite(name=name, runlist_file=suite_runlist,
-                          includes=includes, excludes=excludes,
-                          result=self.result, path=self.testsuitedir,
-                          timeout=self.timeout,
-                          _save_state_callback=self.save_state,
-                          _reboot_callback=self.reboot)
-            self.add_suite(s)
+                # Create a TestSuite
+                s = TestSuite(name=name, runlist_file=suite_runlist,
+                              includes=includes, excludes=excludes,
+                              result=self.result, path=self.testsuitedir,
+                              timeout=self.timeout,
+                              _save_state_callback=self.save_state,
+                              _reboot_callback=self.reboot)
+                self.add_suite(s)
+
 
     def get_next_suite(self):
         """

=== modified file 'utah/client/testcase.py'
--- utah/client/testcase.py	2012-10-17 10:22:00 +0000
+++ utah/client/testcase.py	2012-12-05 15:31:21 +0000
@@ -27,6 +27,7 @@
     """
 
     status = 'NOTRUN'
+    summary = 'PASS' # 'PASS', 'FAIL', of 'ERROR'
     build_cmd = None
     tc_setup = None
     tc_cleanup = None
@@ -181,8 +182,7 @@
         """
 
         if self.is_done():
-            self.run_status = 'PASS'
-            return self.run_status
+            return 'PASS'
 
         # Do not proceed if we are missing data
         if (self.description is None
@@ -203,7 +203,7 @@
 
         # if the build fails don't run the test command
         if result.status != 'PASS':
-            self.run_status = result.status
+            self.summary = 'ERROR'
 
             # reset status
             result.status = 'PASS'
@@ -213,7 +213,7 @@
 
         # if the setup fails don't run the test command
         if result.status != 'PASS':
-            self.run_status = result.status
+            self.summary = 'ERROR'
 
             # reset status
             result.status = 'PASS'
@@ -237,6 +237,11 @@
                                       ),
                                       extra_info=extra_info)
 
+            # only if we haven't failed or errored so far do 
+            # we set the summary for the testcase.
+            if self.summary == 'PASS':
+                self.summary = result.status
+
         # Clean up whether 'command' failed or not.
         self.cleanup(result)
 
@@ -246,7 +251,6 @@
             and result.status == 'PASS'):
             need_reboot = True
 
-        self.run_status = result.status
         result.status = 'PASS'
 
         if self.status == 'CLEANUP':
@@ -285,6 +289,7 @@
                 'command': self.command,
                 'timeout': self.timeout,
                 'status': self.status,
+                'summary': self.summary,
                 'build_cmd': self.build_cmd,
                 'tc_setup': self.tc_setup,
                 'tc_cleanup': self.tc_cleanup,

=== modified file 'utah/client/tests/common.py'
--- utah/client/tests/common.py	2012-10-10 14:44:00 +0000
+++ utah/client/tests/common.py	2012-12-05 15:31:21 +0000
@@ -89,3 +89,30 @@
 
     if os.path.exists(examples_dir):
         shutil.rmtree(examples_dir)
+
+def _get_partial_state_file(filename):
+    """
+    Read a partial state file from disk.
+    """
+
+    state = None
+
+    with open(filename, 'r') as fp:
+        state = fp.read()
+
+    return state
+
+partial_state_file_content = _get_partial_state_file(
+        os.path.join(os.path.dirname(__file__), "..", "examples",
+            "test_state_partial_inprogress.yaml")
+)
+
+partial_state_file_content_run_all = _get_partial_state_file(
+        os.path.join(os.path.dirname(__file__), "..", "examples",
+            "test_state_partial_run_all.yaml")
+)
+
+partial_state_file_content_done_all_failed = _get_partial_state_file(
+        os.path.join(os.path.dirname(__file__), "..", "examples",
+            "test_state_partial_all_failed.yaml")
+)

=== modified file 'utah/client/tests/test_state_agent.py'
--- utah/client/tests/test_state_agent.py	2012-10-03 09:07:18 +0000
+++ utah/client/tests/test_state_agent.py	2012-12-05 15:31:21 +0000
@@ -12,60 +12,11 @@
         setUp,  # Used by nosetests
         tearDown,  # Used by nosetests
         get_module_path,
+        partial_state_file_content,
+        partial_state_file_content_run_all,
+        partial_state_file_content_done_all_failed,
         )
 
-partial_state_file_content = """master_runlist: /var/lib/utah/master.run
-status: RUN
-result_class: !!python/name:utah.client.result.ResultYAML ''
-fetched_suites:
-- utah_tests
-- utah_tests_sample
-suites:
-- build_cmd: null
-  name: utah_tests
-  status: INPROGRESS
-  tests:
-    - build_cmd: echo "building for test_one"
-      command: python test_one.py
-      description: A first sample test case.
-      name: test_one
-      path: utah_tests/test_one
-      status: DONE
-      tc_cleanup: echo "cleanup for test_one"
-      tc_setup: echo "setup for test_one"
-      timeout: 3
-      type: userland
-    - build_cmd: echo "building for test_two"
-      command: python test_two.py
-      description: A second sample test case.
-      name: test_two
-      path: utah_tests/test_two
-      status: SETUP
-      tc_cleanup: echo "cleanup for test_two"
-      tc_setup: echo "setup for test_two"
-      timeout: 300
-      type: userland
-  timeout: 0
-  ts_cleanup: null
-  ts_setup: null
-- build_cmd: null
-  name: utah_tests_sample
-  status: DONE
-  tests:
-    - build_cmd: echo "building for sample_one"
-      command: python sample.py
-      description: A sample test case.
-      name: sample_one
-      path: utah_tests_sample/sample_one
-      status: DONE
-      tc_cleanup: echo "cleanup for sample_one"
-      tc_setup: echo "setup for sample_one"
-      timeout: 200
-      type: userland
-  timeout: 0
-  ts_cleanup: null
-  ts_setup: null
-"""
 
 
 class TestStateAgentYAML(unittest.TestCase):
@@ -83,6 +34,8 @@
         self.runner = Runner(install_type="desktop", result_class=ResultYAML,
                              state_agent=self.state_agent)
         self.partial_state_file_content = partial_state_file_content
+        self.partial_state_file_content_run_all = partial_state_file_content_run_all
+        self.partial_state_file_content_done_all_failed = partial_state_file_content_done_all_failed
 
     def tearDown(self):
         """
@@ -159,14 +112,108 @@
             shutil.copytree(test_src, test_dir)
 
         state_agent = StateAgentYAML(state_file=self.state_file)
-        runner = Runner(install_type="desktop", state_agent=state_agent,
-                        resume=True)
-        runner.load_state()
+        runner = Runner(install_type="desktop",
+                state_agent=state_agent,
+                result_class=ResultYAML,
+                resume=True)
 
         self.assertEqual(runner.status, 'RUN')
+        self.assertEqual(runner.count_suites(), 3)
+        self.assertEqual(runner.count_tests(), 5)
 
         next_test = runner.get_next_test()
 
         self.assertEqual(next_test.name, 'test_two')
 
         runner.run()
+
+        self.assertEqual(runner.passes, 4)
+        self.assertEqual(runner.failures, 1)
+        self.assertEqual(runner.errors, 0)
+        self.assertEqual(runner.passes + runner.failures + runner.errors, 5)
+
+    def test_load_state_partial_run_all(self):
+        """
+        Test that a partially run state file can be resumed.
+        """
+        fp = open(self.state_file, 'w')
+        fp.write(self.partial_state_file_content_run_all)
+        fp.close()
+
+        # Need to set up utah_tests since we're simulating a restart
+        test_src = os.path.join(get_module_path(), 'client',
+                'examples', 'utah_tests')
+        test_dir = os.path.join(UTAH_DIR, 'testsuites',
+                                'utah_tests', 'utah_tests')
+        if not os.path.exists(test_dir):
+            shutil.copytree(test_src, test_dir)
+
+        # Need to set up utah_tests_sample since we're simulating a restart
+        test_src = os.path.join(get_module_path(), 'client',
+                'examples', 'utah_tests_sample')
+        test_dir = os.path.join(UTAH_DIR, 'testsuites', 'utah_tests_sample',
+                'utah_tests_sample')
+        if not os.path.exists(test_dir):
+            shutil.copytree(test_src, test_dir)
+
+        state_agent = StateAgentYAML(state_file=self.state_file)
+        runner = Runner(install_type="desktop",
+                state_agent=state_agent,
+                result_class=ResultYAML,
+                resume=True)
+
+        self.assertEqual(runner.status, 'RUN')
+        self.assertEqual(runner.count_suites(), 3)
+        self.assertEqual(runner.count_tests(), 5)
+
+        next_test = runner.get_next_test()
+
+        self.assertEqual(next_test.name, 'test_one')
+
+        runner.run()
+
+        self.assertEqual(runner.passes, 3)
+        self.assertEqual(runner.failures, 0)
+        self.assertEqual(runner.errors, 2)
+        self.assertEqual(runner.passes + runner.failures + runner.errors, 5)
+
+    def test_load_state_partial_done_all_failed(self):
+        """
+        Test that a partially run state file can be resumed.
+        """
+        fp = open(self.state_file, 'w')
+        fp.write(self.partial_state_file_content_done_all_failed)
+        fp.close()
+
+        # Need to set up utah_tests since we're simulating a restart
+        test_src = os.path.join(get_module_path(), 'client',
+                'examples', 'utah_tests')
+        test_dir = os.path.join(UTAH_DIR, 'testsuites',
+                                'utah_tests', 'utah_tests')
+        if not os.path.exists(test_dir):
+            shutil.copytree(test_src, test_dir)
+
+        # Need to set up utah_tests_sample since we're simulating a restart
+        test_src = os.path.join(get_module_path(), 'client',
+                'examples', 'utah_tests_sample')
+        test_dir = os.path.join(UTAH_DIR, 'testsuites', 'utah_tests_sample',
+                'utah_tests_sample')
+        if not os.path.exists(test_dir):
+            shutil.copytree(test_src, test_dir)
+
+        state_agent = StateAgentYAML(state_file=self.state_file)
+        runner = Runner(install_type="desktop",
+                state_agent=state_agent,
+                result_class=ResultYAML,
+                resume=True)
+
+        self.assertEqual(runner.status, 'RUN')
+        self.assertEqual(runner.count_suites(), 3)
+        self.assertEqual(runner.count_tests(), 5)
+
+        runner.run()
+
+        self.assertEqual(runner.passes, 0)
+        self.assertEqual(runner.failures, 5)
+        self.assertEqual(runner.errors, 0)
+        self.assertEqual(runner.passes + runner.failures + runner.errors, 5)

=== modified file 'utah/client/testsuite.py'
--- utah/client/testsuite.py	2012-10-22 08:15:08 +0000
+++ utah/client/testsuite.py	2012-12-05 15:31:21 +0000
@@ -204,6 +204,9 @@
         Run the test cases; including any build, setup, and cleanup commands.
         """
 
+        # we gather this info from the testcases
+        self.passes = self.errors = self.failures = 0
+
         # Return value to indicate whether processing of a Runner should
         # continue.  This is to avoid a shutdown race on reboot cases.
         keep_going = True
@@ -226,11 +229,11 @@
             for test in self.tests:
                 keep_going = test.run()
 
-                if test.run_status == 'PASS':
+                if test.summary == 'PASS':
                     self.passes += 1
-                elif test.run_status == 'FAIL':
+                elif test.summary == 'FAIL':
                     self.failures += 1
-                elif test.run_status == 'ERROR':
+                elif test.summary == 'ERROR':
                     self.errors += 1
 
                 if not keep_going: