Merge lp:~javier.collado/utah/add_client_include_exclude_tests into lp:utah

[Javier Collado]

This branch verifies include/exclude features by adding some test cases.

- In the runner, a test case is added to make sure that whatever is written
  in the master run list in the include_tests/exclude_tests is properly passed
  to the test suite objects using include/exclude parameters.

- In the test suite, a few test cases make sure that the test case selection
  is based on the information passed through the include/exclude parameters.

[Joe Talbott (joetalbott)]

Looks good to me.

I haven't used mock before so I'm not sure what is happening there. You could just as easily inspect the testsuites in the runner and verify that the tests are what you expect them to be.

review: Approve

[Javier Collado (javier.collado)]

@Joe

The idea of using the mock library is to keep the unit being tested isolated.

While it's true that the test cases I added to the test suite library could be
moved to the runner to check the whole feature, that also makes problems more
difficult to troubleshoot because many things can fail in between. However, if
I just check that the runner passes what it receives from the yaml file to the
test suite, then I know that it's behaving correctly and if something fails in
the suite test cases, then it's the suite where a should look at (actually I
should have mocked the test case class there as well).

I think small test cases are useful not only to make sure things work at a low
level, but also as some kind of documentation of how classes interrelate to
each other.

=== modified file 'utah/client/tests/test_runner.py'
--- utah/client/tests/test_runner.py	2012-10-10 14:44:00 +0000
+++ utah/client/tests/test_runner.py	2012-10-18 17:07:27 +0000
@@ -1,6 +1,7 @@
 import os
 import shutil
 import unittest
+from mock import patch
 
 from utah.client.result import ResultYAML
 from utah.client.state_agent import StateAgentYAML
@@ -12,6 +13,7 @@
         setUp,  # Used by nosetests
         tearDown,  # Used by nosetests
         master_runlist_content,
+        get_module_path,
         )
 
 
@@ -121,9 +123,8 @@
         """
         runlist = '/tmp/master_runlist_test'
 
-        fp = open(runlist, 'w')
-        fp.write(master_runlist_content)
-        fp.close()
+        with open(runlist, 'w') as fp:
+            fp.write(master_runlist_content)
 
         runner = Runner(install_type='desktop',
                         result_class=self.result_class,
@@ -144,3 +145,35 @@
         testdir = self.runner.testdir
 
         self.assertTrue(os.path.exists(testdir))
+
+    @patch('utah.client.runner.TestSuite')
+    def test_include_exclude(self, mock_class):
+        """Test that includes/excludes are passed to the test suite"""
+        include_runlist_content = """---
+testsuites:
+  - name: utah_tests
+    fetch_method: bzr-export
+    fetch_location: {REPO_PATH}/client/examples/utah_tests
+    include_tests:
+      - test_a
+      - test_b
+      - test_c
+    exclude_tests:
+      - test_1
+      - test_2
+      - test_3
+""".format(REPO_PATH=get_module_path())
+
+        runlist = '/tmp/master_runlist_test'
+        with open(runlist, 'w') as fp:
+            fp.write(include_runlist_content)
+
+        Runner(install_type='desktop',
+               result_class=self.result_class,
+               runlist=runlist)
+
+        args, kwargs = mock_class.call_args
+        self.assertListEqual(kwargs['includes'],
+                             ['test_a', 'test_b', 'test_c'])
+        self.assertListEqual(kwargs['excludes'],
+                             ['test_1', 'test_2', 'test_3'])

=== modified file 'utah/client/tests/test_testsuite.py'
--- utah/client/tests/test_testsuite.py	2012-08-08 16:17:12 +0000
+++ utah/client/tests/test_testsuite.py	2012-10-18 17:07:27 +0000
@@ -76,3 +76,54 @@
 
         for test in suite.tests:
             self.assertEqual(timeout, test.timeout)
+
+    def test_include(self):
+        """Included tests are executed"""
+        included_test_name = 'test_one'
+
+        suite = testsuite.TestSuite(name=self.name,
+                                    includes=[included_test_name],
+                                    path=self.path,
+                                    result=ResultYAML())
+
+        # Make sure just one test case has been selected
+        self.assertEqual(len(suite.tests), 1)
+
+        # Make sure that the included test case name matches
+        case = suite.tests[0]
+        self.assertEqual(case.name, included_test_name)
+
+    def test_exclude(self):
+        """Excluded tests are *not* executed"""
+        excluded_test_name = 'test_two'
+
+        suite = testsuite.TestSuite(name=self.name,
+                                    excludes=[excluded_test_name],
+                                    path=self.path,
+                                    result=ResultYAML())
+
+        # Make sure just one test case has been selected
+        self.assertEqual(len(suite.tests), 1)
+
+        # Make sure that the selected test case name matches
+        case = suite.tests[0]
+        self.assertNotEqual(case.name, excluded_test_name)
+
+    def test_include_exclude(self):
+        """Include/exclude combination works fine"""
+        included_test_name = 'test_one'
+        excluded_test_name = 'test_two'
+
+        suite = testsuite.TestSuite(name=self.name,
+                                    includes=[included_test_name],
+                                    excludes=[excluded_test_name],
+                                    path=self.path,
+                                    result=ResultYAML())
+
+        # Make sure just one test case has been selected
+        self.assertEqual(len(suite.tests), 1)
+
+        # Make sure that the selected test case name matches
+        case = suite.tests[0]
+        self.assertEqual(case.name, included_test_name)
+        self.assertNotEqual(case.name, excluded_test_name)