Merge lp:~javier.collado/utah/bug1165175 into lp:utah

[Javier Collado]

This branch:
- Fixes DefaultValidator to raise ValidationError exceptions when a problem is
  found in a runlist/control file.
- Updates python-jsonschema dependencies to the latest version
- Updates schemas to jsonschema draftv4
- Adds unit test cases to verify required properties, default values, etc.

Regarding the schemas updates, note that the old schema for the master runlist
has been removed and that values that in test case properties in the test suite
runlist are all of them under the 'overrides' property.

I've been able to successfully run the that `pass.run` runlist in a VM with
this branch, but I have to reboot the VM manually for the test to complete.
Anyway, this something that I've seen today with other branches, so it's not a
problem introduce in this branch.

Note that to test the changes here, you'll need the latest version (1.3.0) of
the jsonschema library:
https://github.com/Julian/jsonschema

The main advantage in this new version is that the error messages are way
better than in the older one and that the draft4 version of the jsonschema
specification can be used.

I've uploaded a package to my ppa:
https://launchpad.net/~javier.collado/+archive/ppa/

that you can use in case to run the test cases and test your own runlists. I'll
copy it to the UTAH PPA once this branch is merged.

[Andy Doan (doanac)]

591 + def test_empty_dict_invalid(self):
592 + """An empty dictionary is valid."""
593 + with self.assertRaises(jsonschema.ValidationError):
594 + self.validate({})

i think 592 should be "An empty dictionary isn't valid."

[Andy Doan (doanac)]

On 04/26/2013 10:24 AM, Javier Collado wrote:
> https://code.launchpad.net/~javier.collado/utah/bug1165175/+merge/161168

> === modified file 'utah/client/testcase.py'
> CONTROL_SCHEMA = {
> + '$schema': 'http://json-schema.org/draft-04/schema#',

out of curiosity - what does this do?

> + 'required': [
> + 'description',
> + 'dependencies',
> + 'action',
> + 'expected_results',
> + 'command',
> + 'run_as',
> + ],

if you look at our current code. run_as isn't actually being required.
Additionally - a lot of people probably don't want to have to declare
this. ie - do you make it "jenkins", "utah", etc? So to prevent possible
regressions, I think we should make run_as be optional.

> === modified file 'utah/client/tests/test_jsonschema.py'
> === modified file 'utah/client/tests/test_runner.py'

nice set of test cases!

> === modified file 'utah/client/tests/test_testcase.py'

> + def test_run_as_required(self):
> + """Run as is a required property."""
> + with self.assertRaises(jsonschema.ValidationError):
> + self.validate({
> + 'description': 'description',
> + 'dependencies': 'dependencies',
> + 'action': 'action',
> + 'expected_results': 'expected_results',
> + 'command': 'command',
> + })

as per my other comment - this can probably be removed.

[Javier Collado (javier.collado)]

@Andy

> > === modified file 'utah/client/testcase.py'
> > CONTROL_SCHEMA = {
> > + '$schema': 'http://json-schema.org/draft-04/schema#',
>
> out of curiosity - what does this do?
>

From the documentation:
---
 The "$schema" keyword is both used as a JSON Schema version identifier and the location of a resource which is itself a JSON Schema, which describes any schema written for this particular version.
---

You can find the whole information here:
http://json-schema.org/latest/json-schema-core.html#anchor22

> if you look at our current code. run_as isn't actually being required.

I've talked to Joe about this and it seems that the code has a default value for the case in which the value isn't provided, but certainly according to the old version of the schema, it's required:
---
'run_as': {
    'type': 'string',
    'required': True,
},
---

> Additionally - a lot of people probably don't want to have to declare
> this. ie - do you make it "jenkins", "utah", etc? So to prevent possible
> regressions, I think we should make run_as be optional.

Maybe the reason to make it required in the past was to make explicit the user the should be used for each command.
I guess I can make "run_as" optional and default to "utah", but for now there won't be any regression if it's required. We can discuss this and create another merge proposal to address this.

Some other thing that Joe has reminded me that we should think about, is that the build, setup and cleanup commands are run with the same user as the utah client. One proposal at the time we talked about this, was to create a mini-schema for all the commands so that the command and the user could be specified for every command.

> as per my other comment - this can probably be removed.

The safer approach for now should be follow what it was required in the old schema and discuss making it optional and what default value to use.

[lp:~javier.collado/utah/bug1165175]

900. By Javier Collado on 2013-04-29

Fixed typo detected by Andy

901. By Javier Collado on 2013-04-29

Made run_as option in tc_control file

902. By Javier Collado on 2013-04-29

Removed MissingData exception

Now that DefaultValidator is working again and tests cases have been addded to
verify this, there's no need to run custom tests in the code that duplicate the
validator functionality.

[Javier Collado (javier.collado)]

@Andy

After the talk we had, I've updated "run_as" in the schema to be optional and removed the "MissingData" exception code.

review: Needs Resubmitting

[Andy Doan (doanac)]

review: Approve

=== modified file 'debian/changelog'
--- debian/changelog	2013-04-26 12:29:41 +0000
+++ debian/changelog	2013-04-29 21:07:35 +0000
@@ -7,6 +7,7 @@
 
   [ Javier Collado ]
   * Use socket timeout in SSH commands instead of SIGALRM (LP: #1169510)
+  * Fixed DefaultValidator and updated schemas to draftv4 (LP: #1165175)
 
   [ Max Brustkern ]
   * Added syslog output (including real time test output) to client

=== modified file 'debian/control'
--- debian/control	2013-04-19 19:08:00 +0000
+++ debian/control	2013-04-29 21:07:35 +0000
@@ -5,7 +5,7 @@
 Maintainer: Max Brustkern <[e-mail]>
 Build-Depends: debhelper (>= 7.0.50~), devscripts, gawk,
                python-all, python-bzrlib, python-netifaces, python-psutil,
-               python-sphinx, python-jsonschema (>= 0.5~)
+               python-sphinx, python-jsonschema (>= 1.3~)
 Standards-Version: 3.9.3
 Homepage: https://code.launchpad.net/utah
 Vcs-Bzr: https://code.launchpad.net/utah
@@ -49,7 +49,7 @@
 Package: utah-client
 Architecture: all
 Depends: ${misc:Depends}, ${python:Depends},
-         bzr, python-jsonschema (>= 0.5~), python-psutil, python-yaml, sbsigntool, utah-common
+         bzr, python-jsonschema (>= 1.3~), python-psutil, python-yaml, sbsigntool, utah-common
 Recommends: git
 Description: Ubuntu Test Automation Harness Client
  Automation framework for testing in Ubuntu, client portion
@@ -63,6 +63,6 @@
 Package: utah-parser
 Architecture: all
 Depends: ${misc:Depends}, ${python:Depends},
-         python-jsonschema (>= 0.5~), python-yaml, utah-common
+         python-jsonschema (>= 1.3~), python-yaml, utah-common
 Description: Ubuntu Test Automation Harness Parser
  Automation framework for testing in Ubuntu, result parser

=== modified file 'docs/source/development.rst'
--- docs/source/development.rst	2012-10-26 11:01:06 +0000
+++ docs/source/development.rst	2013-04-29 21:07:35 +0000
@@ -85,20 +85,21 @@
 
 ::
 
-    - name: testsuite1
-      fetch_method: git
-      fetch_location: repo
-      include_tests:
-        - t1
-        - t2
-        - t3...
+    - testsuites:
+      - name: testsuite1
+        fetch_method: git
+        fetch_location: repo
+        include_tests:
+          - t1
+          - t2
+          - t3...
 
-    - name: testsuite2
-      fetch_method: bzr
-      fetch_location: lp:utah/dev/
-      exclude_tests:
-        - st4
-        - st5
+      - name: testsuite2
+        fetch_method: bzr
+        fetch_location: lp:utah/dev/
+        exclude_tests:
+          - st4
+          - st5
 
 The only required fields are ``name``, ``fetch_method`` and ``fetch_location``.
 ``name`` must correspond to the name of the top-level testsuite directory.
@@ -136,7 +137,8 @@
 ::
 
     - test: t1                             # directory
-      run_as: nobody                       # user that runs the test
+      overrides:
+      - run_as: nobody                       # user that runs the test
 
     - test: t2                             # directory/binary
       overrides:                           # array of control properties to override

=== modified file 'utah/client/common.py'
--- utah/client/common.py	2013-04-18 18:17:37 +0000
+++ utah/client/common.py	2013-04-29 21:07:35 +0000
@@ -363,7 +363,7 @@
     try:
         pids = subprocess.check_output(['ps', '--no-headers', '-o', 'pid',
                                         '--ppid', str(pid)]).split()
-        return [int(pid) for pid in pids]
+        return [int(process_id) for process_id in pids]
     except subprocess.CalledProcessError:
         return []
 
@@ -404,7 +404,8 @@
 
 # Validator that sets values to defaults as explained in:
 # https://github.com/Julian/jsonschema/issues/4
-class DefaultValidator(jsonschema.Validator):
[e-mail]('draft4')
+class DefaultValidator(jsonschema.Draft4Validator):
 
     """jsonschema validator that sets default values.
 
@@ -413,13 +414,26 @@
 
     """
 
+    @classmethod
+    def check_schema(self, schema):
+        jsonschema.Draft4Validator.check_schema(schema)
+
     def validate_properties(self, properties, instance, schema):
         """Set missing properties to default value."""
-        (super(DefaultValidator, self)
-         .validate_properties(properties, instance, schema))
-        instance.update((k, v['default'])
-                        for k, v in properties.iteritems()
-                        if k not in instance and 'default' in v)
+        if not self.is_type(instance, 'object'):
+            return
+
+        errors = (super(DefaultValidator, self)
+                  .validate_properties(properties, instance, schema))
+        for error in errors:
+            yield error
+
+        default_values = [
+            (k, v['default'])
+            for k, v in properties.iteritems()
+            if k not in instance and 'default' in v]
+
+        instance.update(default_values)
 
 
 def parse_control_file(filename, schema):
@@ -438,8 +452,8 @@
 
     """
     control_data = parse_yaml_file(filename)
-    validator = DefaultValidator()
-    validator.validate(control_data, schema)
+    validator = DefaultValidator(schema)
+    validator.validate(control_data)
     return control_data
 
 

=== modified file 'utah/client/examples/examples/tslist.run'
--- utah/client/examples/examples/tslist.run	2012-08-08 21:10:05 +0000
+++ utah/client/examples/examples/tslist.run	2013-04-29 21:07:35 +0000
@@ -1,11 +1,11 @@
 - test: test_one
-  command: python test_one.py
-  run_as: nobody
   overrides:
     build_cmd: ./build.sh
     timeout: 3
+    command: python test_one.py
+    run_as: nobody
 
 - test: test_two
-  command: python test_two.py
   overrides:
     timeout: 300
+    command: python test_two.py

=== modified file 'utah/client/examples/master.run'
--- utah/client/examples/master.run	2012-10-10 14:44:00 +0000
+++ utah/client/examples/master.run	2013-04-29 21:07:35 +0000
@@ -1,8 +1,4 @@
 ---
-publish:
-  url: http://dashboard.local
-  token: testtoken
-  name: precise-server-amd64
 testsuites:
   - name: utah_tests
     fetch_method: bzr-export

=== modified file 'utah/client/examples/pass.run'
--- utah/client/examples/pass.run	2012-10-26 10:31:36 +0000
+++ utah/client/examples/pass.run	2013-04-29 21:07:35 +0000
@@ -1,8 +1,4 @@
 ---
-publish:
-  url: http://dashboard.local
-  token: testtoken
-  name: example-pass
 testsuites:
   - name: utah_tests_sample
     fetch_method: dev

=== modified file 'utah/client/examples/utah_tests/tslist.run'
--- utah/client/examples/utah_tests/tslist.run	2012-08-13 16:22:08 +0000
+++ utah/client/examples/utah_tests/tslist.run	2013-04-29 21:07:35 +0000
@@ -1,7 +1,7 @@
 - test: test_one
-  run_as: nobody
   overrides:
     timeout: 3
+    run_as: nobody
 
 - test: test_two
   overrides:

=== modified file 'utah/client/examples/utah_tests_sample/tslist.run'
--- utah/client/examples/utah_tests_sample/tslist.run	2012-04-10 20:23:50 +0000
+++ utah/client/examples/utah_tests_sample/tslist.run	2013-04-29 21:07:35 +0000
@@ -1,2 +1,3 @@
 - test: sample_one
-  command: python sample.py
+  overrides:
+    command: python sample.py

=== modified file 'utah/client/exceptions.py'
--- utah/client/exceptions.py	2013-04-08 19:12:25 +0000
+++ utah/client/exceptions.py	2013-04-29 21:07:35 +0000
@@ -73,10 +73,3 @@
     """Used to provide additional information when schema validation fails."""
 
     pass
-
-
-class MissingData(UTAHClientError):
-
-    """Raised when there is missing data required for an object to proceed."""
-
-    pass

=== modified file 'utah/client/result.py'
--- utah/client/result.py	2013-04-05 18:39:43 +0000
+++ utah/client/result.py	2013-04-29 21:07:35 +0000
@@ -33,15 +33,13 @@
     """Result collection class."""
 
     def __init__(self, name=None, testsuite=None, testcase=None,
-                 runlist=None, publish_type=None, install_type=None):
+                 runlist=None, install_type=None):
         self.results = []
         self.status = 'PASS'
         self.name = name
         self.testsuite = testsuite
         self.testcase = testcase
         self.runlist = runlist
-        self.publish_type = publish_type
-        self.publish = None
         self.install_type = install_type
         self.start_battery = None
         self.end_battery = None
@@ -187,9 +185,6 @@
             'name': self.name,
         }
 
-        if self.publish is not None:
-            data['publish'] = self.publish
-
         if self.start_battery or self.end_battery:
             data['battery'] = {}
             if self.start_battery:

=== modified file 'utah/client/runner.py'
--- utah/client/runner.py	2013-04-15 14:11:42 +0000
+++ utah/client/runner.py	2013-04-29 21:07:35 +0000
@@ -73,86 +73,71 @@
     status = "NOTRUN"
     utah_exec_path = '/usr/bin/utah'
 
-    TESTSUITE_ENTRY_SCHEMA = {
-        'type': 'object',
-        'properties': {
-            'name': {
-                'type': 'string',
-                'required': True,
-            },
-            'fetch_method': {
-                'type': 'string',
-                'enum': ['bzr', 'bzr-export', 'dev', 'git'],
-                'required': True,
-            },
-            'fetch_location': {
-                'type': 'string',
-                'required': True,
-            },
-            'include_tests': {
-                'type': 'array',
-                'items': {'type': 'string'},
-            },
-            'exclude_tests': {
-                'type': 'array',
-                'items': {'type': 'string'},
-            },
-        },
-    }
-
-    TESTSUITE_INCLUDE_SCHEMA = {
-        'type': 'object',
-        'properties': {
-            'include': {
-                'type': 'string',
-                'required': True,
-            },
-        },
-    }
-
-    MASTER_RUNLIST_SCHEMA_ORIG = {
-        'type': 'array',
-        'items': {
-            'type': [TESTSUITE_ENTRY_SCHEMA, TESTSUITE_INCLUDE_SCHEMA],
-            'required': True,
-        },
-    }
-
-    MASTER_RUNLIST_SCHEMA_NEW = {
-        'type': 'object',
-        'properties': {
-            "timeout": {
-                "type": "integer",
-                "minimum": 0,
-            },
-            "repeat_count": {
-                "type": "integer",
-                "minimum": 0,
-            },
-            "type": {
-                'type': 'string',
-                'enum': ['smoke', 'kernel-sru', 'bootspeed', 'upgrade'],
-            },
-            "name": {
-                'type': 'string',
-            },
-            "testsuites": {
-                # must be a list to accept a schema rather than a simple type
-                "type": [MASTER_RUNLIST_SCHEMA_ORIG],
-                "required": True,
+    MASTER_RUNLIST_SCHEMA = {
+        'type': 'object',
+        'properties': {
+            'testsuites': {
+                'type': 'array',
+                'items': {
+                    'type': 'object',
+                    'oneOf': [
+                        {'$ref': '#/definitions/testsuite_fetch'},
+                        {'$ref': '#/definitions/testsuite_file'},
+                    ],
+                },
+                'minItems': 1,
             },
             'battery_measurements': {
                 'type': 'boolean',
                 'default': False,
-            }
+            },
+            'timeout': {
+                'type': 'integer',
+                'minimum': 1,
+            },
+            'repeat_count': {
+                'type': 'integer',
+                'minimum': 0,
+                'default': 0,
+            },
+        },
+        'required': ['testsuites'],
+        'additionalProperties': False,
+        'definitions': {
+            'testsuite_fetch': {
+                'type': 'object',
+                'properties': {
+                    'name': {'type': 'string'},
+                    'fetch_method': {
+                        'type': 'string',
+                        'enum': ['dev', 'bzr', 'bzr-export', 'git'],
+                    },
+                    'fetch_location': {'type': 'string'},
+                    'include_tests': {
+                        'type': 'array',
+                        'items': {'type': 'string'},
+                        'minItems': 1,
+                    },
+                    'exclude_tests': {
+                        'type': 'array',
+                        'items': {'type': 'string'},
+                        'minItems': 1,
+                    },
+                },
+                'required': ['name', 'fetch_method', 'fetch_location'],
+                'additionalProperties': False,
+            },
+            'testsuite_file': {
+                'type': 'object',
+                'properties': {
+                    'include': {'type': 'string'},
+                },
+                'required': ['include'],
+                'additionalProperties': False,
+            },
         }
     }
 
-    MASTER_RUNLIST_SCHEMA = {
-        'type': [MASTER_RUNLIST_SCHEMA_ORIG, MASTER_RUNLIST_SCHEMA_NEW],
-        'required': True,
-    }
-
     def __init__(self, install_type, runlist=None, result_class=Result,
                  testdir=UTAH_DIR, state_agent=None,
                  resume=False, old_results=None):
@@ -429,9 +414,9 @@
                 .format(runlist, err))
 
         data = parse_yaml_file(local_filename)
-        validator = DefaultValidator()
+        validator = DefaultValidator(self.MASTER_RUNLIST_SCHEMA)
         try:
-            validator.validate(data, self.MASTER_RUNLIST_SCHEMA)
+            validator.validate(data)
         except jsonschema.ValidationError as exception:
             raise exceptions.ValidationError(
                 'Master runlist failed to validate: {!r}\n'
@@ -446,10 +431,6 @@
         else:
             self.name = 'unnamed'
 
-        # Add publish metatdata to the result object
-        if 'publish' in data:
-            self.result.publish = data['publish']
-
         self.battery_measurements = data['battery_measurements']
 
         seen = []

=== modified file 'utah/client/testcase.py'
--- utah/client/testcase.py	2013-04-15 13:09:55 +0000
+++ utah/client/testcase.py	2013-04-29 21:07:35 +0000
@@ -15,7 +15,6 @@
 
 """Testcase specific code."""
 
-
 import os
 import syslog
 
@@ -31,7 +30,6 @@
     run_cmd,
 )
 from utah.client.exceptions import (
-    MissingData,
     MissingFile,
     ValidationError,
 )
@@ -56,38 +54,24 @@
     type = 'userland'
 
     CONTROL_SCHEMA = {
+        '$schema': 'http://json-schema.org/draft-04/schema#',
         'type': 'object',
         'properties': {
-            'description': {
-                'type': 'string',
-                'required': True,
-            },
-            'dependencies': {
-                'type': 'string',
-                'required': True,
-            },
-            'action': {
-                'type': 'string',
-                'required': True,
-            },
-            'expected_results': {
-                'type': 'string',
-                'required': True,
-            },
+            'description': {'type': 'string'},
+            'dependencies': {'type': 'string'},
+            'action': {'type': 'string'},
+            'expected_results': {'type': 'string'},
             'type': {
                 'type': 'string',
                 'enum': ['userland']
             },
-            'timeout': {'type': 'integer'},
+            'timeout': {
+                'type': 'integer',
+                'minimum': 1,
+            },
             'build_cmd': {'type': 'string'},
-            'command': {
-                'type': 'string',
-                'required': True,
-            },
-            'run_as': {
-                'type': 'string',
-                'required': True,
-            },
+            'command': {'type': 'string'},
+            'run_as': {'type': 'string'},
             'tc_setup': {'type': 'string'},
             'tc_cleanup': {'type': 'string'},
             'reboot': {
@@ -95,6 +79,14 @@
                 'enum': ['always', 'pass', 'never'],
             },
         },
+        'required': [
+            'description',
+            'dependencies',
+            'action',
+            'expected_results',
+            'command',
+        ],
+        'minProperties': 1,
     }
 
     def __init__(self, name, path, result, command=None, timeout=None,
@@ -215,13 +207,6 @@
         if self.is_done():
             return 'PASS'
 
-        # Do not proceed if we are missing data
-        if (self.description is None
-                or self.dependencies is None
-                or self.action is None
-                or self.expected_results is None):
-            raise MissingData
-
         # Return value to indicate whether processing of a TestSuite should
         # continue.  This is to avoid a shutdown race on reboot cases.
         keep_going = True

=== modified file 'utah/client/tests/common.py'
--- utah/client/tests/common.py	2013-04-04 13:42:40 +0000
+++ utah/client/tests/common.py	2013-04-29 21:07:35 +0000
@@ -45,10 +45,6 @@
 master_runlist_content = """# utah/self_test.py master runlist
 # needed for utah/self_test.py runs
 ---
-publish:
-  url: http://dashboard.local/api/add_result/
-  token: testtoken
-  name: precise-server-amd64
 testsuites:
   - name: examples
     fetch_method: bzr-export

=== modified file 'utah/client/tests/test_jsonschema.py'
--- utah/client/tests/test_jsonschema.py	2013-04-04 14:17:14 +0000
+++ utah/client/tests/test_jsonschema.py	2013-04-29 21:07:35 +0000
@@ -25,15 +25,6 @@
 
 
 yaml_content = """---
-- name: testsuite1
-  fetch_method: bzr
-  fetch_location: lp:utah
-- name: testsuite2
-  fetch_method: bzr
-  fetch_location: lp:utah
-"""
-
-yaml_content_new = """---
 timeout: 101
 repeat_count: 99
 testsuites:
@@ -85,15 +76,9 @@
 
     """Test schema validation."""
 
-    def test_orig_schema(self):
-        """Test that the original master.run validates correctly."""
-        data = yaml.load(yaml_content)
-        print("data: {}".format(data))
-        jsonschema.validate(data, Runner.MASTER_RUNLIST_SCHEMA)
-
     def test_multi_schema(self):
         """Test that the new master.run validates correctly."""
-        data_new = yaml.load(yaml_content_new)
+        data_new = yaml.load(yaml_content)
         print("data_new: {}".format(data_new))
         jsonschema.validate(data_new, Runner.MASTER_RUNLIST_SCHEMA)
 

=== modified file 'utah/client/tests/test_runner.py'
--- utah/client/tests/test_runner.py	2013-04-04 16:48:03 +0000
+++ utah/client/tests/test_runner.py	2013-04-29 21:07:35 +0000
@@ -21,10 +21,15 @@
 import unittest
 from mock import patch
 
+import jsonschema
+
 from utah.client.result import ResultYAML
 from utah.client.state_agent import StateAgentYAML
 from utah.client.runner import Runner
-from utah.client.common import ReturnCodes
+from utah.client.common import (
+    DefaultValidator,
+    ReturnCodes,
+)
 from utah.client import exceptions
 
 from utah.client.tests.common import (  # NOQA
@@ -183,3 +188,232 @@
                              ['test_a', 'test_b', 'test_c'])
         self.assertListEqual(kwargs['excludes'],
                              ['test_1', 'test_2', 'test_3'])
+
+
+class TestRunnerMasterRunlistSchema(unittest.TestCase):
+    """Master runlist schema test cases."""
+    def setUp(self):
+        schema = Runner.MASTER_RUNLIST_SCHEMA
+        DefaultValidator.check_schema(schema)
+        self.validator = DefaultValidator(schema)
+
+    def validate(self, data):
+        return self.validator.validate(data)
+
+    def test_empty_string_invalid(self):
+        """An empty string doesn't validate."""
+        with self.assertRaises(jsonschema.ValidationError):
+            self.validate('')
+
+    def test_empty_dict_invalid(self):
+        """An empty dictionary is invalid."""
+        with self.assertRaises(jsonschema.ValidationError):
+            self.validate({})
+
+    def test_valid(self):
+        """An example of valid data."""
+        self.validate({
+            'testsuites': [{
+                'name': 'name',
+                'fetch_method': 'dev',
+                'fetch_location': 'location',
+            }]
+        })
+
+    def test_testsuites_required(self):
+        """A list of test suites is required."""
+        with self.assertRaises(jsonschema.ValidationError):
+            self.validate({
+                'battery_measurements': True,
+            })
+
+    def test_testsuites_empty_invalid(self):
+        """An empty list of test suites is invalid."""
+        with self.assertRaises(jsonschema.ValidationError):
+            self.validate({
+                'testsuites': [],
+            })
+
+    def test_testsuite_name_required(self):
+        """Test suite name is required."""
+        with self.assertRaises(jsonschema.ValidationError):
+            self.validate({
+                'testsuites': [{
+                    'fetch_method': 'dev',
+                    'fetch_location': 'fetch location',
+                }],
+            })
+
+    def test_testsuite_fetch_method_required(self):
+        """Test suite fetch method is required."""
+        with self.assertRaises(jsonschema.ValidationError):
+            self.validate({
+                'testsuites': [{
+                    'name': 'name',
+                    'fetch_location': 'fetch location',
+                }],
+            })
+
+    def test_testsuite_fetch_method_known_values(self):
+        """Test suite fetch method known values are valid."""
+        testsuite = {
+            'name': 'name',
+            'fetch_location': 'fetch location',
+        }
+
+        data = {'testsuites': [testsuite]}
+
+        for fetch_method in ['dev', 'bzr', 'bzr-export', 'git']:
+            testsuite['fetch_method'] = fetch_method
+            self.validate(data)
+
+    def test_testsuite_fetch_method_unknown_value(self):
+        """Test suite fetch method unknown values are valid."""
+        with self.assertRaises(jsonschema.ValidationError):
+            self.validate({
+                'testsuites': [{
+                    'name': 'name',
+                    'fetch_method': 'unknown',
+                    'fetch_location': 'fetch location',
+                }],
+            })
+
+    def test_testsuite_fetch_location_required(self):
+        """Test suite fetch location is required."""
+        with self.assertRaises(jsonschema.ValidationError):
+            self.validate({
+                'testsuites': [{
+                    'name': 'name',
+                    'fetch_method': 'dev',
+                }],
+            })
+
+    def test_testsuite_include_tests_empty_invalid(self):
+        """Empty list of included tests is invalid."""
+        with self.assertRaises(jsonschema.ValidationError):
+            self.validate({
+                'testsuites': [{
+                    'name': 'name',
+                    'fetch_method': 'dev',
+                    'fetch_location': 'fetch location',
+                    'include_tests': [],
+                }],
+            })
+
+    def test_testsuite_include_tests_string(self):
+        """List of strings as included tests is valid."""
+        self.validate({
+            'testsuites': [{
+                'name': 'name',
+                'fetch_method': 'dev',
+                'fetch_location': 'fetch location',
+                'include_tests': ['test_1', 'test_2', 'test_3'],
+            }],
+        })
+
+    def test_testsuite_exclude_tests_empty_invalid(self):
+        """Empty list of excluded tests is invalid."""
+        with self.assertRaises(jsonschema.ValidationError):
+            self.validate({
+                'testsuites': [{
+                    'name': 'name',
+                    'fetch_method': 'dev',
+                    'fetch_location': 'fetch location',
+                    'exclude_tests': [],
+                }],
+            })
+
+    def test_testsuite_exclude_tests_string(self):
+        """List of strings as excluded tests is valid."""
+        self.validate({
+            'testsuites': [{
+                'name': 'name',
+                'fetch_method': 'dev',
+                'fetch_location': 'fetch location',
+                'exclude_tests': ['test_1', 'test_2', 'test_3'],
+            }],
+        })
+
+    def test_battery_measurements_false_by_default(self):
+        """Battery measurements is false by default"""
+        data = {
+            'testsuites': [{
+                'name': 'name',
+                'fetch_method': 'dev',
+                'fetch_location': 'fetch location',
+            }],
+        }
+        self.validate(data)
+        self.assertFalse(data['battery_measurements'])
+
+    def test_battery_measurements_string_invalid(self):
+        """A battery measurements string value is invalid"""
+        with self.assertRaises(jsonschema.ValidationError):
+            self.validate({
+                'testsuites': [{
+                    'name': 'name',
+                    'fetch_method': 'dev',
+                    'fetch_location': 'fetch location',
+                }],
+                'battery_measurements': 'value',
+            })
+
+    def test_battery_measurements_boolean_valid(self):
+        """A battery measurements boolean value is valid"""
+        self.validate({
+            'testsuites': [{
+                'name': 'name',
+                'fetch_method': 'dev',
+                'fetch_location': 'fetch location',
+            }],
+            'battery_measurements': True,
+        })
+
+    def test_zero_timeout_invalid(self):
+        """A zero timeout is invalid"""
+        with self.assertRaises(jsonschema.ValidationError):
+            self.validate({
+                'testsuites': [{
+                    'name': 'name',
+                    'fetch_method': 'dev',
+                    'fetch_location': 'fetch location',
+                }],
+                'timeout': 0,
+            })
+
+    def test_negative_timeout_invalid(self):
+        """A negative timeout is invalid"""
+        with self.assertRaises(jsonschema.ValidationError):
+            self.validate({
+                'testsuites': [{
+                    'name': 'name',
+                    'fetch_method': 'dev',
+                    'fetch_location': 'fetch location',
+                }],
+                'timeout': -1,
+            })
+
+    def test_repeat_count_zero_by_default(self):
+        """Repeat count is zero by default"""
+        data = {
+            'testsuites': [{
+                'name': 'name',
+                'fetch_method': 'dev',
+                'fetch_location': 'fetch location',
+            }],
+        }
+
+        self.validate(data)
+        self.assertEqual(data['repeat_count'], 0)
+
+    def test_negative_repeat_count_invalid(self):
+        """A negative repeat count is invalid"""
+        with self.assertRaises(jsonschema.ValidationError):
+            self.validate({
+                'testsuites': [{
+                    'name': 'name',
+                    'fetch_method': 'dev',
+                    'fetch_location': 'fetch location',
+                }],
+                'repeat_count': -1,
+            })

=== modified file 'utah/client/tests/test_testcase.py'
--- utah/client/tests/test_testcase.py	2013-04-08 19:12:25 +0000
+++ utah/client/tests/test_testcase.py	2013-04-29 21:07:35 +0000
@@ -19,11 +19,13 @@
 import os
 import unittest
 
+import jsonschema
+
 from utah.client import testcase
 from utah.client.common import (
+    DefaultValidator,
     UTAH_DIR,
 )
-from utah.client.exceptions import MissingData
 from utah.client.result import ResultYAML
 from utah.client.tests.common import (  # NOQA
     setUp,  # Used by nosetests
@@ -139,21 +141,169 @@
 
         self.assertTrue(case.is_done())
 
-    def test_bad_control_data(self):
-        """Test that bad control_data does raise an exception."""
-        control_data = {
-            'description': 'a test case',
-            'command': '/bin/true',
+
+class TestTestCaseControlSchema(unittest.TestCase):
+    """Test case control schema test cases."""
+    def setUp(self,):
+        schema = testcase.TestCase.CONTROL_SCHEMA
+        DefaultValidator.check_schema(schema)
+        self.validator = DefaultValidator(schema)
+
+    def validate(self, data):
+        schema = testcase.TestCase.CONTROL_SCHEMA
+        return jsonschema.validate(data, schema)
+
+    def test_empty_string_invalid(self):
+        """An empty string doesn't validate."""
+        with self.assertRaises(jsonschema.ValidationError):
+            self.validate('')
+
+    def test_empty_dict_invalid(self):
+        """An empty dictionary is valid."""
+        with self.assertRaises(jsonschema.ValidationError):
+            self.validate({})
+
+    def test_valid(self):
+        """An example of valid data."""
+        self.validate({
+            'description': 'description',
+            'dependencies': 'dependencies',
+            'action': 'action',
+            'expected_results': 'expected_results',
+            'command': 'command',
+            'run_as': 'run_as',
+        })
+
+    def test_description_required(self):
+        """Description is a required property."""
+        with self.assertRaises(jsonschema.ValidationError):
+            self.validate({
+                'dependencies': 'dependencies',
+                'action': 'action',
+                'expected_results': 'expected_results',
+                'command': 'command',
+                'run_as': 'run_as',
+            })
+
+    def test_dependencies_required(self):
+        """Dependencies is a required property."""
+        with self.assertRaises(jsonschema.ValidationError):
+            self.validate({
+                'description': 'description',
+                'action': 'action',
+                'expected_results': 'expected_results',
+                'command': 'command',
+                'run_as': 'run_as',
+            })
+
+    def test_action_required(self):
+        """Action is a required property."""
+        with self.assertRaises(jsonschema.ValidationError):
+            self.validate({
+                'description': 'description',
+                'dependencies': 'dependencies',
+                'expected_results': 'expected_results',
+                'command': 'command',
+                'run_as': 'run_as',
+            })
+
+    def test_expected_results_required(self):
+        """Expected results is a required property."""
+        with self.assertRaises(jsonschema.ValidationError):
+            self.validate({
+                'description': 'description',
+                'dependencies': 'dependencies',
+                'action': 'action',
+                'command': 'command',
+                'run_as': 'run_as',
+            })
+
+    def test_command_required(self):
+        """Command is a required property."""
+        with self.assertRaises(jsonschema.ValidationError):
+            self.validate({
+                'description': 'description',
+                'dependencies': 'dependencies',
+                'action': 'action',
+                'expected_results': 'expected_results',
+                'run_as': 'run_as',
+            })
+
+    def test_zero_timeout_invalid(self):
+        """A zero timeout is invalid"""
+        with self.assertRaises(jsonschema.ValidationError):
+            self.validate({
+                'description': 'description',
+                'dependencies': 'dependencies',
+                'action': 'action',
+                'expected_results': 'expected_results',
+                'command': 'command',
+                'run_as': 'run_as',
+                'timeout': 0,
+            })
+
+    def test_negative_timeout_invalid(self):
+        """A negative timeout is invalid"""
+        with self.assertRaises(jsonschema.ValidationError):
+            self.validate({
+                'description': 'description',
+                'dependencies': 'dependencies',
+                'action': 'action',
+                'expected_results': 'expected_results',
+                'command': 'command',
+                'run_as': 'run_as',
+                'timeout': -1
+            })
+
+    def test_type_userland_valid(self):
+        """Userland type is valid."""
+        self.validate({
+            'description': 'description',
+            'dependencies': 'dependencies',
+            'action': 'action',
+            'expected_results': 'expected_results',
+            'command': 'command',
+            'run_as': 'run_as',
+            'type': 'userland',
+        })
+
+    def test_type_unknown_invalid(self):
+        """Unknown type is valid."""
+        with self.assertRaises(jsonschema.ValidationError):
+            self.validate({
+                'description': 'description',
+                'dependencies': 'dependencies',
+                'action': 'action',
+                'expected_results': 'expected_results',
+                'command': 'command',
+                'run_as': 'run_as',
+                'type': 'unknown',
+            })
+
+    def test_reboot_known_valid(self):
+        """Reboot known values are valid."""
+        data = {
+            'description': 'description',
+            'dependencies': 'dependencies',
+            'action': 'action',
+            'expected_results': 'expected_results',
+            'command': 'command',
+            'run_as': 'run_as',
         }
 
-        result = ResultYAML()
-
-        case = testcase.TestCase(
-            name=self.name,
-            path=self.path,
-            result=result,
-            _control_data=control_data,
-        )
-
-        with self.assertRaises(MissingData):
-            case.run()
+        for reboot in ['always', 'pass', 'never']:
+            data['reboot'] = reboot
+            self.validate(data)
+
+    def test_unknown_reboot_invalid(self):
+        """Reboot unknown value is invalid."""
+        with self.assertRaises(jsonschema.ValidationError):
+            self.validate({
+                'description': 'description',
+                'dependencies': 'dependencies',
+                'action': 'action',
+                'expected_results': 'expected_results',
+                'command': 'command',
+                'run_as': 'run_as',
+                'reboot': 'unknown',
+            })

=== modified file 'utah/client/tests/test_testsuite.py'
--- utah/client/tests/test_testsuite.py	2013-04-04 17:07:22 +0000
+++ utah/client/tests/test_testsuite.py	2013-04-29 21:07:35 +0000
@@ -19,11 +19,14 @@
 import os
 import unittest
 
+import jsonschema
+
+from utah.client import testsuite
 from utah.client.common import (
     debug_print,
+    DefaultValidator,
 )
 from utah.client.result import ResultYAML
-from utah.client import testsuite
 
 from utah.client.tests.common import (  # NOQA
     setUp,  # Used by nosetests
@@ -157,3 +160,134 @@
         case = suite.tests[0]
         self.assertEqual(case.name, included_test_name)
         self.assertNotEqual(case.name, excluded_test_name)
+
+
+class TestTestSuiteRunlistSchema(unittest.TestCase):
+    """Test suite run list schema test cases."""
+
+    def setUp(self):
+        schema = testsuite.TestSuite.RUNLIST_SCHEMA
+        DefaultValidator.check_schema(schema)
+        self.validator = DefaultValidator(schema)
+
+    def validate(self, data):
+        return self.validator.validate(data)
+
+    def test_empty_string_invalid(self):
+        """An empty string doesn't validate."""
+        with self.assertRaises(jsonschema.ValidationError):
+            self.validate('')
+
+    def test_empty_dict_invalid(self):
+        """An empty dictionary doesn't validate."""
+        with self.assertRaises(jsonschema.ValidationError):
+            self.validate({})
+
+    def test_empty_list_invalid(self):
+        """An empty list doesn't validate."""
+        with self.assertRaises(jsonschema.ValidationError):
+            self.validate([])
+
+    def test_string_list_invalid(self):
+        """A list of strings is invalid."""
+        with self.assertRaises(jsonschema.ValidationError):
+            self.validate(['a', 'b', 'c'])
+
+    def test_empty_dict_list_invalid(self):
+        """A list of empty dictionaries is invalid."""
+        with self.assertRaises(jsonschema.ValidationError):
+            self.validate([{}, {}, {}])
+
+    def test_unknown_test_property_invalid(self):
+        """An unknown property in a test object invalid."""
+        with self.assertRaises(jsonschema.ValidationError):
+            self.validate(
+                [{'test': 'test name',
+                  'unknown-property': 'value',
+                  }])
+
+    def test_empty_overrides_invalid(self):
+        """An empty overrides section is invalid."""
+        with self.assertRaises(jsonschema.ValidationError):
+            self.validate([{'test': 'test name',
+                            'overrides': {}}])
+
+    def test_unknown_property_in_overrides_invalid(self):
+        """An unknown property in overrides section is invalid."""
+        with self.assertRaises(jsonschema.ValidationError):
+            self.validate(
+                [{'test': 'test name',
+                  'overrides': {'unknown-property': 'value'}
+                  }])
+
+    def test_zero_timeout_invalid(self):
+        """A zero timeout is invalid"""
+        with self.assertRaises(jsonschema.ValidationError):
+            self.validate([{'test': 'test name',
+                            'overrides': {'timeout': 0}}])
+
+    def test_negative_timeout_invalid(self):
+        """A negative timeout is invalid"""
+        with self.assertRaises(jsonschema.ValidationError):
+            self.validate([{'test': 'test name',
+                            'overrides': {'timeout': -1}}])
+
+    def test_list_valid(self):
+        """A list of test cases is valid."""
+        self.validate(
+            [{'test': 'test name'},
+             {'test': 'test name 2'}])
+
+    def test_with_overrides_list_valid(self):
+        """A list of test case names with overrides is valid."""
+        self.validate([
+            {'test': 'test name',
+             'overrides': {'timeout': 1},
+             },
+            {'test': 'test name 2',
+             'overrides': {'timeout': 1}
+             },
+        ])
+
+
+class TestTestSuiteControlSchema(unittest.TestCase):
+    """Test suite control schema test cases."""
+    def setUp(self):
+        schema = testsuite.TestSuite.CONTROL_SCHEMA
+        DefaultValidator.check_schema(schema)
+        self.validator = DefaultValidator(schema)
+
+    def validate(self, data):
+        return self.validator.validate(data)
+
+    def test_empty_string_invalid(self):
+        """An empty string doesn't validate."""
+        with self.assertRaises(jsonschema.ValidationError):
+            self.validate('')
+
+    def test_empty_dict_valid(self):
+        """An empty dictionary is valid."""
+        self.validate({})
+
+    def test_unknown_property_invalid(self):
+        """An unknown control property is invalid."""
+        with self.assertRaises(jsonschema.ValidationError):
+            self.validate({'unknown-property': 'value'})
+
+    def test_zero_timeout_invalid(self):
+        """A zero timeout is invalid."""
+        with self.assertRaises(jsonschema.ValidationError):
+            self.validate({'timeout': 0})
+
+    def test_negative_timeout_invalid(self):
+        """A negative timeout is invalid."""
+        with self.assertRaises(jsonschema.ValidationError):
+            self.validate({'timeout': -1})
+
+    def test_valid(self):
+        """An example of valid control file contents."""
+        self.validate({'build_cmd': 'command',
+                       'ts_setup': 'command',
+                       'ts_cleanup': 'command',
+                       'timeout': 1,
+                       })

=== modified file 'utah/client/testsuite.py'
--- utah/client/testsuite.py	2013-04-15 13:09:55 +0000
+++ utah/client/testsuite.py	2013-04-29 21:07:35 +0000
@@ -65,27 +65,48 @@
     save_state_callback = do_nothing
 
     RUNLIST_SCHEMA = {
+        '$schema': 'http://json-schema.org/draft-04/schema#',
         'type': 'array',
+        'minItems': 1,
         'items': {
             'type': 'object',
             'properties': {
                 'test': {
                     'type': 'string',
-                    'required': True,
-                },
-            'overrides': {'type': 'object'},
+                },
+                'overrides': {
+                    'type': 'object',
+                    'properties': {
+                        'timeout': {
+                            'type': 'integer',
+                            'minimum': 1,
+                        },
+                        'build_cmd': {'type': 'string'},
+                        'command': {'type': 'string'},
+                        'run_as': {'type': 'string'},
+                    },
+                    'minProperties': 1,
+                    'additionalProperties': False,
+                },
             },
+            'required': ['test'],
+            'additionalProperties': False,
         },
     }
 
     CONTROL_SCHEMA = {
+        '$schema': 'http://json-schema.org/draft-04/schema#',
         'type': 'object',
         'properties': {
+            'timeout': {
+                'type': 'integer',
+                'minimum': 1,
+            },
             'build_cmd': {'type': 'string'},
-            'timeout': {'type': 'integer'},
             'ts_setup': {'type': 'string'},
             'ts_cleanup': {'type': 'string'},
-        }
+        },
+        'additionalProperties': False,
     }
 
     def __init__(self, name, path, result, runlist_file=DEFAULT_TSLIST,