Merge lp:~milo/lava-tool/lava-169 into lp:~linaro-validation/lava-tool/trunk

[Milo Casagrande]

This is the final step in completing card LAVA-15.
Here there is the "status" command that polls a LAVA server and prints information about the provided job id.

The command can be invoked as:

lava status [ID]
lava job status [ID]

The latter is the actual command and the former just calls it.
If ID is not provided, it will read the stored job IDs from the .lavaconfig file, and ask the user which job to choose.

[lp:~milo/lava-tool/lava-169]

412. By Milo Casagrande on 2013-07-19

Added test for public Config method.

413. By Milo Casagrande on 2013-07-19

Fixed missing argument in method call.

414. By Milo Casagrande on 2013-07-19

Removed a job id from the config file once it is not running anymore.

415. By Milo Casagrande on 2013-07-19

Fixed logic when removing job ids.

416. By Milo Casagrande on 2013-07-22

Merged from parent branch.

417. By Milo Casagrande on 2013-07-22

Merged from parent branch.

418. By Milo Casagrande on 2013-07-22

Fixed logic for URL creation.

419. By Milo Casagrande on 2013-07-22

Fixed problem with value read from config and not deserialized.

420. By Milo Casagrande on 2013-07-24

Merged from parent.

421. By Milo Casagrande on 2013-07-24

Fixed test.

[Antonio Terceiro (terceiro)]

Another great job here, thanks! I have some comments below.

 review needs-fixing

On Thu, Jul 18, 2013 at 04:04:26PM -0000, Milo Casagrande wrote:
> Milo Casagrande has proposed merging lp:~milo/lava-tool/lava-169 into lp:lava-tool with lp:~milo/lava-tool/lava-165 as a prerequisite.
>
> Requested reviews:
> Antonio Terceiro (terceiro)
> Linaro Validation Team (linaro-validation)
>
> For more details, see:
> https://code.launchpad.net/~milo/lava-tool/lava-169/+merge/175622
>
> This is the final step in completing card LAVA-15.
> Here there is the "status" command that polls a LAVA server and prints information about the provided job id.
>
> The command can be invoked as:
>
> lava status [ID]
> lava job status [ID]
>
> The latter is the actual command and the former just calls it.

I'm not sure about the first form. `lava status` might be too generic for
people to memorize that it should be use to get status of jobs. I think we will
be able to do a better UI review after we get all the code merged in though.

> If ID is not provided, it will read the stored job IDs from the .lavaconfig file, and ask the user which job to choose.

(what follows is an updated diff from a few hours ago, not the original diff
from when the MP was created)

> === modified file 'entry_points.ini'
> --- entry_points.ini 2013-07-24 14:41:29 +0000
> +++ entry_points.ini 2013-07-24 14:41:29 +0000
> @@ -13,6 +13,7 @@
> init = lava.commands:init
> submit = lava.commands:submit
> run = lava.commands:run
> +status = lava.commands:status

you can just reference lava.job.commands:status here, you don't need to write a
separate command.

BTW this also applies to a similar comment I did in the previous MP, about
cascading commands: if the arguments are the same, you can just point to the
original command from another scope in entry_points.ini and you don't have to
wrap one command inside another

> update = lava.commands:update
>
> [lava_tool.commands]
> @@ -77,6 +78,7 @@
> [lava.job.commands]
> new = lava.job.commands:new
> submit = lava.job.commands:submit
> +status = lava.job.commands:status
> run = lava.job.commands:run
>
> [lava.device.commands]
>
> === modified file 'lava/commands.py'
> --- lava/commands.py 2013-07-24 14:41:29 +0000
> +++ lava/commands.py 2013-07-24 14:41:29 +0000
> @@ -251,6 +251,29 @@
> submit_cmd.invoke()
>
>
> +class status(BaseCommand):
> +
> + """Checks the status of a job."""
> +
> + @classmethod
> + def register_arguments(cls, parser):
> + super(status, cls).register_arguments(parser)
> + parser.add_argument("JOB_ID",
> + help=("Prints a job status information."),
> + nargs="?",
> + default="-1")
> +
> + def invoke(self):
> + """Runs the job.status command."""
> + from lava.job.commands import status as job_status
> +
> + args = copy.copy(self.args)
> + args.JOB_ID = self.args.JOB_ID
> +
> + status_cmd = job_status(self.parser, args)
> + status_cmd.invoke()
> +
> +
> class update(BaseCommand):
> """Updates a job file with the correct data."""

ditto - this s...

review: Needs Fixing

[lp:~milo/lava-tool/lava-169]

422. By Milo Casagrande on 2013-07-25

Merged from parent branch.

423. By Milo Casagrande on 2013-07-25

Merged from parent branch.

424. By Milo Casagrande on 2013-07-25

Merged from parent branch, fixed conflicts.

425. By Milo Casagrande on 2013-07-25

Merged from parent brach.

426. By Milo Casagrande on 2013-07-25

Merged from parent branch.

427. By Milo Casagrande on 2013-07-25

Merged from parent branch.

428. By Milo Casagrande on 2013-07-25

Merged from parent.

[Milo Casagrande (milo)]

Hi Antonio,

I updated the code.
I also pushed here the "script" commands. Please, take a look at them.

The big part of the refactoring happened on the lava-165 merge.

On Wed, Jul 24, 2013 at 10:46 PM, Antonio Terceiro
<email address hidden> wrote:
>
> I'm not sure about the first form. `lava status` might be too generic for
> people to memorize that it should be use to get status of jobs. I think we will
> be able to do a better UI review after we get all the code merged in though.

Indeed. Also, it would be better to do it with someone that will
actually use the tool.

>> === modified file 'entry_points.ini'
>> --- entry_points.ini 2013-07-24 14:41:29 +0000
>> +++ entry_points.ini 2013-07-24 14:41:29 +0000
>> @@ -13,6 +13,7 @@
>> init = lava.commands:init
>> submit = lava.commands:submit
>> run = lava.commands:run
>> +status = lava.commands:status
>
> you can just reference lava.job.commands:status here, you don't need to write a
> separate command.
>
> BTW this also applies to a similar comment I did in the previous MP, about
> cascading commands: if the arguments are the same, you can just point to the
> original command from another scope in entry_points.ini and you don't have to
> wrap one command inside another

There is actually only the status one, the run and submit are all a
little bit different, in particular when you start from a testdef or a
script file.

> these constants are used in very few places so I think it's better to just use
> the literals directly.

Cleared them out.

> I would use default=None instead, since that -1 is not even a proper number.

Fixed.

> instead of printing the Bundle SHA1 hash, it would be more useful to provide a
> URL to that bundle in the dashboard. Even more useful would be to present the
> results of the tests ... I don't know exactly yet how to do any of those,
> though - presenting the URL should be a lot easier in the short term.

> maybe we could provide a way of listing the latest N jobs submitted by the
> user, together with their results. This would probably require some work on the
> server API side.

I actually removed this from both lava-165 and lava-169.
I want to address that with another merge proposal, but after
everything has been merged.

Will file a bug to keep track of it, and fix the config/cache
"problem" via pyxdg once this code is landed.

Also for the "bundle" one: I prefer to not fix it now, but keep track
of it with a bug. It is tied to the config/cache "problem" since we
need to store information of that job to make the URL (we need to keep
track of the server URL, the stream, and those might change even from
job to job).

--
Milo Casagrande | Automation Engineer
Linaro.org <www.linaro.org> │ Open source software for ARM SoCs

[lp:~milo/lava-tool/lava-169]

429. By Milo Casagrande on 2013-07-25

Merged from parent.

430. By Milo Casagrande on 2013-07-25

Added script commands, completed testdef ones.

    * Refactored and fixes some of the other methods.

431. By Milo Casagrande on 2013-07-25

Added missing entry point commands.

432. By Milo Casagrande on 2013-07-25

Removed TODO, unused import.

433. By Milo Casagrande on 2013-07-25

Fixed string.

434. By Milo Casagrande on 2013-07-25

Fixed the status command.

435. By Milo Casagrande on 2013-07-25

Fixed script entry point.

436. By Milo Casagrande on 2013-07-25

Fixed import.

437. By Milo Casagrande on 2013-07-25

Fixed another import.

438. By Milo Casagrande on 2013-07-25

Fixed template.

439. By Milo Casagrande on 2013-07-26

Fixed error: used def instead of class.

440. By Milo Casagrande on 2013-07-26

PEP8 and pylint fixes.

441. By Milo Casagrande on 2013-07-26

Added qemu device type.

442. By Milo Casagrande on 2013-07-26

Completed script and testdef run and submit commands.

    * Refactored some code.

443. By Milo Casagrande on 2013-07-26

Removed refactored method, added missing edit_file call.

444. By Milo Casagrande on 2013-07-26

Fixed commands, added missing functionalities.

445. By Milo Casagrande on 2013-07-26

Fixed method call.

446. By Milo Casagrande on 2013-07-26

Fixed problem with config saving.

447. By Milo Casagrande on 2013-07-26

Fixed check on non existing files.

448. By Milo Casagrande on 2013-07-26

Fixed error message.

[Antonio Terceiro (terceiro)]

The changes you did seem nice. I don't have any specific remarks.

Let's go ahead and merge this, it's getting too big. Then we
start iterating on the code with all the features in.

 review approve

review: Approve

=== modified file 'entry_points.ini'
--- entry_points.ini	2013-07-26 14:14:25 +0000
+++ entry_points.ini	2013-07-26 14:14:25 +0000
@@ -13,7 +13,9 @@
 init = lava.commands:init
 submit = lava.commands:submit
 run = lava.commands:run
+status = lava.job.commands:status
 update = lava.commands:update
+script = lava.script.commands:script
 
 [lava_tool.commands]
 help = lava.tool.commands.help:help
@@ -77,6 +79,7 @@
 [lava.job.commands]
 new = lava.job.commands:new
 submit = lava.job.commands:submit
+status = lava.job.commands:status
 run = lava.job.commands:run
 
 [lava.device.commands]
@@ -86,3 +89,9 @@
 
 [lava.testdef.commands]
 new = lava.testdef.commands:new
+run = lava.testdef.commands:run
+submit = lava.testdef.commands:submit
+
+[lava.script.commands]
+run = lava.script.commands:run
+submit = lava.script.commands:submit

=== modified file 'lava/commands.py'
--- lava/commands.py	2013-07-26 14:14:25 +0000
+++ lava/commands.py	2013-07-26 14:14:25 +0000
@@ -46,7 +46,6 @@
 import copy
 import json
 import os
-import stat
 import sys
 
 from lava.helper.command import BaseCommand
@@ -63,16 +62,16 @@
 from lava.parameter import (
     Parameter,
 )
-from lava.testdef.templates import (
-    DEFAULT_TESTDEF_FILE,
-    DEFAULT_TESTDEF_SCRIPT,
-    DEFAULT_TESTDEF_SCRIPT_CONTENT,
+from lava.testdef import (
+    DEFAULT_TESTDEF_FILENAME,
 )
 from lava.tool.errors import CommandError
 from lava_tool.utils import (
+    base64_encode,
+    create_dir,
+    create_tar,
     edit_file,
     retrieve_file,
-    create_dir,
     write_file,
 )
 
@@ -110,15 +109,17 @@
                                "file.".format(self.args.DIR))
 
         create_dir(full_path)
-
         data = self._update_data()
 
+        # Create the directory that will contain the test definition and
+        # shell script.
         test_path = create_dir(full_path, TESTS_DIR)
-        # TODO
-        self._create_script(test_path)
+        shell_script = self.create_shell_script(test_path)
+        # Let the user modify the file.
+        edit_file(shell_script)
 
         testdef_file = self.create_test_definition(
-            os.path.join(test_path, DEFAULT_TESTDEF_FILE))
+            os.path.join(test_path, DEFAULT_TESTDEF_FILENAME))
 
         job = data[JOBFILE_ID]
         self.create_tar_repo_job(
@@ -137,25 +138,6 @@
 
         return data
 
-    def _create_script(self, test_path):
-        # This is the default script file as defined in the testdef template.
-        default_script = os.path.join(test_path, DEFAULT_TESTDEF_SCRIPT)
-
-        if not os.path.isfile(default_script):
-            # We do not have the default testdef script. Create it, but
-            # remind the user to update it.
-            print >> sys.stdout, ("\nCreating default test script "
-                                  "'{0}'.".format(DEFAULT_TESTDEF_SCRIPT))
-
-            write_file(default_script, DEFAULT_TESTDEF_SCRIPT_CONTENT)
-
-        # Prompt the user to write the script file.
-        edit_file(default_script)
-
-        # Make sure the script is executable.
-        os.chmod(default_script,
-                 stat.S_IRWXU | stat.S_IRWXG | stat.S_IROTH | stat.S_IXOTH)
-
 
 class run(BaseCommand):
     """Runs a job on the local dispatcher."""
@@ -194,7 +176,7 @@
 
     def invoke(self):
         full_path = os.path.abspath(self.args.JOB)
-        job_file = self.retrieve_file(full_path, JOB_FILE_EXTENSIONS)
+        job_file = retrieve_file(full_path, JOB_FILE_EXTENSIONS)
 
         super(submit, self).submit(job_file)
 
@@ -219,22 +201,27 @@
         tests_dir = os.path.join(job_dir, TESTS_DIR)
 
         if os.path.isdir(tests_dir):
-            # TODO
-            encoded_tests = None
-
-            json_data = None
-            with open(job_file, "r") as json_file:
-                try:
-                    json_data = json.load(json_file)
-                    set_value(
-                        json_data, LAVA_TEST_SHELL_TAR_REPO_KEY, encoded_tests)
-                except Exception:
-                    raise CommandError("Cannot read job file '{0}'.".format(
-                        job_file))
-
-            content = json.dumps(json_data, indent=4)
-            write_file(job_file, content)
-
-            print >> sys.stdout, "Job definition updated."
+            tar_repo = None
+            try:
+                tar_repo = create_tar(tests_dir)
+                encoded_tests = base64_encode(tar_repo)
+
+                json_data = None
+                with open(job_file, "r") as json_file:
+                    try:
+                        json_data = json.load(json_file)
+                        set_value(json_data, LAVA_TEST_SHELL_TAR_REPO_KEY,
+                                  encoded_tests)
+                    except Exception:
+                        raise CommandError("Cannot read job file "
+                                           "'{0}'.".format(job_file))
+
+                content = json.dumps(json_data, indent=4)
+                write_file(job_file, content)
+
+                print >> sys.stdout, "Job definition updated."
+            finally:
+                if tar_repo and os.path.isfile(tar_repo):
+                    os.unlink(tar_repo)
         else:
             raise CommandError("Cannot find tests directory.")

=== modified file 'lava/config.py'
--- lava/config.py	2013-07-26 14:14:25 +0000
+++ lava/config.py	2013-07-26 14:14:25 +0000
@@ -33,7 +33,6 @@
 from lava.parameter import Parameter
 from lava.tool.errors import CommandError
 
-
 __all__ = ['Config', 'InteractiveConfig']
 
 # Store for function calls to be made at exit time.
@@ -55,6 +54,7 @@
         call()
 atexit.register(_run_at_exit)
 
+
 class Config(object):
     """A generic config object."""
 
@@ -119,6 +119,12 @@
             value = self._get_from_backend(parameter, section)
         return value
 
+    def get_from_backend(self, parameter, section=None):
+        """Gets a configuration parameter directly from the config file."""
+        if not section:
+            section = self._calculate_config_section(parameter)
+        return self._get_from_backend(parameter, section)
+
     def _get_from_backend(self, parameter, section):
         """Gets the parameter value from the config backend.
 
@@ -200,8 +206,12 @@
 
     def save(self):
         """Saves the config to file."""
-        with open(self.config_file, "w") as write_file:
-            self.config_backend.write(write_file)
+        # Since we lazy load the config_backend property, this check is needed
+        # when a user enters a wrong command or it will overwrite the 'config'
+        # file with empty contents.
+        if self._config_backend:
+            with open(self.config_file, "w") as write_file:
+                self.config_backend.write(write_file)
 
 
 class InteractiveConfig(Config):

=== modified file 'lava/device/__init__.py'
--- lava/device/__init__.py	2013-07-26 14:14:25 +0000
+++ lava/device/__init__.py	2013-07-26 14:14:25 +0000
@@ -16,6 +16,8 @@
 # You should have received a copy of the GNU Lesser General Public License
 # along with lava-tool.  If not, see <http://www.gnu.org/licenses/>.
 
+"""Device class."""
+
 import re
 
 from copy import deepcopy
@@ -46,9 +48,11 @@
 
 
 class Device(object):
+
     """A generic device."""
-    def __init__(self, template, hostname=None):
-        self.data = deepcopy(template)
+
+    def __init__(self, data, hostname=None):
+        self.data = deepcopy(data)
         self.hostname = hostname
 
     def write(self, conf_file):
@@ -86,7 +90,7 @@
     :return A Device instance.
         """
     instance = Device(DEFAULT_TEMPLATE, hostname=name)
-    for known_dev, (matcher, dev_template) in KNOWN_DEVICES.iteritems():
+    for _, (matcher, dev_template) in KNOWN_DEVICES.iteritems():
         if matcher.match(name):
             instance = Device(dev_template, hostname=name)
             break

=== modified file 'lava/device/templates.py'
--- lava/device/templates.py	2013-07-26 14:14:25 +0000
+++ lava/device/templates.py	2013-07-26 14:14:25 +0000
@@ -55,6 +55,13 @@
 VEXPRESS_CONNECTION_COMMAND = copy(CONNECTION_COMMAND_PARMAETER)
 VEXPRESS_CONNECTION_COMMAND.depends = VEXPRESS_DEVICE_TYPE
 
+QEMU_DEVICE_TYPE = copy(DEVICE_TYPE_PARAMETER)
+QEMU_DEVICE_TYPE.value = "qemu"
+QEMU_DEVICE_TYPE.asked = True
+
+QEMU_CONNECTION_COMMAND = copy(CONNECTION_COMMAND_PARMAETER)
+QEMU_CONNECTION_COMMAND.depends = QEMU_DEVICE_TYPE
+
 # Dictionary with templates of known devices.
 KNOWN_TEMPLATES = {
     'panda': {
@@ -67,4 +74,9 @@
         'device_type': VEXPRESS_DEVICE_TYPE,
         'connection_command': VEXPRESS_CONNECTION_COMMAND,
     },
+    'qemu': {
+        'hostname': HOSTNAME_PARAMETER,
+        'device_type': QEMU_DEVICE_TYPE,
+        'connection_command': QEMU_CONNECTION_COMMAND,
+    }
 }

=== modified file 'lava/helper/command.py'
--- lava/helper/command.py	2013-07-26 14:14:25 +0000
+++ lava/helper/command.py	2013-07-26 14:14:25 +0000
@@ -24,10 +24,25 @@
 
 from lava.config import InteractiveConfig
 from lava.helper.dispatcher import get_devices
+from lava.job import Job
+from lava.job.templates import (
+    LAVA_TEST_SHELL_TAR_REPO,
+    LAVA_TEST_SHELL_TAR_REPO_KEY,
+    LAVA_TEST_SHELL_TESDEF_KEY,
+)
 from lava.parameter import (
     Parameter,
     SingleChoiceParameter,
 )
+from lava.script import (
+    ShellScript,
+    DEFAULT_TESTDEF_SCRIPT,
+)
+from lava.testdef import TestDefinition
+from lava.testdef.templates import (
+    TESTDEF_STEPS_KEY,
+    TESTDEF_TEMPLATE,
+)
 from lava.tool.command import Command
 from lava.tool.errors import CommandError
 from lava_tool.authtoken import (
@@ -35,22 +50,13 @@
     KeyringAuthBackend
 )
 from lava_tool.utils import (
+    base64_encode,
+    create_tar,
     has_command,
+    to_list,
     verify_and_create_url,
-    create_tar,
-    base64_encode,
-)
-from lava.job import Job
-from lava.job.templates import (
-    LAVA_TEST_SHELL_TAR_REPO,
-    LAVA_TEST_SHELL_TAR_REPO_KEY,
-    LAVA_TEST_SHELL_TESDEF_KEY,
 )
 
-from lava.testdef import TestDefinition
-from lava.testdef.templates import (
-    TESTDEF_TEMPLATE,
-)
 CONFIG = InteractiveConfig()
 
 
@@ -66,7 +72,7 @@
     @classmethod
     def register_arguments(cls, parser):
         super(BaseCommand, cls).register_arguments(parser)
-        parser.add_argument("--non-interactive",
+        parser.add_argument("--non-interactive", "-n",
                             action='store_false',
                             help=("Do not ask for input parameters."))
 
@@ -76,6 +82,8 @@
         It will ask the user the necessary parameters to establish the
         connection.
         """
+        print >> sys.stdout, "\nServer connection parameters:"
+
         server_name_parameter = Parameter("server")
         rpc_endpoint_parameter = Parameter("rpc_endpoint",
                                            depends=server_name_parameter)
@@ -134,6 +142,26 @@
             raise CommandError("Job file '{0}' does not exists, or it is not "
                                "a file.".format(job_file))
 
+    def status(self, job_id):
+        """Retrieves the status of a LAVA job.
+
+        :param job_id: The ID of the job to look up.
+        """
+        job_id = str(job_id)
+
+        try:
+            server = self.authenticated_server()
+            job_status = server.scheduler.job_status(job_id)
+
+            status = job_status["job_status"].lower()
+            bundle = job_status["bundle_sha1"]
+
+            print >> sys.stdout, "\nJob id: {0}".format(job_id)
+            print >> sys.stdout, "Status: {0}".format(status)
+            print >> sys.stdout, "Bundle: {0}".format(bundle)
+        except xmlrpclib.Fault, exc:
+            raise CommandError(str(exc))
+
     def create_tar_repo_job(self, job_file, testdef_file, tar_content):
         """Creates a job file based on the tar-repo template.
 
@@ -144,6 +172,9 @@
         :param tar_content: What should go into the tarball repository.
         :return The path of the job file created.
         """
+
+        print >> sys.stdout, "\nCreating job file..."
+
         try:
             tar_repo = create_tar(tar_content)
 
@@ -157,22 +188,55 @@
 
             job_instance.write()
 
+            basename = os.path.basename(job_instance.file_name)
+            print >> sys.stdout, ("\nCreated job file "
+                                  "'{0}'.".format(basename))
+
             return job_instance.file_name
         finally:
             if os.path.isfile(tar_repo):
                 os.unlink(tar_repo)
 
-    def create_test_definition(self, testdef_file, template=TESTDEF_TEMPLATE):
+    def create_test_definition(self, testdef_file, template=TESTDEF_TEMPLATE,
+                               steps=None):
         """Creates a test definition YAML file.
 
         :param testdef_file: The file to create.
         :return The path of the file created.
         """
+
+        print >> sys.stdout, "\nCreating test definition file..."
+
         testdef = TestDefinition(template, testdef_file)
+        if steps:
+            steps = to_list(steps)
+            testdef.set(TESTDEF_STEPS_KEY, steps)
         testdef.update(self.config)
         testdef.write()
 
-        print >> sys.stdout, ("Create test definition "
-                              "'{0}'.".format(testdef.file_name))
+        basename = os.path.basename(testdef.file_name)
+        print >> sys.stdout, ("\nCreated test definition "
+                              "'{0}'.".format(basename))
 
         return testdef.file_name
+
+    def create_shell_script(self, test_path,
+                            script_name=DEFAULT_TESTDEF_SCRIPT):
+        """Creates a shell script with some default content.
+
+        :param test_path: The directory where to create the script.
+        :param script_name: The name of the script.
+        :return The full path to the script file.
+        """
+        default_script = os.path.join(test_path, script_name)
+
+        if not os.path.isfile(default_script):
+            print >> sys.stdout, "Creating shell script..."
+
+            shell_script = ShellScript(default_script)
+            shell_script.write()
+
+            print >> sys.stdout, ("\nCreated shell script "
+                                  "'{0}'.".format(script_name))
+
+        return default_script

=== modified file 'lava/job/__init__.py'
--- lava/job/__init__.py	2013-07-26 14:14:25 +0000
+++ lava/job/__init__.py	2013-07-26 14:14:25 +0000
@@ -16,6 +16,8 @@
 # You should have received a copy of the GNU Lesser General Public License
 # along with lava-tool.  If not, see <http://www.gnu.org/licenses/>.
 
+"""Job class."""
+
 import json
 
 from copy import deepcopy
@@ -30,6 +32,8 @@
     write_file
 )
 
+# A default name for job files.
+DEFAULT_JOB_FILENAME = "lava-tool-job.json"
 # Default job file extension.
 DEFAULT_JOB_EXTENSION = "json"
 # Possible extension for a job file.
@@ -37,6 +41,14 @@
 
 
 class Job(object):
+
+    """A Job object.
+
+    This class should be used to create new job files. The initialization
+    enforces a default file name extension, and makes sure that the file is
+    not already present on the file system.
+    """
+
     def __init__(self, data, file_name):
         self.file_name = verify_file_extension(file_name,
                                                DEFAULT_JOB_EXTENSION,

=== modified file 'lava/job/commands.py'
--- lava/job/commands.py	2013-07-26 14:14:25 +0000
+++ lava/job/commands.py	2013-07-26 14:14:25 +0000
@@ -29,6 +29,7 @@
     JOB_TYPES,
 )
 from lava.tool.command import CommandGroup
+from lava.tool.errors import CommandError
 
 
 class job(CommandGroup):
@@ -84,3 +85,23 @@
 
     def invoke(self):
         super(run, self).run(self.args.FILE)
+
+
+class status(BaseCommand):
+
+    """Retrieves the status of a job."""
+
+    @classmethod
+    def register_arguments(cls, parser):
+        super(status, cls).register_arguments(parser)
+        parser.add_argument("JOB_ID",
+                            help=("Prints status information about the "
+                                  "provided job id."),
+                            nargs="?",
+                            default=None)
+
+    def invoke(self):
+        if self.args.JOB_ID:
+            super(status, self).status(self.args.JOB_ID)
+        else:
+            raise CommandError("It is necessary to specify a job id.")

=== modified file 'lava/job/templates.py'
--- lava/job/templates.py	2013-07-26 14:14:25 +0000
+++ lava/job/templates.py	2013-07-26 14:14:25 +0000
@@ -71,7 +71,7 @@
         },
         {
             "command": "submit_results",
-            "parameters" : {
+            "parameters": {
                 "stream": STREAM_PARAMETER,
                 "server": SERVER_PARAMETER
             }
@@ -97,16 +97,16 @@
             "parameters": {
                 "timeout": 1800,
                 "testdef_repos": [
-                        {
-                            LAVA_TEST_SHELL_TESDEF_KEY: None,
-                            LAVA_TEST_SHELL_TAR_REPO_KEY: None,
-                        }
+                    {
+                        LAVA_TEST_SHELL_TESDEF_KEY: None,
+                        LAVA_TEST_SHELL_TAR_REPO_KEY: None,
+                    }
                 ]
             }
         },
         {
             "command": "submit_results",
-            "parameters" : {
+            "parameters": {
                 "stream": STREAM_PARAMETER,
                 "server": SERVER_PARAMETER
             }

=== modified file 'lava/job/tests/test_commands.py'
--- lava/job/tests/test_commands.py	2013-07-26 14:14:25 +0000
+++ lava/job/tests/test_commands.py	2013-07-26 14:14:25 +0000
@@ -23,7 +23,7 @@
 import json
 import os
 
-from mock import MagicMock, patch
+from mock import patch
 
 from lava.config import Config
 from lava.helper.tests.helper_test import HelperTest
@@ -99,9 +99,10 @@
         command = run(self.parser, self.args)
         self.assertRaises(CommandError, command.invoke)
 
-    @patch("lava_tool.utils.has_command", new=MagicMock(return_value=False))
-    def test_invoke_raises_1(self):
-        # Users passes a valid file to the run command, but she does not have
+    @patch("lava.helper.command.has_command", create=True)
+    def test_invoke_raises_1(self, mocked_has_command):
+        # User passes a valid file to the run command, but she does not have
         # the dispatcher installed.
+        mocked_has_command.return_value = False
         command = run(self.parser, self.args)
         self.assertRaises(CommandError, command.invoke)

=== modified file 'lava/parameter.py'
--- lava/parameter.py	2013-07-26 14:14:25 +0000
+++ lava/parameter.py	2013-07-26 14:14:25 +0000
@@ -20,15 +20,9 @@
 Parameter class and its accessory methods/functions.
 """
 
-import StringIO
-import base64
-import os
 import sys
-import tarfile
-import tempfile
 import types
 
-from lava.tool.errors import CommandError
 from lava_tool.utils import to_list
 
 # Character used to join serialized list parameters.

=== added directory 'lava/script'
=== added file 'lava/script/__init__.py'
--- lava/script/__init__.py	1970-01-01 00:00:00 +0000
+++ lava/script/__init__.py	2013-07-26 14:14:25 +0000
@@ -0,0 +1,51 @@
+# Copyright (C) 2013 Linaro Limited
+#
+# Author: Milo Casagrande <[e-mail]>
+#
+# This file is part of lava-tool.
+#
+# lava-tool is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Lesser General Public License version 3
+# as published by the Free Software Foundation
+#
+# lava-tool is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU Lesser General Public License
+# along with lava-tool.  If not, see <http://www.gnu.org/licenses/>.
+
+"""Scripts handling class."""
+
+import os
+import stat
+
+from lava_tool.utils import write_file
+
+
+DEFAULT_MOD = stat.S_IRWXU | stat.S_IRWXG | stat.S_IROTH | stat.S_IXOTH
+DEFAULT_TESTDEF_SCRIPT_CONTENT = """#!/bin/sh
+# Automatic generated content by lava-tool.
+# Please add your own instructions.
+#
+# You can use all the avialable Bash commands.
+#
+# For the available LAVA commands, see:
+#    http://lava.readthedocs.org/
+#
+"""
+DEFAULT_TESTDEF_SCRIPT = "mytest.sh"
+
+
+class ShellScript(object):
+
+    """Creates a shell script on the file system with some content."""
+
+    def __init__(self, file_name):
+        self.file_name = file_name
+
+    def write(self):
+        write_file(self.file_name, DEFAULT_TESTDEF_SCRIPT_CONTENT)
+        # Make sure the script is executable.
+        os.chmod(self.file_name, DEFAULT_MOD)

=== added file 'lava/script/commands.py'
--- lava/script/commands.py	1970-01-01 00:00:00 +0000
+++ lava/script/commands.py	2013-07-26 14:14:25 +0000
@@ -0,0 +1,115 @@
+# Copyright (C) 2013 Linaro Limited
+#
+# Author: Milo Casagrande <[e-mail]>
+#
+# This file is part of lava-tool.
+#
+# lava-tool is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Lesser General Public License version 3
+# as published by the Free Software Foundation
+#
+# lava-tool is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU Lesser General Public License
+# along with lava-tool.  If not, see <http://www.gnu.org/licenses/>.
+
+"""Commands to run or submit a script."""
+
+import os
+import tempfile
+
+from lava.helper.command import BaseCommand
+from lava.job import DEFAULT_JOB_FILENAME
+from lava.testdef import DEFAULT_TESTDEF_FILENAME
+from lava.tool.command import CommandGroup
+from lava_tool.utils import verify_path_non_existance
+
+
+class script(CommandGroup):
+
+    """LAVA script file handling."""
+
+    namespace = "lava.script.commands"
+
+
+class ScriptBaseCommand(BaseCommand):
+
+    def _create_tmp_job_file(self, script_file):
+        """Creates a temporary job file to run or submit the passed file.
+
+        The temporary job file and its accessory test definition file are
+        not removed by this method.
+
+        :param script_file: The script file that has to be run or submitted.
+        :return A tuple with the job file path, and the test definition path.
+        """
+        script_file = os.path.abspath(script_file)
+        verify_path_non_existance(script_file)
+
+        temp_dir = tempfile.gettempdir()
+
+        # The name of the job and testdef files.
+        job_file = os.path.join(temp_dir, DEFAULT_JOB_FILENAME)
+        testdef_file = os.path.join(temp_dir, DEFAULT_TESTDEF_FILENAME)
+
+        # The steps that the testdef file should have. We need to change it
+        # from the default one, since the users are passing their own file.
+        steps = "./" + os.path.basename(script_file)
+        testdef_file = self.create_test_definition(testdef_file,
+                                                   steps=steps)
+
+        # The content of the tar file.
+        tar_content = [script_file, testdef_file]
+        job_file = self.create_tar_repo_job(job_file, testdef_file,
+                                            tar_content)
+
+        return (job_file, testdef_file)
+
+
+class run(ScriptBaseCommand):
+
+    """Runs the specified shell script on a local device."""
+
+    @classmethod
+    def register_arguments(cls, parser):
+        super(run, cls).register_arguments(parser)
+        parser.add_argument("FILE", help="Shell script file to run.")
+
+    def invoke(self):
+        job_file = ""
+        testdef_file = ""
+
+        try:
+            job_file, testdef_file = self._create_tmp_job_file(self.args.FILE)
+            super(run, self).run(job_file)
+        finally:
+            if os.path.isfile(job_file):
+                os.unlink(job_file)
+            if os.path.isfile(testdef_file):
+                os.unlink(testdef_file)
+
+
+class submit(ScriptBaseCommand):
+
+    """Submits the specified shell script to a LAVA server."""
+
+    @classmethod
+    def register_arguments(cls, parser):
+        super(submit, cls).register_arguments(parser)
+        parser.add_argument("FILE", help="Shell script file to send.")
+
+    def invoke(self):
+        job_file = ""
+        testdef_file = ""
+
+        try:
+            job_file, testdef_file = self._create_tmp_job_file(self.args.FILE)
+            super(submit, self).submit(job_file)
+        finally:
+            if os.path.isfile(job_file):
+                os.unlink(job_file)
+            if os.path.isfile(testdef_file):
+                os.unlink(testdef_file)

=== modified file 'lava/testdef/__init__.py'
--- lava/testdef/__init__.py	2013-07-26 14:14:25 +0000
+++ lava/testdef/__init__.py	2013-07-26 14:14:25 +0000
@@ -16,17 +16,24 @@
 # You should have received a copy of the GNU Lesser General Public License
 # along with lava-tool.  If not, see <http://www.gnu.org/licenses/>.
 
+"""TestDefinition class."""
+
 import yaml
 
 from copy import deepcopy
 
-from lava.helper.template import expand_template
+from lava.helper.template import (
+    expand_template,
+    set_value,
+)
 from lava_tool.utils import (
     write_file,
     verify_path_existance,
-    verify_file_extension
+    verify_file_extension,
 )
 
+# Default name for a test definition.
+DEFAULT_TESTDEF_FILENAME = "lavatest.yaml"
 # Default test def file extension.
 DEFAULT_TESTDEF_EXTENSION = "yaml"
 # Possible extensions for a test def file.
@@ -35,6 +42,13 @@
 
 class TestDefinition(object):
 
+    """A test definition object.
+
+    This class should be used to create test definitions. The initialization
+    enforces a default file name extension, and makes sure that the file is
+    not already present on the file system.
+    """
+
     def __init__(self, data, file_name):
         """Initialize the object.
 
@@ -50,6 +64,14 @@
 
         self.data = deepcopy(data)
 
+    def set(self, key, value):
+        """Set key to the specified value.
+
+        :param key: The key to look in the object data.
+        :param value: The value to set.
+        """
+        set_value(self.data, key, value)
+
     def write(self):
         """Writes the test definition to file."""
         content = yaml.dump(self.data, default_flow_style=False, indent=4)

=== modified file 'lava/testdef/commands.py'
--- lava/testdef/commands.py	2013-07-26 14:14:25 +0000
+++ lava/testdef/commands.py	2013-07-26 14:14:25 +0000
@@ -21,9 +21,12 @@
 """
 
 import os
+import tempfile
 
 from lava.helper.command import BaseCommand
+from lava.job import DEFAULT_JOB_FILENAME
 from lava.tool.command import CommandGroup
+from lava_tool.utils import verify_path_non_existance
 
 
 class testdef(CommandGroup):
@@ -33,7 +36,23 @@
     namespace = "lava.testdef.commands"
 
 
-class new(BaseCommand):
+class TestdefBaseCommand(BaseCommand):
+
+    def _create_tmp_job_file(self, testdef_file):
+        testdef_file = os.path.abspath(testdef_file)
+        verify_path_non_existance(testdef_file)
+
+        job_file = os.path.join(tempfile.gettempdir(),
+                                DEFAULT_JOB_FILENAME)
+
+        tar_content = [testdef_file]
+        job_file = self.create_tar_repo_job(job_file, testdef_file,
+                                            tar_content)
+
+        return job_file
+
+
+class new(TestdefBaseCommand):
 
     """Creates a new test definition file."""
 
@@ -47,7 +66,7 @@
         self.create_test_definition(full_path)
 
 
-class run(BaseCommand):
+class run(TestdefBaseCommand):
 
     """Runs the specified test definition on a local device."""
 
@@ -57,11 +76,18 @@
         parser.add_argument("FILE", help="Test definition file to run.")
 
     def invoke(self):
-        pass
-
-
-def submit(BaseCommand):
-    """Submits the specified test definition to a remove LAVA server."""
+        job_file = ""
+        try:
+            job_file = self._create_tmp_job_file(self.args.FILE)
+            super(run, self).run(job_file)
+        finally:
+            if os.path.isfile(job_file):
+                os.unlink(job_file)
+
+
+class submit(TestdefBaseCommand):
+
+    """Submits the specified test definition to a LAVA server."""
 
     @classmethod
     def register_arguments(cls, parser):
@@ -69,4 +95,10 @@
         parser.add_argument("FILE", help="Test definition file to send.")
 
     def invoke(self):
-        pass
+        job_file = ""
+        try:
+            job_file = self._create_tmp_job_file(self.args.FILE)
+            super(submit, self).submit(job_file)
+        finally:
+            if os.path.isfile(job_file):
+                os.unlink(job_file)

=== modified file 'lava/testdef/templates.py'
--- lava/testdef/templates.py	2013-07-26 14:14:25 +0000
+++ lava/testdef/templates.py	2013-07-26 14:14:25 +0000
@@ -19,23 +19,11 @@
 """Test definition templates."""
 
 from lava.parameter import (
-    ListParameter,
     Parameter,
 )
 
-DEFAULT_TESTDEF_FILE = "lavatest.yaml"
-
 DEFAULT_TESTDEF_VERSION = "1.0"
 DEFAULT_TESTDEF_FORMAT = "Lava-Test Test Definition 1.0"
-
-# This is what will be called by default by the test definition yaml file.
-DEFAULT_TESTDEF_SCRIPT = "mytest.sh"
-DEFAULT_TESTDEF_SCRIPT_CONTENT = """#!/bin/sh
-# Automatic generated content by lava-tool.
-# Please add your own instructions.
-"""
-DEFAULT_TESTDEF_STEP = "./mytest.sh"
-
 DEFAULT_ENVIRONMET_VALUE = "lava_test_shell"
 
 # All these parameters will not be stored on the local config file.
@@ -45,18 +33,7 @@
 DESCRIPTION_PARAMETER = Parameter("description", depends=NAME_PARAMETER)
 DESCRIPTION_PARAMETER.store = False
 
-ENVIRONMENT_PARAMETER = ListParameter("environment",
-                                      depends=NAME_PARAMETER)
-ENVIRONMENT_PARAMETER.add(DEFAULT_ENVIRONMET_VALUE)
-ENVIRONMENT_PARAMETER.asked = True
-ENVIRONMENT_PARAMETER.store = False
-
-# Steps parameter. Default to a local shell script that the user defines.
-# We do not ask this parameter, and we do not store it either.
-STEPS_PARAMETER = ListParameter("steps", depends=NAME_PARAMETER)
-STEPS_PARAMETER.add(DEFAULT_TESTDEF_STEP)
-STEPS_PARAMETER.asked = True
-STEPS_PARAMETER.store = False
+TESTDEF_STEPS_KEY = "steps"
 
 TESTDEF_TEMPLATE = {
     "metadata": {
@@ -64,12 +41,12 @@
         "format": DEFAULT_TESTDEF_FORMAT,
         "version": DEFAULT_TESTDEF_VERSION,
         "description": DESCRIPTION_PARAMETER,
-        "environment": ENVIRONMENT_PARAMETER,
+        "environment": [DEFAULT_ENVIRONMET_VALUE],
     },
     "run": {
-        "steps": STEPS_PARAMETER,
+        TESTDEF_STEPS_KEY: ["./mytest.sh"]
     },
     "parse": {
-        "pattern": '^\s*(?P<test_case_id>\w+)=(?P<result>\w+)\s*$'
+        "pattern": r'^\s*(?P<test_case_id>\w+)=(?P<result>\w+)\s*$'
     }
 }

=== modified file 'lava/testdef/tests/test_commands.py'
--- lava/testdef/tests/test_commands.py	2013-07-26 14:14:25 +0000
+++ lava/testdef/tests/test_commands.py	2013-07-26 14:14:25 +0000
@@ -47,7 +47,7 @@
         self.args.FILE = self.file_path
 
         self.temp_yaml = tempfile.NamedTemporaryFile(suffix=".yaml",
-                                                    delete=False)
+                                                     delete=False)
 
         self.config_file = tempfile.NamedTemporaryFile(delete=False)
         self.config = InteractiveConfig()
@@ -110,7 +110,7 @@
                         'name': ''},
                     'parse': {
                         'pattern':
-                            '^\\s*(?P<test_case_id>\\w+)=(?P<result>\\w+)\\s*$'
+                        '^\\s*(?P<test_case_id>\\w+)=(?P<result>\\w+)\\s*$'
                         },
                     }
         obtained = None
@@ -144,7 +144,7 @@
                     },
                     'parse': {
                         'pattern':
-                            '^\\s*(?P<test_case_id>\\w+)=(?P<result>\\w+)\\s*$'
+                        '^\\s*(?P<test_case_id>\\w+)=(?P<result>\\w+)\\s*$'
                         },
                     }
         obtained = None

=== modified file 'lava/tests/test_commands.py'
--- lava/tests/test_commands.py	2013-07-26 14:14:25 +0000
+++ lava/tests/test_commands.py	2013-07-26 14:14:25 +0000
@@ -64,7 +64,6 @@
         _, args, _ = self.parser.method_calls[1]
         self.assertIn("DIR", args)
 
-
     @patch("lava.commands.edit_file", create=True)
     def test_command_invoke_0(self, mocked_edit_file):
         # Invoke the init command passing a path to a file. Should raise an

=== modified file 'lava/tests/test_config.py'
--- lava/tests/test_config.py	2013-07-26 14:14:25 +0000
+++ lava/tests/test_config.py	2013-07-26 14:14:25 +0000
@@ -148,6 +148,15 @@
             obtained = tmp_file.read()
         self.assertEqual(expected, obtained)
 
+    def test_config_get_from_backend_public(self):
+        # Need to to this, since we want a clean Config instance, with
+        # a config_file with some content.
+        with open(self.config.config_file, "w") as write_config:
+            write_config.write("[DEFAULT]\nfoo=bar\n")
+        param = Parameter("foo")
+        obtained = self.config.get_from_backend(param)
+        self.assertEquals("bar", obtained)
+
 
 class InteractiveConfigTest(ConfigTestCase):
 

=== modified file 'lava/tests/test_parameter.py'
--- lava/tests/test_parameter.py	2013-07-26 14:14:25 +0000
+++ lava/tests/test_parameter.py	2013-07-26 14:14:25 +0000
@@ -27,7 +27,6 @@
     ListParameter,
     Parameter,
 )
-
 from lava_tool.utils import to_list
 
 
@@ -78,7 +77,9 @@
         self.assertIsInstance(obtained, list)
         self.assertEquals(expected, obtained)
 
+
 class ListParameterTest(GeneralParameterTest):
+
     """Tests for the specialized ListParameter class."""
 
     def setUp(self):

=== modified file 'lava_tool/utils.py'
--- lava_tool/utils.py	2013-07-26 14:14:25 +0000
+++ lava_tool/utils.py	2013-07-26 14:14:25 +0000
@@ -191,15 +191,27 @@
 
 
 def verify_path_existance(path):
-    """Verifies if a given path exists or not on the file system.
+    """Verifies if a given path exists on the file system.
 
     Raises a CommandError in case it exists.
 
-    :param path: The path to verify."""
+    :param path: The path to verify.
+    """
     if os.path.exists(path):
         raise CommandError("{0} already exists.".format(path))
 
 
+def verify_path_non_existance(path):
+    """Verifies if a given path does not exist on the file system.
+
+    Raises a CommandError in case it does not exist.
+
+    :param path: The path to verify.
+    """
+    if not os.path.exists(path):
+        raise CommandError("{0} does not exists.".format(path))
+
+
 def write_file(path, content):
     """Creates a file with the specified content.