Merge lp:~stylesen/lava-dispatcher/add-on-to-testdef-yaml into lp:lava-dispatcher

[Senthil Kumaran S]

Add more metadata parameters to the YAML format which could now be understood by lava-test-shell. A test definition with the newly added parameters is available here - http://git.linaro.org/gitweb?p=qa/test-definitions.git;a=blob;f=ubuntu/device-tree.yaml

[lp:~stylesen/lava-dispatcher/add-on-to-testdef-yaml]

539. By Senthil Kumaran S on 2013-01-23

Make certain test definition metadata parameters to be a list in YAML.

[Andy Doan (doanac)]

looks pretty good to me. one thing its probably worth adding now is either a docs entry detailing these YAML fields or an annotated example

[Senthil Kumaran S (stylesen)]

On Thursday 24 January 2013 12:14 AM, Andy Doan wrote:
> looks pretty good to me. one thing its probably worth adding now is either a docs entry detailing these YAML fields or an annotated example

Yep, I shall do that shortly (its in my TODO already).

Thank You.
--
Senthil Kumaran S
http://www.stylesen.org/
http://www.sasenthilkumaran.com/

[Michael Hudson-Doyle (mwhudson)]

Senthil Kumaran S <email address hidden> writes:

> +def _get_testdef_info(testdef):
> + metadata = {}
> + metadata['version'] = testdef['metadata']['version']
> + metadata['description'] = testdef['metadata']['description']
> + metadata['format'] = testdef['metadata']['format']
> + metadata['os'] = ','.join(testdef['metadata']['os'])
> + metadata['devices'] = ','.join(testdef['metadata']['devices'])
> + metadata['environment'] = ','.join(testdef['metadata']['environment'])
> + return metadata

I don't think it's reasonable to assume that the new fields are present
in the testdef. The rest looks fine.

Cheers,
mwh

[lp:~stylesen/lava-dispatcher/add-on-to-testdef-yaml]

540. By Senthil Kumaran S on 2013-01-28

Ensure testdef metadata is attached to each test run and avoid adding it
to context.

541. By Senthil Kumaran S on 2013-01-28

Fix comment to add description about new testdef_metadata file.

542. By Senthil Kumaran S on 2013-01-28

Bring up-to-date with trunk.

[Andy Doan (doanac)]

I think this looks pretty good. I have a concern we'll need some sort of evolution logic as the schema changes. However, maybe we assume this comprises the 1.0 version and hold off doing evultion stuff until we change the schema next time?

[Senthil Kumaran S (stylesen)]

Hi Andy,

On Monday 28 January 2013 10:52 PM, Andy Doan wrote:
> I think this looks pretty good. I have a concern we'll need some sort of evolution logic as the schema changes. However, maybe we assume this comprises the 1.0 version and hold off doing evultion stuff until we change the schema next time?

I thought I can take evolution stuff in next months BP. As of now we
will have the format till date as version 1.0 and lets evolve.

Thank You.
--
Senthil Kumaran S
http://www.stylesen.org/
http://www.sasenthilkumaran.com/

[Michael Hudson-Doyle (mwhudson)]

Senthil Kumaran S <email address hidden> writes:

> === modified file 'lava_dispatcher/test_data.py'
> --- lava_dispatcher/test_data.py 2012-07-31 20:20:02 +0000
> +++ lava_dispatcher/test_data.py 2013-01-28 08:54:19 +0000
> @@ -34,7 +34,8 @@
> def __init__(self, test_id='lava'):
> self.job_status = 'pass'
> self.metadata = {}
> - self._test_run = { 'test_results':[], 'attachments':[], 'tags':[] }
> + self._test_run = { 'test_results':[], 'attachments':[], 'tags':[],
> + 'testdef_metadata':{} }

Does this actually work? The 1.5 bundle format:

http://bazaar.launchpad.net/~linaro-validation/linaro-python-dashboard-bundle/trunk/view/head:/linaro_dashboard_bundle/schemas/dashboard_bundle_format_1.5.json

doesn't allow a testdef_metadata property on elements of the test_run
array.

> self._test_run['test_id'] = test_id
> self._assign_date()
> self._assign_uuid()
> @@ -75,4 +76,3 @@
> def get_test_run(self):
> self.add_result('job_complete', self.job_status)
> return self._test_run
> -
>

[Michael Hudson-Doyle (mwhudson)]

Senthil Kumaran S <email address hidden> writes:

> Hi Andy,
>
> On Monday 28 January 2013 10:52 PM, Andy Doan wrote:
>> I think this looks pretty good. I have a concern we'll need some sort of evolution logic as the schema changes. However, maybe we assume this comprises the 1.0 version and hold off doing evultion stuff until we change the schema next time?
>
> I thought I can take evolution stuff in next months BP. As of now we
> will have the format till date as version 1.0 and lets evolve.

This makes sense to me.

[Senthil Kumaran S (stylesen)]

On Tuesday 29 January 2013 06:46 AM, Michael Hudson-Doyle wrote:
> http://bazaar.launchpad.net/~linaro-validation/linaro-python-dashboard-bundle/trunk/view/head:/linaro_dashboard_bundle/schemas/dashboard_bundle_format_1.5.json
>
> doesn't allow a testdef_metadata property on elements of the test_run
> array.

I did patched linaro-python-dashboard-bundle along with this BP -
https://blueprints.launchpad.net/lava-dispatcher/+spec/add-on-to-testdef-yaml-format

The branch is here -
https://code.launchpad.net/~stylesen/linaro-python-dashboard-bundle/add-on-to-testdef-yaml

Thank You.
--
Senthil Kumaran S
http://www.stylesen.org/
http://www.sasenthilkumaran.com/

[lp:~stylesen/lava-dispatcher/add-on-to-testdef-yaml]

543. By Senthil Kumaran S on 2013-01-29

Bring up-to-date with trunk.

=== modified file 'lava_dispatcher/actions/lava_test_shell.py'
--- lava_dispatcher/actions/lava_test_shell.py	2013-01-15 17:23:13 +0000
+++ lava_dispatcher/actions/lava_test_shell.py	2013-01-29 10:28:22 +0000
@@ -53,6 +53,7 @@
 #          uuid                    The "analyzer_assigned_uuid" of the
 #                                  test_run that is being generated.
 #          testdef.yml             The test definition.
+#          testdef_metadata        Metadata extracted from test definition.
 #          install.sh              The install steps.
 #          run.sh                  The run steps.
 #          [repos]                 The test definition can specify bzr or git
@@ -74,6 +75,7 @@
 #          pkgs.txt                Ditto
 #       ${IDX}_${TEST_ID}-${TIMESTAMP}/
 #          testdef.yml
+#          testdef_metadata
 #          stdout.log
 #          return_code             The exit code of run.sh.
 #          analyzer_assigned_uuid
@@ -173,8 +175,7 @@
     cwd = os.getcwd()
     gitdir = os.path.join(tmpdir, 'gittestrepo')
     try:
-        subprocess.check_call(['git', 'clone', testdef_repo,
-                                  gitdir])
+        subprocess.check_call(['git', 'clone', testdef_repo, gitdir])
         if revision:
             os.chdir(gitdir)
             subprocess.check_call(['git', 'checkout', revision])
@@ -201,6 +202,26 @@
         logging.error('Unable to get test definition from bzr\n' + str(e))
 
 
+def _get_testdef_info(testdef):
+    metadata = {'os': '', 'devices': '', 'environment': ''}
+    metadata['version'] = str(testdef['metadata'].get('version'))
+    metadata['description'] = str(testdef['metadata'].get('description'))
+    metadata['format'] = str(testdef['metadata'].get('format'))
+
+    # Convert list to comma separated string.
+    if testdef['metadata'].get('os'):
+        metadata['os'] = ','.join(testdef['metadata'].get('os'))
+
+    if testdef['metadata'].get('devices'):
+        metadata['devices'] = ','.join(testdef['metadata'].get('devices'))
+
+    if testdef['metadata'].get('environment'):
+        metadata['environment'] = ','.join(
+            testdef['metadata'].get('environment'))
+
+    return metadata
+
+
 class TestDefinitionLoader(object):
     """
     A TestDefinitionLoader knows how to load test definitions from the data
@@ -227,7 +248,10 @@
 
         idx = len(self.testdefs)
 
-        self._append_testdef(URLTestDefinition(self.context, idx, testdef))
+        testdef_metadata = {'url': url, 'location': 'URL'}
+        testdef_metadata.update(_get_testdef_info(testdef))
+        self._append_testdef(URLTestDefinition(self.context, idx, testdef,
+                                               testdef_metadata))
 
     def load_from_repo(self, testdef_repo):
         tmpdir = utils.mkdtemp(self.tmpbase)
@@ -289,9 +313,10 @@
     A test definition that was loaded from a URL.
     """
 
-    def __init__(self, context, idx, testdef):
+    def __init__(self, context, idx, testdef, testdef_metadata):
         self.context = context
         self.testdef = testdef
+        self.testdef_metadata = testdef_metadata
         self.idx = idx
         self.test_run_id = '%s_%s' % (idx, self.testdef['metadata']['name'])
         self.uuid = str(uuid4())
@@ -378,6 +403,9 @@
         with open('%s/uuid' % hostdir, 'w') as f:
             f.write(self.uuid)
 
+        with open('%s/testdef_metadata' % hostdir, 'w') as f:
+            f.write(yaml.dump(self.testdef_metadata))
+
         if 'install' in self.testdef:
             self._create_repos(hostdir)
             self._create_target_install(hostdir, targetdir)
@@ -408,7 +436,14 @@
     """
 
     def __init__(self, context, idx, testdef, repo, info):
-        URLTestDefinition.__init__(self, context, idx, testdef)
+        testdef_metadata = {}
+        testdef_metadata.update({'url': info['branch_url']})
+        testdef_metadata.update({'location': info['branch_vcs'].upper()})
+        testdef_metadata.update({'repo_rev': info['branch_revision']})
+        testdef_metadata.update(_get_testdef_info(testdef))
+
+        URLTestDefinition.__init__(self, context, idx, testdef,
+                                   testdef_metadata)
         self.repo = repo
         self._sw_sources.append(info)
 

=== modified file 'lava_dispatcher/lava_test_shell.py'
--- lava_dispatcher/lava_test_shell.py	2013-01-14 03:08:36 +0000
+++ lava_dispatcher/lava_test_shell.py	2013-01-29 10:28:22 +0000
@@ -292,8 +292,11 @@
     uuid = _read_content(os.path.join(test_run_dir, 'analyzer_assigned_uuid'))
     attachments = _get_run_attachments(test_run_dir, testdef, stdout)
     attributes = _attributes_from_dir(os.path.join(test_run_dir, 'attributes'))
+    testdef_metadata = _read_content(os.path.join(test_run_dir,
+                                                  'testdef_metadata'))
 
     testdef = yaml.load(testdef)
+    testdef_metadata = yaml.load(testdef_metadata)
     if uuid in testdefs_by_uuid:
         sw_sources = testdefs_by_uuid[uuid]._sw_sources
     else:
@@ -311,6 +314,7 @@
         'hardware_context': hwcontext,
         'attachments': attachments,
         'attributes': attributes,
+        'testdef_metadata': testdef_metadata,
     }
 
 

=== modified file 'lava_dispatcher/test_data.py'
--- lava_dispatcher/test_data.py	2012-07-31 20:20:02 +0000
+++ lava_dispatcher/test_data.py	2013-01-29 10:28:22 +0000
@@ -34,7 +34,8 @@
     def __init__(self, test_id='lava'):
         self.job_status = 'pass'
         self.metadata = {}
-        self._test_run = { 'test_results':[], 'attachments':[], 'tags':[] }
+        self._test_run = { 'test_results':[], 'attachments':[], 'tags':[],
+                           'testdef_metadata':{} }
         self._test_run['test_id'] = test_id
         self._assign_date()
         self._assign_uuid()
@@ -75,4 +76,3 @@
     def get_test_run(self):
         self.add_result('job_complete', self.job_status)
         return self._test_run
-

=== modified file 'lava_test_shell/lava-test-runner-android'
--- lava_test_shell/lava-test-runner-android	2013-01-04 13:23:08 +0000
+++ lava_test_shell/lava-test-runner-android	2013-01-29 10:28:22 +0000
@@ -92,6 +92,7 @@
 		mkdir ${odir}
 		mkdir ${odir}/attachments/
 		cp ${line}/testdef.yaml ${odir}/
+		cp ${line}/testdef_metadata ${odir}/
 		cp ${line}/uuid ${odir}/analyzer_assigned_uuid
 		cp ${line}/run.sh ${odir}/attachments/
 		echo 'text/plain' > ${odir}/attachments/run.sh.mimetype

=== modified file 'lava_test_shell/lava-test-runner-ubuntu'
--- lava_test_shell/lava-test-runner-ubuntu	2013-01-14 16:30:05 +0000
+++ lava_test_shell/lava-test-runner-ubuntu	2013-01-29 10:28:22 +0000
@@ -76,6 +76,7 @@
 	mkdir ${odir}/attachments/
 	cp ${line}/uuid ${odir}/analyzer_assigned_uuid
 	cp ${line}/testdef.yaml ${odir}/
+	cp ${line}/testdef_metadata ${odir}/
 	cp ${line}/run.sh ${odir}/attachments/
 	echo 'text/plain' > ${odir}/attachments/run.sh.mimetype
 	if [ -f ${line}/install.sh ]; then