Merge lp:~linaro-infrastructure/tcwg-web/cbuild-lava into lp:tcwg-web

[Stevan Radaković]

Contains:
- support for new lava-backed build queues.
- lava link integration support on scheduler page.
- trivial LAVA API library wrapper.
- .lock file support for LAVA.
- LAVA polling support with multiple functions.

[lp:~linaro-infrastructure/tcwg-web/cbuild-lava]

276. By Milo Casagrande on 2013-01-15

Added tarball file extraction.

277. By Stevan Radaković on 2013-01-15

Remove deletion of .job and .lock files.

278. By Milo Casagrande on 2013-01-15

Fixed regex to retrieve dir name.

279. By Milo Casagrande on 2013-01-15

Write bundle id to lock file to mark job as parsed.

280. By Milo Casagrande on 2013-01-15

Fixed typo in dictionary get operation.

281. By Milo Casagrande on 2013-01-15

Added newline at the end of the string.

[Michael Hudson-Doyle (mwhudson)]

Paul Sokolovsky <email address hidden> writes:

> You have been requested to review the proposed merge of lp:~linaro-infrastructure/tcwg-web/cbuild-lava into lp:tcwg-web.
>
> For more details, see:
> https://code.launchpad.net/~linaro-infrastructure/tcwg-web/cbuild-lava/+merge/143295
>
> Contains:
> - support for new lava-backed build queues.
> - lava link integration support on scheduler page.
> - trivial LAVA API library wrapper.
> - .lock file support for LAVA.
> - LAVA polling support with multiple functions.
>
> --
> https://code.launchpad.net/~linaro-infrastructure/tcwg-web/cbuild-lava/+merge/143295
> Your team Linaro Validation Team is requested to review the proposed
> merge of lp:~linaro-infrastructure/tcwg-web/cbuild-lava into
> lp:tcwg-web.

I'm not really capable of reviewing the overall structure. But here's a
few comments anyway.

> === modified file 'common.py'
> --- common.py 2012-11-15 01:48:14 +0000
> +++ common.py 2013-01-15 12:49:22 +0000
> @@ -17,12 +17,35 @@
> import time
> import json
> import urlparse
> +import urllib2
> import ConfigParser
> import base64
> +import shutil
> +import hashlib
> +import sys
>
> import web
> import memcache
>
> +import lava
> +
> +HOST = "host"
> +TIMESTAMP = "timestamp"
> +JOB_ID = "job_id"
> +LAVA_JOB_ID = "lava_job_id"
> +LAVA_JOB_URL = "lava_job_url"
> +
> +LOCK_FILE_KEYS = (
> + (HOST, 'HOST'),
> + (TIMESTAMP, 'TIMESTAMP'),
> + (JOB_ID, 'JOB_ID'),
> + (LAVA_JOB_ID, 'LAVA_JOB_ID'),
> + (LAVA_JOB_URL, 'LAVA_JOB_URL'),
> + )

This doesn't seem to be used anywhere.

> +class NoBundleFoundException(Exception):
> + """Raised when no bundle result is found with matching job name."""
> + pass
>
> class Cache:
> """In memory cache of file derrived objects."""

Typo: "derrived"

> @@ -292,6 +315,7 @@
> base = special
> abspath = os.path.abspath(os.path.join(base, name))
> else:
> + # TODO need to check this path.
> base = get_config('general', 'results', '/var/www/ex.seabright.co.nz/')
> abspath = os.path.abspath(os.path.join(base, context, name))
>
> @@ -310,7 +334,7 @@
>
> def authenticate(role='admin'):
> if is_development():
> - return 'michaelh1'
> + return 'devel-user'
>
> oid = web.webopenid.status()
>
> @@ -410,7 +434,8 @@
>
> def get_config(group, name, fallback=''):
> parser = ConfigParser.SafeConfigParser()
> - parser.read('tcwg-web.ini')
> + dirname = os.path.dirname(__file__)
> + parser.read(dirname + '/tcwg-web.ini')
>
> if parser.has_option(group, name):
> return parser.get(group, name)
> @@ -438,3 +463,244 @@
> v = base64.b64decode(v)
>
> web.config[name] = v
> +
> +def schedule_job(queue, job, contents=None, use_template_file=False):
> + """Schedule a job. All job scheduling should happen via this
> + function, which encapsulates support for various schdulers in use."""
> + queue_dir = map_filename('queue/%s/' % queue, 'scheduler') + '/'
> +
> + job_file = queue_dir + job + '.job'

Constructing paths with + always make me twitch a bit. What's wrong
with os.path.join?

> +...

[Milo Casagrande (milo)]

Hi,

thanks for taking a look!

On Wed, Jan 16, 2013 at 4:55 AM, Michael Hudson-Doyle
<email address hidden> wrote:
>> +
>> +
>> +def get_log_dir_name(lava_log):
>> + """Returns the dir name found in the LAVA log file.
>> +
>> + :param lava_log: The name of the saved LAVA log file.
>> + :type str
>> + """
>> + # We need to match strings in the form of:
>> + # + lava-test-case-attach [...] results[...]
>> + matcher = re.compile("\s*\+?\s*lava-test-case-attach\s*.*\s*results/")
>> + path = None
>> + if lava_log:
>> + lines = None
>> + try:
>> + # XXX Probably not the most efficient way of parsing the file.
>> + # Might be better to read it backwards, since look like the strings
>> + # we need are closer to the bottom of the file.
>> + with open(lava_log, 'r') as log:
>> + lines = log.readlines()
>> + for line in lines:
>> + if matcher.match(line):
>> + match = line.strip().split(' ')[-1]
>> + break
>> + if match:
>> + # We treat it like a path.
>> + path = os.path.dirname(match).split('/')[-1]
>> + finally:
>> + os.unlink(lava_log)
>> + return path
>
> Wow, this seems SUPER fragile.

It is, indeed. At the moment that was the first thing, and actually
the only place, where we could find that information.

> Can't you create an attachment that has
> the name of this directory in it?

I honestly have no idea, for I have yet to find out in cbuild code
where that directory name is created.

> Test cases can have attributes too,
> which would be more appropriate somehow, but there is no api for
> lava-test-shell tests to create them yet. Adding one will be easier
> than working out what this function is doing :-)

It would indeed, but I guess first we need to find out where that
directory name comes out.

Some of the other comments have been addressed, some are in the process.
Thanks again!

--
Milo Casagrande | Infrastructure Team
Linaro.org <www.linaro.org> │ Open source software for ARM SoCs

[lp:~linaro-infrastructure/tcwg-web/cbuild-lava]

282. By Milo Casagrande on 2013-01-16

Fixed code as per review comments.

283. By Milo Casagrande on 2013-01-16

Fixed as per review comments.

[Milo Casagrande (milo)]

Hello Michael,

all the comments have been addressed in the code.
The only missing part is the "weak" one, unfortunately: we need to
find another way of exporting that directory name, but we also need to
find where that directory name is defined in the first place.

Thanks again for your review!

--
Milo Casagrande | Infrastructure Team
Linaro.org <www.linaro.org> │ Open source software for ARM SoCs

[lp:~linaro-infrastructure/tcwg-web/cbuild-lava]

284. By Paul Sokolovsky on 2013-01-17

lava_polling_cron: Add verbose operation mode.

285. By Paul Sokolovsky on 2013-01-17

Update parse_lock_file() to support arbitrary fields, simplify naming scheme.

This was triggered by the fact that FINISHED field wasn't parsed, so
lava_polling_cron.py processed same jobs again and again.

286. By Stevan Radaković on 2013-01-17

Fix infinite .lock file update with 'FINISHED'.

287. By Paul Sokolovsky on 2013-01-17

Fix case where literal field value instead of constant was used previously.

288. By Paul Sokolovsky on 2013-01-17

Add debug logging for lava_polling_cron.py diagnostics.

289. By Paul Sokolovsky on 2013-01-17

Few minor fixes for get_result_bundle_logs().

290. By Paul Sokolovsky on 2013-01-17

Debug logging for save_lava_log_file().

291. By Paul Sokolovsky on 2013-01-17

Fix thinko from os.path.join() refactor.

292. By Paul Sokolovsky on 2013-01-17

We also need to pass job content down to builder, as it contains extra config.

293. By Paul Sokolovsky on 2013-01-17

Strip CRs from webform's string.

294. By Paul Sokolovsky on 2013-01-17

Typo fix in var name.

295. By Paul Sokolovsky on 2013-01-17

update_lock_file() no longer needs extra check.

296. By Paul Sokolovsky on 2013-01-17

Rename FINISHED -> LAVA_BUNDLE.

297. By Paul Sokolovsky on 2013-01-18

Calculate lava_test_shell timeout dynamically based on job.

It's not ok to have the same timeout as gcc (>10hr) for every job, it's
hard to even test stuff which hangs and locks up regularly.

298. By Paul Sokolovsky on 2013-01-18

Use relative URLs, as this is supposed to be on same server in out setup.

This allows for faithful and comfortable tesing on sandboxes.

[Matthew Gretton-Dann (matthew-gretton-dann)]

Approved.

review: Approve

=== modified file 'common.py'
--- common.py	2012-11-15 01:48:14 +0000
+++ common.py	2013-01-18 12:55:25 +0000
@@ -17,15 +17,30 @@
 import time
 import json
 import urlparse
+import urllib2
 import ConfigParser
 import base64
+import shutil
+import hashlib
+import sys
+import tarfile
+import logging
 
 import web
 import memcache
 
+import lava
+
+
+log = logging.getLogger("common")
+
+class NoBundleFoundException(Exception):
+    """Raised when no bundle result is found with matching job name."""
+    pass
+
 
 class Cache:
-    """In memory cache of file derrived objects."""
+    """In memory cache of file derived objects."""
     def __init__(self):
         self.items = {}
 
@@ -52,7 +67,7 @@
 
 def shorten(v):
     if '. ' in v:
-        v = v[:v.index('. ')+1]
+        v = v[:v.index('. ') + 1]
 
     if '\n' in v:
         v = v[:v.index('\n')]
@@ -292,6 +307,7 @@
         base = special
         abspath = os.path.abspath(os.path.join(base, name))
     else:
+        # TODO need to check this path.
         base = get_config('general', 'results', '/var/www/ex.seabright.co.nz/')
         abspath = os.path.abspath(os.path.join(base, context, name))
 
@@ -310,7 +326,7 @@
 
 def authenticate(role='admin'):
     if is_development():
-        return 'michaelh1'
+        return 'devel-user'
 
     oid = web.webopenid.status()
 
@@ -410,7 +426,8 @@
 
 def get_config(group, name, fallback=''):
     parser = ConfigParser.SafeConfigParser()
-    parser.read('tcwg-web.ini')
+    dirname = os.path.dirname(__file__)
+    parser.read(dirname + '/tcwg-web.ini')
 
     if parser.has_option(group, name):
         return parser.get(group, name)
@@ -428,6 +445,7 @@
     else:
         return template.replace('.SERVER.', '.' + server + '.')
 
+
 def setup_email():
     """Copy the email configuration into web.py"""
     for name in 'smtp_server smtp_username smtp_password smtp_starttls smtp_debuglevel'.split():
@@ -438,3 +456,290 @@
                 v = base64.b64decode(v)
 
             web.config[name] = v
+
+
+def schedule_job(queue, job, contents=None, use_template_file=False):
+    """Schedule a job. All job scheduling should happen via this
+    function, which encapsulates support for various schdulers in use."""
+    queue_dir = map_filename('queue/%s/' % queue, 'scheduler') + '/'
+
+    job_file = os.path.join(queue_dir, job + '.job')
+    if use_template_file and os.path.exists(queue_dir + 'template.txt'):
+        shutil.copyfile(queue_dir + 'template.txt', job_file)
+    else:
+        if contents is None:
+            # Preserve old contents (but make sure file exists)
+            with open(job_file, 'a'):
+                os.utime(job_file, None)
+        else:
+            with open(job_file, 'w') as f:
+                f.write(contents)
+
+    lava_template = queue_dir + 'lava-job-template.json'
+    if os.path.exists(lava_template):
+        request_env = None
+        if hasattr(web.ctx, 'env'):
+            request_env = web.ctx.env
+        lava_job_id, lava_job_url = lava.submit_job(lava_template, queue, job, request_env)
+
+        jobfile = '%s%s.job' % (queue_dir, job)
+        lockfile = '%s.lock' % jobfile
+
+        job_id = create_job_id(jobfile)
+
+        write_lock_file(lockfile, 'lava', job_id, lava_job_id, lava_job_url)
+
+        print "Queue %s: scheduled %s for LAVA, job id: %d, url: %s" % \
+        (queue, job, lava_job_id, lava_job_url)
+
+
+def create_job_id(jobfile):
+
+    with open(jobfile) as f:
+        details = [x.rstrip() for x in f.readlines()]
+
+    stat = os.stat(jobfile)
+    sha = hashlib.sha1()
+    sha.update('%s %d %.3f %r' % (jobfile, stat.st_size,
+                                  stat.st_mtime, details))
+    digest = int(sha.hexdigest(), 16)
+    job_id = digest % 2 ** 31
+
+    return job_id
+
+
+def write_lock_file(lockfile, host, job_id, lava_job_id=None, lava_job_url=None):
+    """Write relevant info into the .lock file."""
+
+    with open(lockfile, 'w') as f:
+        f.write('%s\n' % host)
+        f.write('%s\n' % datetime.datetime.utcnow().isoformat())
+        f.write('%s=%s\n' % ("JOB_ID", job_id))
+
+        if lava_job_id:
+            f.write("%s=%d\n" % ("LAVA_JOB_ID", lava_job_id))
+        if lava_job_url:
+            f.write("%s=%s\n" % ("LAVA_JOB_URL", lava_job_url))
+
+
+def parse_lock_file(lockfile):
+    """Parse info from the .lock file and return dict."""
+
+    result = {}
+    if os.path.exists(lockfile):
+        with open(lockfile, 'r') as f:
+            lines = [x.rstrip() for x in f.readlines()]
+            result["HOST"] = lines[0]
+            result["TIMESTAMP"] = lines[1]
+            for l in lines[2:]:
+                k, v = l.split("=", 1)
+                result[k] = v
+
+    return result
+
+
+def fetch_lava_log(lava_job_id):
+    """Downloads the lava log file for the specified job id.
+
+    :param lava_job_id: The id of the LAVA job.
+    :return Tha temporary file name.
+    """
+    log_file_url = create_job_url(lava_job_id)
+    log_f = urllib2.urlopen(log_file_url)
+    import tempfile
+    handle, name = tempfile.mkstemp()
+    with open(name, 'w+b') as tmp_f:
+        tmp_f.write(log_f.read())
+    return name
+
+
+def get_bundle_sha1(lava_job_id):
+    """Gets the bundle code for the job id.
+
+    :param lava_job_id: The id of the LAVA job.
+    :return The bundle code associated with the job.
+    """
+    bundle_sha1 = None
+    # Get the JSON struct of the job.
+    url = create_job_url(lava_job_id, "json")
+    log.debug("Fetching job info from url: %s", url)
+    if url:
+        try:
+            json_str = json.load(urllib2.urlopen(url))
+            results = json_str.get("results_link", None)
+            if results:
+                # The last is an empty string, since the URL ends with a /.
+                bundle_sha1 = results.split('/')[-2]
+        except ValueError, ex:
+            print "Error retrieving bundle SHA1 for job %s." % str(lava_job_id)
+            print "Reported error is: %s" + str(ex)
+    return bundle_sha1
+
+
+def create_job_url(lava_job_id, what='log_file/plain'):
+    """Creates the URL to the log file in LAVA scheduler for specific job."""
+    url = get_config('lava', 'plain_url', None)
+    if url:
+        url = urlparse.urljoin(url, "scheduler/job/")
+        # We need the trailing slash, or the join doesn't work.
+        url = urlparse.urljoin(url, str(lava_job_id) + "/")
+        url = urlparse.urljoin(url, what)
+    return url
+
+
+def get_result_bundle_logs(job_name, bundle_sha1, lava_job_id):
+    """Gets attachments from LAVA test result bundle.
+
+    This functions iterates over test runs and downloads all attachments
+    to the correct location where the job logs are supposed to be saved.
+    """
+    bundle = lava.get_result_bundle(bundle_sha1)
+    bundle = json.loads(bundle['content'])
+    # Find the test ID that matches the job name.
+    bundle_test_run = None
+    for test_run in bundle['test_runs']:
+        if test_run['test_id'] == job_name:
+            bundle_test_run = test_run
+            break
+
+    if not bundle_test_run:
+        raise NoBundleFoundException("No testcase with the name %s in bundle" % job_name)
+
+    for test_result in bundle_test_run['test_results']:
+        for attachment in test_result['attachments']:
+            save_lava_log_file(job_name, attachment['pathname'],
+                               base64.b64decode(attachment['content']),
+                               attachment['mime_type'],
+                               lava_job_id)
+
+
+def get_log_dir_name(lava_log):
+    """Returns the dir name found in the LAVA log file.
+
+    :param lava_log: The name of the saved LAVA log file.
+    :type str
+    """
+    # We need to match strings in the form of:
+    # + tar -cf [...]
+    matcher = re.compile("\s*\+?\s*tar -cf")
+    path = None
+    if lava_log:
+        lines = None
+        try:
+            # XXX Probably not the most efficient way of parsing the file.
+            # Might be better to read it backwards, since look like the strings
+            # we need are closer to the bottom of the file.
+            with open(lava_log, 'r') as log:
+                lines = log.readlines()
+                for line in lines:
+                    if matcher.match(line):
+                        split_line = line.strip().split(' ')
+                        # Sending the tarball file, the last value is a dot (.)
+                        # We need to make sure that we are getting the right
+                        # one.
+                        match = split_line[-2]
+                        if not split_line[-1] == ".":
+                            match = split_line[-1]
+                        break
+            if match:
+                # We need to treat the name as a path in order to split it
+                # up: check if it has a trailing slash, otherwise add it.
+                dir_matcher = re.compile("/$")
+                if not dir_matcher.match(match):
+                    match += "/"
+                path = os.path.dirname(match).split('/')[-1]
+        finally:
+            os.unlink(lava_log)
+    return path
+
+
+def extract_and_remove_tar_file(path):
+    """Extracts the content of a tarball into the file directory, and removes
+    it when done.
+
+    :param path: The path of the tarball to extract."""
+    if os.getcwd() != os.path.dirname(path):
+        os.chdir(os.path.dirname(path))
+    with tarfile.open(path) as tarball:
+        for member in tarball.getmembers():
+            # Check that it is a file, since we might have directories in it.
+            if member.isfile():
+                tarball.extract(member)
+    try:
+        # Remove the tarball once done with it.
+        os.unlink(path)
+    except:
+        print "Impossible to remove tarball file '%s'." % path
+
+
+def save_lava_log_file(job_name, filename, content, mime_type, lava_job_id):
+    # Files should be saved under:
+    # /space/build/$job_name/logs/$log_dir/
+    # TODO Check if we can get the path from the web config.
+    directory = "/space/build/%(job_name)s/logs/%(log_dir)s"
+    log_dir = get_log_dir_name(fetch_lava_log(lava_job_id))
+    directory %= dict(job_name=job_name, log_dir=log_dir)
+    log.debug("Writing CBuild log file %s to %s", filename, directory)
+
+    if not os.path.exists(directory):
+        try:
+            os.makedirs(directory)
+        except OSError:
+            # We do not have permissions to write.
+            print "Cannot create directory %s." % directory
+            sys.exit(1)
+
+    file_path = os.path.join(directory, filename)
+    # Double check that we can write.
+    try:
+        fp = open(file_path, "w+b")
+    except IOError:
+        print "Cannot write file '%s' into %s." % (filename, directory)
+        sys.exit(1)
+    else:
+        with fp:
+            fp.write(content)
+
+    if tarfile.is_tarfile(file_path):
+        extract_and_remove_tar_file(file_path)
+
+
+def find_lava_jobs():
+    """Go through the queue directory and recognize all LAVA jobs."""
+    # This might become obsolete once we switch to LAVA completely.
+
+    lava_jobs = []
+
+    # TODO: this is relative path, so we must take into consideration where
+    # is this function called from.
+    for filename in find_files('hosts.txt', 'queue', 'scheduler'):
+        hosts = []
+        dirname = os.path.dirname(filename)
+
+        with open(filename) as f:
+            for line in f:
+                hosts.extend(line.split())
+
+        if hosts:
+            for lock_file in find_files('.job.lock', dirname, 'scheduler'):
+                lock_results = parse_lock_file(lock_file)
+                if lock_results["HOST"] == 'lava' and not lock_results.get("LAVA_BUNDLE", None):
+                    job = {}
+                    job["LAVA_JOB_ID"] = lock_results["LAVA_JOB_ID"]
+                    job["FILE_PATH"] = os.path.abspath(lock_file)
+                    job["JOB_NAME"] = os.path.basename(lock_file).replace('.job.lock', '')
+                    lava_jobs.append(job)
+
+    return lava_jobs
+
+
+def update_lock_file(path, bundle_id):
+    """Updates the lock file for a job.
+    Writes the bundle id in order to mark the job as already parsed.
+
+    :param path: The path to the lock file.
+    :bundle_id: The bundle id to write.
+    """
+    with open(path, 'a') as lock:
+        content = "%s=%s\n" % ("LAVA_BUNDLE", str(bundle_id))
+        lock.write(content)

=== modified file 'index.py'
--- index.py	2013-01-03 09:56:26 +0000
+++ index.py	2013-01-18 12:55:25 +0000
@@ -58,7 +58,7 @@
     web.ctx.user = None
 
     if common.is_development():
-        web.ctx.user = 'michaelh1'
+        web.ctx.user = 'devel-user'
 
     if web.ctx.oid:
         match = re.match(r'https://launchpad.net/~(.+)', web.ctx.oid)

=== added file 'lava.py'
--- lava.py	1970-01-01 00:00:00 +0000
+++ lava.py	2013-01-18 12:55:25 +0000
@@ -0,0 +1,79 @@
+import json
+import xmlrpclib
+
+import common
+
+
+def get_api(url):
+    user = common.get_config('lava:' + url, 'user')
+    if not user:
+        user = common.get_config('lava', 'user')
+    token = common.get_config('lava:' + url, 'token')
+    if not token:
+        token = common.get_config('lava', 'token')
+    proto, rest = url.split('://')
+    url = "%s://%s:%s@%s" % (proto, user, token, rest)
+    return xmlrpclib.ServerProxy(url)
+
+
+def calc_job_timeout(job_name):
+    if job_name.startswith("gcc"):
+        return 15 * 60 * 60
+    if job_name.startswith("cortex-str"):
+        return 0.5 * 60 * 60
+    # Our aim is to avoid lockups, so keep it low for now
+    return 2 * 60 * 60
+
+
+def submit_job(jobdef_template, queue, job_name, request_env):
+    f = open(jobdef_template)
+    job_def = f.read()
+    f.close()
+    # Allow jobdef to "inherit" lava server from tcwg-web's config
+    # by using %(lava_server)s substituation in jobdef template
+    api_url = common.get_config('lava', 'url')
+    substs = {'queue': queue, 'job': job_name, 'lava_server': api_url,
+              'timeout': calc_job_timeout(job_name)}
+    # If this is executed from web context, get this server's
+    # address from request. Otherwise, fallback to preconfigured
+    # values (applies when running from command-line for cbuild-tools
+    # for example).
+    if request_env:
+        substs['SERVER_NAME'] = request_env['SERVER_NAME']
+        substs['SERVER_PORT'] = request_env['SERVER_PORT']
+    else:
+        substs['SERVER_NAME'] = common.get_config('web', 'server_name')
+        substs['SERVER_PORT'] = common.get_config('web', 'server_port')
+
+    job_def = job_def % substs
+
+
+    submit_url = ''
+    job_dict = json.loads(job_def)
+    submit_dict = filter(lambda a: a['command'] == 'submit_results',
+                             job_dict['actions'])
+    if submit_dict:
+        submit_url = submit_dict[0]["parameters"]["server"]
+
+    # If lava url in config was not specified, let's try to use the same
+    # url as jobdef's submit_results action.
+    if not api_url:
+        api_url = submit_url
+
+    lava_api = get_api(api_url)
+    job_id = lava_api.scheduler.submit_job(job_def)
+    job_url = ''
+    if submit_url:
+        if submit_url[-1] == '/':
+            submit_url = submit_url[:-1]
+        job_url = submit_url.rsplit('/', 1)[0] + '/scheduler/job/%d' % job_id
+    return job_id, job_url
+
+def get_result_bundle(bundle_sha1):
+    """Returns the json format from lava dashboard 'get' call."""
+
+    api_url = common.get_config('lava', 'url')
+    lava_api = get_api(api_url)
+
+    bundle_result = lava_api.dashboard.get(bundle_sha1)
+    return bundle_result

=== added file 'lava_polling_cron.py'
--- lava_polling_cron.py	1970-01-01 00:00:00 +0000
+++ lava_polling_cron.py	2013-01-18 12:55:25 +0000
@@ -0,0 +1,44 @@
+#!/usr/bin/env python
+# Copyright (C) 2012 Linaro
+
+import common
+import sys
+import optparse
+import logging
+
+
+def main(args, options):
+    jobs = common.find_lava_jobs()
+    if not jobs:
+        logging.info("No active LAVA jobs found")
+        return
+
+    for job in jobs:
+            logging.info("Processing: %r" % job)
+            bundle_sha1 = common.get_bundle_sha1(job.get("LAVA_JOB_ID"))
+            if bundle_sha1:
+                logging.info("Job result bundle: %r", bundle_sha1)
+                # Fetch cbuild logs and put them where cbuild webapp expects
+                try:
+                    common.get_result_bundle_logs(job.get("JOB_NAME"),
+                                                  bundle_sha1,
+                                                  job.get("LAVA_JOB_ID"))
+                    common.update_lock_file(job.get("FILE_PATH"), bundle_sha1)
+                except:
+                    print "Error saving log files for job %s." % job.get("LAVA_JOB_ID")
+            else:
+                # No bundle code means that the job is still running.
+                logging.info("No result bundle available")
+
+
+if __name__ == '__main__':
+
+    optparser = optparse.OptionParser("%prog <options>")
+    optparser.add_option(
+        "-v", "--verbose", action="store_true")
+    options, args = optparser.parse_args(sys.argv[1:])
+    if options.verbose:
+        logging.basicConfig(level=logging.DEBUG)
+    else:
+        logging.basicConfig(level=logging.ERROR)
+    main(args, options)

=== added file 'schedulejob.py'
--- schedulejob.py	1970-01-01 00:00:00 +0000
+++ schedulejob.py	2013-01-18 12:55:25 +0000
@@ -0,0 +1,10 @@
+# This is standalone script intended to be called by other scripts/cronjobs
+# (including from outside components) to schedule a build.
+# Usage:
+# python -m schedulejob <queue> <job>
+import sys
+
+import common
+
+
+common.schedule_job(sys.argv[1], sys.argv[2], use_template_file=True, contents='')

=== modified file 'scheduler.py'
--- scheduler.py	2012-12-09 20:18:41 +0000
+++ scheduler.py	2013-01-18 12:55:25 +0000
@@ -18,10 +18,12 @@
 import urllib
 import hashlib
 import collections
+from pipes import quote
 
 import web
 
 import common
+import lava
 
 hostre = r'([\w\-]+)'
 
@@ -38,6 +40,8 @@
     '/job/' + hostre + '/(\w[\w.+\^\-~]+)\.job/release', 'job_release',
     '/job/' + hostre + '/(\w[\w.+\^\-~]+)\.job/bump', 'job_bump',
     '/job/' + hostre + '/(\w[\w.+\^\-~]+)\.job/drop', 'job_drop',
+    # Serve YAML "test definition" for LAVA build
+    '/lava/testdef/([^/]+)/([^/]+)', 'lava_testdef',
 )
 
 render = web.template.render('templates/',
@@ -53,7 +57,7 @@
 
 db = web.database(dbn='sqlite', db=common.map_filename('scheduler.sqlite', 'run'))
 
-Candidate = collections.namedtuple('Candidate', 'queue name jobfname lockfname jobtime host started locked locktime scaledage')
+Candidate = collections.namedtuple('Candidate', 'queue name jobfname lockfname jobtime host started locked locktime scaledage lava_url')
 
 def log(host, state, arg, msg=''):
     now = time.time()
@@ -246,8 +250,11 @@
                     if len(lines) >= 2:
                         started = lines[1].strip()
 
+                job_info = common.parse_lock_file(job + '.lock')
+                lava_url = job_info.get("LAVA_JOB_URL")
+
                 if with_locked or locktime < mtime:
-                    candidates.append([queue, os.path.basename(job), job, lock, mtime, ahost, started, locktime >= mtime, locktime, 0])
+                    candidates.append([queue, os.path.basename(job), job, lock, mtime, ahost, started, locktime >= mtime, locktime, 0, lava_url])
 
     if len(candidates) > 1:
         # Normalise the mtimes
@@ -255,7 +262,7 @@
         yongest = max(x[4] for x in candidates)
 
         for candidate in candidates:
-            candidate[-1] = (yongest - candidate[4]) / (yongest - oldest + 1)
+            candidate[-2] = (yongest - candidate[4]) / (yongest - oldest + 1)
 
     candidates = [Candidate(*x) for x in candidates]
     # Now have a list of candidates
@@ -279,19 +286,13 @@
                     with open(jobfile) as f:
                         details = [x.rstrip() for x in f.readlines()]
 
-                    stat = os.stat(jobfile)
-                    sha = hashlib.sha1()
-                    sha.update('%s %d %.3f %r' % (jobfile, stat.st_size, stat.st_mtime, details))
-                    digest = int(sha.hexdigest(), 16)
-                    job_id = digest % 2**31
-
+                    job_id = common.create_job_id(jobfile)
                     details.append('JOB_ID=%s' % job_id)
 
-                    with open(lock, 'w') as f:
-                        print >> f, host
-                        print >> f, '%s' % datetime.datetime.utcnow().isoformat()
-                        print >> f, 'JOB_ID=%s' % job_id
-                        return '%s\n\n%s' % (job, '\n'.join(details))
+                    common.write_lock_file(lock, host, job_id)
+
+                    return '%s\n\n%s' % (job, '\n'.join(details))
+
             finally:
                 fcntl.lockf(lockf.fileno(), fcntl.LOCK_UN)
         finally:
@@ -365,11 +366,8 @@
     def GET(self, queue, job):
         common.authenticate()
 
-        filename = common.map_filename('queue/%s/%s.job' % (queue, job), 'scheduler')
-
         try:
-            with file(filename, 'a'):
-                os.utime(filename, None)
+            common.schedule_job(queue, job)
         finally:
             pass
 
@@ -460,12 +458,28 @@
         extras = form.d.extras
 
         for queue in selected:
-            filename = common.map_filename('queue/%s/%s.job' % (queue, job), 'scheduler')
-
-            with open(filename, 'w') as f:
-                f.write(extras)
+            common.schedule_job(queue, job, contents=extras)
 
         msg = 'Spawned %s into %s' % (job, ', '.join(selected))
         return web.seeother('/helpers/scheduler?%s' % urllib.urlencode({'msg': msg}), absolute=True)
 
+
+class lava_testdef:
+
+    def GET(self, queue, job):
+        template = common.map_filename('queue/%s/lava-test-shell-template.yaml' % queue, 'scheduler')
+        queue_dir = common.map_filename('queue/%s/' % queue, 'scheduler') + '/'
+        job_file = os.path.join(queue_dir, job + '.job')
+        job_content = ""
+        with open(job_file) as f:
+            job_content = f.read()
+        job_content = quote(job_content)
+        # Yes, CR can be there %)
+        job_content = job_content.replace("\r", "")
+        job_content = job_content.replace("\n", "\\n")
+
+        with open(template) as f:
+            return f.read() % {'job': job, 'job_content': job_content}
+
+
 app = web.application(urls, globals())

=== added file 'tcwg-web.ini.local'
--- tcwg-web.ini.local	1970-01-01 00:00:00 +0000
+++ tcwg-web.ini.local	2013-01-18 12:55:25 +0000
@@ -0,0 +1,34 @@
+# Sample tcwg-web.ini for local development
+
+[general]
+development=True
+# Path below matches production install, it's recommended to follow it
+results=/home/cbuild/var
+
+[web]
+# Name, etc. of the server running web frontend
+# this is required for LAVA integration
+server_name=localhost
+server_port=8080
+
+[lava]
+user = user_here
+token = token_here
+# There're 2 choices:
+# a) if url is defined, LAVA jobs will be always submitted against this url.
+# Also, job definition may refer to this url for submitting results to
+# (by using "%(lava_server)s" substitution in "submit_results" action).
+# This allows to centrally control location of LAVA instance to work with
+# (and potentially, exotic scenarios like queuing job against one LAVA server
+# and  let it post results to another server).
+# b) if url is not defined, a url from jobdef's "submit_results" action
+# is used to submit job to. This allows to work with different LAVA servers
+# at the same time (one server per queue). This is useful for development/
+# testing for example (local LAVA install vs production).
+#url = http://validation.linaro.org/lava-server/RPC2/
+#plain_url = https://validation.linaro.org/lava-server/
+
+# username/token overrides for particular LAVA server
+[lava:https://validation.linaro.org/lava-server/RPC2/]
+user = user_here2
+token = token_here2

=== modified file 'templates/buildlog.html'
--- templates/buildlog.html	2012-11-30 02:31:03 +0000
+++ templates/buildlog.html	2013-01-18 12:55:25 +0000
@@ -4,7 +4,7 @@
 $ hosts = db['hosts']
 $ host_names = sorted(list(hosts), key=lambda x: hosts[x]+x)
 $ width = 100/(len(host_names)+1)
-$ hroot = 'http://cbuild.validation.linaro.org/build/'
+$ hroot = '/build/'
 
 $ builds = sorted(db['versions'])
 $ versions = db['versions']

=== modified file 'templates/proposals/index.html'
--- templates/proposals/index.html	2012-11-30 02:31:03 +0000
+++ templates/proposals/index.html	2013-01-18 12:55:25 +0000
@@ -5,7 +5,7 @@
 $ projects = db['lp'].projects.values()
 $ projects = [x for x in projects if x.get('proposals') and any(y.age < age_limit for y in x.proposals)]
 $ projects = sorted(projects, key=lambda x: x.name)
-$ root = 'http://cbuild.validation.linaro.org/build/'
+$ root = '/build/'
 $ ssh_root = 'ssh://cbuild@cbuild-master:/home/cbuild/public_html/build/'
 
 $ states = { 'Pending': 'white', 'Finished': 'lightgreen', 'Failed': 'pink', 'Regressed': 'gold' }

=== modified file 'templates/recent/index.html'
--- templates/recent/index.html	2012-11-30 02:31:03 +0000
+++ templates/recent/index.html	2013-01-18 12:55:25 +0000
@@ -1,7 +1,7 @@
 $def with (records, dbm)
 $var title = 'Recent'
 
-$ base = 'http://cbuild.validation.linaro.org/'
+$ base = ''
 $ spawn = '/helpers/scheduler/spawn'
 $ klasses = { 'benchmarks': 'aliceblue' }
 $ colours = (('gcc-4', '#e0e0ff'),)

=== modified file 'templates/scheduler.html'
--- templates/scheduler.html	2012-12-09 20:17:37 +0000
+++ templates/scheduler.html	2013-01-18 12:55:25 +0000
@@ -114,7 +114,11 @@
             bgcolor="#fbfbfb"
     >
         <td>$job.queue</td>
-        <td>$job.name</td>
+        <td>
+            $job.name
+            $if job.lava_url:
+                <a href="$job.lava_url">lava</a>
+        </td>
         <td>$elapsed_str(job.jobtime) ago</td>
         <td>
           $if job.host:

=== modified file 'templates/testcompare.html'
--- templates/testcompare.html	2012-11-30 02:31:03 +0000
+++ templates/testcompare.html	2013-01-18 12:55:25 +0000
@@ -1,6 +1,6 @@
 $def with (a, b, base, bases)
 $var title = 'Test Compare'
-$ builds = 'http://cbuild.validation.linaro.org/build/'
+$ builds = '/build/'
 
 <h1>Test result comparison</h1>
 Compare against

=== modified file 'templates/testlog.html'
--- templates/testlog.html	2012-11-30 02:31:03 +0000
+++ templates/testlog.html	2013-01-18 12:55:25 +0000
@@ -3,7 +3,7 @@
 
 <h1>Test Log</h1>
 
-$ root = 'http://cbuild.validation.linaro.org/build/' + path
+$ root = '/build/' + path
 
 Original log: <a href="$root">$root</a>.
 

=== modified file 'templates/testtimeline/index.html'
--- templates/testtimeline/index.html	2012-11-30 02:31:03 +0000
+++ templates/testtimeline/index.html	2013-01-18 12:55:25 +0000
@@ -8,7 +8,7 @@
 $for c in groups.values():
     $ configs |= set(c)
 $ configs = sorted(configs)
-$ root = 'http://cbuild.validation.linaro.org/build/'
+$ root = '/build/'
 
 $ configname = None
 $ binname = None