Merge lp:~pfalcon/linaro-ci-dashboard/kernel_chain into lp:linaro-ci-dashboard

[Paul Sokolovsky]

This is preliminary merge proposal for Kernel->Hwpack loop chaining implementation. This so far lacks tests, on which I'm working in the meantime, but I'd like to collect any early feedback. As usual with more or less big changesets, it may be more comfortable to review the changes commit by commit, instead of aggregated diff.

Otherwise, these changes already underwent some refactoring comparing to the initial implementation I coded, and should be pretty architecturally clean IMHO. The main change is to introduce Loop method process_build_results() which is called at the end of build to process build outcome and prepare build results which can be used by downsteam builds, and to add LoopBuild API which process_build_results() can use to query artifacts of a build (because artifacts is the only outcome of a build in Jenkins, so any "build results" would be produced by analyzing artifacts). Some other convenience changes (like providing default implementation of Loop.preconfigure()) were also introduced.

[lp:~pfalcon/linaro-ci-dashboard/kernel_chain]

69. By Paul Sokolovsky on 2012-09-21

[merge] Merge from trunk.

70. By Paul Sokolovsky on 2012-09-21

Typo fix.

[Stevan Radaković (stevanr)]

Nice work Paul, thanks!!

I have no minor things to note, architecture looks pretty good.
One thing I'd like to see though are tests. There aren't any for one single method we implemented here.

review: Needs Fixing

[Milo Casagrande (milo)]

Hi Paul!

Some comments follow inline.

On Fri, Sep 21, 2012 at 10:34 AM, Paul Sokolovsky
<email address hidden> wrote:
> https://code.launchpad.net/~pfalcon/linaro-ci-dashboard/kernel_chain/+merge/125646
> Your team Linaro Infrastructure is requested to review the proposed merge of lp:~pfalcon/linaro-ci-dashboard/kernel_chain into lp:linaro-ci-dashboard.
>
> === modified file 'config_template/jenkins-jobs/AndroidLoop.xml'
> --- config_template/jenkins-jobs/AndroidLoop.xml 2012-09-18 15:08:05 +0000
> +++ config_template/jenkins-jobs/AndroidLoop.xml 2012-09-21 08:33:27 +0000
> @@ -178,7 +178,7 @@
> </builders>
> <publishers>
> <hudson.tasks.ArtifactArchiver>
> - <artifacts>build/out/target/*/*/*.img.bz2,build/out/target/*/*/*.tar.bz2,build/out/target/*/*/MD5SUMS,build/out/*.tar.bz2,build/out/*.xml,build/out/*_config,build/out/lava-job-info,build/out/linaro_kernel_build_cmds.sh,build/out/linaro_android_build_cmds.sh,build/out/**/*EULA*</artifacts>
> + <artifacts>build/out/target/*/*/*.img.bz2,build/out/target/*/*/*.tar.bz2,build/out/target/*/*/MD5SUMS,build/out/*.tar.bz2,build/out/*.xml,build/out/*_config,build/out/lava-job-info,build/out/linaro_kernel_build_cmds.sh,build/out/linaro_android_build_cmds.sh,build/out/**/*EULA*,MANIFEST</artifacts>

Is it possible to be more greedy here? Like "*.img.*", so to avoid
file suffix, or there might be other file types that we do not want?
Minor suggestion though...

> @models.permalink
> def get_detail_url(self):
> return 'KernelLoopDetail', (), {'slug': self.name}
> @@ -135,3 +141,17 @@
> @models.permalink
> def get_update_url(self):
> return 'KernelLoopUpdate', (), {'slug': self.name}
> +
> + def process_build_results(self, loop_build):
> + """This method gets called with list of artifact files from a build.
> + Its responsibility is to categorize artifacts, and prepare result data
> + to be passed to downstream loop in chain. If needed, this method can
> + access content of artifacts (not just list of them).
> + """
> + files = loop_build.list_artifacts()
> + kernel_debs = filter(lambda fname: "linux-image" in fname, files)
> + if len(kernel_debs) == 1:
> + return {"kernel_deb_url": loop_build.get_artifact_url(kernel_debs[0])}
> + else:
> + self.log.warning("No single kernel deb found among: %s", files)
> + return {}

This is more a personal thing than anything else: why not having only
a single exit point? Just using a single return statement.

Overall it looks good and clean, maybe a couple of tests wouldn't hurt.
Thanks!

--
Milo Casagrande
Infrastructure Engineer
Linaro.org <www.linaro.org> │ Open source software for ARM SoCs

[lp:~pfalcon/linaro-ci-dashboard/kernel_chain]

71. By Paul Sokolovsky on 2012-09-21

preconfigure(): Check that argument is not None before iterating over it.

72. By Paul Sokolovsky on 2012-09-24

Factor out variable with Jenkins URL to ease debugging.

This regulalrly needs to be dumped to find out what's wrong with Jenkins.

73. By Paul Sokolovsky on 2012-09-24

Add integration test for for artifact querying methods of JenkinsServer.

This required making IntegrationLoop to export artifacts (also exposes
shell code injection with loop build scripts). TODO: Provide special Loop
just for integration testing.

74. By Paul Sokolovsky on 2012-10-04

Update another reference self.get_build_info() -> self.update_build_info()

75. By Paul Sokolovsky on 2012-10-04

self.remote_number can be None during tests, so print it as string.

76. By Paul Sokolovsky on 2012-10-04

Add high-level unit test for kernel => hwpack loop chaining.

77. By Paul Sokolovsky on 2012-10-04

[merge] Merge from trunk.

[Paul Sokolovsky (pfalcon)]

Milo:

> Is it possible to be more greedy here? Like "*.img.*", so to avoid
file suffix, or there might be other file types that we do not want?
Minor suggestion though...

Well, if you looked carefully, what I did to that line was adding "MANIFEST" file import. The original line likely was added by yourself while working on Android loop. But as it apparently was taken from a build config as set up be me, here's reply to the very question:

What about "1.img.something.completely.different" file? So yes, it's grounded conservatism imho. Of course, if there're good reasons to change that, we can.

>This is more a personal thing than anything else: why not having only
a single exit point? Just using a single return statement.

Single statement of course would be possible, but that would program transformation skewing the original idea of the code. It's because 2 branches have different semantics: one executes if data is valid and returns expected processed data, while another executes when data invalid and returns "data invalid" token. To make the point more clear, in the next incarnation of this function, return in "data invalid" branch may become raise.

[Paul Sokolovsky (pfalcon)]

Ok, so I finally get to complete tests for this. 2 tests were added:

1. Integrational test for new JenkinsServer methods to query artifacts.
2. Unit tests which mocks JenkinsServer methods and assures proper data flow between upstream and downstream tests in chain.

[Stevan Radaković (stevanr)]

Looks good to me Paul, thanks!
Approve +1

review: Approve

[Milo Casagrande (milo)]

Hi Paul,

thanks for fixing this, and also for the previous comments.
It looks good to me, I only have some errors when running the tests, but overall all the tests pass (total number is 96). Is that normal?

This is the kind of Python errors I see:
https://pastebin.linaro.org/991/

[Stevan Radaković (stevanr)]

Strange I don't get such errors. Let's check tomorrow with Paul.

[Paul Sokolovsky (pfalcon)]

Milo:

Here's my story: I didn't do work on ci-dashboard for a while, so when I restarted with work on these tested, I was greeted with the need to upgrade jenkins.war. I did, and then on first "python dashboard/manage.py test" run I got errors similar to those you quoted, except that in my case there were more errors "JenkinsException: create[testjob_integration] failed" and no "JenkinsException: build[2] does not exist for job test". I almost prepared to scratch my head and blame spurious jenkins upgrades that we have so far, but then I decided just to rerun the testsuite. Those errors were gone. There was another one though:

====
FAIL: test_login_teams (django_openid_auth.tests.test_views.RelyingPartyTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/pymodules/python2.7/django_openid_auth/tests/test_views.py", line 359, in test_login_teams
    self.assertTrue(group in user.groups.all())
AssertionError: False is not true
====

But that's known by all of us (?) heizenbug, except that it has much lower oscillation frequency (i.e. I have it stable, including with the trunk now even with several reruns, tomorrow it may be gone, day after appear again).

So, answering your question: it's "normal" (in quotes), in the sense that it's matter of fact that it does happen for more than one developer. It's not normal in other senses of this word, and we should take measures to investigate and fix those non-deterministic failures. First measure would be to use well-known jenkins version, instead of non-deterministic one - I proposed that some time ago, but didn't make that change at that time, so now it seems time has come.

=== modified file 'config_template/jenkins-jobs/AndroidLoop.xml'
--- config_template/jenkins-jobs/AndroidLoop.xml	2012-09-18 15:08:05 +0000
+++ config_template/jenkins-jobs/AndroidLoop.xml	2012-10-04 15:43:21 +0000
@@ -178,7 +178,7 @@
   </builders>
   <publishers>
     <hudson.tasks.ArtifactArchiver>
-      <artifacts>build/out/target/*/*/*.img.bz2,build/out/target/*/*/*.tar.bz2,build/out/target/*/*/MD5SUMS,build/out/*.tar.bz2,build/out/*.xml,build/out/*_config,build/out/lava-job-info,build/out/linaro_kernel_build_cmds.sh,build/out/linaro_android_build_cmds.sh,build/out/**/*EULA*</artifacts>
+      <artifacts>build/out/target/*/*/*.img.bz2,build/out/target/*/*/*.tar.bz2,build/out/target/*/*/MD5SUMS,build/out/*.tar.bz2,build/out/*.xml,build/out/*_config,build/out/lava-job-info,build/out/linaro_kernel_build_cmds.sh,build/out/linaro_android_build_cmds.sh,build/out/**/*EULA*,MANIFEST</artifacts>
       <latestOnly>false</latestOnly>
     </hudson.tasks.ArtifactArchiver>
     <hudson.plugins.logparser.LogParserPublisher>

=== modified file 'config_template/jenkins-jobs/HwpackLoop.xml'
--- config_template/jenkins-jobs/HwpackLoop.xml	2012-09-18 14:44:17 +0000
+++ config_template/jenkins-jobs/HwpackLoop.xml	2012-10-04 15:43:21 +0000
@@ -19,7 +19,7 @@
   </builders>
   <publishers>
     <hudson.tasks.ArtifactArchiver>
-      <artifacts>*.tar.gz</artifacts>
+      <artifacts>*.tar.gz,MANIFEST</artifacts>
       <latestOnly>false</latestOnly>
     </hudson.tasks.ArtifactArchiver>
   </publishers>

=== modified file 'config_template/jenkins-jobs/IntegrationLoop.xml'
--- config_template/jenkins-jobs/IntegrationLoop.xml	2012-09-17 10:27:00 +0000
+++ config_template/jenkins-jobs/IntegrationLoop.xml	2012-10-04 15:43:21 +0000
@@ -17,6 +17,11 @@
       <command>{build_script}</command>
     </hudson.tasks.Shell>
   </builders>
-  <publishers/>
+  <publishers>
+    <hudson.tasks.ArtifactArchiver>
+      <artifacts>*.result,MANIFEST</artifacts>
+      <latestOnly>false</latestOnly>
+    </hudson.tasks.ArtifactArchiver>
+  </publishers>
   <buildWrappers/>
 </project>

=== modified file 'config_template/jenkins-jobs/KernelLoop.xml'
--- config_template/jenkins-jobs/KernelLoop.xml	2012-09-18 13:29:37 +0000
+++ config_template/jenkins-jobs/KernelLoop.xml	2012-10-04 15:43:21 +0000
@@ -19,7 +19,7 @@
   </builders>
   <publishers>
     <hudson.tasks.ArtifactArchiver>
-      <artifacts>*.deb</artifacts>
+      <artifacts>*.deb,MANIFEST</artifacts>
       <latestOnly>false</latestOnly>
     </hudson.tasks.ArtifactArchiver>
   </publishers>

=== modified file 'dashboard/frontend/hwpack_loop/models/hwpack_loop.py'
--- dashboard/frontend/hwpack_loop/models/hwpack_loop.py	2012-09-17 13:00:42 +0000
+++ dashboard/frontend/hwpack_loop/models/hwpack_loop.py	2012-10-04 15:43:21 +0000
@@ -32,7 +32,7 @@
 
     # Version of hwpack tarball to build (passed directly to
     # linaro-hwpack-create)
-    hwpack_build_version = models.CharField(max_length=15)
+    hwpack_build_version = models.CharField(max_length=15, default="1")
 
     # URL of kernel deb package to use for hwpack creation
     kernel_deb_url = models.CharField(max_length=255, default="", blank=True)

=== modified file 'dashboard/frontend/kernel_build/models/kernel_loop.py'
--- dashboard/frontend/kernel_build/models/kernel_loop.py	2012-09-12 12:48:46 +0000
+++ dashboard/frontend/kernel_build/models/kernel_loop.py	2012-10-04 15:43:21 +0000
@@ -19,6 +19,7 @@
 from django.db import models
 from frontend.models.loop import Loop
 from jenkinsserver.models.jenkins_server import JenkinsServer
+from frontend.models.loop_reference import HWPACK_LOOP
 
 
 LLT_URL = (r'git://git.linaro.org/kernel/linux-linaro-tracking.git',
@@ -124,6 +125,11 @@
             # TODO: Log error.
             pass
 
+    @staticmethod
+    def can_chain_into():
+        chain = [HWPACK_LOOP]
+        return chain
+
     @models.permalink
     def get_detail_url(self):
         return 'KernelLoopDetail', (), {'slug': self.name}
@@ -135,3 +141,17 @@
     @models.permalink
     def get_update_url(self):
         return 'KernelLoopUpdate', (), {'slug': self.name}
+
+    def process_build_results(self, loop_build):
+        """This method gets called with list of artifact files from a build.
+        Its responsibility is to categorize artifacts, and prepare result data
+        to be passed to downstream loop in chain. If needed, this method can
+        access content of artifacts (not just list of them).
+        """
+        files = loop_build.list_artifacts()
+        kernel_debs = filter(lambda fname: "linux-image" in fname, files)
+        if len(kernel_debs) == 1:
+            return {"kernel_deb_url": loop_build.get_artifact_url(kernel_debs[0])}
+        else:
+            self.log.warning("No single kernel deb found among: %s", files)
+            return {}

=== modified file 'dashboard/frontend/kernel_build/tests/__init__.py'
--- dashboard/frontend/kernel_build/tests/__init__.py	2012-08-23 11:22:21 +0000
+++ dashboard/frontend/kernel_build/tests/__init__.py	2012-10-04 15:43:21 +0000
@@ -1,9 +1,11 @@
 from dashboard.frontend.kernel_build.tests.test_kernel_build_views import *
 from dashboard.frontend.kernel_build.tests.test_kernel_build_clientresponse \
     import *
+from dashboard.frontend.kernel_build.tests.test_kernel_chaining import *
 
 
 __test__ = {
     'KernelBuildViewsTest': KernelBuildViewsTest,
     'KernelBuildClientResponseTests': KernelBuildClientResponseTests,
+    'KernelHwpackChainingTest': KernelHwpackChainingTest,
 }

=== added file 'dashboard/frontend/kernel_build/tests/test_kernel_chaining.py'
--- dashboard/frontend/kernel_build/tests/test_kernel_chaining.py	1970-01-01 00:00:00 +0000
+++ dashboard/frontend/kernel_build/tests/test_kernel_chaining.py	2012-10-04 15:43:21 +0000
@@ -0,0 +1,70 @@
+# Copyright (C) 2012 Linaro
+#
+# This file is part of linaro-ci-dashboard.
+#
+# linaro-ci-dashboard is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# linaro-ci-dashboard is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with linaro-ci-dashboard.  If not, see <http://www.gnu.org/licenses/>.
+
+from django.test import TestCase
+from mock import patch
+
+from jenkinsserver.models.jenkins_server import JenkinsServer
+from frontend.models.loop_build import LoopBuild
+from frontend.kernel_build.models.kernel_loop import KernelLoop
+from frontend.hwpack_loop.models.hwpack_loop import HwpackLoop
+
+
+class KernelHwpackChainingTest(TestCase):
+
+    def setUp(self):
+        self.server = JenkinsServer.objects.get(id=1)
+
+    @patch('jenkinsserver.models.jenkins_server.JenkinsServer.create_or_update_loop')
+    @patch('jenkinsserver.models.jenkins_server.JenkinsServer.get_build_artifact_content')
+    @patch('lib.jenkins_dashboard.JenkinsDashboard.get_build_info')
+    def test_chaining(self, get_build_info_m, get_build_artifact_content_m, mck1):
+        kernel_package = "linux-image.deb"
+        get_build_info_m.return_value = {"result": LoopBuild.SUCCESS}
+        get_build_artifact_content_m.return_value = kernel_package + "\n"
+
+        hwpack_loop = HwpackLoop()
+        hwpack_loop.name = "hwpack_test"
+        hwpack_loop.server = self.server
+        hwpack_loop.save()
+        self.assertFalse(hwpack_loop.kernel_deb_url)
+
+        loop = KernelLoop()
+        loop.name = "kernel_test"
+        loop.server = self.server
+        loop.next_loop = hwpack_loop
+        loop.save()
+        build = loop.schedule_build()
+
+        self.server.sync_unfinished_builds()
+
+        # Reload object from DB in Django-speak
+        build = LoopBuild.objects.get(id=build.id)
+
+        # Make sure kernel build status/results were updated
+        self.assertEquals(build.status, LoopBuild.SUCCESS)
+        self.assertTrue(kernel_package in build.result_xml)
+
+        # Make sure results from kernel loop were propagated to hwpack
+        hwpack_loop = HwpackLoop.objects.get(id=hwpack_loop.id)
+        self.assertTrue(kernel_package in hwpack_loop.kernel_deb_url)
+
+        # Make sure that hwpack build was scheduled
+        self.assertEquals(len(LoopBuild.objects.all()), 2)
+        hwpack_build = LoopBuild.objects.get(loop=hwpack_loop)
+        self.assertTrue(hwpack_build)
+        self.assertEquals(hwpack_build.status, LoopBuild.SCHEDULED)

=== modified file 'dashboard/frontend/models/loop.py'
--- dashboard/frontend/models/loop.py	2012-09-20 13:51:27 +0000
+++ dashboard/frontend/models/loop.py	2012-10-04 15:43:21 +0000
@@ -259,8 +259,26 @@
 
     def preconfigure(self, configuration=None):
         """Makes a build parameter dict from build result configuration.
-
-        This method should be overridden by subclasses.
+        Default implementation just assigns loop's attributes from dictionary
+        keys. This method may be overridden by subclasses.
+
+        :return Extra Jenkins build scheduling parameters. TODO: breaks separation
+        """
+        if configuration:
+            for key, value in configuration.iteritems():
+                if hasattr(self, key):
+                    setattr(self, key, value)
+                else:
+                    self.log.warning("Input build parameters for %s contain "\
+                            "unknown key/value pair %s=%s", self, key, value)
+            self.save()
+        return {}
+
+    def process_build_results(self, loop_build):
+        """This method gets called with list of artifact files from a build.
+        Its responsibility is to categorize artifacts, and prepare result data
+        to be passed to downstream loop in chain. If needed, this method can
+        access content of artifacts (not just list of them).
         """
         return {}
 

=== modified file 'dashboard/frontend/models/loop_build.py'
--- dashboard/frontend/models/loop_build.py	2012-09-19 10:50:22 +0000
+++ dashboard/frontend/models/loop_build.py	2012-10-04 15:43:21 +0000
@@ -84,3 +84,41 @@
     def build_finished(self):
         """Method to be called after the build is executed."""
         self.loop.continue_down_the_chain(self.get_build_result())
+
+    def list_artifacts(self):
+        """
+        List artifacts as produced by the build.
+
+        :return List containing full paths of artifacts files
+        """
+        return self.loop.server.list_build_artifacts(self.loop.name,
+                 self.remote_number)
+
+    def get_artifact_url(self, artifact_name):
+        """
+        Get a URL from which given artifact can be downloaded (for example,
+        for passing to a build script).
+
+        :param artifact_name Full artifact path, as returned by
+                             list_artifacts()
+        :return A URL
+        """
+        return self.loop.server.get_build_artifact_url(self.loop.name,
+                 self.remote_number, artifact_name)
+
+    def get_artifact_content(self, artifact_name):
+        """
+        Get contents of an artifact. This function loads entire artifact
+        into memory, so should not be used with big artifacts.
+
+        :param artifact_name Full artifact path, as returned by
+                             list_artifacts()
+        :return Raw binary data of artifact
+        """
+        return self.loop.server.get_build_artifact_url(self.loop.name,
+                 self.remote_number, artifact_name)
+
+    def __repr__(self):
+        return "%s(id=%s, loop.name=%s, remote_numer=%s)" % \
+            (self.__class__.__name__, self.id, self.loop.name,
+             self.remote_number)

=== modified file 'dashboard/frontend/models/loop_reference.py'
--- dashboard/frontend/models/loop_reference.py	2012-08-31 11:53:01 +0000
+++ dashboard/frontend/models/loop_reference.py	2012-10-04 15:43:21 +0000
@@ -22,3 +22,4 @@
 KERNEL_LOOP = ('KernelLoop')
 INTEGRATION_LOOP = ('IntegrationLoop')
 ANDROID_TEXTFIELD_LOOP = ('AndroidTextFieldLoop')
+HWPACK_LOOP = ('HwpackLoop')

=== modified file 'dashboard/jenkinsserver/models/jenkins_server.py'
--- dashboard/jenkinsserver/models/jenkins_server.py	2012-09-18 16:01:01 +0000
+++ dashboard/jenkinsserver/models/jenkins_server.py	2012-10-04 15:43:21 +0000
@@ -16,11 +16,14 @@
 # You should have received a copy of the GNU Affero General Public License
 # along with linaro-ci-dashboard.  If not, see <http://www.gnu.org/licenses/>.
 
+import urllib2
+
 from django.db import models
 from django.conf import settings
 from lib.jenkins_dashboard import JenkinsDashboard
 from lib.logger import Logger
 from lib.template import TextTemplate, XMLTemplate
+from dashboard.lib.xml_to_dict import DictToXml
 
 
 class JenkinsServer(models.Model):
@@ -71,9 +74,9 @@
                         # update the build info in CI.
                         if build.remote_number and \
                             jenkins_build["number"] == build.remote_number:
-                            self.get_build_info(build)
+                            self.update_build_info(build)
 
-    def get_build_info(self, build):
+    def update_build_info(self, build):
         jenkins_build = self.jenkins.get_build_info(build.loop.name,
                                                     build.remote_number)
         # Duration in jenkins is presented in milliseconds
@@ -82,18 +85,22 @@
 
         if jenkins_build.get("result", None):
             build.status = jenkins_build.get("result", None).lower()
-        # TODO: Add build result xml, but from where?
+        # Let loop process the build outcome and extract anything relevant
+        # to store as build results.
+        loop = build.loop.get_child_object()
+        values = loop.process_build_results(build)
+        build.result_xml = DictToXml(values).dict_to_tree()
         build.save()
 
     def sync_unfinished_builds(self):
         from frontend.models.loop_build import LoopBuild
         builds = LoopBuild.objects.filter(
-            status__in=LoopBuild.NON_FINISHED_STATUSES)
+            status__in=LoopBuild.NON_FINISHED_STATUSES).order_by('id')
         for build in builds:
             # We don't wont the sync proces to fail if there is no particular
             # loop present in jenkins. Log the error and continue.
             try:
-                self.get_build_info(build)
+                self.update_build_info(build)
             except:
                 self.log.exception("build[%s] does not exist for job %s" % (build.remote_number, build.loop.name))
                 pass
@@ -148,3 +155,27 @@
         self.jenkins.build_job(jobname, parameters)
         job_info = self.jenkins.get_job_info(jobname)
         return job_info["nextBuildNumber"]
+
+    def get_build_artifact_url(self, jobname, build_no, artifact_name):
+        url = settings.ARTIFACT_TEMPLATE_URL % locals()
+        return url
+
+    def get_build_artifact_content(self, jobname, build_no, artifact_name):
+        "Get contents of a build artifact."
+        url = self.get_build_artifact_url(jobname, build_no, artifact_name)
+        response = urllib2.urlopen(url)
+        contents = response.read()
+        response.close()
+        return contents
+
+    def list_build_artifacts(self, jobname, build_no):
+        try:
+            manifest = self.get_build_artifact_content(jobname, build_no, "MANIFEST")
+        except urllib2.HTTPError, e:
+            if e.code == 404:
+                self.log.warn("MANIFEST artifact not found for %s/%s, assumed empty", jobname, build_no)
+                manifest = ""
+            else:
+                raise
+        files = [f for f in manifest.split("\n") if f]
+        return files

=== modified file 'dashboard/jenkinsserver/tests/test_jenkins_server.py'
--- dashboard/jenkinsserver/tests/test_jenkins_server.py	2012-09-18 16:01:01 +0000
+++ dashboard/jenkinsserver/tests/test_jenkins_server.py	2012-10-04 15:43:21 +0000
@@ -15,9 +15,12 @@
 #
 # You should have received a copy of the GNU Affero General Public License
 # along with linaro-ci-dashboard.  If not, see <http://www.gnu.org/licenses/>.
+import time
+
 from django.test import TestCase
 from jenkinsserver.models.jenkins_server import JenkinsServer
 import jenkins
+from frontend.models.loop_build import LoopBuild
 from frontend.integration_loop.models.integration_loop import IntegrationLoop
 
 
@@ -118,3 +121,35 @@
             self.assertIsNotNone(build.duration)
             self.assertIsNot(build.duration, 0.0)
         self.server.delete_job(self.test_job_name)
+
+    def test_build_artifacts(self):
+        integration_loop = IntegrationLoop()
+        integration_loop.name = self.test_job_name
+        integration_loop.server = self.server
+        integration_loop.branch = 'none:none || mkdir -p '
+        integration_loop.precommand = 'cd ..; echo "foobar" >file.result'
+        integration_loop.command = 'ls -1 *.result >MANIFEST'
+        integration_loop.save()
+        build = integration_loop.schedule_build()
+        for i in range(10):
+            self.server.sync_unfinished_builds()
+            # Reload object from DB in Django-speak
+            build = LoopBuild.objects.get(id=build.id)
+            if build.status not in (LoopBuild.SCHEDULED, LoopBuild.RUNNING):
+                break
+            time.sleep(2)
+        self.assertEquals(build.status, LoopBuild.SUCCESS)
+        # We could test LoopBuild methods here, that would give us even more
+        # coverage, but - we're testing JenkinsServer here, so let's test it.
+        artifacts = self.server.list_build_artifacts(integration_loop.name,
+            build.remote_number)
+        self.assertEquals(artifacts, ["file.result"])
+
+        url = self.server.get_build_artifact_url(integration_loop.name,
+            build.remote_number, artifacts[0])
+        self.assertTrue(url.startswith("http://"))
+        self.assertTrue(artifacts[0] in url)
+
+        data = self.server.get_build_artifact_content(integration_loop.name,
+            build.remote_number, artifacts[0])
+        self.assertEquals(data, "foobar\n")

=== modified file 'dashboard/lib/jenkins_dashboard.py'
--- dashboard/lib/jenkins_dashboard.py	2012-09-18 15:06:02 +0000
+++ dashboard/lib/jenkins_dashboard.py	2012-10-04 15:43:21 +0000
@@ -30,8 +30,9 @@
     def get_build_info(self, name, build_number):
 
         try:
+            url = self.server + self.BUILD_INFO % locals()
             response = self.jenkins_open(
-                urllib2.Request(self.server + self.BUILD_INFO % locals()))
+                urllib2.Request(url))
             if response:
                 return json.loads(response)
             else:

=== modified file 'dashboard/settings.py'
--- dashboard/settings.py	2012-09-25 08:07:10 +0000
+++ dashboard/settings.py	2012-10-04 15:43:21 +0000
@@ -227,6 +227,10 @@
 JENKINS_LOG_NAME = 'jenkins.log'
 JENKINS_DB_ID = 1
 
+# URL from which build artifacts can be downloaded
+ARTIFACT_TEMPLATE_URL = "http://localhost:" + JENKINS_PORT + \
+    "/jenkins/job/%(jobname)s/%(build_no)s/artifact/%(artifact_name)s"
+
 # LAVA specific values
 LAVA_URL = 'validation.linaro.org/lava-server/RPC2/'
 LAVA_ACCEPTING_ADDRESS = '0.0.0.0'