Merge lp:~stevanr/linaro-ci-dashboard/poll_jenkins into lp:linaro-ci-dashboard

[Stevan Radaković]

Add a jenkins server custom command to sync the unfinished builds to frontend.
We should now have a ready command to put it in the cronjob, and can quite easy implement command to do periodic sync from jenkins. This other proposal should be discussed and maybe implemented in this MP.

[Milo Casagrande (milo)]

=== modified file 'dashboard/frontend/models/loop_build.py'
--- dashboard/frontend/models/loop_build.py 2012-09-10 12:25:33 +0000
+++ dashboard/frontend/models/loop_build.py 2012-09-10 17:16:19 +0000
@@ -34,6 +34,7 @@
         ('aborted', 'ABORTED'),
     )
     FINISHED_STATUSES = ["failure", "success"]
+ NON_FINISHED_STATUSES = ["scheduled", "running"]

I was thinking about defining single variables for each status, and then use that instead of writing the "real" value each time:
FAILURE = 'failure'
...
FINISHED_STATUSES = [FAILURE, ...]

@@ -73,14 +71,23 @@
                         # update the build info in CI.
                         if build.remote_number and \
                             jenkins_build["number"] == build.remote_number:
-
- request = urllib2.Request(jenkins_build["url"] +
- "api/json")
- response = urllib2.urlopen(request).read()
- response_dict = json.loads(response)
- build.status = response_dict["result"]
- build.duration = response_dict["duration"]
- build.save()
+ self.get_build_info()
+
+ def get_build_info(self, build):
+ jenkins_build = self.jenkins.get_build_info(build.loop.name,
+ build.remote_number)
+ # Duration in jenkins is presented in milliseconds
+ build.duration = float(jenkins_build["duration"]) / 1000
+ build.status = jenkins_build["result"].lower()
+ # TODO: Add build result xml, but from where?

This is a good point to discuss: the build.result_xml is probably different from the XML we want to store here, since it is coming from Jenkins and that will hold results valid or for another build, or valid by themselves for this build. If they are different, probably it makes sense to store the XML in a different field of the model (jenkins_xml), maybe using the same XML structure or extending our XML DTD, or we can update the XML we already have with Jenkins information. But with the latter I would say to extend the DTD and have a specific jenkins element in there to retrieve the necessary values in one shot.

=== added file 'dashboard/lib/jenkins_dashboard.py'
--- dashboard/lib/jenkins_dashboard.py 1970-01-01 00:00:00 +0000
+++ dashboard/lib/jenkins_dashboard.py 2012-09-10 17:16:19 +0000
@@ -0,0 +1,56 @@
+# Copyright (C) 2012 Linaro
+#
+# This file is part of linaro-ci-dashboard.
+#
+# linaro-ci-dashboard is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# linaro-ci-dashboard is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU Affero General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public Li...

review: Needs Fixing

[lp:~stevanr/linaro-ci-dashboard/poll_jenkins]

47. By Stevan Radaković on 2012-09-11

Fixes from code review.

48. By Stevan Radaković on 2012-09-11

Fixes from code review.

[Stevan Radaković (stevanr)]

Agreed to push this now for demo purposes, whether or not we have result_xml present. Additional MP will be created for that.

[Paul Sokolovsky (pfalcon)]

Sorry for the late comment, but there're more Jenkins build statuses than added so far, e.g. "NOT_BUILT" (used by current android-build for example), may have made sense to add them all.

[Stevan Radaković (stevanr)]

Well we will not care about Jenkins statuses in the end, but the statuses that make sense to us (which should be part of a different discussion and MP as well). Anyway those that we do not have and Jenkins does should be 'translated' as corresponding statuses in ci dashboard app.

=== modified file 'README'
--- README	2012-07-31 14:43:21 +0000
+++ README	2012-09-11 15:52:19 +0000
@@ -34,6 +34,9 @@
 To sync the jenkins and CI Loop builds run:
   $ python dashboard/manage.py jenkins sync
 
+To get the latest build information from jenkins to frontend app, run:
+  $ python dashboard/manage.py jenkins_build_sync
+
 PS: Please go through the settings.py JENKINS_* params to change
 jenkins related configurations
 
 
=== modified file 'dashboard/frontend/models/loop_build.py'
--- dashboard/frontend/models/loop_build.py	2012-09-10 12:25:33 +0000
+++ dashboard/frontend/models/loop_build.py	2012-09-11 15:52:19 +0000
@@ -26,14 +26,21 @@
         app_label = 'frontend'
         ordering = ['-id']
 
+    SCHEDULED = "scheduled"
+    RUNNING = "running"
+    FAILURE = "failure"
+    SUCCESS = "success"
+    ABORTED = "aborted"
+
     BUILD_STATUS = (
-        ('scheduled', 'SCHEDULED'),
-        ('running', 'RUNNING'),
-        ('failure', 'FAILURE'),
-        ('success', 'SUCCESS'),
-        ('aborted', 'ABORTED'),
+        (SCHEDULED, 'SCHEDULED'),
+        (RUNNING, 'RUNNING'),
+        (FAILURE, 'FAILURE'),
+        (SUCCESS, 'SUCCESS'),
+        (ABORTED, 'ABORTED'),
     )
-    FINISHED_STATUSES = ["failure", "success"]
+    FINISHED_STATUSES = [FAILURE, SUCCESS]
+    NON_FINISHED_STATUSES = [SCHEDULED, RUNNING]
 
     loop = models.ForeignKey(Loop)
     build_number = models.IntegerField(default=None)
 
=== added directory 'dashboard/jenkinsserver/management'
=== added file 'dashboard/jenkinsserver/management/__init__.py'
=== added directory 'dashboard/jenkinsserver/management/commands'
=== added file 'dashboard/jenkinsserver/management/commands/__init__.py'
=== added file 'dashboard/jenkinsserver/management/commands/jenkins_build_sync.py'
--- dashboard/jenkinsserver/management/commands/jenkins_build_sync.py	1970-01-01 00:00:00 +0000
+++ dashboard/jenkinsserver/management/commands/jenkins_build_sync.py	2012-09-11 15:52:19 +0000
@@ -0,0 +1,30 @@
+# Copyright (C) 2012 Linaro
+#
+# This file is part of linaro-ci-dashboard.
+#
+# linaro-ci-dashboard is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# linaro-ci-dashboard is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with linaro-ci-dashboard.  If not, see <http://www.gnu.org/licenses/>.
+# Django settings for dashboard project.
+
+from django.core.management.base import BaseCommand, CommandError
+from jenkinsserver.models.jenkins_server import JenkinsServer
+from lib.jenkins_dashboard import JenkinsDashboard
+
+
+class Command(BaseCommand):
+    help = 'Executes the jenkins non-finished build sync process once.'
+
+    def handle(self, *args, **options):
+
+        server = JenkinsServer.objects.get(id=1)
+        server.sync_unfinished_builds()
 
=== modified file 'dashboard/jenkinsserver/models/jenkins_server.py'
--- dashboard/jenkinsserver/models/jenkins_server.py	2012-09-10 17:15:24 +0000
+++ dashboard/jenkinsserver/models/jenkins_server.py	2012-09-11 15:52:19 +0000
@@ -18,9 +18,7 @@
 
 from django.db import models
 from django.conf import settings
-import jenkins
-import json
-import urllib2
+from lib.jenkins_dashboard import JenkinsDashboard
 from lib.logger import Logger
 from lib.template import TextTemplate, XMLTemplate
 
@@ -36,7 +34,7 @@
     def __init__(self, *args, **kwargs):
         super(JenkinsServer, self).__init__(*args, **kwargs)
         self.log = Logger.getClassLogger(self)
-        self.jenkins = jenkins.Jenkins(self.url,
+        self.jenkins = JenkinsDashboard(self.url,
                                        self.username.encode('utf-8'),
                                        self.password.encode('utf-8'))
 
@@ -73,14 +71,23 @@
                         # update the build info in CI.
                         if build.remote_number and \
                             jenkins_build["number"] == build.remote_number:
-
-                            request = urllib2.Request(jenkins_build["url"] +
-                                                      "api/json")
-                            response = urllib2.urlopen(request).read()
-                            response_dict = json.loads(response)
-                            build.status = response_dict["result"]
-                            build.duration = response_dict["duration"]
-                            build.save()
+                            self.get_build_info()
+
+    def get_build_info(self, build):
+        jenkins_build = self.jenkins.get_build_info(build.loop.name,
+                                                    build.remote_number)
+        # Duration in jenkins is presented in milliseconds
+        build.duration = float(jenkins_build["duration"]) / 1000
+        build.status = jenkins_build["result"].lower()
+        # TODO: Add build result xml, but from where?
+        build.save()
+
+    def sync_unfinished_builds(self):
+        from frontend.models.loop_build import LoopBuild
+        builds = LoopBuild.objects.filter(
+            status__in=LoopBuild.NON_FINISHED_STATUSES)
+        for build in builds:
+            self.get_build_info(build)
 
     def create_job(self, jname, jxml):
         """ Creates a new job on jenkins if its  not already present """
 
=== added file 'dashboard/jenkinsserver/tests/test_custom_commands.py'
--- dashboard/jenkinsserver/tests/test_custom_commands.py	1970-01-01 00:00:00 +0000
+++ dashboard/jenkinsserver/tests/test_custom_commands.py	2012-09-11 15:52:19 +0000
@@ -0,0 +1,33 @@
+#!/usr/bin/env python
+# Copyright (C) 2012 Linaro
+#
+# This file is part of linaro-ci-dashboard.
+#
+# linaro-ci-dashboard is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# linaro-ci-dashboard is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with linaro-ci-dashboard.  If not, see <http://www.gnu.org/licenses/>.
+
+from django.core.management.base import CommandError
+from django.conf import settings
+from django.test import TestCase
+
+
+class PollJenkinsTest(TestCase):
+
+    def setUp(self):
+        pass
+
+    def test_find_and_update_user_non_existing(self):
+        pass
+
+    def test_find_and_update_user(self):
+        pass
 
=== added file 'dashboard/lib/jenkins_dashboard.py'
--- dashboard/lib/jenkins_dashboard.py	1970-01-01 00:00:00 +0000
+++ dashboard/lib/jenkins_dashboard.py	2012-09-11 15:52:19 +0000
@@ -0,0 +1,58 @@
+# Copyright (C) 2012 Linaro
+#
+# This file is part of linaro-ci-dashboard.
+#
+# linaro-ci-dashboard is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# linaro-ci-dashboard is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with linaro-ci-dashboard.  If not, see <http://www.gnu.org/licenses/>.
+# Django settings for dashboard project.
+
+import jenkins
+import json
+import urllib2
+
+
+class JenkinsDashboard(jenkins.Jenkins):
+    """ Extension to the python-jenkins library."""
+
+    BUILD_INFO = 'job/%(name)s/%(build_number)s/api/json'
+    BUILD_CONSOLE = 'job/%(name)s/%(build_number)s/consoleText'
+
+    def get_build_info(self, name, build_number):
+
+        try:
+            response = self.jenkins_open(
+                urllib2.Request(self.server + self.BUILD_INFO % locals()))
+            if response:
+                return json.loads(response)
+            else:
+                raise JenkinsException('job[%s] does not exist' % name)
+        except urllib2.HTTPError:
+            raise JenkinsException('job[%s] does not exist' % name)
+        except ValueError:
+            raise JenkinsException(
+                "Could not parse JSON info for job[%s]" % name)
+
+    def get_build_output(self, name, build_number):
+
+        try:
+            response = self.jenkins_open(
+                urllib2.Request(self.server + self.BUILD_CONSOLE % locals()))
+            if response:
+                return response
+            else:
+                raise JenkinsException('job[%s] does not exist' % name)
+        except urllib2.HTTPError:
+            raise JenkinsException('job[%s] does not exist' % name)
+        except ValueError:
+            raise JenkinsException(
+                "Could not parse JSON info for job[%s]" % name)