Merge lp:~dooferlad/linaro-ci-dashboard/get-command into lp:linaro-ci-dashboard

[James Tunnicliffe]

API changes + tests to support the CLI get command.

Also changed the Jenkins update behaviour so the user is told about a Jenkins update rather than required to perform the update. This is partially because updates can be slow and inconvenient (especially during a demo), but also because sometimes the Jenkins API changes and it can break how the dashboard interacts with it. A better fix would be to have a tested Jenkins version listed in the branch which is the one downloaded and checked against. That version can be bumped as a separate downlaod + test procedure.

[Milo Casagrande (milo)]

Hello James,

looks good to me. It took me a while to go through, also for fancy tastypie terminology for serializing objects.

Randomly, I have one error running the tests, but have no idea if it is related to Jenkins or something else. Last time I ran the tests I didn't have any.
Speaking of Jenkins, we should try to stick with what we will probably be using in ci.l.o, I guess we will be based on that.

review: Approve

=== modified file 'HACKING'
--- HACKING	2012-09-21 09:24:01 +0000
+++ HACKING	2013-02-26 12:48:40 +0000
@@ -49,6 +49,18 @@
 
 Tests should be written and defined inside each sub-application.
 
+When defining a new application, make sure the model app_label matches what is
+listed in settings.py:
+
+class CIJob(Loop):
+    """Defines the base class for a CI Job."""
+    class Meta:
+        app_label = 'ci_job'
+
+INSTALLED_APPS = (
+...
+    'frontend.ci_job',
+
 == Class Hierarchy ==
 
 === Templates & Views ===

=== modified file 'Makefile'
--- Makefile	2012-09-24 13:18:02 +0000
+++ Makefile	2013-02-26 12:48:40 +0000
@@ -24,6 +24,8 @@
 			echo "No changes to android_textfield_loop models."
 	python dashboard/manage.py schemamigration hwpack_loop --auto || \
 			echo "No changes to hwpack_loop models."
+	python dashboard/manage.py schemamigration ci_job --auto || \
+    			echo "No changes to ci_job models."
 	python dashboard/manage.py migrate
 
 migrate: syncdb

=== modified file 'README'
--- README	2013-02-01 11:49:24 +0000
+++ README	2013-02-26 12:48:40 +0000
@@ -17,6 +17,8 @@
   build-essential
   openjdk-6-jre-headless
   python-tastypie
+  python-simplejson
+  python-requests
 
 To unit test the API that tastypie provides you need Django version of at least
 1.4. If you have Django 1.3 the API will still work, but not be unit tested.

=== added file 'config_template/build_scripts/CIJob.sh'
--- config_template/build_scripts/CIJob.sh	1970-01-01 00:00:00 +0000
+++ config_template/build_scripts/CIJob.sh	2013-02-26 12:48:40 +0000
@@ -0,0 +1,2 @@
+touch done
+echo "done" > MANIFEST

=== modified file 'config_template/config.xml'
--- config_template/config.xml	2012-09-20 08:46:37 +0000
+++ config_template/config.xml	2013-02-26 12:48:40 +0000
@@ -50,7 +50,7 @@
   </views>
   <primaryView>All</primaryView>
   <slaveAgentPort>0</slaveAgentPort>
-  <label>IntegrationLoop AndroidLoop KernelLoop AndroidTextFieldLoop HwpackLoop</label>
+  <label>IntegrationLoop AndroidLoop KernelLoop AndroidTextFieldLoop HwpackLoop CIJob</label>
   <nodeProperties/>
   <globalNodeProperties/>
-</hudson>
\ No newline at end of file
+</hudson>

=== added file 'config_template/jenkins-jobs/CIJob.xml'
--- config_template/jenkins-jobs/CIJob.xml	1970-01-01 00:00:00 +0000
+++ config_template/jenkins-jobs/CIJob.xml	2013-02-26 12:48:40 +0000
@@ -0,0 +1,27 @@
+<?xml version='1.0' encoding='UTF-8'?>
+<project>
+  <actions/>
+  <description></description>
+  <keepDependencies>false</keepDependencies>
+  <properties/>
+  <scm class="hudson.scm.NullSCM"/>
+  <assignedNode>{assigned_node}</assignedNode>
+  <canRoam>false</canRoam>
+  <disabled>false</disabled>
+  <blockBuildWhenDownstreamBuilding>false</blockBuildWhenDownstreamBuilding>
+  <blockBuildWhenUpstreamBuilding>false</blockBuildWhenUpstreamBuilding>
+  <triggers class="vector"/>
+  <concurrentBuild>false</concurrentBuild>
+  <builders>
+    <hudson.tasks.Shell>
+      <command>{build_script}</command>
+    </hudson.tasks.Shell>
+  </builders>
+  <publishers>
+    <hudson.tasks.ArtifactArchiver>
+      <artifacts>*</artifacts>
+      <latestOnly>false</latestOnly>
+    </hudson.tasks.ArtifactArchiver>
+  </publishers>
+  <buildWrappers/>
+</project>

=== modified file 'dashboard/frontend/api/api.py'
--- dashboard/frontend/api/api.py	2013-02-04 10:59:49 +0000
+++ dashboard/frontend/api/api.py	2013-02-26 12:48:40 +0000
@@ -16,9 +16,13 @@
 # along with linaro-ci-dashboard.  If not, see <http://www.gnu.org/licenses/>.
 
 from tastypie.resources import ModelResource
-from tastypie.authorization import DjangoAuthorization
-from tastypie.authentication import ApiKeyAuthentication
+from tastypie.authorization import DjangoAuthorization, Authorization
+from tastypie.authentication import ApiKeyAuthentication, Authentication
 from dashboard.frontend.models.loop import Loop
+from frontend.ci_job.models.ci_job import CIJob
+from frontend.models.loop_build import LoopBuild
+from tastypie.constants import ALL, ALL_WITH_RELATIONS
+from tastypie import fields
 
 """
 Add API endpoint logic here. The documentation to do this can be found here:
@@ -54,3 +58,107 @@
         resource_name = 'loops'
         authorization = DjangoAuthorization()
         authentication = ApiKeyAuthentication()
+
+        filtering = {'name': ALL_WITH_RELATIONS}
+
+
+class CIJobResource(ModelResource):
+    class Meta:
+        """Returns a list of all loops."""
+        queryset = CIJob.objects.all()
+        resource_name = 'jobs'
+        list_allowed_methods = ['get', 'post']
+        detail_allowed_methods = ['get', 'post', 'put', 'delete']
+
+        #http://django-tastypie.readthedocs.org/en/latest/authorization.html
+        # Currently everyone who has an API key is equal. Suggest we tie this
+        # down using a custom Authorization class (see above link) so anyone
+        # can create and modify their own jobs, but only some users have
+        # the ability to delete and modify other users' jobs.
+        authorization = Authorization() #DjangoAuthorization()
+        authentication = ApiKeyAuthentication()
+
+
+class CIRunResource(ModelResource):
+    loop = fields.ForeignKey(LoopsResource, 'loop')
+
+    class Meta:
+        """Returns a list of all loops."""
+        queryset = LoopBuild.objects.all()
+        resource_name = 'runs'
+        list_allowed_methods = ['get', 'post']
+        detail_allowed_methods = ['get', 'post', 'put', 'delete']
+        filtering = {'name': 'exact',
+                     'build_number': 'exact',
+                     'loop': ALL_WITH_RELATIONS}
+
+        #http://django-tastypie.readthedocs.org/en/latest/authorization.html
+        # Currently everyone who has an API key is equal. Suggest we tie this
+        # down using a custom Authorization class (see above link) so anyone
+        # can create and modify their own jobs, but only some users have
+        # the ability to delete and modify other users' jobs.
+        authorization = Authorization() #DjangoAuthorization()
+        authentication = ApiKeyAuthentication()
+
+    def dehydrate(self, bundle):
+        data = {}
+        # We entirely rewrite this...
+        data["status"] = bundle.data["status"]
+        data["get_build_result"] = bundle.obj.get_build_result()
+        data["build_finished"] = bundle.obj.build_finished()
+        data["list_artifacts"] = bundle.obj.list_artifacts()
+        data["build_number"] = bundle.data["build_number"]
+        data["name"] = bundle.obj.loop.name
+        #data["get_artifact_url"] = bundle.obj.get_artifact_url()
+        #data["get_artifact_content"] = bundle.obj.get_artifact_content()
+        bundle.data = data
+        return bundle
+
+    def full_hydrate(self, bundle):
+        """
+        Given a populated bundle, distill it and turn it back into
+        a full-fledged object instance.
+
+        This is a copy of the tastypie full_hydrate code with a single
+        modification to not try to fill in LoopBuild.loop from the API supplied
+        data.
+
+        Note that this won't work because we won't create a full object:
+        loop_object = self.fields["loop"]
+        del(self.fields["loop"])
+        super(self.__class__, self).full_hydrate(bundle)
+        self.fields["loop"] = loop_object
+        """
+        if bundle.obj is None:
+            bundle.obj = self._meta.object_class()
+
+        for field_name, field_object in self.fields.items():
+            # Fill in all attributes apart from loop, which is created during
+            # the (later) hydrate phase.
+            if field_object.attribute and field_object.attribute != "loop":
+                value = field_object.hydrate(bundle)
+
+                if value is not None or field_object.null:
+                    # We need to avoid populating M2M data here as that will
+                    # cause things to blow up.
+                    if not getattr(field_object, 'is_related', False):
+                        setattr(bundle.obj, field_object.attribute, value)
+                    elif not getattr(field_object, 'is_m2m', False):
+                        if value is not None:
+                            setattr(bundle.obj, field_object.attribute,
+                                    value.obj)
+
+            # Check for an optional method to do further hydration.
+            method = getattr(self, "hydrate_%s" % field_name, None)
+
+            if method:
+                bundle = method(bundle)
+
+        bundle = self.hydrate(bundle)
+        return bundle
+
+    def hydrate(self, bundle):
+        bundle.obj.loop = Loop.objects.get(name=bundle.data["name"])
+        bundle.obj.status = 'unscheduled'
+        bundle.obj.duration = 0.0
+        return bundle

=== modified file 'dashboard/frontend/api/urls.py'
--- dashboard/frontend/api/urls.py	2013-02-04 17:55:49 +0000
+++ dashboard/frontend/api/urls.py	2013-02-26 12:48:40 +0000
@@ -7,6 +7,8 @@
 # Register new resources here (probably defined in api.py)
 v1_api.register(LoginTestResource())
 v1_api.register(LoopsResource())
+v1_api.register(CIJobResource())
+v1_api.register(CIRunResource())
 
 urlpatterns = patterns('',
    (r'^api/', include(v1_api.urls)),

=== added directory 'dashboard/frontend/ci_job'
=== added file 'dashboard/frontend/ci_job/__init__.py'
=== added directory 'dashboard/frontend/ci_job/migrations'
=== added file 'dashboard/frontend/ci_job/migrations/0001_initial.py'
--- dashboard/frontend/ci_job/migrations/0001_initial.py	1970-01-01 00:00:00 +0000
+++ dashboard/frontend/ci_job/migrations/0001_initial.py	2013-02-26 12:48:40 +0000
@@ -0,0 +1,60 @@
+# -*- coding: utf-8 -*-
+import datetime
+from south.db import db
+from south.v2 import SchemaMigration
+from django.db import models
+
+
+class Migration(SchemaMigration):
+
+    def forwards(self, orm):
+        # Adding model 'CIJob'
+        db.create_table('ci_job_cijob', (
+            ('loop_ptr', self.gf('django.db.models.fields.related.OneToOneField')(to=orm['frontend.Loop'], unique=True, primary_key=True)),
+            ('vcs_url', self.gf('django.db.models.fields.CharField')(max_length=255)),
+            ('vcs_branch', self.gf('django.db.models.fields.CharField')(default='', max_length=255)),
+            ('vcs_tag', self.gf('django.db.models.fields.CharField')(default='', max_length=255)),
+            ('file_name', self.gf('django.db.models.fields.CharField')(max_length=255)),
+            ('class_name', self.gf('django.db.models.fields.CharField')(max_length=255)),
+            ('run_when_ready', self.gf('django.db.models.fields.BooleanField')(default=False)),
+        ))
+        db.send_create_signal('ci_job', ['CIJob'])
+
+
+    def backwards(self, orm):
+        # Deleting model 'CIJob'
+        db.delete_table('ci_job_cijob')
+
+
+    models = {
+        'ci_job.cijob': {
+            'Meta': {'object_name': 'CIJob', '_ormbases': ['frontend.Loop']},
+            'class_name': ('django.db.models.fields.CharField', [], {'max_length': '255'}),
+            'file_name': ('django.db.models.fields.CharField', [], {'max_length': '255'}),
+            'loop_ptr': ('django.db.models.fields.related.OneToOneField', [], {'to': "orm['frontend.Loop']", 'unique': 'True', 'primary_key': 'True'}),
+            'run_when_ready': ('django.db.models.fields.BooleanField', [], {'default': 'False'}),
+            'vcs_branch': ('django.db.models.fields.CharField', [], {'default': "''", 'max_length': '255'}),
+            'vcs_tag': ('django.db.models.fields.CharField', [], {'default': "''", 'max_length': '255'}),
+            'vcs_url': ('django.db.models.fields.CharField', [], {'max_length': '255'})
+        },
+        'frontend.loop': {
+            'Meta': {'object_name': 'Loop'},
+            'id': ('django.db.models.fields.AutoField', [], {'primary_key': 'True'}),
+            'is_official': ('django.db.models.fields.BooleanField', [], {'default': 'False'}),
+            'is_restricted': ('django.db.models.fields.BooleanField', [], {'default': 'False'}),
+            'lava_tests': ('django.db.models.fields.CharField', [], {'max_length': '255', 'null': 'True', 'blank': 'True'}),
+            'name': ('django.db.models.fields.CharField', [], {'unique': 'True', 'max_length': '200'}),
+            'next_loop': ('django.db.models.fields.related.OneToOneField', [], {'related_name': "'previous_loop'", 'null': 'True', 'on_delete': 'models.SET_NULL', 'to': "orm['frontend.Loop']", 'blank': 'True', 'unique': 'True'}),
+            'server': ('django.db.models.fields.related.ForeignKey', [], {'default': 'None', 'to': "orm['jenkinsserver.JenkinsServer']", 'null': 'True', 'on_delete': 'models.SET_NULL'}),
+            'type': ('django.db.models.fields.CharField', [], {'max_length': '50', 'blank': 'True'})
+        },
+        'jenkinsserver.jenkinsserver': {
+            'Meta': {'object_name': 'JenkinsServer'},
+            'id': ('django.db.models.fields.AutoField', [], {'primary_key': 'True'}),
+            'password': ('django.db.models.fields.CharField', [], {'max_length': '100'}),
+            'url': ('django.db.models.fields.CharField', [], {'max_length': '255'}),
+            'username': ('django.db.models.fields.CharField', [], {'max_length': '100'})
+        }
+    }
+
+    complete_apps = ['ci_job']
\ No newline at end of file

=== added file 'dashboard/frontend/ci_job/migrations/__init__.py'
=== added directory 'dashboard/frontend/ci_job/models'
=== added file 'dashboard/frontend/ci_job/models/__init__.py'
--- dashboard/frontend/ci_job/models/__init__.py	1970-01-01 00:00:00 +0000
+++ dashboard/frontend/ci_job/models/__init__.py	2013-02-26 12:48:40 +0000
@@ -0,0 +1,1 @@
+from frontend.ci_job.models.ci_job import *

=== added file 'dashboard/frontend/ci_job/models/ci_job.py'
--- dashboard/frontend/ci_job/models/ci_job.py	1970-01-01 00:00:00 +0000
+++ dashboard/frontend/ci_job/models/ci_job.py	2013-02-26 12:48:40 +0000
@@ -0,0 +1,135 @@
+# Copyright (C) 2012 Linaro
+#
+# This file is part of linaro-ci-dashboard.
+#
+# linaro-ci-dashboard is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# linaro-ci-dashboard is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with linaro-ci-dashboard.  If not, see <http://www.gnu.org/licenses/>.
+
+from django.db import models
+from frontend.models.loop import Loop
+from jenkinsserver.models.jenkins_server import JenkinsServer
+from frontend.models.loop_build import LoopBuild
+#from django.contrib.auth.models import User
+
+
+class CIJob(Loop):
+    """Defines the base class for a CI Job."""
+    class Meta:
+        app_label = 'ci_job'
+
+    vcs_url = models.CharField(max_length=255,
+                                   verbose_name='Configuration repository')
+    vcs_branch = models.CharField(max_length=255,
+                                     verbose_name='Configuration branch',
+                                     default='')
+    vcs_tag = models.CharField(max_length=255,
+                                  verbose_name='Configuration tag',
+                                  default='')
+    file_name = models.CharField(max_length=255,
+                                   verbose_name='Configuration file name')
+    class_name = models.CharField(max_length=255,
+                                    verbose_name='Configuration class name')
+    run_when_ready = models.BooleanField(default=False)
+
+    #job_owner = models.ForeignKey(User, editable=False,
+    #                              verbose_name='Job Owner')
+
+    def save(self, *args, **kwargs):
+        self.server = JenkinsServer.objects.get(id=1)
+        self.type = self.__class__.__name__
+        super(self.__class__, self).save(*args, **kwargs)
+        # We do not want a failure in remote server to affect the
+        # CI Loop flow.
+
+        self.server.create_or_update_loop(self)
+
+
+    def schedule_build(self, parameters=None):
+        parameters = {'parameter': [{'delay': '0'}]}
+        return super(self.__class__, self).schedule_build(parameters)
+
+    def base64_config(self,
+                      include=None,
+                      exclude=None,
+                      capitalize=False,
+                      upper=False):
+        exclude = ['id', 'server', 'loop_ptr', 'next_loop', 'name', 'type']
+        return super(CIJob, self).base64_config(include=include,
+                                                      exclude=exclude,
+                                                      capitalize=capitalize,
+                                                      upper=True)
+
+    def preconfigure(self, configuration=None):
+        """Makes a build parameter dict from build result configuration.
+
+        This method should be overridden by subclasses.
+        """
+        if isinstance(configuration, dict):
+            # The name is the name of the loop that is calling the chained one,
+            # not the one we want to actually configure, we drop it.
+            configuration.pop('name')
+            unmatched_values = u''
+            for key, value in configuration.iteritems():
+                if hasattr(self, key.lower()):
+                    setattr(self, key.lower(), value)
+                else:
+                    # If the attr cannot be found, we throw everything in
+                    # user_defined_values since the text field is free form.
+                    unmatched_values = u'%s=%s\n' % (key.upper(), str(value))
+
+            self.user_defined_values = unmatched_values
+            self.save()
+        else:
+            self.log.error("Parameter was not a 'dict' instance.")
+        return {}
+
+    @models.permalink
+    def get_detail_url(self):
+        return 'CIJobDetail', (), {'slug': self.name}
+
+    @models.permalink
+    def get_build_url(self):
+        return 'CIJobBuild', (), {'slug': self.name}
+
+    @models.permalink
+    def get_update_url(self):
+        return 'CIJobUpdate', (), {'slug': self.name}
+
+    @classmethod
+    def schedule_unscheduled_jobs(self):
+        unscheduled_builds = LoopBuild.objects.filter(status='unscheduled')
+        for build in unscheduled_builds:
+            build.status = 'scheduled'
+
+            build.remote_number = build.loop.server.schedule_job_build(
+                                    build.loop.name)
+            build.save()
+
+    def process_build_results(self, loop_build):
+        """This method gets called with list of artifact files from a build.
+        Its responsibility is to categorize artifacts, and prepare result data
+        to be passed to downstream loop in chain. If needed, this method can
+        access content of artifacts (not just list of them).
+        """
+
+        all_files_zip = "*zip*/archive.zip"
+        urls = {
+            all_files_zip:
+                loop_build.get_artifact_url(all_files_zip),
+        }
+
+        files = loop_build.list_artifacts()
+        for filename in files:
+            urls[filename] = loop_build.get_artifact_url(filename)
+
+        return urls

=== added file 'dashboard/frontend/ci_job/urls.py'
--- dashboard/frontend/ci_job/urls.py	1970-01-01 00:00:00 +0000
+++ dashboard/frontend/ci_job/urls.py	2013-02-26 12:48:40 +0000
@@ -0,0 +1,31 @@
+# Copyright (C) 2012 Linaro
+#
+# This file is part of linaro-ci-dashboard.
+#
+# linaro-ci-dashboard is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# linaro-ci-dashboard is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with linaro-ci-dashboard.  If not, see <http://www.gnu.org/licenses/>.
+
+from django.conf.urls.defaults import patterns, url
+
+from frontend.ci_job.views.ci_job_detail_view \
+    import CIJobDetailView
+from frontend.ci_job.views.ci_job_build_view \
+    import CIJobBuildView
+
+
+urlpatterns = patterns('',
+    url(r'^ci_job/detail/(?P<slug>[\w|\W]+)/$',
+        CIJobDetailView.as_view(), name='CIJobDetail'),
+    url(r'^ci_job/build/(?P<slug>[\w|\W]+)/$',
+        CIJobBuildView.as_view(), name='CIJobBuild'),
+)

=== added directory 'dashboard/frontend/ci_job/views'
=== added file 'dashboard/frontend/ci_job/views/__init__.py'
=== added file 'dashboard/frontend/ci_job/views/ci_job_build_view.py'
--- dashboard/frontend/ci_job/views/ci_job_build_view.py	1970-01-01 00:00:00 +0000
+++ dashboard/frontend/ci_job/views/ci_job_build_view.py	2013-02-26 12:48:40 +0000
@@ -0,0 +1,25 @@
+#!/usr/bin/env python
+# Copyright (C) 2012 Linaro
+#
+# This file is part of linaro-ci-dashboard.
+#
+# linaro-ci-dashboard is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# linaro-ci-dashboard is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with linaro-ci-dashboard.  If not, see <http://www.gnu.org/licenses/>.
+
+from frontend.ci_job.models.ci_job import CIJob
+from frontend.views.loop_build_view import LoopBuildView
+
+
+class CIJobBuildView(LoopBuildView):
+
+    model = CIJob

=== added file 'dashboard/frontend/ci_job/views/ci_job_detail_view.py'
--- dashboard/frontend/ci_job/views/ci_job_detail_view.py	1970-01-01 00:00:00 +0000
+++ dashboard/frontend/ci_job/views/ci_job_detail_view.py	2013-02-26 12:48:40 +0000
@@ -0,0 +1,24 @@
+# Copyright (C) 2012 Linaro
+#
+# This file is part of linaro-ci-dashboard.
+#
+# linaro-ci-dashboard is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# linaro-ci-dashboard is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with linaro-ci-dashboard.  If not, see <http://www.gnu.org/licenses/>.
+
+from frontend.ci_job.models.ci_job import CIJob
+from frontend.views.loop_detail_view import LoopDetailView
+
+
+class CIJobDetailView(LoopDetailView):
+
+    model = CIJob

=== modified file 'dashboard/frontend/management/commands/jenkins.py'
--- dashboard/frontend/management/commands/jenkins.py	2012-09-18 13:06:35 +0000
+++ dashboard/frontend/management/commands/jenkins.py	2013-02-26 12:48:40 +0000
@@ -144,9 +144,9 @@
         downloaded_size = int(os.stat(filename).st_size)
         response.close()
         if actual_size != downloaded_size:
-            raise CommandError("Size mismatch between remote and local "\
-                               "jenkins.war, remove %s to upgrade to new "\
-                               "version" % filename)
+            print "Size mismatch between remote and local "\
+                  "jenkins.war, remove %s to upgrade to new "\
+                  "version" % filename
 
         return 0
 

=== added file 'dashboard/frontend/management/commands/tick.py'
--- dashboard/frontend/management/commands/tick.py	1970-01-01 00:00:00 +0000
+++ dashboard/frontend/management/commands/tick.py	2013-02-26 12:48:40 +0000
@@ -0,0 +1,36 @@
+# Copyright (C) 2013 Linaro
+#
+# This file is part of linaro-ci-dashboard.
+#
+# linaro-ci-dashboard is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# linaro-ci-dashboard is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with linaro-ci-dashboard.  If not, see <http://www.gnu.org/licenses/>.
+# Django settings for dashboard project.
+
+from django.core.management.base import BaseCommand
+from frontend.ci_job.models.ci_job import CIJob
+from jenkinsserver.models.jenkins_server import JenkinsServer
+from django.conf import settings
+
+
+class Command(BaseCommand):
+    args = ""
+    help = "Run background tasks. Needs to be run periodically."
+
+    def __init__(self):
+        super(self.__class__, self).__init__()
+        self.jenkins_id = settings.JENKINS_DB_ID
+
+    def handle(self, *args, **options):
+        CIJob.schedule_unscheduled_jobs()
+        server = JenkinsServer.objects.get(id=self.jenkins_id)
+        server.sync_jobs_and_builds()

=== modified file 'dashboard/frontend/models/loop_reference.py'
--- dashboard/frontend/models/loop_reference.py	2012-09-18 12:16:59 +0000
+++ dashboard/frontend/models/loop_reference.py	2013-02-26 12:48:40 +0000
@@ -23,3 +23,4 @@
 INTEGRATION_LOOP = ('IntegrationLoop')
 ANDROID_TEXTFIELD_LOOP = ('AndroidTextFieldLoop')
 HWPACK_LOOP = ('HwpackLoop')
+CI_JOB = ('CIJob')

=== modified file 'dashboard/frontend/tests/test_api.py'
--- dashboard/frontend/tests/test_api.py	2013-02-04 17:55:49 +0000
+++ dashboard/frontend/tests/test_api.py	2013-02-26 12:48:40 +0000
@@ -19,9 +19,12 @@
 from django.test import LiveServerTestCase
 from tastypie.models import ApiKey
 from dashboard.frontend.models.loop import Loop
+from dashboard.frontend.models.loop_build import LoopBuild
 from dashboard.frontend.android_build.models.android_loop import AndroidLoop
+from dashboard.frontend.ci_job.models.ci_job import CIJob
 import requests
 from urlparse import urljoin
+import simplejson
 
 
 class ApiTests(LiveServerTestCase):
@@ -32,13 +35,18 @@
     def setUp(self):
         super(ApiTests, self).setUp()
 
+        self.settings = {}
+        self.settings["ci_server"] = "http://localhost:8081"
+        self.settings["api_prefix"] = "/api/dev/"
+        self.settings["username"] = 'daniel'
+        self.settings["password"] = 'pass'
+        self.settings["api_key"] = "0"
+
         # Create a user.
-        self.username = 'daniel'
-        self.password = 'pass'
         self.user = User.objects.create_user(
-                        self.username, '[e-mail]', self.password)
-
-        self.api_key = "0"
+            self.settings["username"],
+            '[e-mail]',
+            self.settings["password"])
 
         # Add a couple of jobs to test against.
         self.android_loop = AndroidLoop()
@@ -51,16 +59,35 @@
         self.loop.type = Loop.__class__.__name__
         self.loop.save()
 
-    def get(self, url):
-        payload = {"username": self.username,
-                   "api_key": self.api_key,
-                   "format": "json"}
-        url = urljoin("http://localhost:8081", "/api/dev/" + url)
-        return requests.get(url, params=payload)
+    def get(self, url, in_params=None):
+        url, params = self._prep_request(url, in_params)
+        return requests.get(url, params=params)
+
+    def post(self, url, data, in_params=None):
+        url, params = self._prep_request(url, in_params)
+        return requests.post(url, data=data, params=params,
+                             headers={"content-type": "application/json"})
+
+    def _prep_request(self, url, in_params=None):
+        params = {"username": self.settings["username"],
+                  "api_key": self.settings["api_key"],
+                  "format": "json"}
+
+        url = urljoin(self.settings["ci_server"],
+                      self.settings["api_prefix"] + url)
+
+        # If we don't have a trailing slash, we will just be redirected. Avoid
+        # the extra call.
+        if url[-1] != "/":
+            url += "/"
+
+        if in_params:
+            params.update(in_params)
+
+        return url, params
 
     def login(self):
-        self.api_key = ApiKey.objects.create(user=self.user).key
-        #self.create_apikey(self.username, self.api_key)
+        self.settings["api_key"] = ApiKey.objects.create(user=self.user).key
 
     def test_login_test_unauthorzied(self):
         r = self.get("login_test")
@@ -93,3 +120,71 @@
         api_test_names = {json[0]['name'], json[1]['name']}
         test_names = {self.android_loop.name, self.loop.name}
         self.assertEqual(api_test_names, test_names)
+
+    def test_add_job(self):
+        self.login()
+        data = {
+            "vcs_url": "vcsurl",
+            "file_name": "filename",
+            "class_name": "classname",
+            "run": False,
+            "name": "jobname"
+        }
+        json_data = simplejson.dumps(data)
+
+        r = self.post("jobs", json_data, {})
+        self.assertEqual(r.status_code, requests.codes.created)
+
+        jobs = CIJob.objects.all()
+
+        self.assertEqual(len(jobs), 1)
+        self.assertEqual(jobs[0].name, data["name"])
+
+    def test_run_job(self):
+        self.login()
+
+        # First, create a dummy job
+        job = CIJob()
+        job.vcs_url = "vcsurl"
+        job.name = "jobname"
+        job.save()
+
+        # Now run it
+        data = {
+            "name": "jobname"
+        }
+        json_data = simplejson.dumps(data)
+
+        r = self.post("runs", json_data, {})
+        self.assertEqual(r.status_code, requests.codes.created)
+
+        runs = LoopBuild.objects.all()
+
+        self.assertEqual(len(runs), 1)
+        self.assertEqual(runs[0].loop.name, data["name"])
+
+    def test_get_job(self):
+        self.login()
+
+        # Create a dummy job
+        job = CIJob()
+        job.vcs_url = "vcsurl"
+        job.name = "jobname"
+        job.save()
+
+        # Create a dummy run
+        run = LoopBuild()
+        run.build_number = 1
+        run.status = "SUCCESS"
+        run.duration = 0
+        run.loop = job
+        run.save()
+
+        r = self.get("runs")
+        self.assertEqual(r.status_code, requests.codes.ok)
+
+        run_data = r.json()["objects"]
+        self.assertEqual(len(run_data), 1)
+        self.assertEqual(run_data[0]["build_number"], run.build_number)
+        self.assertEqual(run_data[0]["status"], run.status)
+        self.assertEqual(run_data[0]["name"], job.name)

=== modified file 'dashboard/frontend/views/index_view.py'
--- dashboard/frontend/views/index_view.py	2012-09-13 11:37:13 +0000
+++ dashboard/frontend/views/index_view.py	2013-02-26 12:48:40 +0000
@@ -25,6 +25,7 @@
 from frontend.android_textfield_loop.models.android_textfield_loop \
     import AndroidTextFieldLoop
 from frontend.hwpack_loop.models.hwpack_loop import HwpackLoop
+from frontend.ci_job.models.ci_job import CIJob
 
 class IndexView(TemplateView):
 
@@ -42,4 +43,5 @@
         context['kernel_loops'] = KernelLoop.objects.all()
         context['android_textfield_loops'] = AndroidTextFieldLoop.objects.all()
         context['hwpack_loops'] = HwpackLoop.objects.all()
+        context['ci_jobs'] = CIJob.objects.all()
         return context

=== modified file 'dashboard/jenkinsserver/models/jenkins_server.py'
--- dashboard/jenkinsserver/models/jenkins_server.py	2012-10-04 15:26:28 +0000
+++ dashboard/jenkinsserver/models/jenkins_server.py	2013-02-26 12:48:40 +0000
@@ -24,6 +24,7 @@
 from lib.logger import Logger
 from lib.template import TextTemplate, XMLTemplate
 from dashboard.lib.xml_to_dict import DictToXml
+import requests
 
 
 class JenkinsServer(models.Model):
@@ -89,6 +90,8 @@
         # to store as build results.
         loop = build.loop.get_child_object()
         values = loop.process_build_results(build)
+        values["*console_url*"] = self.jenkins.get_build_output_url(
+                                        build.loop.name, build.remote_number)
         build.result_xml = DictToXml(values).dict_to_tree()
         build.save()
 
@@ -152,7 +155,15 @@
 
     def schedule_job_build(self, jobname, parameters=None):
         """Trigger build job on jenkins. Return the next build number."""
-        self.jenkins.build_job(jobname, parameters)
+
+        # Note that python-jenkins uses urllib2 in such a way that new job
+        # API requests will always be an HTTP GET. In recent versions of
+        # Jenkins this is invalid. Using requests to work around this, but
+        # using the python-jenkins library to generate the job URL.
+        requests.post(self.jenkins.build_job_url(jobname),
+                      auth=(settings.JENKINS_ADMIN_USER,
+                            settings.JENKINS_ADMIN_PASSWD),
+                      params=parameters)
         job_info = self.jenkins.get_job_info(jobname)
         return job_info["nextBuildNumber"]
 

=== modified file 'dashboard/lib/jenkins_dashboard.py'
--- dashboard/lib/jenkins_dashboard.py	2012-09-24 08:21:17 +0000
+++ dashboard/lib/jenkins_dashboard.py	2013-02-26 12:48:40 +0000
@@ -43,11 +43,14 @@
             raise jenkins.JenkinsException(
                 "Could not parse JSON info for job[%s]" % name)
 
+    def get_build_output_url(self, name, build_number):
+        return self.server + self.BUILD_CONSOLE % locals()
+
     def get_build_output(self, name, build_number):
 
         try:
             response = self.jenkins_open(
-                urllib2.Request(self.server + self.BUILD_CONSOLE % locals()))
+                urllib2.Request(self.get_build_output_url(name, build_number)))
             if response:
                 return response
             else:

=== modified file 'dashboard/lib/xml_to_dict.py'
--- dashboard/lib/xml_to_dict.py	2012-09-07 12:48:04 +0000
+++ dashboard/lib/xml_to_dict.py	2013-02-26 12:48:40 +0000
@@ -48,7 +48,7 @@
         :param xml: The XML tree as a string.
         :type xml str
         """
-        if not isinstance(xml, str):
+        if not (isinstance(xml, str) or isinstance(xml, unicode)):
             raise XmlToDictException("Type of parameter is not 'str', "
                                      "got '%s' instead." % type(xml).__name__)
         super(XmlToDict, self).__init__()

=== modified file 'dashboard/settings.py'
--- dashboard/settings.py	2013-01-30 17:20:40 +0000
+++ dashboard/settings.py	2013-02-26 12:48:40 +0000
@@ -158,6 +158,7 @@
     'frontend.android_textfield_loop',
     'frontend.hwpack_loop',
     'frontend.api',
+    'frontend.ci_job',
     'tastypie',
     'south',
     # Uncomment the next line to enable admin documentation:

=== modified file 'dashboard/urls.py'
--- dashboard/urls.py	2013-01-21 16:16:03 +0000
+++ dashboard/urls.py	2013-02-26 12:48:40 +0000
@@ -40,4 +40,5 @@
     url(r'^', include('dashboard.frontend.android_textfield_loop.urls')),
     url(r'^', include('dashboard.frontend.hwpack_loop.urls')),
     url(r'^', include('dashboard.frontend.api.urls')),
+    url(r'^', include('dashboard.frontend.ci_job.urls')),
 )

=== modified file 'requirements.txt'
--- requirements.txt	2013-02-01 11:39:17 +0000
+++ requirements.txt	2013-02-26 12:48:40 +0000
@@ -12,4 +12,5 @@
 python-jenkins==0.2
 python-openid==2.2.5
 requests==1.1.0
+simplejson==3.0.7
 wsgiref==0.1.2