Merge lp:~linaro-infrastructure/linaro-ci-dashboard/create_job_jenkins_changes into lp:linaro-ci-dashboard

[Deepti B. Kalakeri]

Adding changes to create /delete job on jenkins server

[lp:~linaro-infrastructure/linaro-ci-dashboard/create_job_jenkins_changes]

15. By Deepti B. Kalakeri on 2012-07-20

Fix to check if the values are Null

[Stevan Radaković (stevanr)]

Hey Deepti, thanks for the good work..
Comments following:

initial_data.json is missing the jenkinsserver record as we talked. It is in my branch as well so you can leave it as it is, but someone else trying to run the application will be confused.

Also, create job method breaks when the name of the method has 'whitespace' in it. It creates job on jenkins, but only with the first string. I guess some escaping needs to be done here.

Basically we can't consider this done until we have a build actually working on the jenkins when it is scheduled to run. The following problems and possible solutions follow:
1. I also get a 'permission denied(public key)' on the job build when it tries to 'branch' the code from LP.
I've tried it an putting HOME=/tmp/ bzr branch... worked.
2. Also after the code is checked out, the script needs to position itself in the directory which is checkout.
i.e. cd branch_dir
3. We need to clean up the branch_dir either before the build or after (i think it's better before)

Let's get these few problems resolved then we can take a look at the code implementation.

review: Needs Fixing (code)

[Deepti B. Kalakeri (deeptik)]

> Hey Deepti, thanks for the good work..
> Comments following:
>
> initial_data.json is missing the jenkinsserver record as we talked. It is in
> my branch as well so you can leave it as it is, but someone else trying to run
> the application will be confused.

Yes, I did not wanted to overlap your changes, so did not make it part of the MP.
>
> Also, create job method breaks when the name of the method has 'whitespace' in
> it. It creates job on jenkins, but only with the first string. I guess some
> escaping needs to be done here.

I will fix this.
>
> Basically we can't consider this done until we have a build actually working
> on the jenkins when it is scheduled to run. The following problems and
> possible solutions follow:
> 1. I also get a 'permission denied(public key)' on the job build when it tries
> to 'branch' the code from LP.
I was able to get the jobs started to build once I Clicked on the Build button.
I did not see this error though. How does your input in the loopcreate form looks like?

> I've tried it an putting HOME=/tmp/ bzr branch... worked.

> 2. Also after the code is checked out, the script needs to position itself in
> the directory which is checkout.

Yes, correct, so my form inputs looks like this:
Name: Job_name
branch: linaro-ci-dashboard
precommand: cd linaro-ci-dashboard; make syncdb
command: make test

> i.e. cd branch_dir
> 3. We need to clean up the branch_dir either before the build or after (i
> think it's better before)
We can make it part of the command.
>
> Let's get these few problems resolved then we can take a look at the code
> implementation.

[Stevan Radaković (stevanr)]

> > Hey Deepti, thanks for the good work..
> > Comments following:
> >
> > initial_data.json is missing the jenkinsserver record as we talked. It is in
> > my branch as well so you can leave it as it is, but someone else trying to
> run
> > the application will be confused.
>
> Yes, I did not wanted to overlap your changes, so did not make it part of the
> MP.
> >
> > Also, create job method breaks when the name of the method has 'whitespace'
> in
> > it. It creates job on jenkins, but only with the first string. I guess some
> > escaping needs to be done here.
>
> I will fix this.
> >
> > Basically we can't consider this done until we have a build actually working
> > on the jenkins when it is scheduled to run. The following problems and
> > possible solutions follow:
> > 1. I also get a 'permission denied(public key)' on the job build when it
> tries
> > to 'branch' the code from LP.
> I was able to get the jobs started to build once I Clicked on the Build
> button.
> I did not see this error though. How does your input in the loopcreate form
> looks like?
>
>
> > I've tried it an putting HOME=/tmp/ bzr branch... worked.
>
> > 2. Also after the code is checked out, the script needs to position itself
> in
> > the directory which is checkout.
>
> Yes, correct, so my form inputs looks like this:
> Name: Job_name
> branch: linaro-ci-dashboard
> precommand: cd linaro-ci-dashboard; make syncdb
> command: make test
>

We don't want user to think about these things. If we can do it easy (and we can), why shouldn't we. Furthermore we can use some different directory name to check out to (it does not to be same as branch name).
We dont want to end up writing scripts in our command or precommand field.
Same goes for 1. and 3.
We need to add the environment prefix in 1. from code, but what I gave is just an example, it should be some /tmp/tmp_dir/ from i.e. gettempdir() in python.

>
> > i.e. cd branch_dir
> > 3. We need to clean up the branch_dir either before the build or after (i
> > think it's better before)
> We can make it part of the command.
> >
> > Let's get these few problems resolved then we can take a look at the code
> > implementation.

[lp:~linaro-infrastructure/linaro-ci-dashboard/create_job_jenkins_changes]

16. By Deepti B. Kalakeri on 2012-07-20

Addressing the review comments

[Deepti B. Kalakeri (deeptik)]

Changes are already submitted with the changes.

[Stevan Radaković (stevanr)]

Thanks for addressing the review comments.
Looks good now.

review: Approve

=== added directory 'config_template/sample-job'
=== added file 'config_template/sample-job/config.xml'
--- config_template/sample-job/config.xml	1970-01-01 00:00:00 +0000
+++ config_template/sample-job/config.xml	2012-07-20 15:53:17 +0000
@@ -0,0 +1,22 @@
+<?xml version='1.0' encoding='UTF-8'?>
+<project>
+  <actions/>
+  <description></description>
+  <keepDependencies>false</keepDependencies>
+  <properties/>
+  <scm class="hudson.scm.NullSCM"/>
+  <canRoam>true</canRoam>
+  <disabled>false</disabled>
+  <blockBuildWhenDownstreamBuilding>false</blockBuildWhenDownstreamBuilding>
+  <blockBuildWhenUpstreamBuilding>false</blockBuildWhenUpstreamBuilding>
+  <triggers class="vector"/>
+  <concurrentBuild>false</concurrentBuild>
+  <builders>
+    <hudson.tasks.Shell>
+      <command> # Put your command here
+      </command>
+    </hudson.tasks.Shell>
+  </builders>
+  <publishers/>
+  <buildWrappers/>
+</project>

=== modified file 'dashboard/frontend/fixtures/initial_data.json'
--- dashboard/frontend/fixtures/initial_data.json	2012-07-16 11:14:43 +0000
+++ dashboard/frontend/fixtures/initial_data.json	2012-07-20 15:53:17 +0000
@@ -12,5 +12,14 @@
     "fields": {
       "name": "linaro-infrastructure"
     }
+  },
+  {
+    "model": "jenkinsserver.JenkinsServer",
+    "pk": 1,
+    "fields": {
+      "url": "http://localhost:9090/jenkins",
+      "username": "admin",
+      "password": "admin"
+    }
   }
 ]
\ No newline at end of file

=== modified file 'dashboard/frontend/models/loop.py'
--- dashboard/frontend/models/loop.py	2012-07-18 17:48:08 +0000
+++ dashboard/frontend/models/loop.py	2012-07-20 15:53:17 +0000
@@ -32,7 +32,8 @@
 
     def save(self, *args, **kwargs):
         server = JenkinsServer.objects.get(id=1)
-        server.add_integration_loop()
+        self.server = server
+        server.add_integration_loop(self)
         if (True):
             super(Loop, self).save(*args, **kwargs)
         else:

=== modified file 'dashboard/jenkinsserver/models/jenkins_server.py'
--- dashboard/jenkinsserver/models/jenkins_server.py	2012-07-19 08:00:53 +0000
+++ dashboard/jenkinsserver/models/jenkins_server.py	2012-07-20 15:53:17 +0000
@@ -17,6 +17,8 @@
 # along with linaro-ci-dashboard.  If not, see <http://www.gnu.org/licenses/>.
 
 from django.db import models
+from django.conf import settings
+from xml.dom.minidom import parse, parseString
 import jenkins
 
 
@@ -42,8 +44,9 @@
         loop.server = self
         loop.save()
 
-    def add_integration_loop(self):
-        pass
+    def add_integration_loop(self, loop):
+        jxml = self.create_job_xml(loop)
+        self.create_job(loop.name, jxml) 
 
     def sync_jobs(self):
         """Sync jobs in dashboard DB with jobs in jenkins server.
@@ -82,3 +85,46 @@
     def unique_items(self, list_1, list_2):
         """Return a list of items that are present in list_1 but not list_2."""
         return [item for item in list_1 if item not in list_2]
+
+
+    def create_job(self, jname, jxml):
+        """ Creates a new job on jenkins if its  not already present """
+        jname = jname.replace(' ', '%20').strip()
+        if not self.jenkins.job_exists(jname):
+            self.jenkins.create_job(jname, jxml)
+            self.jenkins.enable_job(jname)
+
+    def delete_job(self, jname):
+        """ Deleted a job on jenkins if present """
+        if self.jenkins.job_exists(jname):
+            self.jenkins.delete_job(jname)
+       
+    def create_job_xml(self, loop):
+        """ Prepares the config.xml to be used for new job"""
+
+        dest_dir = 'dest_dir'
+        if not 'lp:' in loop.branch:
+            jbranch = 'bzr branch lp:%s %s '% (loop.branch, dest_dir)
+        else:
+            jbranch = 'bzr branch %s %s '% (loop.branch, dest_dir)
+
+        jcommands ='' 
+        rm_old_dest = 'rm -rf %s' % dest_dir
+        dest_command = 'cur_dir=$WORKSPACE; cd ./%s' % dest_dir
+        for command in [rm_old_dest, jbranch, dest_command, loop.precommand, \
+                        loop.command]:
+            if command: 
+                jcommands = '\n'.join([jcommands, command])
+
+        dest_command = "cd \"$cur_dir\""
+        jcommands = '\n'.join([jcommands, dest_command]) 
+
+        file = open(settings.JENKINS_JOB_CONFIG)
+        jdata = file.read()
+        file.close()
+
+        doc = parseString(jdata)
+        node = doc.getElementsByTagName('command')
+        node[0].firstChild.nodeValue = jcommands
+        jxml =  doc.toprettyxml()
+        return jxml

=== modified file 'dashboard/jenkinsserver/tests/test_jenkins_server.py'
--- dashboard/jenkinsserver/tests/test_jenkins_server.py	2012-07-19 10:44:27 +0000
+++ dashboard/jenkinsserver/tests/test_jenkins_server.py	2012-07-20 15:53:17 +0000
@@ -98,3 +98,30 @@
         loops = [loop.name.encode("utf8") for loop in
                           self.server.loop_set.all()]
         self.assertNotIn(self.test_job_name, loops)
+
+    def test_create_job_xml(self):
+        loop = Loop()
+        loop.name = 'sample-test'
+        loop.server = self.server
+        loop.branch = 'linaro-ci-dashboard'
+        loop.precommand = 'cd %s' % loop.branch
+        loop.command = 'make syncdb; make test'
+        jbranch = 'bzr branch lp:%s' % loop.branch
+        jcommands = '\n'.join([jbranch,  loop.precommand, loop.command])
+        jxml = self.server.create_job_xml(loop)
+        self.assertIn(jcommands, jxml)
+        
+    def test_create_new_job_delete_job(self):
+        loop = Loop()
+        loop.name = 'sample-test'
+        loop.server = self.server
+        loop.branch = 'linaro-ci-dashboard'
+        loop.precommand = 'cd %s' % loop.branch
+        loop.command = 'make syncdb; make test'
+        jbranch = 'bzr branch lp:%s' % loop.branch
+        jcommands = '\n'.join([jbranch,  loop.precommand, loop.command])
+        jxml = self.server.create_job_xml(loop)
+        self.server.create_job(loop.name, jxml)
+        self.assertTrue(self.test_jenkins.job_exists(loop.name) == True)
+        self.server.delete_job(loop.name)
+        self.assertTrue(self.test_jenkins.job_exists(loop.name) == False)

=== modified file 'dashboard/settings.py'
--- dashboard/settings.py	2012-07-19 10:44:27 +0000
+++ dashboard/settings.py	2012-07-20 15:53:17 +0000
@@ -188,5 +188,8 @@
 JENKINS_USER_CONFIG = os.path.join(ROOT_PATH,
                                    '../config_template/users/',
                                    'config.xml')
+JENKINS_JOB_CONFIG = os.path.join(ROOT_PATH,
+                                   '../config_template/sample-job/',
+                                   'config.xml')
 JENKINS_HOME = os.path.join(os.getenv('HOME'), ".jenkins_test")
 JENKINS_LOG_NAME = 'jenkins.log'