linaro-ci-dashboard

Merge lp:~linaro-infrastructure/linaro-ci-dashboard/run_stop_jenkins_command into lp:linaro-ci-dashboard

run_stop_jenkins_command
Merge into trunk

Proposed by Deepti B. Kalakeri on 2012-07-17

Status:	Merged
Approved by:	Stevan Radaković on 2012-07-18
Approved revision:	17
Merged at revision:	10
Proposed branch:	lp:~linaro-infrastructure/linaro-ci-dashboard/run_stop_jenkins_command
Merge into:	lp:linaro-ci-dashboard
Diff against target:	468 lines (+395/-3) 6 files modified README (+15/-1) config_template/config.xml (+36/-0) dashboard/frontend/management/commands/jenkins.py (+183/-0) dashboard/frontend/tests/__init__.py (+2/-0) dashboard/frontend/tests/test_custom_commands.py (+148/-0) dashboard/settings.py (+11/-2)
To merge this branch:	bzr merge lp:~linaro-infrastructure/linaro-ci-dashboard/run_stop_jenkins_command
Related bugs:	Link a bug report

Reviewer	Date Requested	Status
Stevan Radaković	2012-07-17	Approve on 2012-07-18
Данило Шеган	2012-07-17	Pending
Review via email: mp+115381@code.launchpad.net

Description of the change

Adding the custom command to start/stop the jenkins locally for development.
Please see the README for instruction on how to use the commands.

lp:~linaro-infrastructure/linaro-ci-dashboard/run_stop_jenkins_command updated on 2012-07-18

13. By Deepti B. Kalakeri on 2012-07-18: Adding code to handle the situation where jenkins is already running on the same port
14. By Deepti B. Kalakeri on 2012-07-18: some more changes for check where jenkins is already running on the same port

Revision history for this message

Stevan Radaković (stevanr) wrote on 2012-07-18:

Download full text (18.2 KiB)

One general comment. Code documentation is a bit scarce, it would be great to have method descriptions as py docs in the main code.

=== modified file 'README'
--- README 2012-07-11 10:20:55 +0000
+++ README 2012-07-18 08:37:18 +0000
@@ -14,7 +14,7 @@
   python-setuptools
   python-dev
   build-essential
-
+ openjdk-6-jre-headless

Running the application
-----------------------
@@ -22,6 +22,18 @@
$ python dashboard/manage.py build # only for first time run
$ python dashboard/manage.py runserver

+Other commands useful for development
+---------------------------------------
+To experiment with jenkins for development activities use the
+command below to run jenkins locally
+ $ python dashboard/manage.py runjenkins
+
+To stop the running jenkins use the command below:
+ $ python dashboard/manage.py stopjenkins <jenkins pid>
+
+PS: Please go through the settings.py JENKINS_* params to change
+jenkins related configurations
+

We basically want this to be part of the Makefile as well. This description is OK, but please add Jenkins run server as a target in make and set it to be mandatory for running tests, because our tests will be running on local Jenkins ran by this command. So we should run jenkins, run the tests, stop jenkins.

Database migration with South
-----------------------------
@@ -44,6 +56,8 @@

python-mock

+ $ python dashboard/manager.py install_configure jenkins
+
To run all tests:

$ python dashboard/manage.py test

Let's also put this in the make file.. If it's necessary for run jenkins command then let's put it in 'dependencies' target or similar.

=== added directory 'config_template'
=== added file 'config_template/config.xml'
--- config_template/config.xml 1970-01-01 00:00:00 +0000
+++ config_template/config.xml 2012-07-18 08:37:18 +0000
@@ -0,0 +1,36 @@
+<?xml version='1.0' encoding='UTF-8'?>
+<hudson>
+ <disabledAdministrativeMonitors/>
+ <numExecutors>2</numExecutors>
+ <mode>NORMAL</mode>
+ <useSecurity>true</useSecurity>
+ <authorizationStrategy class="hudson.security.LegacyAuthorizationStrategy"/>
+ <securityRealm class="hudson.security.LegacySecurityRealm"/>
+ <projectNamingStrategy class="jenkins.model.ProjectNamingStrategy$DefaultProjectNamingStrategy"/>
+ <workspaceDir>${JENKINS_HOME}/workspace/${ITEM_FULLNAME}</workspaceDir>
+ <buildsDir>${ITEM_ROOTDIR}/builds</buildsDir>
+ <markupFormatter class="hudson.markup.RawHtmlMarkupFormatter">
+ <disableSyntaxHighlighting>false</disableSyntaxHighlighting>
+ </markupFormatter>
+ <jdks/>
+ <viewsTabBar class="hudson.views.DefaultViewsTabBar"/>
+ <myViewsTabBar class="hudson.views.DefaultMyViewsTabBar"/>
+ <clouds/>
+ <slaves/>
+ <quietPeriod>5</quietPeriod>
+ <scmCheckoutRetryCount>0</scmCheckoutRetryCount>
+ <views>
+ <hudson.model.AllView>
+ <owner class="hudson" reference="../../.."/>
+ <name>All</name>
+ <filterExecutors>false</filterExecutors>
+ <filterQueue>false</filterQueue>
+ <properties class="hudson.model.View$PropertyList"/>
+ </hudson.model.AllView>
+ </views>
+ <primaryView>All</primaryView>
+ <slaveAgentPort>0</slaveAgentPort>
+ <label></label>
+ <nodeProperties/...

One general comment. Code documentation is a bit scarce, it would be great to have method descriptions as py docs in the main code.

=== modified file 'README'
--- README	2012-07-11 10:20:55 +0000
+++ README	2012-07-18 08:37:18 +0000
@@ -14,7 +14,7 @@
   python-setuptools
   python-dev
   build-essential
-
+  openjdk-6-jre-headless
 
 Running the application
 -----------------------
@@ -22,6 +22,18 @@
   $ python dashboard/manage.py build  # only for first time run
   $ python dashboard/manage.py runserver
 
+Other commands useful for development
+---------------------------------------
+To experiment with jenkins for development activities use the 
+command below to run jenkins locally
+  $ python dashboard/manage.py runjenkins  
+
+To stop the running jenkins use the command below:
+  $ python dashboard/manage.py stopjenkins <jenkins pid>
+
+PS: Please go through the settings.py JENKINS_* params to change
+jenkins related configurations
+

Database migration with South
 -----------------------------
@@ -44,6 +56,8 @@
 
   python-mock
 
+  $ python dashboard/manager.py install_configure jenkins
+
 To run all tests:
 
   $ python dashboard/manage.py test

Let's also put this in the make file.. If it's necessary for run jenkins command then let's put it in 'dependencies' target or similar.

=== added directory 'config_template'
=== added file 'config_template/config.xml'
--- config_template/config.xml	1970-01-01 00:00:00 +0000
+++ config_template/config.xml	2012-07-18 08:37:18 +0000
@@ -0,0 +1,36 @@
+<?xml version='1.0' encoding='UTF-8'?>
+<hudson>
+  <disabledAdministrativeMonitors/>
+  <numExecutors>2</numExecutors>
+  <mode>NORMAL</mode>
+  <useSecurity>true</useSecurity>
+  <authorizationStrategy class="hudson.security.LegacyAuthorizationStrategy"/>
+  <securityRealm class="hudson.security.LegacySecurityRealm"/>
+  <projectNamingStrategy class="jenkins.model.ProjectNamingStrategy$DefaultProjectNamingStrategy"/>
+  <workspaceDir>${JENKINS_HOME}/workspace/${ITEM_FULLNAME}</workspaceDir>
+  <buildsDir>${ITEM_ROOTDIR}/builds</buildsDir>
+  <markupFormatter class="hudson.markup.RawHtmlMarkupFormatter">
+    <disableSyntaxHighlighting>false</disableSyntaxHighlighting>
+  </markupFormatter>
+  <jdks/>
+  <viewsTabBar class="hudson.views.DefaultViewsTabBar"/>
+  <myViewsTabBar class="hudson.views.DefaultMyViewsTabBar"/>
+  <clouds/>
+  <slaves/>
+  <quietPeriod>5</quietPeriod>
+  <scmCheckoutRetryCount>0</scmCheckoutRetryCount>
+  <views>
+    <hudson.model.AllView>
+      <owner class="hudson" reference="../../.."/>
+      <name>All</name>
+      <filterExecutors>false</filterExecutors>
+      <filterQueue>false</filterQueue>
+      <properties class="hudson.model.View$PropertyList"/>
+    </hudson.model.AllView>
+  </views>
+  <primaryView>All</primaryView>
+  <slaveAgentPort>0</slaveAgentPort>
+  <label></label>
+  <nodeProperties/>
+  <globalNodeProperties/>
+</hudson>

=== added file 'dashboard/frontend/management/commands/runjenkins.py'
--- dashboard/frontend/management/commands/runjenkins.py	1970-01-01 00:00:00 +0000
+++ dashboard/frontend/management/commands/runjenkins.py	2012-07-18 08:37:18 +0000
@@ -0,0 +1,134 @@
+# Copyright (C) 2012 Linaro
+#
+# This file is part of linaro-ci-dashboard.
+#
+# linaro-ci-dashboard is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# linaro-ci-dashboard is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with linaro-ci-dashboard.  If not, see <http://www.gnu.org/licenses/>.
+# Django settings for dashboard project.
+
+import os
+import sys
+import shutil
+import urllib2
+import urlparse
+import shlex
+from django.conf import settings 
+from django.core import management
+from subprocess import check_output, CalledProcessError, Popen, STDOUT
+from django.core.management.base import NoArgsCommand, CommandError
+
+class Command(NoArgsCommand):
+    help = 'Run the jenkins service for development/testing'
+  
+    def __init__(self):
+        self.jenkins_war = None
+        self.jenkins_pid = None
+        self.port = settings.JENKINS_PORT
+        self.jenkins_user = settings.JENKINS_ADMIN_USER
+        self.jenkins_passwd = settings.JENKINS_ADMIN_PASSWD
+        self.jenkins_home = settings.JENKINS_HOME
+        self.jenkins_log = settings.JENKINS_LOG_NAME
+        self.jenkins_war_url = settings.JENKKINS_WAR_URL
+
+    def handle_noargs(self, **options):
+        self.install_package()
+        if self.jenkins_pid:
+            msg = "Successfully configured jenkins with pid %s. \n"\
+                   % self.jenkins_pid
+            self.stdout.write(msg)
+
+        jenkins_url='http://127.0.0.1:%s/jenkins' % self.port
+        msg = "Use '%s' in your browser \n" %  jenkins_url
+        self.stdout.write(msg)
+

Is install_package method also starting jenkins instance? If yes let's rename it.. 'install&run' or any similar name would do the trick.

+    def install_package(self):
+        try:
+            # Download openid plugin
+            cmd = 'mkdir -p %s' % self.jenkins_home
+            self.run(cmd)
+
+            # Stop any jenkins which is already running on the same port
+            cmd = "lsof -w -n -i tcp:%s | awk -F ' ' '{print $2}' | tail -n 1 | "\
+               "sed '/^$/d'"  % self.port
+            pid = self.run(cmd)
+            if pid:
+                msg = "Jenkins with pid '%s' is already running on port %s\n" \
+                      % (pid.strip(), self.port)
+                self.stdout.write(msg)
+                return 
+
+            # start jenkins service  
+            self.jenkins_start()
+  
+            cmd = 'cp %s %s' % (settings.JENKINS_CONFIG, self.jenkins_home)
+            self.run(cmd)
+
+        except CalledProcessError, e:
+            msg = "Cmd %s failed \n" % e.cmd
+            self.stdout.write(msg) 
+            sys.exit(1)
+        return
+
+    def download(self, url, filename):
+        try:
+            response = urllib2.urlopen(urllib2.quote(url, safe=":/"))
+            if not (os.path.exists(self.jenkins_war) and \
+               os.path.isfile(self.jenkins_war)):
+                fd = open(filename, 'wb')
+                shutil.copyfileobj(response, fd)
+                fd.close()
+
+            actual_size = int(response.headers['Content-Length'])
+            downloaded_size = int(os.stat(filename).st_size)
+            response.close()
+            if actual_size != downloaded_size:
+                raise
+        except:
+            raise CommandError("Could not retrieve %s" % url)
+            sys.exit(1)
+        return 0
+
+    def jenkins_start(self):
+        # Download Jenkins war file
+        self.jenkins_war = os.path.join(self.jenkins_home, 
+                           os.path.basename(self.jenkins_war_url))
+
+        self.download(self.jenkins_war_url, self.jenkins_war)
+
+        # Start the jenkins service on port 9090
+        cmd = '/usr/bin/java -jar %s --httpPort=%s --prefix=/jenkins \
+              --argumentsRealm.passwd.%s=%s \
+              --argumentsRealm.roles.%s=admin' \
+              % (self.jenkins_war, self.port, 
+                 self.jenkins_user, self.jenkins_passwd,
+                 self.jenkins_user)
+
+        cmd = shlex.split(cmd)
+        log_fn = os.path.join(self.jenkins_home, self.jenkins_log)
+        log_fd = open(log_fn, 'a')
+        ret = Popen(cmd, stdout=log_fd, stderr=STDOUT)
+
+        if ret.returncode:  
+            raise CalledProcessError(1, cmd, 'cmd Failed')
+        log_fd.close()
+        
+        self.jenkins_pid = ret.pid
+        return self.jenkins_pid
+
+    def run(self, cmd):
+        try:
+            ret = check_output(cmd, shell=True)
+            return ret
+        except CalledProcessError:
+            raise CalledProcessError(1, cmd, "Command Error")
+            sys.exit(1)

System exit? I think this should be removed. Exception will do this for us.

=== added file 'dashboard/frontend/management/commands/stopjenkins.py'
--- dashboard/frontend/management/commands/stopjenkins.py	1970-01-01 00:00:00 +0000
+++ dashboard/frontend/management/commands/stopjenkins.py	2012-07-18 08:37:18 +0000
@@ -0,0 +1,39 @@
+# Copyright (C) 2012 Linaro
+#
+# This file is part of linaro-ci-dashboard.
+#
+# linaro-ci-dashboard is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# linaro-ci-dashboard is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with linaro-ci-dashboard.  If not, see <http://www.gnu.org/licenses/>.
+# Django settings for dashboard project.
+
+import sys
+from subprocess import check_call, CalledProcessError
+from django.core.management.base import BaseCommand, CommandError
+
+class Command(BaseCommand):
+    args = 'pid pid ...'
+    help = 'Specify the jenkins pid to stop'
+
+    def handle(self, *args, **options):
+       for pid in args:
+           self.terminate(pid)
+           self.stdout.write('Successfully terminated jenkins with pid %s\n'\
+                             % pid)
+         
+    def terminate(self, pid):
+        try:
+            cmd = 'kill -9 %s' % pid
+            check_call(cmd, shell=True)
+        except CalledProcessError:
+            raise CalledProcessError(1, cmd, "Command Error")
+            sys.exit(1)

This should be in the same command as 'runjenkins'. Lets call it just 'jenkins' and have an argument with it. So it can be run like this: "python manage.py jenkins start" and stopped like "python manage.py jenkins stop __pid__"

=== modified file 'dashboard/frontend/tests/__init__.py'
--- dashboard/frontend/tests/__init__.py	2012-07-09 19:29:31 +0000
+++ dashboard/frontend/tests/__init__.py	2012-07-18 08:37:18 +0000
@@ -1,5 +1,6 @@
 from dashboard.frontend.tests.test_views import *  
 from dashboard.frontend.tests.test_clientresponse import *  
+from dashboard.frontend.tests.test_custom_commands import *  
 
       
 #starts the test suite  
@@ -7,4 +8,6 @@
            'HomePageViewTests': HomePageViewTest,
            'IndexViewTests': IndexViewTest,
            'ClientResponseTests': ClientResponseTests,
+           'RunJenkinsCommandTest': RunJenkinsCommandTest,
+           'StopJenkinsCommandTest': StopJenkinsCommandTest,
            }

Also, let's merge tests into one class as per suggestion above.

=== added file 'dashboard/frontend/tests/test_custom_commands.py'
--- dashboard/frontend/tests/test_custom_commands.py	1970-01-01 00:00:00 +0000
+++ dashboard/frontend/tests/test_custom_commands.py	2012-07-18 08:37:18 +0000
@@ -0,0 +1,147 @@
+#!/usr/bin/env python
+# Copyright (C) 2012 Linaro
+#
+# This file is part of linaro-ci-dashboard.
+#
+# linaro-ci-dashboard is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# linaro-ci-dashboard is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with linaro-ci-dashboard.  If not, see <http://www.gnu.org/licenses/>.
+
+import frontend.management.commands.runjenkins as runjenkins
+import frontend.management.commands.stopjenkins as stopjenkins
+from django.core.management.base import CommandError
+from django.conf import settings
+from subprocess import  CalledProcessError
+from django.core import management
+from StringIO import StringIO
+from django.test import TestCase
+import os, sys, shutil
+
+
+class RunJenkinsCommandTest(TestCase):
+     
+    orig_dir = os.getcwd()
+       
+    def setUp(self):
+        self.command = runjenkins.Command()
+        self.command.port = '9191'
+        settings.JENKINS_PORT = self.command.port 
+        self.url = 'http://updates.jenkins-ci.org/latest/bazaar.hpi'
+        os.environ["JENKINS_HOME"] = os.path.join(self.orig_dir, 'tmp/', 
+                                     'jenkins_home') 
+        self.command.jenkins_home = os.getenv("JENKINS_HOME") 
+        self.command.jenkins_war = os.path.join(self.command.jenkins_home,
+                              os.path.basename(self.command.jenkins_war_url))
+        self.download_filename = os.path.join(self.command.jenkins_home, 
+                                 os.path.basename(self.url))
+
+        if not os.path.exists(self.command.jenkins_home):
+            os.makedirs(self.command.jenkins_home)
+
+        super(RunJenkinsCommandTest, self).setUp()
+
+    def tearDown(self):
+        os.chdir(self.orig_dir)
+        if os.path.exists(self.download_filename):
+             cmd = 'rm  %s' % self.download_filename
+             self.command.run(cmd)
+
+        parent_dir = os.path.dirname(self.command.jenkins_home)
+        if os.path.isdir(parent_dir):
+            shutil.rmtree(parent_dir)
+
+        super(RunJenkinsCommandTest, self).tearDown()
+
+    def test_download_non_existing_package(self):
+        url = 'http://mirrors.jenkins-ci.org/war/latest/jenkins1.war'
+        filename = os.path.join('/tmp/', os.path.basename(url))
+        self.assertRaises(CommandError, self.command.download, url, filename)
+
+    def test_download_existing_package(self):
+        ret = self.command.download(self.url, self.download_filename)
+        self.assertTrue(ret == 0)
+        
+    def test_install_package(self):
+        orig_stdout = sys.stdout
+        stdout = sys.stdout = StringIO()
+        cmd = "lsof -w -n -i tcp:%s | awk -F ' ' '{print $2}' | tail -n 1 | "\
+               "sed '/^$/d'"  % self.command.port
+        self.command.jenkins_pid = self.command.run(cmd)
+        if len(self.command.jenkins_pid) != 0:
+            msg = "Jenkins with pid '%s' is already running on port %s"\
+                  % (self.command.jenkins_pid, self.command.port)
+        else:
+            msg = "Successfully configured jenkins with pid"
+        management.call_command('runjenkins', interactive=False)
+        sys.stdout = orig_stdout
+        stdout.seek(0)
+        self.assertIn(msg, stdout.read())
+
+    def test_invalid_run(self):
+        cmd = 'rm /tmp/dummy'
+        self.assertRaises(CalledProcessError, self.command.run, cmd)
+
+    def test_valid_run(self):
+        ret = self.command.run(['whoami'])
+        self.assertTrue(ret.strip() == os.getenv('USER'))
+
+class StopJenkinsCommandTest(TestCase):
+      
+    orig_dir = os.getcwd()
+
+    def setUp(self):
+        self.runcommand = runjenkins.Command()
+        self.stopcommand = stopjenkins.Command()
+        self.runcommand.port = '9191'
+        settings.JENKINS_PORT = self.runcommand.port 
+        os.environ["JENKINS_HOME"] = os.path.join(self.orig_dir, 'tmp/', 
+                                     'jenkins_home') 
+        self.runcommand.jenkins_home = os.getenv("JENKINS_HOME") 
+        self.runcommand.jenkins_war = os.path.join(self.runcommand.jenkins_home,
+                              os.path.basename(self.runcommand.jenkins_war_url))
+
+        if not os.path.exists(self.runcommand.jenkins_home):
+            os.makedirs(self.runcommand.jenkins_home)
+
+        super(StopJenkinsCommandTest, self).setUp()
+
+    def tearDown(self):
+        os.chdir(self.orig_dir)
+
+        parent_dir = os.path.dirname(self.runcommand.jenkins_home)
+        if os.path.isdir(parent_dir):
+            shutil.rmtree(parent_dir)
+
+        super(StopJenkinsCommandTest, self).tearDown()
+
+ 
+    def test_jenkins_terminate_non_existing_pid(self):
+        pid = '123456'
+        self.assertRaises(CalledProcessError, self.stopcommand.terminate, pid)
+
+    def test_jenkins_terminate(self):
+        orig_stdout = sys.stdout
+        stdout = sys.stdout = StringIO()
+        cmd = "lsof -w -n -i tcp:%s | awk -F ' ' '{print $2}' | tail -n 1 | "\
+               "sed '/^$/d'"  % self.runcommand.port
+        pid = self.runcommand.run(cmd)
+        management.call_command('runjenkins', interactive=False)
+        if len(pid) != 0:
+            msg = "Jenkins with pid '%s' is already running on port %s"\
+                  % (pid.strip(), self.runcommand.port)
+        else:
+            pid = self.runcommand.jenkins_start()
+            management.call_command('stopjenkins', pid, interactive=False)
+            msg = "Successfully terminated jenkins with pid" 
+        sys.stdout = orig_stdout
+        stdout.seek(0)
+        self.assertIn(msg, stdout.read())

=== modified file 'dashboard/settings.py'
--- dashboard/settings.py	2012-07-11 10:20:55 +0000
+++ dashboard/settings.py	2012-07-18 08:37:18 +0000
@@ -21,8 +21,8 @@
 DEBUG = True
 TEMPLATE_DEBUG = DEBUG
 
-DEPS_INSTALL_PATH = os.path.join(os.path.dirname(os.path.abspath(__file__)),
-                                "..", "lib")
+ROOT_PATH = os.path.dirname(os.path.abspath(__file__))
+DEPS_INSTALL_PATH = os.path.join(ROOT_PATH, "..", "lib")
 
 ADMINS = (
     # ('Your Name', 'your_email@example.com'),
@@ -175,3 +175,12 @@
         },
     }
 }
+
+JENKKINS_WAR_URL = 'http://mirrors.jenkins-ci.org/war/latest/jenkins.war'
+JENKINS_ADMIN_USER = 'admin'
+JENKINS_ADMIN_PASSWD = 'admin'
+JENKINS_PORT = '9090'
+JENKINS_CONFIG = os.path.join(ROOT_PATH, '../config_template/', 'config.xml')
+JENKINS_HOME = os.path.join(os.getenv('HOME'), ".jenkins")
+JENKINS_LOG_NAME = 'jenkins.log'
+

review: Needs Fixing (code)

lp:~linaro-infrastructure/linaro-ci-dashboard/run_stop_jenkins_command updated on 2012-07-18

15. By Deepti B. Kalakeri on 2012-07-18: Merged the jenkins stop command to the same class as start.
Renamed the runjenkins command to jenkins and made changes to handle
commands like :
python manage.py jenkins start
python manage.py jenkins stop <pid>
Merged the tests into one class
Added/modified new tests to handle the changes made
16. By Deepti B. Kalakeri on 2012-07-18: README changes for jenkins

Revision history for this message

Deepti B. Kalakeri (deeptik) wrote on 2012-07-18:

Download full text (19.4 KiB)

> One general comment. Code documentation is a bit scarce, it would be great to
> have method descriptions as py docs in the main code.
>
>
> === modified file 'README'
> --- README 2012-07-11 10:20:55 +0000
> +++ README 2012-07-18 08:37:18 +0000
> @@ -14,7 +14,7 @@
> python-setuptools
> python-dev
> build-essential
> -
> + openjdk-6-jre-headless
>
> Running the application
> -----------------------
> @@ -22,6 +22,18 @@
> $ python dashboard/manage.py build # only for first time run
> $ python dashboard/manage.py runserver
>
> +Other commands useful for development
> +---------------------------------------
> +To experiment with jenkins for development activities use the
> +command below to run jenkins locally
> + $ python dashboard/manage.py runjenkins
> +
> +To stop the running jenkins use the command below:
> + $ python dashboard/manage.py stopjenkins <jenkins pid>
> +
> +PS: Please go through the settings.py JENKINS_* params to change
> +jenkins related configurations
> +
>
>
>
> We basically want this to be part of the Makefile as well. This description is
> OK, but please add Jenkins run server as a target in make and set it to be
> mandatory for running tests, because our tests will be running on local
> Jenkins ran by this command. So we should run jenkins, run the tests, stop
> jenkins.
>
>

> One general comment. Code documentation is a bit scarce, it would be great to
> have method descriptions as py docs in the main code.
> 
> 
> === modified file 'README'
> --- README      2012-07-11 10:20:55 +0000
> +++ README      2012-07-18 08:37:18 +0000
> @@ -14,7 +14,7 @@
>    python-setuptools
>    python-dev
>    build-essential
> -
> +  openjdk-6-jre-headless
> 
>  Running the application
>  -----------------------
> @@ -22,6 +22,18 @@
>    $ python dashboard/manage.py build  # only for first time run
>    $ python dashboard/manage.py runserver
> 
> +Other commands useful for development
> +---------------------------------------
> +To experiment with jenkins for development activities use the
> +command below to run jenkins locally
> +  $ python dashboard/manage.py runjenkins
> +
> +To stop the running jenkins use the command below:
> +  $ python dashboard/manage.py stopjenkins <jenkins pid>
> +
> +PS: Please go through the settings.py JENKINS_* params to change
> +jenkins related configurations
> +
> 
> 
> 
> We basically want this to be part of the Makefile as well. This description is
> OK, but please add Jenkins run server as a target in make and set it to be
> mandatory for running tests, because our tests will be running on local
> Jenkins ran by this command. So we should run jenkins, run the tests, stop
> jenkins.
> 
>

Well, we do not have Makefile yet in the branch and I dont want to redo your work. 
So will add this once we have your changes in the branch.
>  Database migration with South
>  -----------------------------
> @@ -44,6 +56,8 @@
> 
>    python-mock
> 
> +  $ python dashboard/manager.py install_configure jenkins
> +
>  To run all tests:
> 
>    $ python dashboard/manage.py test
> 
> 
> Let's also put this in the make file.. If it's necessary for run jenkins
> command then let's put it in 'dependencies' target or similar.
> 
same as above. All the other changes are addressed
> 
> === added directory 'config_template'
> === added file 'config_template/config.xml'
> --- config_template/config.xml  1970-01-01 00:00:00 +0000
> +++ config_template/config.xml  2012-07-18 08:37:18 +0000
> @@ -0,0 +1,36 @@
> +<?xml version='1.0' encoding='UTF-8'?>
> +<hudson>
> +  <disabledAdministrativeMonitors/>
> +  <numExecutors>2</numExecutors>
> +  <mode>NORMAL</mode>
> +  <useSecurity>true</useSecurity>
> +  <authorizationStrategy
> class="hudson.security.LegacyAuthorizationStrategy"/>
> +  <securityRealm class="hudson.security.LegacySecurityRealm"/>
> +  <projectNamingStrategy
> class="jenkins.model.ProjectNamingStrategy$DefaultProjectNamingStrategy"/>
> +  <workspaceDir>${JENKINS_HOME}/workspace/${ITEM_FULLNAME}</workspaceDir>
> +  <buildsDir>${ITEM_ROOTDIR}/builds</buildsDir>
> +  <markupFormatter class="hudson.markup.RawHtmlMarkupFormatter">
> +    <disableSyntaxHighlighting>false</disableSyntaxHighlighting>
> +  </markupFormatter>
> +  <jdks/>
> +  <viewsTabBar class="hudson.views.DefaultViewsTabBar"/>
> +  <myViewsTabBar class="hudson.views.DefaultMyViewsTabBar"/>
> +  <clouds/>
> +  <slaves/>
> +  <quietPeriod>5</quietPeriod>
> +  <scmCheckoutRetryCount>0</scmCheckoutRetryCount>
> +  <views>
> +    <hudson.model.AllView>
> +      <owner class="hudson" reference="../../.."/>
> +      <name>All</name>
> +      <filterExecutors>false</filterExecutors>
> +      <filterQueue>false</filterQueue>
> +      <properties class="hudson.model.View$PropertyList"/>
> +    </hudson.model.AllView>
> +  </views>
> +  <primaryView>All</primaryView>
> +  <slaveAgentPort>0</slaveAgentPort>
> +  <label></label>
> +  <nodeProperties/>
> +  <globalNodeProperties/>
> +</hudson>
> 
> === added file 'dashboard/frontend/management/commands/runjenkins.py'
> --- dashboard/frontend/management/commands/runjenkins.py        1970-01-01
> 00:00:00 +0000
> +++ dashboard/frontend/management/commands/runjenkins.py        2012-07-18
> 08:37:18 +0000
> @@ -0,0 +1,134 @@
> +# Copyright (C) 2012 Linaro
> +#
> +# This file is part of linaro-ci-dashboard.
> +#
> +# linaro-ci-dashboard is free software: you can redistribute it and/or modify
> +# it under the terms of the GNU Affero General Public License as published by
> +# the Free Software Foundation, either version 3 of the License, or
> +# (at your option) any later version.
> +#
> +# linaro-ci-dashboard is distributed in the hope that it will be useful,
> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> +# GNU Affero General Public License for more details.
> +#
> +# You should have received a copy of the GNU Affero General Public License
> +# along with linaro-ci-dashboard.  If not, see
> <http://www.gnu.org/licenses/>.
> +# Django settings for dashboard project.
> +
> +import os
> +import sys
> +import shutil
> +import urllib2
> +import urlparse
> +import shlex
> +from django.conf import settings
> +from django.core import management
> +from subprocess import check_output, CalledProcessError, Popen, STDOUT
> +from django.core.management.base import NoArgsCommand, CommandError
> +
> +class Command(NoArgsCommand):
> +    help = 'Run the jenkins service for development/testing'
> +
> +    def __init__(self):
> +        self.jenkins_war = None
> +        self.jenkins_pid = None
> +        self.port = settings.JENKINS_PORT
> +        self.jenkins_user = settings.JENKINS_ADMIN_USER
> +        self.jenkins_passwd = settings.JENKINS_ADMIN_PASSWD
> +        self.jenkins_home = settings.JENKINS_HOME
> +        self.jenkins_log = settings.JENKINS_LOG_NAME
> +        self.jenkins_war_url = settings.JENKKINS_WAR_URL
> +
> +    def handle_noargs(self, **options):
> +        self.install_package()
> +        if self.jenkins_pid:
> +            msg = "Successfully configured jenkins with pid %s. \n"\
> +                   % self.jenkins_pid
> +            self.stdout.write(msg)
> +
> +        jenkins_url='http://127.0.0.1:%s/jenkins' % self.port
> +        msg = "Use '%s' in your browser \n" %  jenkins_url
> +        self.stdout.write(msg)
> +
> 
> 
> Is install_package method also starting jenkins instance? If yes let's rename
> it.. 'install&run' or any similar name would do the trick.
> 
> +    def install_package(self):
> +        try:
> +            # Download openid plugin
> +            cmd = 'mkdir -p %s' % self.jenkins_home
> +            self.run(cmd)
> +
> +            # Stop any jenkins which is already running on the same port
> +            cmd = "lsof -w -n -i tcp:%s | awk -F ' ' '{print $2}' | tail -n 1
> | "\
> +               "sed '/^$/d'"  % self.port
> +            pid = self.run(cmd)
> +            if pid:
> +                msg = "Jenkins with pid '%s' is already running on port %s\n"
> \
> +                      % (pid.strip(), self.port)
> +                self.stdout.write(msg)
> +                return
> +
> +            # start jenkins service
> +            self.jenkins_start()
> +
> +            cmd = 'cp %s %s' % (settings.JENKINS_CONFIG, self.jenkins_home)
> +            self.run(cmd)
> +
> +        except CalledProcessError, e:
> +            msg = "Cmd %s failed \n" % e.cmd
> +            self.stdout.write(msg)
> +            sys.exit(1)
> +        return
> +
> +    def download(self, url, filename):
> +        try:
> +            response = urllib2.urlopen(urllib2.quote(url, safe=":/"))
> +            if not (os.path.exists(self.jenkins_war) and \
> +               os.path.isfile(self.jenkins_war)):
> +                fd = open(filename, 'wb')
> +                shutil.copyfileobj(response, fd)
> +                fd.close()
> +
> +            actual_size = int(response.headers['Content-Length'])
> +            downloaded_size = int(os.stat(filename).st_size)
> +            response.close()
> +            if actual_size != downloaded_size:
> +                raise
> +        except:
> +            raise CommandError("Could not retrieve %s" % url)
> +            sys.exit(1)
> +        return 0
> +
> +    def jenkins_start(self):
> +        # Download Jenkins war file
> +        self.jenkins_war = os.path.join(self.jenkins_home,
> +                           os.path.basename(self.jenkins_war_url))
> +
> +        self.download(self.jenkins_war_url, self.jenkins_war)
> +
> +        # Start the jenkins service on port 9090
> +        cmd = '/usr/bin/java -jar %s --httpPort=%s --prefix=/jenkins \
> +              --argumentsRealm.passwd.%s=%s \
> +              --argumentsRealm.roles.%s=admin' \
> +              % (self.jenkins_war, self.port,
> +                 self.jenkins_user, self.jenkins_passwd,
> +                 self.jenkins_user)
> +
> +        cmd = shlex.split(cmd)
> +        log_fn = os.path.join(self.jenkins_home, self.jenkins_log)
> +        log_fd = open(log_fn, 'a')
> +        ret = Popen(cmd, stdout=log_fd, stderr=STDOUT)
> +
> +        if ret.returncode:
> +            raise CalledProcessError(1, cmd, 'cmd Failed')
> +        log_fd.close()
> +
> +        self.jenkins_pid = ret.pid
> +        return self.jenkins_pid
> +
> +    def run(self, cmd):
> +        try:
> +            ret = check_output(cmd, shell=True)
> +            return ret
> +        except CalledProcessError:
> +            raise CalledProcessError(1, cmd, "Command Error")
> +            sys.exit(1)
> 
> 
> System exit? I think this should be removed. Exception will do this for us.
> 
> === added file 'dashboard/frontend/management/commands/stopjenkins.py'
> --- dashboard/frontend/management/commands/stopjenkins.py       1970-01-01
> 00:00:00 +0000
> +++ dashboard/frontend/management/commands/stopjenkins.py       2012-07-18
> 08:37:18 +0000
> @@ -0,0 +1,39 @@
> +# Copyright (C) 2012 Linaro
> +#
> +# This file is part of linaro-ci-dashboard.
> +#
> +# linaro-ci-dashboard is free software: you can redistribute it and/or modify
> +# it under the terms of the GNU Affero General Public License as published by
> +# the Free Software Foundation, either version 3 of the License, or
> +# (at your option) any later version.
> +#
> +# linaro-ci-dashboard is distributed in the hope that it will be useful,
> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> +# GNU Affero General Public License for more details.
> +#
> +# You should have received a copy of the GNU Affero General Public License
> +# along with linaro-ci-dashboard.  If not, see
> <http://www.gnu.org/licenses/>.
> +# Django settings for dashboard project.
> +
> +import sys
> +from subprocess import check_call, CalledProcessError
> +from django.core.management.base import BaseCommand, CommandError
> +
> +class Command(BaseCommand):
> +    args = 'pid pid ...'
> +    help = 'Specify the jenkins pid to stop'
> +
> +    def handle(self, *args, **options):
> +       for pid in args:
> +           self.terminate(pid)
> +           self.stdout.write('Successfully terminated jenkins with pid %s\n'\
> +                             % pid)
> +
> +    def terminate(self, pid):
> +        try:
> +            cmd = 'kill -9 %s' % pid
> +            check_call(cmd, shell=True)
> +        except CalledProcessError:
> +            raise CalledProcessError(1, cmd, "Command Error")
> +            sys.exit(1)
> 
> 
> This should be in the same command as 'runjenkins'. Lets call it just
> 'jenkins' and have an argument with it. So it can be run like this: "python
> manage.py jenkins start" and stopped like "python manage.py jenkins stop
> __pid__"
> 
> === modified file 'dashboard/frontend/tests/__init__.py'
> --- dashboard/frontend/tests/__init__.py        2012-07-09 19:29:31 +0000
> +++ dashboard/frontend/tests/__init__.py        2012-07-18 08:37:18 +0000
> @@ -1,5 +1,6 @@
>  from dashboard.frontend.tests.test_views import *
>  from dashboard.frontend.tests.test_clientresponse import *
> +from dashboard.frontend.tests.test_custom_commands import *
> 
> 
>  #starts the test suite
> @@ -7,4 +8,6 @@
>             'HomePageViewTests': HomePageViewTest,
>             'IndexViewTests': IndexViewTest,
>             'ClientResponseTests': ClientResponseTests,
> +           'RunJenkinsCommandTest': RunJenkinsCommandTest,
> +           'StopJenkinsCommandTest': StopJenkinsCommandTest,
>             }
> 
> Also, let's merge tests into one class as per suggestion above.
> 
> 
> === added file 'dashboard/frontend/tests/test_custom_commands.py'
> --- dashboard/frontend/tests/test_custom_commands.py    1970-01-01 00:00:00
> +0000
> +++ dashboard/frontend/tests/test_custom_commands.py    2012-07-18 08:37:18
> +0000
> @@ -0,0 +1,147 @@
> +#!/usr/bin/env python
> +# Copyright (C) 2012 Linaro
> +#
> +# This file is part of linaro-ci-dashboard.
> +#
> +# linaro-ci-dashboard is free software: you can redistribute it and/or modify
> +# it under the terms of the GNU Affero General Public License as published by
> +# the Free Software Foundation, either version 3 of the License, or
> +# (at your option) any later version.
> +#
> +# linaro-ci-dashboard is distributed in the hope that it will be useful,
> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> +# GNU Affero General Public License for more details.
> +#
> +# You should have received a copy of the GNU Affero General Public License
> +# along with linaro-ci-dashboard.  If not, see
> <http://www.gnu.org/licenses/>.
> +
> +import frontend.management.commands.runjenkins as runjenkins
> +import frontend.management.commands.stopjenkins as stopjenkins
> +from django.core.management.base import CommandError
> +from django.conf import settings
> +from subprocess import  CalledProcessError
> +from django.core import management
> +from StringIO import StringIO
> +from django.test import TestCase
> +import os, sys, shutil
> +
> +
> +class RunJenkinsCommandTest(TestCase):
> +
> +    orig_dir = os.getcwd()
> +
> +    def setUp(self):
> +        self.command = runjenkins.Command()
> +        self.command.port = '9191'
> +        settings.JENKINS_PORT = self.command.port
> +        self.url = 'http://updates.jenkins-ci.org/latest/bazaar.hpi'
> +        os.environ["JENKINS_HOME"] = os.path.join(self.orig_dir, 'tmp/',
> +                                     'jenkins_home')
> +        self.command.jenkins_home = os.getenv("JENKINS_HOME")
> +        self.command.jenkins_war = os.path.join(self.command.jenkins_home,
> +                              os.path.basename(self.command.jenkins_war_url))
> +        self.download_filename = os.path.join(self.command.jenkins_home,
> +                                 os.path.basename(self.url))
> +
> +        if not os.path.exists(self.command.jenkins_home):
> +            os.makedirs(self.command.jenkins_home)
> +
> +        super(RunJenkinsCommandTest, self).setUp()
> +
> +    def tearDown(self):
> +        os.chdir(self.orig_dir)
> +        if os.path.exists(self.download_filename):
> +             cmd = 'rm  %s' % self.download_filename
> +             self.command.run(cmd)
> +
> +        parent_dir = os.path.dirname(self.command.jenkins_home)
> +        if os.path.isdir(parent_dir):
> +            shutil.rmtree(parent_dir)
> +
> +        super(RunJenkinsCommandTest, self).tearDown()
> +
> +    def test_download_non_existing_package(self):
> +        url = 'http://mirrors.jenkins-ci.org/war/latest/jenkins1.war'
> +        filename = os.path.join('/tmp/', os.path.basename(url))
> +        self.assertRaises(CommandError, self.command.download, url, filename)
> +
> +    def test_download_existing_package(self):
> +        ret = self.command.download(self.url, self.download_filename)
> +        self.assertTrue(ret == 0)
> +
> +    def test_install_package(self):
> +        orig_stdout = sys.stdout
> +        stdout = sys.stdout = StringIO()
> +        cmd = "lsof -w -n -i tcp:%s | awk -F ' ' '{print $2}' | tail -n 1 |
> "\
> +               "sed '/^$/d'"  % self.command.port
> +        self.command.jenkins_pid = self.command.run(cmd)
> +        if len(self.command.jenkins_pid) != 0:
> +            msg = "Jenkins with pid '%s' is already running on port %s"\
> +                  % (self.command.jenkins_pid, self.command.port)
> +        else:
> +            msg = "Successfully configured jenkins with pid"
> +        management.call_command('runjenkins', interactive=False)
> +        sys.stdout = orig_stdout
> +        stdout.seek(0)
> +        self.assertIn(msg, stdout.read())
> +
> +    def test_invalid_run(self):
> +        cmd = 'rm /tmp/dummy'
> +        self.assertRaises(CalledProcessError, self.command.run, cmd)
> +
> +    def test_valid_run(self):
> +        ret = self.command.run(['whoami'])
> +        self.assertTrue(ret.strip() == os.getenv('USER'))
> +
> +class StopJenkinsCommandTest(TestCase):
> +
> +    orig_dir = os.getcwd()
> +
> +    def setUp(self):
> +        self.runcommand = runjenkins.Command()
> +        self.stopcommand = stopjenkins.Command()
> +        self.runcommand.port = '9191'
> +        settings.JENKINS_PORT = self.runcommand.port
> +        os.environ["JENKINS_HOME"] = os.path.join(self.orig_dir, 'tmp/',
> +                                     'jenkins_home')
> +        self.runcommand.jenkins_home = os.getenv("JENKINS_HOME")
> +        self.runcommand.jenkins_war =
> os.path.join(self.runcommand.jenkins_home,
> +
> os.path.basename(self.runcommand.jenkins_war_url))
> +
> +        if not os.path.exists(self.runcommand.jenkins_home):
> +            os.makedirs(self.runcommand.jenkins_home)
> +
> +        super(StopJenkinsCommandTest, self).setUp()
> +
> +    def tearDown(self):
> +        os.chdir(self.orig_dir)
> +
> +        parent_dir = os.path.dirname(self.runcommand.jenkins_home)
> +        if os.path.isdir(parent_dir):
> +            shutil.rmtree(parent_dir)
> +
> +        super(StopJenkinsCommandTest, self).tearDown()
> +
> +
> +    def test_jenkins_terminate_non_existing_pid(self):
> +        pid = '123456'
> +        self.assertRaises(CalledProcessError, self.stopcommand.terminate,
> pid)
> +
> +    def test_jenkins_terminate(self):
> +        orig_stdout = sys.stdout
> +        stdout = sys.stdout = StringIO()
> +        cmd = "lsof -w -n -i tcp:%s | awk -F ' ' '{print $2}' | tail -n 1 |
> "\
> +               "sed '/^$/d'"  % self.runcommand.port
> +        pid = self.runcommand.run(cmd)
> +        management.call_command('runjenkins', interactive=False)
> +        if len(pid) != 0:
> +            msg = "Jenkins with pid '%s' is already running on port %s"\
> +                  % (pid.strip(), self.runcommand.port)
> +        else:
> +            pid = self.runcommand.jenkins_start()
> +            management.call_command('stopjenkins', pid, interactive=False)
> +            msg = "Successfully terminated jenkins with pid"
> +        sys.stdout = orig_stdout
> +        stdout.seek(0)
> +        self.assertIn(msg, stdout.read())
> 
> === modified file 'dashboard/settings.py'
> --- dashboard/settings.py       2012-07-11 10:20:55 +0000
> +++ dashboard/settings.py       2012-07-18 08:37:18 +0000
> @@ -21,8 +21,8 @@
>  DEBUG = True
>  TEMPLATE_DEBUG = DEBUG
> 
> -DEPS_INSTALL_PATH = os.path.join(os.path.dirname(os.path.abspath(__file__)),
> -                                "..", "lib")
> +ROOT_PATH = os.path.dirname(os.path.abspath(__file__))
> +DEPS_INSTALL_PATH = os.path.join(ROOT_PATH, "..", "lib")
> 
>  ADMINS = (
>      # ('Your Name', 'your_email@example.com'),
> @@ -175,3 +175,12 @@
>          },
>      }
>  }
> +
> +JENKKINS_WAR_URL = 'http://mirrors.jenkins-ci.org/war/latest/jenkins.war'
> +JENKINS_ADMIN_USER = 'admin'
> +JENKINS_ADMIN_PASSWD = 'admin'
> +JENKINS_PORT = '9090'
> +JENKINS_CONFIG = os.path.join(ROOT_PATH, '../config_template/', 'config.xml')
> +JENKINS_HOME = os.path.join(os.getenv('HOME'), ".jenkins")
> +JENKINS_LOG_NAME = 'jenkins.log'
> +

lp:~linaro-infrastructure/linaro-ci-dashboard/run_stop_jenkins_command updated on 2012-07-18

17. By Stevan Radaković on 2012-07-18: Last minute change to use test JENKINS_HOME directory so existing jenkins instance is not jeopardised.

Revision history for this message

Stevan Radaković (stevanr) wrote on 2012-07-18:

Thanks for the additional fixes Deepti.
Great work, once again.
Approve +1

review: Approve

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk

Subscribers

People subscribed via source and target branches

to all changes:

Linaro Infrastructure

 === modified file 'README'
 --- README	2012-07-11 10:20:55 +0000
 +++ README	2012-07-18 17:24:18 +0000
@@ -14,7 +14,7 @@
    python-setuptools
    python-dev
    build-essential
--
++  openjdk-6-jre-headless
  Running the application
  -----------------------
@@ -22,6 +22,18 @@
    $ python dashboard/manage.py build  # only for first time run
    $ python dashboard/manage.py runserver
++Other commands useful for development
++---------------------------------------
++To experiment with jenkins for development activities use the
++command below to run jenkins locally
++  $ python dashboard/manage.py jenkins start
++
++To stop the running jenkins use the command below:
++  $ python dashboard/manage.py jenkins stop <jenkins pid>
++
++PS: Please go through the settings.py JENKINS_* params to change
++jenkins related configurations
++
  Database migration with South
  -----------------------------
@@ -44,6 +56,8 @@
    python-mock
++  $ python dashboard/manager.py install_configure jenkins
++
  To run all tests:
    $ python dashboard/manage.py test
 === added directory 'config_template'
 === added file 'config_template/config.xml'
 --- config_template/config.xml	1970-01-01 00:00:00 +0000
 +++ config_template/config.xml	2012-07-18 17:24:18 +0000
@@ -0,0 +1,36 @@
++<?xml version='1.0' encoding='UTF-8'?>
++<hudson>
++  <disabledAdministrativeMonitors/>
++  <numExecutors>2</numExecutors>
++  <mode>NORMAL</mode>
++  <useSecurity>true</useSecurity>
++  <authorizationStrategy class="hudson.security.LegacyAuthorizationStrategy"/>
++  <securityRealm class="hudson.security.LegacySecurityRealm"/>
++  <projectNamingStrategy class="jenkins.model.ProjectNamingStrategy$DefaultProjectNamingStrategy"/>
++  <workspaceDir>${JENKINS_HOME}/workspace/${ITEM_FULLNAME}</workspaceDir>
++  <buildsDir>${ITEM_ROOTDIR}/builds</buildsDir>
++  <markupFormatter class="hudson.markup.RawHtmlMarkupFormatter">
++    <disableSyntaxHighlighting>false</disableSyntaxHighlighting>
++  </markupFormatter>
++  <jdks/>
++  <viewsTabBar class="hudson.views.DefaultViewsTabBar"/>
++  <myViewsTabBar class="hudson.views.DefaultMyViewsTabBar"/>
++  <clouds/>
++  <slaves/>
++  <quietPeriod>5</quietPeriod>
++  <scmCheckoutRetryCount>0</scmCheckoutRetryCount>
++  <views>
++    <hudson.model.AllView>
++      <owner class="hudson" reference="../../.."/>
++      <name>All</name>
++      <filterExecutors>false</filterExecutors>
++      <filterQueue>false</filterQueue>
++      <properties class="hudson.model.View$PropertyList"/>
++    </hudson.model.AllView>
++  </views>
++  <primaryView>All</primaryView>
++  <slaveAgentPort>0</slaveAgentPort>
++  <label></label>
++  <nodeProperties/>
++  <globalNodeProperties/>
++</hudson>
 === added file 'dashboard/frontend/management/commands/jenkins.py'
 --- dashboard/frontend/management/commands/jenkins.py	1970-01-01 00:00:00 +0000
 +++ dashboard/frontend/management/commands/jenkins.py	2012-07-18 17:24:18 +0000
@@ -0,0 +1,183 @@
++# Copyright (C) 2012 Linaro
++#
++# This file is part of linaro-ci-dashboard.
++#
++# linaro-ci-dashboard is free software: you can redistribute it and/or modify
++# it under the terms of the GNU Affero General Public License as published by
++# the Free Software Foundation, either version 3 of the License, or
++# (at your option) any later version.
++#
++# linaro-ci-dashboard is distributed in the hope that it will be useful,
++# but WITHOUT ANY WARRANTY; without even the implied warranty of
++# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
++# GNU Affero General Public License for more details.
++#
++# You should have received a copy of the GNU Affero General Public License
++# along with linaro-ci-dashboard.  If not, see <http://www.gnu.org/licenses/>.
++# Django settings for dashboard project.
++
++import os
++import sys
++import shutil
++import urllib2
++import urlparse
++import shlex
++from django.conf import settings
++from django.core import management
++from subprocess import check_output, CalledProcessError, Popen, STDOUT
++from django.core.management.base import BaseCommand, CommandError
++
++class Command(BaseCommand):
++    args = 'start or stop <pid>'
++    help = 'Starts/Stops Jenkins service on machine.\n'\
++            'You need to specify pid of the jenkins process to stop'
++
++    def __init__(self):
++        self.jenkins_war = None
++        self.jenkins_pid = None
++        self.port = settings.JENKINS_PORT
++        self.jenkins_user = settings.JENKINS_ADMIN_USER
++        self.jenkins_passwd = settings.JENKINS_ADMIN_PASSWD
++        self.jenkins_home = settings.JENKINS_HOME
++        self.jenkins_log = settings.JENKINS_LOG_NAME
++        self.jenkins_war_url = settings.JENKKINS_WAR_URL
++
++
++    def handle(self, *args, **options):
++        try:
++            if len(args) < 1:
++                sys.stdout.write("Need to specify either start/stop\n")
++                raise
++
++            if args[0] == 'start':
++                self.install_run_jenkins()
++                if self.jenkins_pid:
++                    msg = "Successfully configured jenkins with pid %s. \n"\
++                           % self.jenkins_pid
++                    self.stdout.write(msg)
++
++                jenkins_url='http://127.0.0.1:%s/jenkins' % self.port
++                msg = "Use '%s' in your browser \n" %  jenkins_url
++            elif args[0] == 'stop':
++                if len(args[1:]):
++                    for pid in args[1:]:
++                       self.stop_jenkins(pid)
++                       msg = 'Successfully terminated jenkins with pid %s\n' \
++                              % pid
++                else:
++                    msg = "Missing pid value. Specify pid after stop argument\n"
++            else:
++                msg = "Invalid option. Supports only start/stop.\n"
++
++            self.stdout.write(msg)
++        except:
++            sys.stdout.write("Error processing the command\n")
++
++
++    def install_run_jenkins(self):
++        """
++        Installs/configure/start the jenkins
++        making sure another instance of jenkins is not running on the same self.port.
++        """
++        try:
++            # Download openid plugin
++            cmd = 'mkdir -p %s' % self.jenkins_home
++            self.run(cmd)
++
++            # Stop any jenkins which is already running on the same port
++            cmd = "lsof -w -n -i tcp:%s | awk -F ' ' '{print $2}' | tail -n 1 | "\
++               "sed '/^$/d'"  % self.port
++            pid = self.run(cmd)
++            if pid:
++                msg = "Jenkins with pid '%s' is already running on port %s\n" \
++                      % (pid.strip(), self.port)
++                self.stdout.write(msg)
++                return
++
++            # start jenkins service
++            self.jenkins_start()
++
++            cmd = 'cp %s %s' % (settings.JENKINS_CONFIG, self.jenkins_home)
++            self.run(cmd)
++
++        except CalledProcessError:
++            raise CalledProcessError(1, cmd, "Command Error")
++
++
++    def stop_jenkins(self, pid):
++        """
++        Stops the jenkins process running with the pid specified.
++        param :
++        pid: pid of the jenkins service to stop
++        """
++        cmd = 'kill -9 %s' % pid
++        try:
++            self.run(cmd)
++        except CalledProcessError:
++            raise CalledProcessError(1, cmd, "Command Error")
++
++    def download(self, url, filename):
++        """
++        Downloads the jenkins.war file if it is not already
++        present in the JENKINS_HOME dir
++        param:
++        filename: contains the location of the jenkins.war file
++        """
++        try:
++            response = urllib2.urlopen(urllib2.quote(url, safe=":/"))
++            if not (os.path.exists(filename) and  os.path.isfile(filename)):
++                fd = open(filename, 'wb')
++                shutil.copyfileobj(response, fd)
++                fd.close()
++
++            actual_size = int(response.headers['Content-Length'])
++            downloaded_size = int(os.stat(filename).st_size)
++            response.close()
++            if actual_size != downloaded_size:
++                raise
++        except:
++            raise CommandError("Could not retrieve %s" % url)
++            sys.exit(1)
++        return 0
++
++    def jenkins_start(self):
++        """
++        Start the jenkins service using the downloaded jenkins.war
++        available in the JENKINS_HOME dir.
++        Jenkins is started with admin JENKINS_USER/JENKINS_PASSWD
++        credentials. Copy the template config.xml which enables
++        options for user to login with JENKINS_USER/JENKINS_PASSWD credentials
++        from browser.
++        """
++        # Download Jenkins war file
++        self.jenkins_war = os.path.join(self.jenkins_home,
++                           os.path.basename(self.jenkins_war_url))
++
++        self.download(self.jenkins_war_url, self.jenkins_war)
++
++        # Start the jenkins service on port 9090
++        cmd = '/usr/bin/java -jar %s --httpPort=%s --prefix=/jenkins \
++              --argumentsRealm.passwd.%s=%s \
++              --argumentsRealm.roles.%s=admin' \
++              % (self.jenkins_war, self.port,
++                 self.jenkins_user, self.jenkins_passwd,
++                 self.jenkins_user)
++
++        cmd = shlex.split(cmd)
++        log_fn = os.path.join(self.jenkins_home, self.jenkins_log)
++        log_fd = open(log_fn, 'a')
++        ret = Popen(cmd, stdout=log_fd, stderr=STDOUT, env={"JENKINS_HOME":self.jenkins_home})
++
++        if ret.returncode:
++            raise CalledProcessError(1, cmd, 'cmd Failed')
++        log_fd.close()
++
++        self.jenkins_pid = ret.pid
++        return self.jenkins_pid
++
++    def run(self, cmd):
++        try:
++            ret = check_output(cmd, shell=True)
++            return ret
++        except CalledProcessError:
++            raise CalledProcessError(1, cmd, "Command Error")
 === modified file 'dashboard/frontend/tests/__init__.py'
 --- dashboard/frontend/tests/__init__.py	2012-07-09 19:29:31 +0000
 +++ dashboard/frontend/tests/__init__.py	2012-07-18 17:24:18 +0000
@@ -1,5 +1,6 @@
  from dashboard.frontend.tests.test_views import *
  from dashboard.frontend.tests.test_clientresponse import *
++from dashboard.frontend.tests.test_custom_commands import *
  #starts the test suite
@@ -7,4 +8,5 @@
             'HomePageViewTests': HomePageViewTest,
             'IndexViewTests': IndexViewTest,
             'ClientResponseTests': ClientResponseTests,
++           'JenkinsCommandTest': JenkinsCommandTest,
+            }
 === added file 'dashboard/frontend/tests/test_custom_commands.py'
 --- dashboard/frontend/tests/test_custom_commands.py	1970-01-01 00:00:00 +0000
 +++ dashboard/frontend/tests/test_custom_commands.py	2012-07-18 17:24:18 +0000
@@ -0,0 +1,148 @@
++#!/usr/bin/env python
++# Copyright (C) 2012 Linaro
++#
++# This file is part of linaro-ci-dashboard.
++#
++# linaro-ci-dashboard is free software: you can redistribute it and/or modify
++# it under the terms of the GNU Affero General Public License as published by
++# the Free Software Foundation, either version 3 of the License, or
++# (at your option) any later version.
++#
++# linaro-ci-dashboard is distributed in the hope that it will be useful,
++# but WITHOUT ANY WARRANTY; without even the implied warranty of
++# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
++# GNU Affero General Public License for more details.
++#
++# You should have received a copy of the GNU Affero General Public License
++# along with linaro-ci-dashboard.  If not, see <http://www.gnu.org/licenses/>.
++
++import frontend.management.commands.jenkins as jenkins
++from django.core.management.base import CommandError
++from django.conf import settings
++from subprocess import  CalledProcessError
++from django.core import management
++from StringIO import StringIO
++from django.test import TestCase
++import os, sys, shutil
++
++
++class JenkinsCommandTest(TestCase):
++
++    orig_dir = os.getcwd()
++
++    def setUp(self):
++        self.command = jenkins.Command()
++        self.command.port = '9191'
++        settings.JENKINS_PORT = self.command.port
++        self.url = 'http://updates.jenkins-ci.org/latest/bazaar.hpi'
++        os.environ["JENKINS_HOME"] = os.path.join(self.orig_dir, 'tmp/',
++                                     'jenkins_home')
++        self.command.jenkins_home = os.getenv("JENKINS_HOME")
++        self.command.jenkins_war = os.path.join(self.command.jenkins_home,
++                              os.path.basename(self.command.jenkins_war_url))
++        self.download_filename = os.path.join(self.command.jenkins_home,
++                                 os.path.basename(self.url))
++
++        if not os.path.exists(self.command.jenkins_home):
++            os.makedirs(self.command.jenkins_home)
++
++        super(JenkinsCommandTest, self).setUp()
++
++    def tearDown(self):
++        os.chdir(self.orig_dir)
++        if os.path.exists(self.download_filename):
++             cmd = 'rm  %s' % self.download_filename
++             self.command.run(cmd)
++
++        parent_dir = os.path.dirname(self.command.jenkins_home)
++        if os.path.isdir(parent_dir):
++            shutil.rmtree(parent_dir)
++
++        super(JenkinsCommandTest, self).tearDown()
++
++    def test_download_non_existing_package(self):
++        url = 'http://mirrors.jenkins-ci.org/war/latest/jenkins1.war'
++        filename = os.path.join('/tmp/', os.path.basename(url))
++        self.assertRaises(CommandError, self.command.download, url, filename)
++
++    def test_download_existing_package(self):
++        ret = self.command.download(self.url, self.download_filename)
++        self.assertTrue(ret == 0)
++
++    def test_install_run_jenkins(self):
++        orig_stdout = sys.stdout
++        stdout = sys.stdout = StringIO()
++        cmd = "lsof -w -n -i tcp:%s | awk -F ' ' '{print $2}' | tail -n 1 | "\
++               "sed '/^$/d'"  % self.command.port
++        self.command.jenkins_pid = self.command.run(cmd)
++        management.call_command('jenkins', 'start', interactive=False)
++        if len(self.command.jenkins_pid) != 0:
++            msg = "Jenkins with pid '%s' is already running on port %s"\
++                  % (self.command.jenkins_pid.strip(), self.command.port)
++        else:
++            msg = "Successfully configured jenkins with pid"
++        sys.stdout = orig_stdout
++        stdout.seek(0)
++        self.assertIn(msg, stdout.read())
++
++    def test_invalid_run(self):
++        cmd = 'rm /tmp/dummy'
++        self.assertRaises(CalledProcessError, self.command.run, cmd)
++
++    def test_valid_run(self):
++        ret = self.command.run(['whoami'])
++        self.assertTrue(ret.strip() == os.getenv('USER'))
++
++
++    def test_jenkins_stop_jenkins_non_existing_pid(self):
++        pid = '123456'
++        self.assertRaises(CalledProcessError, self.command.stop_jenkins, pid)
++
++
++    def test_jenkins_stop_jenkins(self):
++        orig_stdout = sys.stdout
++        stdout = sys.stdout = StringIO()
++        cmd = "lsof -w -n -i tcp:%s | awk -F ' ' '{print $2}' | tail -n 1 | "\
++               "sed '/^$/d'"  % self.command.port
++        pid = self.command.run(cmd)
++        management.call_command('jenkins', 'start', interactive=False)
++        if len(pid) != 0:
++            msg = "Jenkins with pid '%s' is already running on port %s"\
++                  % (pid.strip(), self.command.port)
++        else:
++            pid = self.command.jenkins_start()
++            management.call_command('jenkins', 'stop', pid, interactive=False)
++            msg = "Successfully terminated jenkins with pid"
++        sys.stdout = orig_stdout
++        stdout.seek(0)
++        self.assertIn(msg, stdout.read())
++
++
++    def test_jenkins_stop_missing_value(self):
++        orig_stdout = sys.stdout
++        stdout = sys.stdout = StringIO()
++        management.call_command('jenkins', 'stop', interactive=False)
++        msg = "Missing pid value. Specify pid after stop argument"
++        sys.stdout = orig_stdout
++        stdout.seek(0)
++        self.assertIn(msg, stdout.read())
++
++
++    def test_jenkins_invalid_option(self):
++        orig_stdout = sys.stdout
++        stdout = sys.stdout = StringIO()
++        management.call_command('jenkins', 'Invalid', interactive=False)
++        msg = "Invalid option. Supports only start/stop."
++        sys.stdout = orig_stdout
++        stdout.seek(0)
++        self.assertIn(msg, stdout.read())
++
++
++    def test_jenkins_invalid_option(self):
++        orig_stdout = sys.stdout
++        stdout = sys.stdout = StringIO()
++        management.call_command('jenkins', interactive=False)
++        msg = "Need to specify either start/stop"
++        sys.stdout = orig_stdout
++        stdout.seek(0)
++        self.assertIn(msg, stdout.read())
 === modified file 'dashboard/settings.py'
 --- dashboard/settings.py	2012-07-11 10:20:55 +0000
 +++ dashboard/settings.py	2012-07-18 17:24:18 +0000
@@ -21,8 +21,8 @@
  DEBUG = True
  TEMPLATE_DEBUG = DEBUG
--DEPS_INSTALL_PATH = os.path.join(os.path.dirname(os.path.abspath(__file__)),
--                                "..", "lib")
++ROOT_PATH = os.path.dirname(os.path.abspath(__file__))
++DEPS_INSTALL_PATH = os.path.join(ROOT_PATH, "..", "lib")
  ADMINS = (
      # ('Your Name', 'your_email@example.com'),
@@ -175,3 +175,12 @@
          },
+     }
+ }
++
++JENKKINS_WAR_URL = 'http://mirrors.jenkins-ci.org/war/latest/jenkins.war'
++JENKINS_ADMIN_USER = 'admin'
++JENKINS_ADMIN_PASSWD = 'admin'
++JENKINS_PORT = '9090'
++JENKINS_CONFIG = os.path.join(ROOT_PATH, '../config_template/', 'config.xml')
++JENKINS_HOME = os.path.join(os.getenv('HOME'), ".jenkins_test")
++JENKINS_LOG_NAME = 'jenkins.log'
++

linaro-ci-dashboard

Merge lp:~linaro-infrastructure/linaro-ci-dashboard/run_stop_jenkins_command into lp:linaro-ci-dashboard

Commit message

Description of the change

Preview Diff

Subscribers