Merge lp:~mrazik/cupstream2distro-config/jenkins-per-project into lp:cupstream2distro-config

[Martin Mrazik]

This implements a per-job jenkins ci server (with default to ci-jenkins from the current configuration) and moves stacks/phablet/ubuntu-touch-coreapps.cfg to "touch-coreapps-jenkins" jenkins.

This implements a per-job jenkins ci server (with default to ci-jenkins from the current configuration) and moves stacks/phablet/ubuntu-touch-coreapps.cfg to "touch-coreapps-jenkins" jenkins.

[PS Jenkins bot (ps-jenkins)]

PASSED: Continuous integration, rev:139
http://jenkins.qa.ubuntu.com/job/cupstream2distro-config-ci/63/
Executed test runs:
    SUCCESS: http://jenkins.qa.ubuntu.com/job/cupstream2distro-config-raring-amd64-ci/63

Click here to trigger a rebuild:
http://s-jenkins:8080/job/cupstream2distro-config-ci/63/rebuild

review: Approve (continuous-integration)

[Francis Ginther (fginther)]

I would prefer to see
63 + credentials = load_jenkins_credentials(
64 + os.path.expanduser(self.credentialsfile),
65 + jenkins_name)
66 + if not credentials:
67 + logging.error(
68 + 'Credentials for "{}" not found.'.format(jenkins_name))
69 + return None
kept inside of __call__ and store the parsed credentials as 'self.credentials'.

This will cause a bad credentials file to fail the script before any stack processing is done. Should also make testing easier as you can use a pre-made credentials dictionary instead of patching load_jenkins_credentials.

Otherwise I like this change.

review: Needs Fixing

[Martin Mrazik (mrazik)]

> I would prefer to see
> 63 + credentials = load_jenkins_credentials(
> 64 + os.path.expanduser(self.credentialsfile),
> 65 + jenkins_name)
> 66 + if not credentials:
> 67 + logging.error(
> 68 + 'Credentials for "{}" not
> found.'.format(jenkins_name))
> 69 + return None
> kept inside of __call__ and store the parsed credentials as
> 'self.credentials'.
>
> This will cause a bad credentials file to fail the script before any stack
> processing is done. Should also make testing easier as you can use a pre-made
> credentials dictionary instead of patching load_jenkins_credentials.

I would need to modify load_jenkins_credentials to load _all_ jenkins credentials and not just the specified. The thing is that you also don't know the credentials name _before_ you process the stack.

[Francis Ginther (fginther)]

My mistake. I was thinking that 'credentials' was simply the entire contents of the credentials file, but now I see that 'credentials' is just the portion specific to the jenkins_name. So, my suggestion would require more refactoring which isn't really necessary.

Approving as is.

review: Approve

[PS Jenkins bot (ps-jenkins)]

FAILED: Autolanding.
More details in the following jenkins job:
http://jenkins.qa.ubuntu.com/job/cupstream2distro-config-autolanding/34/
Executed test runs:
    FAILURE: http://jenkins.qa.ubuntu.com/job/cupstream2distro-config-raring-amd64-autolanding/34/console

review: Needs Fixing (continuous-integration)

[Francis Ginther (fginther)]

Dropping to a work in progress. This just needs to be re-done from scratch.

[Loïc Minier (lool)]

(Rejecting to drop out of active reviews)

Unmerged revisions

139. By Martin Mrazik on 2013-04-03

minor change to improve logging

138. By Martin Mrazik on 2013-04-03

added tests for _get_jenkins_handle

137. By Martin Mrazik on 2013-04-03

fixed a test

136. By Martin Mrazik on 2013-04-03

Some adjustments

135. By Martin Mrazik on 2013-04-02

first attempt to make jenkins server configurable per project

=== modified file 'c2dconfigutils/cu2dUpdateCi.py'
--- c2dconfigutils/cu2dUpdateCi.py	2013-03-21 12:51:16 +0000
+++ c2dconfigutils/cu2dUpdateCi.py	2013-04-03 09:23:22 +0000
@@ -53,6 +53,11 @@
 class UpdateCi(object):
     DEFAULT_CREDENTIALS = os.path.expanduser('~/.cu2d.cred')
     JENKINS_CI_CONFIG_NAME = 'ci-jenkins'
+    # directory of handles for different jenkins servers
+    # this is in the form of jenkins_name: jenkins_handle
+    # the directory is lazy-initialized. When a stack/job needs
+    # a specific jenkins for the first time the handle is created.
+    JENKINS_HANDLES = {}
 
     # Please keep list sorted
     TEMPLATE_CONTEXT_KEYS = [
@@ -75,6 +80,7 @@
         'team',
         'use_description_for_commit',
         'log_rotator',
+        'jenkins_ci_server',
         'days_to_keep_builds',
         'num_to_keep_builds',
     ]
@@ -284,23 +290,49 @@
                                    autolanding_dict, autolanding_template,
                                    build_template)
 
-    def update_jenkins(self, jenkins_handle, jjenv, stack, update=False):
+    def update_jenkins(self, jjenv, stack, update=False):
         """ Add/update jenkins jobs
 
-        :param jenkins_handle: jenkins access handle
         :param jjenv: jinja2 template environment handle
         :param stack: dictionary with configuration of the stack
         :param update: Update existing jobs if true
 
         :return: True on success
         """
+
+        result = True
         if stack['projects']:
             job_list = []
             self.process_stack(job_list, stack)
             for job in job_list:
-                setup_job(jenkins_handle, jjenv, job['name'], job['template'],
-                          job['ctx'], update)
-        return True
+                jenkins_handle = self._get_jenkins_handle(job)
+                if jenkins_handle:
+                    setup_job(jenkins_handle, jjenv, job['name'],
+                              job['template'], job['ctx'], update)
+                else:
+                    result = False
+                    logging.error(
+                        'Unable to acquire jenkins. Skipping this job.')
+        return result
+
+    def _get_jenkins_handle(self, job_config):
+        jenkins_name = job_config['ctx']['jenkins_ci_server']
+        if jenkins_name in self.JENKINS_HANDLES:
+            return self.JENKINS_HANDLES[jenkins_name]
+
+        credentials = load_jenkins_credentials(
+            os.path.expanduser(self.credentialsfile),
+            jenkins_name)
+        if not credentials:
+            logging.error(
+                'Credentials for "{}" not found.'.format(jenkins_name))
+            return None
+        jenkins_handle = get_jenkins_handle(credentials)
+        if not jenkins_handle:
+            logging.error('Could not acquire connection to jenkins.')
+            return None
+        self.JENKINS_HANDLES[jenkins_name] = jenkins_handle
+        return jenkins_handle
 
     def __call__(self, default_config_path):
         """Entry point for cu2d-update-ci"""
@@ -315,23 +347,10 @@
             logging.error('Stack configuration failed to load. Aborting!')
             return 1
 
-        credentials = None
         if args.credentials:
-            credentialsfile = args.credentials
-            credentials = load_jenkins_credentials(
-                os.path.expanduser(credentialsfile),
-                self.JENKINS_CI_CONFIG_NAME)
-            if not credentials:
-                logging.error('Credentials not found. Aborting!')
-                return 1
-            jenkins_handle = get_jenkins_handle(credentials)
-            if not jenkins_handle:
-                logging.error('Could not acquire connection to jenkins. ' +
-                              'Aborting!')
-                return 1
+            self.credentialsfile = args.credentials
             jjenv = get_jinja_environment(default_config_path, stackcfg)
-            if not self.update_jenkins(jenkins_handle, jjenv, stackcfg,
-                                       args.update_jobs):
+            if not self.update_jenkins(jjenv, stackcfg, args.update_jobs):
                 logging.error('Failed to configure jenkins jobs. Aborting!')
                 return 2
         return 0

=== modified file 'ci/config/defaults.conf'
--- ci/config/defaults.conf	2013-03-21 12:51:16 +0000
+++ ci/config/defaults.conf	2013-04-03 09:23:22 +0000
@@ -1,5 +1,6 @@
 stack:
   ci_default:
+    jenkins_ci_server: 'ci-jenkins'
     ci_template: ci-config.xml.tmpl
     autolanding_template: autolanding-config.xml.tmpl
     build_template: pbuilder-config.xml.tmpl

=== modified file 'credentials'
--- credentials	2013-02-28 16:31:58 +0000
+++ credentials	2013-04-03 09:23:22 +0000
@@ -8,3 +8,7 @@
   password: JENKINS_PASSWORD
   url: JENKINS_URL
   token: JENKINS_TOKEN
+touch-coreapps-jenkins:
+  username: JENKINS_USERNAME
+  password: JENKINS_PASSWORD
+  url: JENKINS_URL

=== modified file 'stacks/phablet/ubuntu-touch-coreapps.cfg'
--- stacks/phablet/ubuntu-touch-coreapps.cfg	2013-03-26 13:37:50 +0000
+++ stacks/phablet/ubuntu-touch-coreapps.cfg	2013-04-03 09:23:22 +0000
@@ -9,6 +9,7 @@
   daily_release_default:
     daily_release: False # Disable all for daily release process
   ci_default:
+    jenkins_ci_server: 'touch-coreapps-jenkins'
     autolanding:
       ppa_target: ppa:ubuntu-touch-coreapps-drivers/daily
       distributions: quantal,raring

=== modified file 'tests/test_cu2dUpdateCi.py'
--- tests/test_cu2dUpdateCi.py	2013-03-15 15:27:32 +0000
+++ tests/test_cu2dUpdateCi.py	2013-04-03 09:23:22 +0000
@@ -220,6 +220,54 @@
         self.assertEqual(expected, actual)
 
 
+class TestGetJenkinsHandle(TestCase):
+    def setUp(self):
+        self.update_ci = UpdateCi()
+        self.update_ci.credentialsfile = '~/.cu2d.cred'
+
+    def test_handle_exists(self):
+        jenkins_name = 'test_handle_exists'
+        self.update_ci.JENKINS_HANDLES[jenkins_name] = MagicMock()
+        jenkins_handle = self.update_ci._get_jenkins_handle(
+            {'ctx': {'jenkins_ci_server': jenkins_name}})
+        self.assertEqual(jenkins_handle,
+                         self.update_ci.JENKINS_HANDLES[jenkins_name])
+
+    def test_no_credentials(self):
+        jenkins_name = 'test_no_credentials'
+        load_jenkins_credentials = lambda x, y: None
+        with patch('c2dconfigutils.cu2dUpdateCi.load_jenkins_credentials',
+                   load_jenkins_credentials):
+            jenkins_handle = self.update_ci._get_jenkins_handle(
+                {'ctx': {'jenkins_ci_server': jenkins_name}})
+            self.assertIsNone(jenkins_handle)
+
+    def test_no_jenkins_handle(self):
+        jenkins_name = 'test_no_jenkins_handle'
+        load_jenkins_credentials = lambda x, y: object()
+        get_jenkins_handle = lambda x: None
+        with patch('c2dconfigutils.cu2dUpdateCi.load_jenkins_credentials',
+                   load_jenkins_credentials), \
+            patch('c2dconfigutils.cu2dUpdateCi.get_jenkins_handle',
+                  get_jenkins_handle):
+            jenkins_handle = self.update_ci._get_jenkins_handle(
+                {'ctx': {'jenkins_ci_server': jenkins_name}})
+            self.assertIsNone(jenkins_handle)
+
+    def test_get_jenkins_handle(self):
+        jenkins_name = 'test_get_jenkins_handle'
+        jenkins_handle = object()
+        load_jenkins_credentials = lambda x, y: object()
+        get_jenkins_handle = lambda x: jenkins_handle
+        with patch('c2dconfigutils.cu2dUpdateCi.load_jenkins_credentials',
+                   load_jenkins_credentials), \
+            patch('c2dconfigutils.cu2dUpdateCi.get_jenkins_handle',
+                  get_jenkins_handle):
+            result = self.update_ci._get_jenkins_handle(
+                {'ctx': {'jenkins_ci_server': jenkins_name}})
+            self.assertEqual(jenkins_handle, result)
+
+
 class TestUpdateJenkins(TestCase):
     def setUp(self):
         self.jenkins = MagicMock()
@@ -233,7 +281,7 @@
 
     def test_empty_stack(self):
         stack = {'projects': None}
-        self.update_ci.update_jenkins(None, None, stack)
+        self.update_ci.update_jenkins(None, stack)
 
     def test_stack(self):
         stack = {