Merge lp:~matsubara/charms/trusty/jenkins/jenkins-bundle into lp:charms/trusty/jenkins

[Diogo Matsubara]

Hi, this branch ports a few features from the shell based precise jenkins charm into the python based jenkins charm.

- Add a way to deploy jenkins from a bundled jenkins.deb package. Useful for environments where the jenkins node can't reach http://jenkins-ci.org;
- add extension-relation-* so the charm can be used together with the ci-configurator charm;
- export credentials to jenkins slave as part of the fix for bug 1232886. I'll merge propose the jenkins-slave charm fix separately;
- fix an UnboundLocalError: local variable 'PASSWORD' referenced before assignment.

[Diogo Matsubara (matsubara)]

jenkins-slave part of the fix is MP'ed here: https://code.launchpad.net/~matsubara/charms/trusty/jenkins-slave/trusty-jenkins-slave/+merge/270418

[Cory Johns (johnsca)]

Diogo,

Thank you for this submission. On the whole, this looks good, but I have some inline comments I've made below that need to be addressed. Nothing major, a couple of small lint errors introduced, some questions / comments in the handling of the extension relation, and what looks to be a carry-over of a bug with the password handling in one spot. Again, thanks for this work!

review: Needs Fixing

[Review Queue (review-queue)]

The results (PASS) are in and available here: http://juju-ci.vapour.ws:8080/job/charm-bundle-test-lxc/1429/

review: Approve (automated testing)

[Review Queue (review-queue)]

The results (PASS) are in and available here: http://juju-ci.vapour.ws:8080/job/charm-bundle-test-aws/1417/

review: Approve (automated testing)

[Diogo Matsubara (matsubara)]

Hi Cory,

thanks for the review. I think I've addressed all your points and made comments inline below. Please take another look.

Cheers,

Diogo

[lp:~matsubara/charms/trusty/jenkins/jenkins-bundle]

42. By Diogo Matsubara on 2015-11-11

address review comments: improve documentation of bundle option, fix logic of existing password, fix lint errors, add zuul interface

43. By Diogo Matsubara on 2015-11-14

port zuul-relation-joined hook from bash to python

[Cory Johns (johnsca)]

Diogo,

Thanks for addressing the comments in my previous review! However, I see an issue with the Zuul addition, which I mentioned on IRC and added as a line comment below. I'd also like some direction, if not an actual test case in the charm, on how to test the extension and zuul relation additions.

review: Needs Fixing

[lp:~matsubara/charms/trusty/jenkins/jenkins-bundle]

44. By Diogo Matsubara on 2015-11-17

oops forgot to pass jenkins_uid and jenkins_gid to install_jenkins_plugins()

[Diogo Matsubara (matsubara)]

Hi Cory,

thanks for catching that. I updated the code to pass the correct args to the install_jenkins_plugin() function.

I'll give it a try on full deployment and comment back with instructions.

[lp:~matsubara/charms/trusty/jenkins/jenkins-bundle]

45. By Diogo Matsubara on 2015-11-17

required_plugins needs to be a string rather than a list

46. By Diogo Matsubara on 2015-11-17

pass a list to hooks.execute

[Diogo Matsubara (matsubara)]

Hi Cory,

you'll need to grab lp:~paulgear/charms/precise/zuul/trunk and deploy from it. I'm using this bundle[1] to deploy and had to run make installdeps in zuul charm. It deployed successfully for me on canonistack.

[1] https://pastebin.canonical.com/144291/

[Cory Johns (johnsca)]

Diogo,

Thank you for your work on this. I have finished my review and testing, and this looks good. It is now merged.

As an aside, to get Zuul to deploy without the `make installdeps` step (which I missed), I had to make a change, which I submitted as an MP against that charm: https://code.launchpad.net/~johnsca/charms/precise/zuul/install-403/+merge/278072 (in case anyone else runs into this issue).

review: Approve

=== modified file 'config.yaml'
--- config.yaml	2014-11-18 23:06:36 +0000
+++ config.yaml	2015-11-17 21:39:31 +0000
@@ -10,6 +10,10 @@
      Source of Jenkins, options include:
      - lts: use the most recent Jenkins LTS release.
      - trunk: use the most recent Jenkins release.
+     - bundle: use a bundled deb package. The jenkins deb package needs to be
+               manually copied over to the charm files/ dir before deployment
+               and named jenkins.deb. It can be downloaded from
+               http://pkg.jenkins-ci.org/debian/
   username:
     type: string
     default: admin

=== added symlink 'hooks/extension-relation-changed'
=== target is u'jenkins_hooks.py'
=== added symlink 'hooks/extension-relation-joined'
=== target is u'jenkins_hooks.py'
=== modified file 'hooks/jenkins_hooks.py'
--- hooks/jenkins_hooks.py	2015-01-23 09:36:49 +0000
+++ hooks/jenkins_hooks.py	2015-11-17 21:39:31 +0000
@@ -12,6 +12,7 @@
     UnregisteredHookError,
     config,
     remote_unit,
+    related_units,
     relation_get,
     relation_set,
     relation_ids,
@@ -26,6 +27,7 @@
     apt_update,
 )
 from charmhelpers.core.host import (
+    service_restart,
     service_start,
     service_stop,
 )
@@ -36,7 +38,9 @@
     TEMPLATES_DIR,
     add_node,
     del_node,
+    get_jenkins_password,
     setup_source,
+    install_from_bundle,
     install_jenkins_plugins,
 )
 
@@ -46,9 +50,13 @@
 @hooks.hook('install')
 def install():
     execd_preinstall('hooks/install.d')
-    # Only setup the source if jenkins is not already installed i.e. makes the
-    # config 'release' immutable so you can't change source once deployed
-    setup_source(config('release'))
+    if config('release') == 'bundle':
+        install_from_bundle()
+    else:
+        # Only setup the source if jenkins is not already installed i.e. makes
+        # the config 'release' immutable so you can't change source once
+        # deployed.
+        setup_source(config('release'))
     config_changed()
     open_port(8080)
 
@@ -160,10 +168,12 @@
 
 @hooks.hook('master-relation-changed')
 def master_relation_changed():
-    PASSWORD = config('password')
-    if PASSWORD:
-        with open('/var/lib/jenkins/.admin_password', 'r') as fd:
-            PASSWORD = fd.read()
+    password = get_jenkins_password()
+    # Once we have the password, export credentials to the slave so it can
+    # download slave-agent.jnlp from the master.
+    username = config('username')
+    relation_set(username=username)
+    relation_set(password=password)
 
     required_settings = ['executors', 'labels', 'slavehost']
     settings = relation_get()
@@ -183,7 +193,7 @@
         return
 
     log("Adding slave with hostname %s." % (slavehost), level=DEBUG)
-    add_node(slavehost, executors, labels, config('username'), PASSWORD)
+    add_node(slavehost, executors, labels, username, password)
     log("Node slave %s added." % (slavehost), level=DEBUG)
 
 
@@ -198,16 +208,12 @@
 
 @hooks.hook('master-relation-broken')
 def master_relation_broken():
-    password = config('password')
-    if not password:
-        passwd_file = os.path.join(JENKINS_HOME, '.admin_password')
-        with open(passwd_file, 'w+') as fd:
-            PASSWORD = fd.read()
+    password = get_jenkins_password()
 
     for member in relation_ids():
         member = member.replace('/', '-')
         log("Removing node %s from Jenkins master." % (member), level=DEBUG)
-        del_node(member, config('username'), PASSWORD)
+        del_node(member, config('username'), password)
 
 
 @hooks.hook('website-relation-joined')
@@ -217,6 +223,85 @@
     relation_set(port=8080, hostname=hostname)
 
 
[e-mail]('extension-relation-joined')
+def extension_relation_joined():
+    log("Updating extension interface with up-to-date data.")
+    # Fish out the current zuul address from any relation we have.
+    zuul_address = ""
+    for rid in relation_ids('zuul'):
+        for unit in related_units(rid):
+            zuul_address = relation_get(
+                rid=rid, unit=unit, attribute='private-address')
+
+    for rid in relation_ids('extension'):
+        r_settings = {
+            'admin_username': config('username'),
+            'admin_password': get_jenkins_password(),
+            'jenkins_url': 'http://%s:8080' % unit_get('private-address'),
+            'jenkins-admin-user': config('jenkins-admin-user'),
+            'jenkins-token': config('jenkins-token')
+        }
+        relation_set(relation_id=rid, relation_settings=r_settings)
+        if zuul_address:
+            relation_set(relation_id=rid, zuul_address=zuul_address)
+
+
[e-mail]('extension-relation-changed')
+def extension_relation_changed():
+    # extension subordinates may request the principle service install
+    # specified jenkins plugins
+    if relation_get('required_plugins'):
+        log("Installing required plugins as requested by jenkins-extension "
+            "subordinate.")
+        jenkins_uid = pwd.getpwnam('jenkins').pw_uid
+        jenkins_gid = grp.getgrnam('jenkins').gr_gid
+        install_jenkins_plugins(
+            jenkins_uid, jenkins_gid,
+            plugins=relation_get('required_plugins'))
+
+
+ZUUL_CONFIG_SNIPPET = """
+<hudson.plugins.gearman.GearmanPluginConfig>
+  <enablePlugin>true</enablePlugin>
+    <host>{}</host>
+    <port>4730</port>
+</hudson.plugins.gearman.GearmanPluginConfig>
+"""
+
+
[e-mail]('zuul-relation-joined')
+def zuul_relation_joined():
+    log("Installing and configuring gearman-plugin for Zuul communication")
+    # zuul relation requires we install the required plugins and set the
+    # address of the remote zuul/gearman service in the plugin setting.
+    required_plugins = (
+        "credentials ssh-credentials ssh-agent gearman-plugin git-client git")
+
+    # Grab jenkins uid and gid.
+    jenkins_uid = pwd.getpwnam('jenkins').pw_uid
+    jenkins_gid = grp.getgrnam('jenkins').gr_gid
+    log("Installing and configuring gearman-plugin for Zuul communication")
+    install_jenkins_plugins(
+        jenkins_uid, jenkins_gid, plugins=required_plugins)
+    # Generate plugin config with address of remote unit.
+    zuul_config = ZUUL_CONFIG_SNIPPET.format(relation_get('private-address'))
+    config_path = os.path.join(
+        JENKINS_HOME, "hudson.plugins.gearman.GearmanPluginConfig.xml")
+    with open(config_path, 'w') as f:
+        f.write(zuul_config)
+
+    # Change permission of config file.
+    nogroup_gid = grp.getgrnam('nogroup').gr_gid
+    os.chown(config_path, jenkins_uid, nogroup_gid)
+
+    # Restart jenkins so changes will take efect.
+    service_restart('jenkins')
+
+    # Trigger the extension hook to update it with zuul relation data, if its
+    # coded to do so.
+    hooks.execute(['extension-relation-joined'])
+
+
 if __name__ == '__main__':
     try:
         hooks.execute(sys.argv)

=== modified file 'hooks/jenkins_utils.py'
--- hooks/jenkins_utils.py	2015-01-20 18:33:34 +0000
+++ hooks/jenkins_utils.py	2015-11-17 21:39:31 +0000
@@ -6,6 +6,7 @@
 import tempfile
 
 from charmhelpers.core.hookenv import (
+    charm_dir,
     config,
     log,
     DEBUG,
@@ -13,6 +14,7 @@
     WARNING,
 )
 from charmhelpers.fetch import (
+    apt_install,
     apt_update,
     add_source,
 )
@@ -60,6 +62,21 @@
         log("Node '%s' does not exist - not deleting" % (host), level=INFO)
 
 
+def install_from_bundle():
+    """Install Jenkins from bundled package."""
+    # Check bundled package exists.
+    bundle_path = os.path.join(charm_dir(), 'files', 'jenkins.deb')
+    if not os.path.isfile(bundle_path):
+        errmsg = "'%s' doesn't exist. No package bundled." % (bundle_path)
+        raise Exception(errmsg)
+    log('Installing from bundled Jenkins package: %s' % bundle_path)
+    # Install bundle deps.
+    apt_install(['daemon', 'adduser', 'psmisc', 'default-jre'], fatal=True)
+    # Run dpkg to install bundled deb.
+    env = os.environ.copy()
+    subprocess.call(['dpkg', '-i', bundle_path], env=env)
+
+
 def setup_source(release):
     """Install Jenkins archive."""
     log("Configuring source of jenkins as %s" % release, level=INFO)
@@ -108,8 +125,8 @@
     apt_update(fatal=True)
 
 
-def install_jenkins_plugins(jenkins_uid, jenkins_gid):
-    plugins = config('plugins')
+def install_jenkins_plugins(jenkins_uid, jenkins_gid, plugins=None):
+    plugins = plugins or config('plugins')
     if plugins:
         plugins = plugins.split()
     else:
@@ -176,3 +193,13 @@
     finally:
         # Delete install refs
         shutil.rmtree(track_dir)
+
+
+def get_jenkins_password():
+    """Return password from the config or the one saved on file."""
+    password = config('password')
+    if not password:
+        passwd_file = os.path.join(JENKINS_HOME, '.admin_password')
+        with open(passwd_file, 'r') as fd:
+            password = fd.read()
+    return password

=== added symlink 'hooks/zuul-relation-joined'
=== target is u'jenkins_hooks.py'
=== modified file 'metadata.yaml'
--- metadata.yaml	2015-03-16 20:37:46 +0000
+++ metadata.yaml	2015-11-17 21:39:31 +0000
@@ -25,3 +25,6 @@
   master:
     interface: jenkins-slave
     optional: true
+  zuul:
+    interface: zuul
+    optional: true

=== modified file 'test-requirements.txt'
--- test-requirements.txt	2015-01-26 11:19:23 +0000
+++ test-requirements.txt	2015-11-17 21:39:31 +0000
@@ -1,3 +1,4 @@
+coverage
 distribute
 flake8
 nose