Merge lp:~doanac/charms/precise/ubuntu-ci-services-itself/lander into lp:~canonical-ci-engineering/charms/precise/ubuntu-ci-services-itself/lander

[Andy Doan]

This updates the master and ppa-assigner jobs to handle parameters properly. They now are able to determine things like the ppa-assigner's host ip via juju relation ships. It might be easiest to read this commit by commit.

[Andy Doan (doanac)]

FYI - I have a private juju deployer file that can do a full deployment of the ppa-assigner and lander. Let me know if you want a copy (it includes oauth tokens so i can't share publically).

[Francis Ginther (fginther)]

The jenkins jobs are supposed to pass the params.json file from one job to the next, it's not intended to be created as is being done in templates/jobs/lander-ppa-assigner.xml. I confirmed that this is broken and am investigating.

[Andy Doan (doanac)]

On 01/06/2014 01:32 PM, Francis Ginther wrote:
> The jenkins jobs are supposed to pass the params.json file from one job to the next, it's not intended to be created as is being done in templates/jobs/lander-ppa-assigner.xml. I confirmed that this is broken and am investigating.

I think I've got this fixed in a branch based on this:

<http://bazaar.launchpad.net/~doanac/charms/precise/ubuntu-ci-services-itself/lander-bsb/revision/27>

if that looks sane, we can just fix it in that MP if that's okay?

[lp:~doanac/charms/precise/ubuntu-ci-services-itself/lander]

22. By Francis Ginther on 2014-01-08

Fix jenkins jobs so that the accumulated json data is passed as a job parameter, update lander_archiver to work with updated script, add datastore config parameters.

23. By Andy Doan on 2014-01-08

store relationship information

the jenkins jobs will need to know the hostname of each service
connected to it like the ppa-assigner. This stores them under
a fix directory jobs can load dynamically

24. By Andy Doan on 2014-01-08

add ppa-assigner url to config

25. By Andy Doan on 2014-01-08

allow charm to interface with amqp

26. By Andy Doan on 2014-01-08

add bsbuilder config to master

27. By Andy Doan on 2014-01-08

clarify parameter passing and add progress trigger

parameter passing was being done via a file named child.json. However,
this is really a jenkins parameters file and not really json. This
renames that file and creates it with a few less lines of code.

[Andy Doan (doanac)]

I've rebased on your branch with a couple of minor updates. I changed your naming of "child.json" to "child.params" since it isn't really a json file. I then added the relationship handling logic to get the ppa-assigner and bsbuilder working.

review: Needs Resubmitting

[Francis Ginther (fginther)]

Thanks for cleaning up my child.json. I got the first one updated in lander_master, but failed to update the rest. The params file works just as well.

[Francis Ginther (fginther)]

I'm ok with the relation hooks, if we need to separate them later, we'll do it then.

review: Approve

=== modified file 'config.yaml'
--- config.yaml	2014-01-08 15:04:30 +0000
+++ config.yaml	2014-01-08 21:09:33 +0000
@@ -51,3 +51,13 @@
     type: string
     default: ""
     description: Data store tenant name.
+
+# required for rabbitmq-server charm:
+  amqp-user:
+    type: string
+    default: workerbee
+    description: The user to log into the rabbitMQ server.
+  amqp-vhost:
+    type: string
+    default: '/'
+    description: The vhost in the rabbitMQ server.

=== added symlink 'hooks/amqp-relation-broken'
=== target is u'hooks.py'
=== added symlink 'hooks/amqp-relation-changed'
=== target is u'hooks.py'
=== added symlink 'hooks/amqp-relation-joined'
=== target is u'hooks.py'
=== modified file 'hooks/hooks.py'
--- hooks/hooks.py	2013-12-19 22:20:21 +0000
+++ hooks/hooks.py	2014-01-08 21:09:33 +0000
@@ -4,12 +4,10 @@
 import json
 import subprocess
 import sys
-import time
 
 
 def juju_info(msg):
     subprocess.check_call(['juju-log', '-l', 'INFO', msg])
-    pass
 
 
 def _config():
@@ -51,6 +49,11 @@
     return os.path.join(config['install_root'], unit)
 
 
+def _relation_file(config):
+    relation = os.environ['JUJU_RELATION_ID']
+    return os.path.join(_service_dir(config), 'juju-relations', relation)
+
+
 def lander_jenkins_relation_joined(config):
     juju_info("Config: %s" % config)
     hostname = subprocess.check_output(['unit-get', 'private-address']).strip()
@@ -61,13 +64,24 @@
         'port': 8080, # the jenkins charm hardcodes 8080
     })
 
+    rfile = _relation_file(config)
+    rdir = os.path.dirname(rfile)
+    if not os.path.exists(rdir):
+        os.mkdir(rdir)
+    with open(rfile, 'w') as f:
+        unit = os.environ['JUJU_REMOTE_UNIT'].split('/')[0]
+        f.write('%s: %s' % (unit, _relation_get('private-address')))
+
 
 def lander_jenkins_relation_changed(config):
-    pass
+    lander_jenkins_relation_joined(config)
 
 
 def lander_jenkins_relation_broken(config):
-    pass
+    rfile = _relation_file(config)
+    juju_info('removing relation file: %s' % rfile)
+    if os.path.exists(rfile):
+        os.unlink(rfile)
 
 
 def jenkins_extension_relation_joined(config):
@@ -85,6 +99,26 @@
     juju_info("Config: %s" % config)
 
 
+def amqp_relation_joined(config):
+    _relation_set({
+        'username': config['amqp-user'],
+        'vhost': config['amqp-vhost'],
+    })
+
+
+def amqp_relation_changed(config):
+    with open(os.path.join(_service_dir(config), 'amqp_config.py'), 'w') as f:
+        f.write('# DO NOT EDIT. Generated by lander-jenkins charm hook\n')
+        f.write('AMQP_USER = "%s"\n' % config['amqp-user'])
+        f.write('AMQP_VHOST = "%s"\n' % config['amqp-vhost'])
+        f.write('AMQP_HOST = "%s"\n' % _relation_get('private-address'))
+        f.write('AMQP_PASSWORD = "%s"\n' % _relation_get('password'))
+
+
+def amqp_relation_broken(config):
+    os.unlink(os.path.join(_service_dir(config), 'amqp_config.py'))
+
+
 def main():
     hook = os.path.basename(sys.argv[0])
     juju_info("Running hook: %s" % hook)
@@ -105,4 +139,3 @@
 
 if __name__ == '__main__':
     exit(main())
-

=== modified file 'metadata.yaml'
--- metadata.yaml	2013-12-19 22:20:21 +0000
+++ metadata.yaml	2014-01-08 21:09:33 +0000
@@ -9,6 +9,8 @@
   extension:
     interface: jenkins-extension
     scope: container
+  amqp:
+    interface: rabbitmq
 provides:
   lander-jenkins:
     interface: lander-jenkins

=== modified file 'templates/jobs/lander_master.xml'
--- templates/jobs/lander_master.xml	2014-01-08 15:04:30 +0000
+++ templates/jobs/lander_master.xml	2014-01-08 21:09:33 +0000
@@ -31,23 +31,29 @@
       <command><![CDATA[#!/bin/bash
 set -ex
 
+ppa_ip=$(grep ppa-django /srv/lander_jenkins/juju-relations/* | awk '{print $NF}' | head -n1)
+bsb_ip=$(grep bsb-restish /srv/lander_jenkins/juju-relations/* | awk '{print $NF}' | head -n1)
+
 # Convert to input file
 request_parameters=$(echo $request_parameters | sed -e 's/^"//' -e 's/"$//')
 cat > params.json <<EOF
 {
   "request_id": "$request_id",
+  "progress_trigger": "${JOB_NAME}-${BUILD_NUMBER}",
+  "ppa_assigner_url": "http://${ppa_ip}:8080",
+  "bsb_url": "http://${bsb_ip}:8080",
   "request_parameters": $request_parameters
 }
 EOF
 cat params.json
 
 /srv/lander_jenkins/lander/bin/lander_merge_parameters.py --result-file params.json --service master --output-file all.json
-request_parameters=$(echo $request_parameters | sed -e 's/^"//' -e 's/"$//')
+
 # Convert to a format that can be passed to the child jobs
-all_params=`cat all.json`
-cat > child.json <<EOF
-"request_parameters": $all_params
-EOF]]></command>
+cat > child.params <<EOF
+request_parameters=$(cat all.json)
+EOF
+      ]]></command>
     </hudson.tasks.Shell>
     <hudson.plugins.parameterizedtrigger.TriggerBuilder>
       <configs>
@@ -59,7 +65,7 @@
           </configs>
           <configFactories>
             <hudson.plugins.parameterizedtrigger.FileBuildParameterFactory>
-              <filePattern>child.json</filePattern>
+              <filePattern>child.params</filePattern>
             </hudson.plugins.parameterizedtrigger.FileBuildParameterFactory>
           </configFactories>
           <projects>lander_ppa_assigner</projects>
@@ -97,10 +103,10 @@
 /srv/lander_jenkins/lander/bin/lander_merge_parameters.py --result-file params.json --service ppa_assigner --output-file all.json --prior-file all.json
 
 # Convert to a format that can be passed to the child jobs
-all_params=`cat all.json`
-cat > child.json <<EOF
-request_parameters=$all_params
-EOF]]></command>
+cat > child.params <<EOF
+request_parameters=$(cat all.json)
+EOF
+      ]]></command>
     </hudson.tasks.Shell>
     <hudson.plugins.parameterizedtrigger.TriggerBuilder>
       <configs>
@@ -112,7 +118,7 @@
           </configs>
           <configFactories>
             <hudson.plugins.parameterizedtrigger.FileBuildParameterFactory>
-              <filePattern>child.json</filePattern>
+              <filePattern>child.params</filePattern>
             </hudson.plugins.parameterizedtrigger.FileBuildParameterFactory>
           </configFactories>
           <projects>lander_branch_source_builder</projects>
@@ -150,10 +156,10 @@
 /srv/lander_jenkins/lander/bin/lander_merge_parameters.py --result-file params.json --service bsbuilder --output-file all.json --prior-file all.json
 
 # Convert to a format that can be passed to the child jobs
-all_params=`cat all.json`
-cat > child.json <<EOF
-request_parameters=$all_params
-EOF]]></command>
+cat > child.params <<EOF
+request_parameters=$(cat all.json)
+EOF
+      ]]></command>
     </hudson.tasks.Shell>
     <hudson.plugins.parameterizedtrigger.TriggerBuilder>
       <configs>
@@ -165,7 +171,7 @@
           </configs>
           <configFactories>
             <hudson.plugins.parameterizedtrigger.FileBuildParameterFactory>
-              <filePattern>child.json</filePattern>
+              <filePattern>child.params</filePattern>
             </hudson.plugins.parameterizedtrigger.FileBuildParameterFactory>
           </configFactories>
           <projects>lander_image_builder</projects>
@@ -203,10 +209,10 @@
 /srv/lander_jenkins/lander/bin/lander_merge_parameters.py --result-file params.json --service image_builder --output-file all.json --prior-file all.json
 
 # Convert to a format that can be passed to the child jobs
-all_params=`cat all.json`
-cat > child.json <<EOF
-request_parameters=$all_params
-EOF]]></command>
+cat > child.params <<EOF
+request_parameters=$(cat all.json)
+EOF
+      ]]></command>
     </hudson.tasks.Shell>
     <hudson.plugins.parameterizedtrigger.TriggerBuilder>
       <configs>
@@ -218,7 +224,7 @@
           </configs>
           <configFactories>
             <hudson.plugins.parameterizedtrigger.FileBuildParameterFactory>
-              <filePattern>child.json</filePattern>
+              <filePattern>child.params</filePattern>
             </hudson.plugins.parameterizedtrigger.FileBuildParameterFactory>
           </configFactories>
           <projects>lander_test_runner</projects>
@@ -256,10 +262,10 @@
 /srv/lander_jenkins/lander/bin/lander_merge_parameters.py --result-file params.json --service test_runner --output-file all.json --prior-file all.json
 
 # Convert to a format that can be passed to the child jobs
-all_params=`cat all.json`
-cat > child.json <<EOF
-request_parameters=$all_params
-EOF]]></command>
+cat > child.params <<EOF
+request_parameters=$(cat all.json)
+EOF
+      ]]></command>
     </hudson.tasks.Shell>
     <hudson.plugins.parameterizedtrigger.TriggerBuilder>
       <configs>
@@ -271,7 +277,7 @@
           </configs>
           <configFactories>
             <hudson.plugins.parameterizedtrigger.FileBuildParameterFactory>
-              <filePattern>child.json</filePattern>
+              <filePattern>child.params</filePattern>
             </hudson.plugins.parameterizedtrigger.FileBuildParameterFactory>
           </configFactories>
           <projects>lander_publisher</projects>
@@ -309,10 +315,10 @@
 /srv/lander_jenkins/lander/bin/lander_merge_parameters.py --result-file params.json --service publisher --output-file all.json --prior-file all.json
 
 # Convert to a format that can be passed to the child jobs
-all_params=`cat all.json`
-cat > child.json <<EOF
-request_parameters=$all_params
-EOF]]></command>
+cat > child.params <<EOF
+request_parameters=$(cat all.json)
+EOF
+      ]]></command>
     </hudson.tasks.Shell>
   </builders>
   <publishers>