Merge lp:~doanac/charms/precise/ubuntu-ci-services-itself/lander into lp:~canonical-ci-engineering/charms/precise/ubuntu-ci-services-itself/lander
- Precise Pangolin (12.04)
- lander
- Merge into lander
Status: | Merged |
---|---|
Approved by: | Francis Ginther |
Approved revision: | 27 |
Merged at revision: | 23 |
Proposed branch: | lp:~doanac/charms/precise/ubuntu-ci-services-itself/lander |
Merge into: | lp:~canonical-ci-engineering/charms/precise/ubuntu-ci-services-itself/lander |
Diff against target: |
283 lines (+86/-35) 4 files modified
config.yaml (+10/-0) hooks/hooks.py (+38/-5) metadata.yaml (+2/-0) templates/jobs/lander_master.xml (+36/-30) |
To merge this branch: | bzr merge lp:~doanac/charms/precise/ubuntu-ci-services-itself/lander |
Related bugs: |
Reviewer | Review Type | Date Requested | Status |
---|---|---|---|
Francis Ginther | Approve | ||
Andy Doan (community) | Needs Resubmitting | ||
Review via email: mp+200449@code.launchpad.net |
Commit message
Description of the change
This updates the master and ppa-assigner jobs to handle parameters properly. They now are able to determine things like the ppa-assigner's host ip via juju relation ships. It might be easiest to read this commit by commit.
Andy Doan (doanac) wrote : | # |
Francis Ginther (fginther) wrote : | # |
The jenkins jobs are supposed to pass the params.json file from one job to the next, it's not intended to be created as is being done in templates/
Andy Doan (doanac) wrote : | # |
On 01/06/2014 01:32 PM, Francis Ginther wrote:
> The jenkins jobs are supposed to pass the params.json file from one job to the next, it's not intended to be created as is being done in templates/
I think I've got this fixed in a branch based on this:
<http://
if that looks sane, we can just fix it in that MP if that's okay?
- 22. By Francis Ginther
-
Fix jenkins jobs so that the accumulated json data is passed as a job parameter, update lander_archiver to work with updated script, add datastore config parameters.
- 23. By Andy Doan
-
store relationship information
the jenkins jobs will need to know the hostname of each service
connected to it like the ppa-assigner. This stores them under
a fix directory jobs can load dynamically - 24. By Andy Doan
-
add ppa-assigner url to config
- 25. By Andy Doan
-
allow charm to interface with amqp
- 26. By Andy Doan
-
add bsbuilder config to master
- 27. By Andy Doan
-
clarify parameter passing and add progress trigger
parameter passing was being done via a file named child.json. However,
this is really a jenkins parameters file and not really json. This
renames that file and creates it with a few less lines of code.
Andy Doan (doanac) wrote : | # |
I've rebased on your branch with a couple of minor updates. I changed your naming of "child.json" to "child.params" since it isn't really a json file. I then added the relationship handling logic to get the ppa-assigner and bsbuilder working.
Francis Ginther (fginther) wrote : | # |
Thanks for cleaning up my child.json. I got the first one updated in lander_master, but failed to update the rest. The params file works just as well.
Francis Ginther (fginther) wrote : | # |
I'm ok with the relation hooks, if we need to separate them later, we'll do it then.
Preview Diff
1 | === modified file 'config.yaml' | |||
2 | --- config.yaml 2014-01-08 15:04:30 +0000 | |||
3 | +++ config.yaml 2014-01-08 21:09:33 +0000 | |||
4 | @@ -51,3 +51,13 @@ | |||
5 | 51 | type: string | 51 | type: string |
6 | 52 | default: "" | 52 | default: "" |
7 | 53 | description: Data store tenant name. | 53 | description: Data store tenant name. |
8 | 54 | |||
9 | 55 | # required for rabbitmq-server charm: | ||
10 | 56 | amqp-user: | ||
11 | 57 | type: string | ||
12 | 58 | default: workerbee | ||
13 | 59 | description: The user to log into the rabbitMQ server. | ||
14 | 60 | amqp-vhost: | ||
15 | 61 | type: string | ||
16 | 62 | default: '/' | ||
17 | 63 | description: The vhost in the rabbitMQ server. | ||
18 | 54 | 64 | ||
19 | === added symlink 'hooks/amqp-relation-broken' | |||
20 | === target is u'hooks.py' | |||
21 | === added symlink 'hooks/amqp-relation-changed' | |||
22 | === target is u'hooks.py' | |||
23 | === added symlink 'hooks/amqp-relation-joined' | |||
24 | === target is u'hooks.py' | |||
25 | === modified file 'hooks/hooks.py' | |||
26 | --- hooks/hooks.py 2013-12-19 22:20:21 +0000 | |||
27 | +++ hooks/hooks.py 2014-01-08 21:09:33 +0000 | |||
28 | @@ -4,12 +4,10 @@ | |||
29 | 4 | import json | 4 | import json |
30 | 5 | import subprocess | 5 | import subprocess |
31 | 6 | import sys | 6 | import sys |
32 | 7 | import time | ||
33 | 8 | 7 | ||
34 | 9 | 8 | ||
35 | 10 | def juju_info(msg): | 9 | def juju_info(msg): |
36 | 11 | subprocess.check_call(['juju-log', '-l', 'INFO', msg]) | 10 | subprocess.check_call(['juju-log', '-l', 'INFO', msg]) |
37 | 12 | pass | ||
38 | 13 | 11 | ||
39 | 14 | 12 | ||
40 | 15 | def _config(): | 13 | def _config(): |
41 | @@ -51,6 +49,11 @@ | |||
42 | 51 | return os.path.join(config['install_root'], unit) | 49 | return os.path.join(config['install_root'], unit) |
43 | 52 | 50 | ||
44 | 53 | 51 | ||
45 | 52 | def _relation_file(config): | ||
46 | 53 | relation = os.environ['JUJU_RELATION_ID'] | ||
47 | 54 | return os.path.join(_service_dir(config), 'juju-relations', relation) | ||
48 | 55 | |||
49 | 56 | |||
50 | 54 | def lander_jenkins_relation_joined(config): | 57 | def lander_jenkins_relation_joined(config): |
51 | 55 | juju_info("Config: %s" % config) | 58 | juju_info("Config: %s" % config) |
52 | 56 | hostname = subprocess.check_output(['unit-get', 'private-address']).strip() | 59 | hostname = subprocess.check_output(['unit-get', 'private-address']).strip() |
53 | @@ -61,13 +64,24 @@ | |||
54 | 61 | 'port': 8080, # the jenkins charm hardcodes 8080 | 64 | 'port': 8080, # the jenkins charm hardcodes 8080 |
55 | 62 | }) | 65 | }) |
56 | 63 | 66 | ||
57 | 67 | rfile = _relation_file(config) | ||
58 | 68 | rdir = os.path.dirname(rfile) | ||
59 | 69 | if not os.path.exists(rdir): | ||
60 | 70 | os.mkdir(rdir) | ||
61 | 71 | with open(rfile, 'w') as f: | ||
62 | 72 | unit = os.environ['JUJU_REMOTE_UNIT'].split('/')[0] | ||
63 | 73 | f.write('%s: %s' % (unit, _relation_get('private-address'))) | ||
64 | 74 | |||
65 | 64 | 75 | ||
66 | 65 | def lander_jenkins_relation_changed(config): | 76 | def lander_jenkins_relation_changed(config): |
68 | 66 | pass | 77 | lander_jenkins_relation_joined(config) |
69 | 67 | 78 | ||
70 | 68 | 79 | ||
71 | 69 | def lander_jenkins_relation_broken(config): | 80 | def lander_jenkins_relation_broken(config): |
73 | 70 | pass | 81 | rfile = _relation_file(config) |
74 | 82 | juju_info('removing relation file: %s' % rfile) | ||
75 | 83 | if os.path.exists(rfile): | ||
76 | 84 | os.unlink(rfile) | ||
77 | 71 | 85 | ||
78 | 72 | 86 | ||
79 | 73 | def jenkins_extension_relation_joined(config): | 87 | def jenkins_extension_relation_joined(config): |
80 | @@ -85,6 +99,26 @@ | |||
81 | 85 | juju_info("Config: %s" % config) | 99 | juju_info("Config: %s" % config) |
82 | 86 | 100 | ||
83 | 87 | 101 | ||
84 | 102 | def amqp_relation_joined(config): | ||
85 | 103 | _relation_set({ | ||
86 | 104 | 'username': config['amqp-user'], | ||
87 | 105 | 'vhost': config['amqp-vhost'], | ||
88 | 106 | }) | ||
89 | 107 | |||
90 | 108 | |||
91 | 109 | def amqp_relation_changed(config): | ||
92 | 110 | with open(os.path.join(_service_dir(config), 'amqp_config.py'), 'w') as f: | ||
93 | 111 | f.write('# DO NOT EDIT. Generated by lander-jenkins charm hook\n') | ||
94 | 112 | f.write('AMQP_USER = "%s"\n' % config['amqp-user']) | ||
95 | 113 | f.write('AMQP_VHOST = "%s"\n' % config['amqp-vhost']) | ||
96 | 114 | f.write('AMQP_HOST = "%s"\n' % _relation_get('private-address')) | ||
97 | 115 | f.write('AMQP_PASSWORD = "%s"\n' % _relation_get('password')) | ||
98 | 116 | |||
99 | 117 | |||
100 | 118 | def amqp_relation_broken(config): | ||
101 | 119 | os.unlink(os.path.join(_service_dir(config), 'amqp_config.py')) | ||
102 | 120 | |||
103 | 121 | |||
104 | 88 | def main(): | 122 | def main(): |
105 | 89 | hook = os.path.basename(sys.argv[0]) | 123 | hook = os.path.basename(sys.argv[0]) |
106 | 90 | juju_info("Running hook: %s" % hook) | 124 | juju_info("Running hook: %s" % hook) |
107 | @@ -105,4 +139,3 @@ | |||
108 | 105 | 139 | ||
109 | 106 | if __name__ == '__main__': | 140 | if __name__ == '__main__': |
110 | 107 | exit(main()) | 141 | exit(main()) |
111 | 108 | |||
112 | 109 | 142 | ||
113 | === modified file 'metadata.yaml' | |||
114 | --- metadata.yaml 2013-12-19 22:20:21 +0000 | |||
115 | +++ metadata.yaml 2014-01-08 21:09:33 +0000 | |||
116 | @@ -9,6 +9,8 @@ | |||
117 | 9 | extension: | 9 | extension: |
118 | 10 | interface: jenkins-extension | 10 | interface: jenkins-extension |
119 | 11 | scope: container | 11 | scope: container |
120 | 12 | amqp: | ||
121 | 13 | interface: rabbitmq | ||
122 | 12 | provides: | 14 | provides: |
123 | 13 | lander-jenkins: | 15 | lander-jenkins: |
124 | 14 | interface: lander-jenkins | 16 | interface: lander-jenkins |
125 | 15 | 17 | ||
126 | === modified file 'templates/jobs/lander_master.xml' | |||
127 | --- templates/jobs/lander_master.xml 2014-01-08 15:04:30 +0000 | |||
128 | +++ templates/jobs/lander_master.xml 2014-01-08 21:09:33 +0000 | |||
129 | @@ -31,23 +31,29 @@ | |||
130 | 31 | <command><![CDATA[#!/bin/bash | 31 | <command><![CDATA[#!/bin/bash |
131 | 32 | set -ex | 32 | set -ex |
132 | 33 | 33 | ||
133 | 34 | ppa_ip=$(grep ppa-django /srv/lander_jenkins/juju-relations/* | awk '{print $NF}' | head -n1) | ||
134 | 35 | bsb_ip=$(grep bsb-restish /srv/lander_jenkins/juju-relations/* | awk '{print $NF}' | head -n1) | ||
135 | 36 | |||
136 | 34 | # Convert to input file | 37 | # Convert to input file |
137 | 35 | request_parameters=$(echo $request_parameters | sed -e 's/^"//' -e 's/"$//') | 38 | request_parameters=$(echo $request_parameters | sed -e 's/^"//' -e 's/"$//') |
138 | 36 | cat > params.json <<EOF | 39 | cat > params.json <<EOF |
139 | 37 | { | 40 | { |
140 | 38 | "request_id": "$request_id", | 41 | "request_id": "$request_id", |
141 | 42 | "progress_trigger": "${JOB_NAME}-${BUILD_NUMBER}", | ||
142 | 43 | "ppa_assigner_url": "http://${ppa_ip}:8080", | ||
143 | 44 | "bsb_url": "http://${bsb_ip}:8080", | ||
144 | 39 | "request_parameters": $request_parameters | 45 | "request_parameters": $request_parameters |
145 | 40 | } | 46 | } |
146 | 41 | EOF | 47 | EOF |
147 | 42 | cat params.json | 48 | cat params.json |
148 | 43 | 49 | ||
149 | 44 | /srv/lander_jenkins/lander/bin/lander_merge_parameters.py --result-file params.json --service master --output-file all.json | 50 | /srv/lander_jenkins/lander/bin/lander_merge_parameters.py --result-file params.json --service master --output-file all.json |
151 | 45 | request_parameters=$(echo $request_parameters | sed -e 's/^"//' -e 's/"$//') | 51 | |
152 | 46 | # Convert to a format that can be passed to the child jobs | 52 | # Convert to a format that can be passed to the child jobs |
157 | 47 | all_params=`cat all.json` | 53 | cat > child.params <<EOF |
158 | 48 | cat > child.json <<EOF | 54 | request_parameters=$(cat all.json) |
159 | 49 | "request_parameters": $all_params | 55 | EOF |
160 | 50 | EOF]]></command> | 56 | ]]></command> |
161 | 51 | </hudson.tasks.Shell> | 57 | </hudson.tasks.Shell> |
162 | 52 | <hudson.plugins.parameterizedtrigger.TriggerBuilder> | 58 | <hudson.plugins.parameterizedtrigger.TriggerBuilder> |
163 | 53 | <configs> | 59 | <configs> |
164 | @@ -59,7 +65,7 @@ | |||
165 | 59 | </configs> | 65 | </configs> |
166 | 60 | <configFactories> | 66 | <configFactories> |
167 | 61 | <hudson.plugins.parameterizedtrigger.FileBuildParameterFactory> | 67 | <hudson.plugins.parameterizedtrigger.FileBuildParameterFactory> |
169 | 62 | <filePattern>child.json</filePattern> | 68 | <filePattern>child.params</filePattern> |
170 | 63 | </hudson.plugins.parameterizedtrigger.FileBuildParameterFactory> | 69 | </hudson.plugins.parameterizedtrigger.FileBuildParameterFactory> |
171 | 64 | </configFactories> | 70 | </configFactories> |
172 | 65 | <projects>lander_ppa_assigner</projects> | 71 | <projects>lander_ppa_assigner</projects> |
173 | @@ -97,10 +103,10 @@ | |||
174 | 97 | /srv/lander_jenkins/lander/bin/lander_merge_parameters.py --result-file params.json --service ppa_assigner --output-file all.json --prior-file all.json | 103 | /srv/lander_jenkins/lander/bin/lander_merge_parameters.py --result-file params.json --service ppa_assigner --output-file all.json --prior-file all.json |
175 | 98 | 104 | ||
176 | 99 | # Convert to a format that can be passed to the child jobs | 105 | # Convert to a format that can be passed to the child jobs |
181 | 100 | all_params=`cat all.json` | 106 | cat > child.params <<EOF |
182 | 101 | cat > child.json <<EOF | 107 | request_parameters=$(cat all.json) |
183 | 102 | request_parameters=$all_params | 108 | EOF |
184 | 103 | EOF]]></command> | 109 | ]]></command> |
185 | 104 | </hudson.tasks.Shell> | 110 | </hudson.tasks.Shell> |
186 | 105 | <hudson.plugins.parameterizedtrigger.TriggerBuilder> | 111 | <hudson.plugins.parameterizedtrigger.TriggerBuilder> |
187 | 106 | <configs> | 112 | <configs> |
188 | @@ -112,7 +118,7 @@ | |||
189 | 112 | </configs> | 118 | </configs> |
190 | 113 | <configFactories> | 119 | <configFactories> |
191 | 114 | <hudson.plugins.parameterizedtrigger.FileBuildParameterFactory> | 120 | <hudson.plugins.parameterizedtrigger.FileBuildParameterFactory> |
193 | 115 | <filePattern>child.json</filePattern> | 121 | <filePattern>child.params</filePattern> |
194 | 116 | </hudson.plugins.parameterizedtrigger.FileBuildParameterFactory> | 122 | </hudson.plugins.parameterizedtrigger.FileBuildParameterFactory> |
195 | 117 | </configFactories> | 123 | </configFactories> |
196 | 118 | <projects>lander_branch_source_builder</projects> | 124 | <projects>lander_branch_source_builder</projects> |
197 | @@ -150,10 +156,10 @@ | |||
198 | 150 | /srv/lander_jenkins/lander/bin/lander_merge_parameters.py --result-file params.json --service bsbuilder --output-file all.json --prior-file all.json | 156 | /srv/lander_jenkins/lander/bin/lander_merge_parameters.py --result-file params.json --service bsbuilder --output-file all.json --prior-file all.json |
199 | 151 | 157 | ||
200 | 152 | # Convert to a format that can be passed to the child jobs | 158 | # Convert to a format that can be passed to the child jobs |
205 | 153 | all_params=`cat all.json` | 159 | cat > child.params <<EOF |
206 | 154 | cat > child.json <<EOF | 160 | request_parameters=$(cat all.json) |
207 | 155 | request_parameters=$all_params | 161 | EOF |
208 | 156 | EOF]]></command> | 162 | ]]></command> |
209 | 157 | </hudson.tasks.Shell> | 163 | </hudson.tasks.Shell> |
210 | 158 | <hudson.plugins.parameterizedtrigger.TriggerBuilder> | 164 | <hudson.plugins.parameterizedtrigger.TriggerBuilder> |
211 | 159 | <configs> | 165 | <configs> |
212 | @@ -165,7 +171,7 @@ | |||
213 | 165 | </configs> | 171 | </configs> |
214 | 166 | <configFactories> | 172 | <configFactories> |
215 | 167 | <hudson.plugins.parameterizedtrigger.FileBuildParameterFactory> | 173 | <hudson.plugins.parameterizedtrigger.FileBuildParameterFactory> |
217 | 168 | <filePattern>child.json</filePattern> | 174 | <filePattern>child.params</filePattern> |
218 | 169 | </hudson.plugins.parameterizedtrigger.FileBuildParameterFactory> | 175 | </hudson.plugins.parameterizedtrigger.FileBuildParameterFactory> |
219 | 170 | </configFactories> | 176 | </configFactories> |
220 | 171 | <projects>lander_image_builder</projects> | 177 | <projects>lander_image_builder</projects> |
221 | @@ -203,10 +209,10 @@ | |||
222 | 203 | /srv/lander_jenkins/lander/bin/lander_merge_parameters.py --result-file params.json --service image_builder --output-file all.json --prior-file all.json | 209 | /srv/lander_jenkins/lander/bin/lander_merge_parameters.py --result-file params.json --service image_builder --output-file all.json --prior-file all.json |
223 | 204 | 210 | ||
224 | 205 | # Convert to a format that can be passed to the child jobs | 211 | # Convert to a format that can be passed to the child jobs |
229 | 206 | all_params=`cat all.json` | 212 | cat > child.params <<EOF |
230 | 207 | cat > child.json <<EOF | 213 | request_parameters=$(cat all.json) |
231 | 208 | request_parameters=$all_params | 214 | EOF |
232 | 209 | EOF]]></command> | 215 | ]]></command> |
233 | 210 | </hudson.tasks.Shell> | 216 | </hudson.tasks.Shell> |
234 | 211 | <hudson.plugins.parameterizedtrigger.TriggerBuilder> | 217 | <hudson.plugins.parameterizedtrigger.TriggerBuilder> |
235 | 212 | <configs> | 218 | <configs> |
236 | @@ -218,7 +224,7 @@ | |||
237 | 218 | </configs> | 224 | </configs> |
238 | 219 | <configFactories> | 225 | <configFactories> |
239 | 220 | <hudson.plugins.parameterizedtrigger.FileBuildParameterFactory> | 226 | <hudson.plugins.parameterizedtrigger.FileBuildParameterFactory> |
241 | 221 | <filePattern>child.json</filePattern> | 227 | <filePattern>child.params</filePattern> |
242 | 222 | </hudson.plugins.parameterizedtrigger.FileBuildParameterFactory> | 228 | </hudson.plugins.parameterizedtrigger.FileBuildParameterFactory> |
243 | 223 | </configFactories> | 229 | </configFactories> |
244 | 224 | <projects>lander_test_runner</projects> | 230 | <projects>lander_test_runner</projects> |
245 | @@ -256,10 +262,10 @@ | |||
246 | 256 | /srv/lander_jenkins/lander/bin/lander_merge_parameters.py --result-file params.json --service test_runner --output-file all.json --prior-file all.json | 262 | /srv/lander_jenkins/lander/bin/lander_merge_parameters.py --result-file params.json --service test_runner --output-file all.json --prior-file all.json |
247 | 257 | 263 | ||
248 | 258 | # Convert to a format that can be passed to the child jobs | 264 | # Convert to a format that can be passed to the child jobs |
253 | 259 | all_params=`cat all.json` | 265 | cat > child.params <<EOF |
254 | 260 | cat > child.json <<EOF | 266 | request_parameters=$(cat all.json) |
255 | 261 | request_parameters=$all_params | 267 | EOF |
256 | 262 | EOF]]></command> | 268 | ]]></command> |
257 | 263 | </hudson.tasks.Shell> | 269 | </hudson.tasks.Shell> |
258 | 264 | <hudson.plugins.parameterizedtrigger.TriggerBuilder> | 270 | <hudson.plugins.parameterizedtrigger.TriggerBuilder> |
259 | 265 | <configs> | 271 | <configs> |
260 | @@ -271,7 +277,7 @@ | |||
261 | 271 | </configs> | 277 | </configs> |
262 | 272 | <configFactories> | 278 | <configFactories> |
263 | 273 | <hudson.plugins.parameterizedtrigger.FileBuildParameterFactory> | 279 | <hudson.plugins.parameterizedtrigger.FileBuildParameterFactory> |
265 | 274 | <filePattern>child.json</filePattern> | 280 | <filePattern>child.params</filePattern> |
266 | 275 | </hudson.plugins.parameterizedtrigger.FileBuildParameterFactory> | 281 | </hudson.plugins.parameterizedtrigger.FileBuildParameterFactory> |
267 | 276 | </configFactories> | 282 | </configFactories> |
268 | 277 | <projects>lander_publisher</projects> | 283 | <projects>lander_publisher</projects> |
269 | @@ -309,10 +315,10 @@ | |||
270 | 309 | /srv/lander_jenkins/lander/bin/lander_merge_parameters.py --result-file params.json --service publisher --output-file all.json --prior-file all.json | 315 | /srv/lander_jenkins/lander/bin/lander_merge_parameters.py --result-file params.json --service publisher --output-file all.json --prior-file all.json |
271 | 310 | 316 | ||
272 | 311 | # Convert to a format that can be passed to the child jobs | 317 | # Convert to a format that can be passed to the child jobs |
277 | 312 | all_params=`cat all.json` | 318 | cat > child.params <<EOF |
278 | 313 | cat > child.json <<EOF | 319 | request_parameters=$(cat all.json) |
279 | 314 | request_parameters=$all_params | 320 | EOF |
280 | 315 | EOF]]></command> | 321 | ]]></command> |
281 | 316 | </hudson.tasks.Shell> | 322 | </hudson.tasks.Shell> |
282 | 317 | </builders> | 323 | </builders> |
283 | 318 | <publishers> | 324 | <publishers> |
FYI - I have a private juju deployer file that can do a full deployment of the ppa-assigner and lander. Let me know if you want a copy (it includes oauth tokens so i can't share publically).