Merge lp:~psivaa/uci-engine/lander-jenkins-sub-charm-with-plugin into lp:uci-engine/mthood

Proposed by Para Siva
Status: Rejected
Rejected by: Para Siva
Proposed branch: lp:~psivaa/uci-engine/lander-jenkins-sub-charm-with-plugin
Merge into: lp:uci-engine/mthood
Diff against target: 2276 lines (+611/-298)
70 files modified
TRICKS (+5/-9)
branch-source-builder/bsbuilder/__init__.py (+1/-1)
branch-source-builder/bsbuilder/resources/v1.py (+2/-3)
branch-source-builder/bsbuilder/run_worker.py (+1/-1)
branch-source-builder/bsbuilder/tests/test_upload.py (+1/-1)
charms/precise/lander-jenkins/hooks/hooks.py (+61/-34)
charms/precise/python-django/hooks/hooks.py (+12/-7)
charms/precise/rabbitmq-worker/hooks/hooks.py (+38/-15)
charms/precise/restish/hooks/hooks.py (+24/-10)
charms/precise/webui/hooks/hooks.py (+9/-0)
ci-utils/ci_utils/__init__.py (+1/-1)
ci-utils/ci_utils/amqp_utils.py (+13/-1)
ci-utils/ci_utils/create_db.py (+1/-1)
ci-utils/ci_utils/restish_utils.py (+1/-1)
ci-utils/ci_utils/setup_helper.py (+1/-1)
ci-utils/ci_utils/tastypie/resource.py (+1/-1)
ci-utils/ci_utils/tastypie/test.py (+1/-1)
ci-utils/ci_utils/tests/test_amqp.py (+8/-1)
ci-utils/ci_utils/tests/test_restish.py (+1/-1)
ci-utils/ci_utils/ticket_states.py (+1/-1)
cli/ci_libs/__init__.py (+1/-1)
cli/tests/test_file_handler.py (+1/-1)
debian/control (+1/-1)
debian/copyright (+1/-1)
image-builder/imagebuilder/__init__.py (+1/-1)
image-builder/imagebuilder/cloud_image.py (+28/-2)
image-builder/imagebuilder/resources/v1.py (+2/-3)
image-builder/imagebuilder/run_worker.py (+1/-1)
image-builder/imagebuilder/tests/test_api_v1.py (+1/-1)
image-builder/imagebuilder/tests/test_modify_cloud_image.py (+32/-1)
juju-deployer/branch-source-builder.yaml.tmpl (+2/-2)
juju-deployer/configs/unit_config.yaml.tmpl (+2/-0)
juju-deployer/deploy.py (+81/-15)
juju-deployer/image-builder.yaml.tmpl (+2/-2)
juju-deployer/lander.yaml.tmpl (+4/-4)
juju-deployer/ppa-assigner.yaml.tmpl (+1/-1)
juju-deployer/production-only.yaml (+1/-1)
juju-deployer/test-runner.yaml.tmpl (+2/-3)
juju-deployer/test_deploy.py (+1/-0)
juju-deployer/ticket-system.yaml.tmpl (+1/-1)
lander/bin/lander_merge_parameters.py (+1/-1)
lander/bin/lander_service_wrapper.py (+7/-14)
lander/bin/ticket_api.py (+1/-1)
lander/lander/__init__.py (+1/-1)
lander/lander/resources/v1.py (+1/-1)
lander/lander/tests/test_service_wrapper.py (+23/-17)
lander/lander/tests/test_v1.py (+1/-1)
ppa-assigner/local_settings.py.example (+1/-1)
ppa-assigner/ppa_assigner/__init__.py (+1/-1)
ppa-assigner/ppa_assigner/launchpad.py (+1/-1)
run-tests (+1/-1)
tarmac.sh (+1/-1)
test_runner/run_test.py (+19/-13)
test_runner/tstrun/__init__.py (+0/-6)
test_runner/tstrun/resources/root.py (+1/-1)
test_runner/tstrun/resources/v1.py (+2/-4)
test_runner/tstrun/run_worker.py (+65/-43)
test_runner/tstrun/testbed.py (+31/-14)
test_runner/tstrun/wsgi.py (+1/-1)
tests/image_builder/test.py (+9/-2)
tests/run.py (+65/-34)
tests/test_run.py (+21/-0)
tests/test_runner/test.py (+1/-0)
tests/ticket_system/test.py (+1/-0)
ticket_system/project/admin.py (+1/-1)
ticket_system/project/api.py (+1/-1)
ticket_system/project/models.py (+1/-1)
ticket_system/ticket/admin.py (+1/-1)
ticket_system/ticket/tests/test_write_api.py (+1/-1)
ticket_system/ticket_system/__init__.py (+1/-1)
To merge this branch: bzr merge lp:~psivaa/uci-engine/lander-jenkins-sub-charm-with-plugin
Reviewer Review Type Date Requested Status
Canonical CI Engineering Pending
Review via email: mp+219013@code.launchpad.net

Commit message

Change to include parameterized-trigger plugin inside the charm instead of downloading it from the jenkins site to be made suitable for private mthood cloud

Note: For this to work well https://code.launchpad.net/~psivaa/charms/precise/jenkins/bundle-jenkins/+merge/215635 needs merging first

Description of the change

Change to include parameterized-trigger plugin inside the charm instead of downloading it from the jenkins site to be made suitable for private mthood cloud.

Proposing against the right branch now.

To post a comment you must log in.
Revision history for this message
Para Siva (psivaa) wrote :

Wrong source branch

Unmerged revisions

431. By Para Siva

Include plugins in lander-jenkins-sub charm pep8 fixes

430. By Para Siva

Include plugins in lander-jenkins-sub charm

429. By Andy Doan

[r=Evan Dandrea, PS Jenkins bot] lander: convert to using rabbit queues directly instead of the REST API. from Andy Doan

428. By Vincent Ladeuil

[r=Evan Dandrea, PS Jenkins bot] Update copyrights. from Vincent Ladeuil

427. By Andy Doan

[r=PS Jenkins bot, Evan Dandrea] use common set of constants for queue names

The lander is about to need these, so lets keep them in a single
consistent place.
  from Andy Doan

426. By Vincent Ladeuil

[r=PS Jenkins bot, Chris Johnston] Any test failure should make the run fail. from Vincent Ladeuil

425. By Vincent Ladeuil

[r=Evan Dandrea, PS Jenkins bot] Remove the need for OS_REQUIRES_TUNNEL by checking if the bootstrap IP address is private before starting a tunnel. 1301502 from Vincent Ladeuil

424. By Chris Johnston

[r=Andy Doan, PS Jenkins bot] Fix juju-deployer not being on sys.path from Chris Johnston

423. By Andy Doan

[r=Vincent Ladeuil, PS Jenkins bot] Add the ability to upgrade an existing deployment using deploy.py. A new option was added:

 ./juju-deployer/deploy.py --upgrade

This code path will:

1) ensure the deployment has been completed (classic path through our code)
2) all options defined in our deployer yaml files are updated (in case a setting was changed)
3) call upgrade-charm on every deployed service

upgrade-charm runs async and the script currently just exits, so you must run juju-status a few times afterwards to ensure your upgrade worked as expected. from Andy Doan

422. By Vincent Ladeuil

[r=PS Jenkins bot, Evan Dandrea] Fix image builder and test runner test failures. 1292159 from Vincent Ladeuil

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1=== modified file 'TRICKS'
2--- TRICKS 2014-03-14 10:49:19 +0000
3+++ TRICKS 2014-05-09 16:00:26 +0000
4@@ -58,12 +58,6 @@
5 empty.
6
7
8-amulet thinks it has charms in the current directory
9-====================================================
10-
11-We need:
12-vila:~/ci/ubuntu-ci-services-itself/trunk :) $ ln -s precise/python-django django
13-
14 OMG ! I can't connect with ssh to my nova fresh instances !!!
15 =============================================================
16
17@@ -168,7 +162,7 @@
18 Airline.
19
20 export CI_PPA=ppa:canonical-ci-engineering/ci-airline-phase-0
21-export CI_BRANCH=lp:ubuntu-ci-services-itself
22+export CI_BRANCH=lp:uci-engine
23 export CI_CODE_SOURCE=branch
24
25 cheetah fill --env juju-deployer/configs/unit_config.yaml.tmpl -p > juju-deployer/configs/unit_config.yaml
26@@ -294,6 +288,8 @@
27 - merge branches into the payload,
28 - restart the service.
29
30+NOTE: Since we use the test runner has an example, some paths need to be
31+adjusted for other components. Like-wise, some command outputs may vary.
32
33 Turning the payload into a branch
34 ---------------------------------
35@@ -309,7 +305,7 @@
36 # bzr whoami "bugfixer <bugfixer@example.com>"
37 # cd /srv
38 # bzr init-repo .
39-# bzr branch lp:ubuntu-ci-services-itself trunk
40+# bzr branch lp:uci-engine trunk
41
42 Really turn the payload into a branch:
43
44@@ -335,7 +331,7 @@
45 ---------------------------
46
47 $ cd /srv
48-$ bzr branch lp:~vila/ubuntu-ci-services-itself/hotfix
49+$ bzr branch lp:~vila/uci-engine/hotfix
50
51 Merge branches into the payload
52 -------------------------------
53
54=== modified file 'branch-source-builder/bsbuilder/__init__.py'
55--- branch-source-builder/bsbuilder/__init__.py 2014-03-10 22:25:00 +0000
56+++ branch-source-builder/bsbuilder/__init__.py 2014-05-09 16:00:26 +0000
57@@ -1,5 +1,5 @@
58 # Ubuntu CI Engine
59-# Copyright 2013 Canonical Ltd.
60+# Copyright 2013, 2014 Canonical Ltd.
61
62 # This program is free software: you can redistribute it and/or modify it
63 # under the terms of the GNU Affero General Public License version 3, as
64
65=== modified file 'branch-source-builder/bsbuilder/resources/v1.py'
66--- branch-source-builder/bsbuilder/resources/v1.py 2014-03-14 10:37:28 +0000
67+++ branch-source-builder/bsbuilder/resources/v1.py 2014-05-09 16:00:26 +0000
68@@ -21,13 +21,12 @@
69 from ci_utils import amqp_utils, json_status, restish_utils
70
71 log = logging.getLogger(__name__)
72-WORKER_QUEUE = 'bsbuilder'
73
74
75 def _status():
76 status = json_status.JSONStatus()
77 status.add_rabbit_configured()
78- status.add_rabbit_worker_health(WORKER_QUEUE)
79+ status.add_rabbit_worker_health(amqp_utils.BSBUILDER_QUEUE)
80 return restish_utils.json_ok(status.results)
81
82
83@@ -48,7 +47,7 @@
84 log.error('Unable to notify progress trigger, aborting build_source')
85 return http.service_unavailable(body=r)
86
87- r = amqp_utils.send(WORKER_QUEUE, json.dumps(params))
88+ r = amqp_utils.send(amqp_utils.BSBUILDER_QUEUE, json.dumps(params))
89 if r:
90 # send only returns something if it an error message
91 r = http.service_unavailable(body=r)
92
93=== modified file 'branch-source-builder/bsbuilder/run_worker.py'
94--- branch-source-builder/bsbuilder/run_worker.py 2014-03-14 10:37:28 +0000
95+++ branch-source-builder/bsbuilder/run_worker.py 2014-05-09 16:00:26 +0000
96@@ -154,4 +154,4 @@
97
98
99 if __name__ == '__main__':
100- BSBuilderWorker().main('bsbuilder')
101+ BSBuilderWorker().main(amqp_utils.BSBUILDER_QUEUE)
102
103=== modified file 'branch-source-builder/bsbuilder/tests/test_upload.py'
104--- branch-source-builder/bsbuilder/tests/test_upload.py 2014-03-16 18:58:01 +0000
105+++ branch-source-builder/bsbuilder/tests/test_upload.py 2014-05-09 16:00:26 +0000
106@@ -1,5 +1,5 @@
107 # Ubuntu CI Engine
108-# Copyright 2013 Canonical Ltd.
109+# Copyright 2014 Canonical Ltd.
110
111 # This program is free software: you can redistribute it and/or modify it
112 # under the terms of the GNU Affero General Public License version 3, as
113
114=== added directory 'charms/precise/lander-jenkins/files'
115=== added directory 'charms/precise/lander-jenkins/files/plugins'
116=== added file 'charms/precise/lander-jenkins/files/plugins/parameterized-trigger.hpi'
117Binary files charms/precise/lander-jenkins/files/plugins/parameterized-trigger.hpi 1970-01-01 00:00:00 +0000 and charms/precise/lander-jenkins/files/plugins/parameterized-trigger.hpi 2014-05-09 16:00:26 +0000 differ
118=== modified file 'charms/precise/lander-jenkins/hooks/hooks.py' (properties changed: +x to -x)
119--- charms/precise/lander-jenkins/hooks/hooks.py 2014-03-10 22:25:00 +0000
120+++ charms/precise/lander-jenkins/hooks/hooks.py 2014-05-09 16:00:26 +0000
121@@ -14,12 +14,12 @@
122 sys.path.insert(0, os.path.join(os.environ['CHARM_DIR'], 'lib'))
123 from charmhelpers import core
124 from charmhelpers import fetch
125-from charmhelpers.fetch import bzrurl
126
127 JENKINS_HOME = '/var/lib/jenkins'
128 JENKINS_USER = 'jenkins'
129 JENKINS_GROUP = 'nogroup'
130 NUM_EXECUTORS_LINE = ' <numExecutors>{}</numExecutors>\n'
131+PLUGIN_LOCAL = os.path.join(os.environ['CHARM_DIR'], 'files/plugins')
132
133
134 def juju_info(msg, level='INFO'):
135@@ -149,49 +149,76 @@
136 plugin_list = config['plugins'].split()
137 plugins_site = config['plugins-site']
138 for plugin in plugin_list:
139- if ':' in plugin:
140- (name, version) = plugin.split(':')
141- # A specific version is being requested
142- url = '{}/download/plugins/{}/{}/{}.hpi'.format(
143- plugins_site, name, version, name)
144- plugin_file = '{}-{}.hpi'.format(name, version)
145- else:
146- url = '{}/latest/{}.hpi'.format(plugins_site, plugin)
147- plugin_file = '{}.hpi'.format(name)
148- juju_info('Installing {} from {}'.format(plugin_file, url))
149- plugin_file = os.path.join(JENKINS_HOME, 'plugins', plugin_file)
150- cmd = ['wget', '-O', plugin_file, url]
151- subprocess.check_call(cmd)
152+ plugin = '{}.hpi'.format(plugin)
153+ plugin_file = os.path.join(PLUGIN_LOCAL, plugin)
154+ if not os.path.exists(plugin_file):
155+ if ':' in plugin:
156+ (name, version) = plugin.split(':')
157+ # A specific version is being requested
158+ url = '{}/download/plugins/{}/{}/{}.hpi'.format(
159+ plugins_site, name, version, name)
160+ plugin_file = '{}-{}.hpi'.format(name, version)
161+ else:
162+ url = '{}/latest/{}.hpi'.format(plugins_site, plugin)
163+ plugin_file = plugin
164+ juju_info('Installing {} from {}'.format(plugin_file, url))
165+ plugin_file = os.path.join(JENKINS_HOME, 'plugins', plugin_file)
166+ cmd = ['wget', '-O', plugin_file, url]
167+ subprocess.check_call(cmd)
168+
169 cmd = ['chown', '-R', '{}.{}'.format(JENKINS_USER, JENKINS_GROUP),
170 plugin_file]
171 subprocess.check_call(cmd)
172
173
174-def _install_from_tarball(config):
175+def _install_from_tarball(config, upgrade):
176 juju_info('grabbing service from tarball...')
177 sdir = _service_dir(config)
178+ if not upgrade:
179+ if os.path.exists(sdir):
180+ juju_info('deleting pre-existing service directory: %s' % sdir)
181+ shutil.rmtree(sdir)
182+ os.mkdir(sdir)
183+ cmd = 'curl %s | tar -xzC %s' % (config['tarball'], sdir)
184+ subprocess.check_call(cmd, shell=True)
185+
186+
187+def _install_from_bzr(config, upgrade):
188+ juju_info('grabbing service from bzr...')
189+
190+ sdir = _service_dir(config)
191+ rev = config.get('revno', '')
192+
193+ if upgrade:
194+ args = ['bzr', 'pull']
195+ if rev:
196+ args.extend(['-r', rev])
197+ subprocess.check_call(args, cwd=sdir)
198+ return
199+
200 if os.path.exists(sdir):
201 juju_info('deleting pre-existing service directory: %s' % sdir)
202 shutil.rmtree(sdir)
203- os.mkdir(sdir)
204- cmd = 'curl %s | tar -xzC %s' % (config['tarball'], sdir)
205- subprocess.check_call(cmd, shell=True)
206-
207-
208-def _install_from_bzr(config):
209- service_dir = _service_dir(config)
210- bzr = bzrurl.BzrUrlFetchHandler()
211- if os.path.exists(service_dir):
212- juju_info('deleting pre-existing service directory: %s' % service_dir)
213- shutil.rmtree(service_dir)
214- bzr.branch(config['branch'], service_dir)
215-
216-
217-def install(config):
218+
219+ args = ['bzr', 'branch']
220+ if rev:
221+ args.extend(['-r', rev])
222+ args.append(config['branch'])
223+ args.append(sdir)
224+ subprocess.check_call(args)
225+
226+
227+def install(config, upgrade=False):
228 if config.get('vcs') == 'tarball':
229- _install_from_tarball(config)
230+ _install_from_tarball(config, upgrade)
231 else:
232- _install_from_bzr(config)
233+ _install_from_bzr(config, upgrade)
234+
235+
236+def upgrade_charm(config):
237+ install(config, True)
238+ restart_jenkins(config)
239+ restart_upstart(config)
240
241
242 def create_unit_config(config):
243@@ -210,9 +237,9 @@
244 start on (local-filesystems and net-device-up IFACE=eth0)
245 stop on runlevel [!12345]
246
247- # If the process quits unexpectadly trigger respawn it, unless it
248- # fails 15 times within 5 seconds
249+ # If the process quits unexpectedly trigger respawn it
250 respawn
251+ # unless it fails 15 times within 5 seconds
252 respawn limit 15 5
253
254 setuid {uid}
255
256=== added symlink 'charms/precise/lander-jenkins/hooks/upgrade-charm'
257=== target is u'hooks.py'
258=== modified file 'charms/precise/python-django/hooks/hooks.py'
259--- charms/precise/python-django/hooks/hooks.py 2014-03-10 22:25:00 +0000
260+++ charms/precise/python-django/hooks/hooks.py 2014-05-09 16:00:26 +0000
261@@ -454,13 +454,14 @@
262 inject_file.write(str(template))
263
264
265-def _install_from_tarball():
266+def _install_from_tarball(upgrade=False):
267 juju_log(MSG_INFO, 'grabbing service from tarball...')
268- if os.path.exists(vcs_clone_dir):
269- juju_log(MSG_INFO,
270- 'deleting pre-existing service directory: %s' % vcs_clone_dir)
271- shutil.rmtree(vcs_clone_dir)
272- os.mkdir(vcs_clone_dir)
273+ if not upgrade:
274+ if os.path.exists(vcs_clone_dir):
275+ juju_log(MSG_INFO,
276+ 'deleting pre-existing service directory: %s' % vcs_clone_dir)
277+ shutil.rmtree(vcs_clone_dir)
278+ os.mkdir(vcs_clone_dir)
279 cmd = 'curl %s | tar -xzC %s' % (repos_url, vcs_clone_dir)
280 subprocess.check_call(cmd, shell=True)
281
282@@ -616,7 +617,7 @@
283 elif vcs == 'svn' or vcs == 'subversion':
284 run('svn up %s %s' % (repos_url, vcs_clone_dir))
285 elif vcs == 'tarball':
286- _install_from_tarball()
287+ _install_from_tarball(True)
288 else:
289 juju_log(MSG_ERROR, "Unknown version control")
290 sys.exit(1)
291@@ -638,6 +639,10 @@
292 install_dir(settings_dir_path, owner=wsgi_user, group=wsgi_group, mode=0755)
293 install_dir(urls_dir_path, owner=wsgi_user, group=wsgi_group, mode=0755)
294
295+ if unit_config:
296+ with open(os.path.join(vcs_clone_dir, 'unit_config'), 'w') as f:
297+ f.write(base64.b64decode(unit_config))
298+
299
300 def django_settings_relation_joined_changed():
301 os.environ['DJANGO_SETTINGS_MODULE'] = django_settings_modules
302
303=== modified file 'charms/precise/rabbitmq-worker/hooks/hooks.py'
304--- charms/precise/rabbitmq-worker/hooks/hooks.py 2014-03-10 22:25:00 +0000
305+++ charms/precise/rabbitmq-worker/hooks/hooks.py 2014-05-09 16:00:26 +0000
306@@ -53,27 +53,38 @@
307 return(subprocess.call(cmd_line))
308
309
310-def _install_from_tarball(config):
311- juju_info('grabbing service from tarball...')
312+def _install_from_tarball(config, upgrade):
313+ tarball_url = config['tarball']
314+ juju_info('grabbing service from tarball {}...'.format(tarball_url))
315 sdir = _service_dir(config)
316- if os.path.exists(sdir):
317- juju_info('deleting pre-existing service directory: %s' % sdir)
318- shutil.rmtree(sdir)
319- os.mkdir(sdir)
320- cmd = 'curl %s | tar -xzC %s' % (config['tarball'], sdir)
321+ if not upgrade:
322+ if os.path.exists(sdir):
323+ juju_info(
324+ 'deleting pre-existing service directory: {}'.format(sdir))
325+ shutil.rmtree(sdir)
326+ os.mkdir(sdir)
327+ cmd = 'curl %s | tar -xzC %s' % (tarball_url, sdir)
328 subprocess.check_call(cmd, shell=True)
329
330
331-def _install_from_bzr(config):
332+def _install_from_bzr(config, upgrade):
333 juju_info('grabbing service from bzr...')
334
335 sdir = _service_dir(config)
336+ rev = config.get('revno', '')
337+
338+ if upgrade:
339+ args = ['bzr', 'pull']
340+ if rev:
341+ args.extend(['-r', rev])
342+ subprocess.check_call(args, cwd=sdir)
343+ return
344+
345 if os.path.exists(sdir):
346 juju_info('deleting pre-existing service directory: %s' % sdir)
347 shutil.rmtree(sdir)
348
349 args = ['bzr', 'branch']
350- rev = config.get('revno', '')
351 if rev:
352 args.extend(['-r', rev])
353 args.append(config['branch'])
354@@ -81,8 +92,11 @@
355 subprocess.check_call(args)
356
357
358-def install(config):
359- charmhelpers.fetch.configure_sources(update=True)
360+def install(config, upgrade=False):
361+ if upgrade:
362+ charmhelpers.fetch.apt_update()
363+ else:
364+ charmhelpers.fetch.configure_sources(update=True)
365
366 pkgs = [x for x in config.get('packages', '').split(' ') if x]
367 pkgs.append('python-amqplib')
368@@ -99,9 +113,9 @@
369 pip_install(package.strip())
370
371 if config.get('vcs') == 'tarball':
372- _install_from_tarball(config)
373+ _install_from_tarball(config, upgrade)
374 else:
375- _install_from_bzr(config)
376+ _install_from_bzr(config, upgrade)
377
378 unit_config = config.get('unit-config')
379 if unit_config:
380@@ -109,6 +123,15 @@
381 f.write(base64.b64decode(unit_config))
382
383
384+def upgrade_charm(config):
385+ install(config, True)
386+ try:
387+ subprocess.check_call(['/sbin/restart', _service_name(config)])
388+ except subprocess.CalledProcessError:
389+ # if it wasn't running, restart fails, so just try to start
390+ subprocess.check_call(['/sbin/start', _service_name(config)])
391+
392+
393 def _create_upstart(config):
394 template = textwrap.dedent('''
395 #--------------------------------------------------------------
396@@ -121,9 +144,9 @@
397 # use sigint so python code just needs to catch KeyboardInterrupt
398 kill signal SIGINT
399
400- # If the process quits unexpectadly trigger respawn it, unless it
401- # fails 15 times within 5 seconds
402+ # If the process quits unexpectedly trigger respawn it
403 respawn
404+ # unless it fails 15 times within 5 seconds
405 respawn limit 15 5
406
407 setuid {uid}
408
409=== added symlink 'charms/precise/rabbitmq-worker/hooks/upgrade-charm'
410=== target is u'hooks.py'
411=== modified file 'charms/precise/restish/hooks/hooks.py'
412--- charms/precise/restish/hooks/hooks.py 2014-03-10 22:25:00 +0000
413+++ charms/precise/restish/hooks/hooks.py 2014-05-09 16:00:26 +0000
414@@ -54,27 +54,36 @@
415 return os.path.join(config['install_root'], unit)
416
417
418-def _install_from_tarball(config):
419+def _install_from_tarball(config, upgrade):
420 juju_info('grabbing service from tarball...')
421 sdir = _service_dir(config)
422- if os.path.exists(sdir):
423- juju_info('deleting pre-existing service directory: %s' % sdir)
424- shutil.rmtree(sdir)
425- os.mkdir(sdir)
426+ if not upgrade:
427+ if os.path.exists(sdir):
428+ juju_info('deleting pre-existing service directory: %s' % sdir)
429+ shutil.rmtree(sdir)
430+ os.mkdir(sdir)
431 cmd = 'curl %s | tar -xzC %s' % (config['tarball'], sdir)
432 subprocess.check_call(cmd, shell=True)
433
434
435-def _install_from_bzr(config):
436+def _install_from_bzr(config, upgrade):
437 juju_info('grabbing service from bzr...')
438
439 sdir = _service_dir(config)
440+ rev = config.get('revno', '')
441+
442+ if upgrade:
443+ args = ['bzr', 'pull']
444+ if rev:
445+ args.extend(['-r', rev])
446+ subprocess.check_call(args, cwd=sdir)
447+ return
448+
449 if os.path.exists(sdir):
450 juju_info('deleting pre-existing service directory: %s' % sdir)
451 shutil.rmtree(sdir)
452
453 args = ['bzr', 'branch']
454- rev = config.get('revno', '')
455 if rev:
456 args.extend(['-r', rev])
457 args.append(config['branch'])
458@@ -82,7 +91,7 @@
459 subprocess.check_call(args)
460
461
462-def install(config):
463+def install(config, upgrade=False):
464 pkgs = [x for x in config['packages'].split(' ') if x]
465 pkgs.append('bzr')
466 pkgs.append('python-restish')
467@@ -92,9 +101,14 @@
468 charmhelpers.fetch.apt_install(pkgs)
469
470 if config.get('vcs') == 'tarball':
471- _install_from_tarball(config)
472+ _install_from_tarball(config, upgrade)
473 else:
474- _install_from_bzr(config)
475+ _install_from_bzr(config, upgrade)
476+
477+
478+def upgrade_charm(config):
479+ install(config, True)
480+ start(config)
481
482
483 def start(config):
484
485=== added symlink 'charms/precise/restish/hooks/upgrade-charm'
486=== target is u'hooks.py'
487=== modified file 'charms/precise/webui/hooks/hooks.py'
488--- charms/precise/webui/hooks/hooks.py 2014-03-10 22:25:00 +0000
489+++ charms/precise/webui/hooks/hooks.py 2014-05-09 16:00:26 +0000
490@@ -69,6 +69,11 @@
491
492 @hooks.hook()
493 def install():
494+ status_contents = None
495+ if os.path.exists(json_status_file):
496+ with open(json_status_file) as f:
497+ status_contents = f.read()
498+
499 if os.path.exists(site_path):
500 shutil.rmtree(site_path)
501
502@@ -97,6 +102,10 @@
503 if regex.match(f):
504 os.remove(os.path.join(site_path, f))
505
506+ if status_contents:
507+ with open(json_status_file, 'w') as f:
508+ f.write(status_contents)
509+
510
511 @hooks.hook()
512 def upgrade_charm():
513
514=== modified file 'ci-utils/ci_utils/__init__.py'
515--- ci-utils/ci_utils/__init__.py 2014-03-10 22:25:00 +0000
516+++ ci-utils/ci_utils/__init__.py 2014-05-09 16:00:26 +0000
517@@ -1,5 +1,5 @@
518 # Ubuntu CI Engine
519-# Copyright 2013 Canonical Ltd.
520+# Copyright 2013, 2014 Canonical Ltd.
521
522 # This program is free software: you can redistribute it and/or modify it
523 # under the terms of the GNU Affero General Public License version 3, as
524
525=== modified file 'ci-utils/ci_utils/amqp_utils.py'
526--- ci-utils/ci_utils/amqp_utils.py 2014-01-31 10:59:22 +0000
527+++ ci-utils/ci_utils/amqp_utils.py 2014-05-09 16:00:26 +0000
528@@ -21,6 +21,10 @@
529
530 from amqplib import client_0_8 as amqp
531
532+BSBUILDER_QUEUE = 'bsbuilder'
533+IMAGE_BUILDER_QUEUE = 'imagebuilder'
534+TEST_RUNNER_QUEUE = 'test_runner'
535+
536 log = logging.getLogger(__name__)
537
538
539@@ -49,7 +53,13 @@
540 )
541
542
543-def send(queue, msg):
544+def send(queue, msg, raise_errors=False):
545+ '''send a message to the given queue.
546+
547+ :param raise_errors: There are two users of this function. Some users are
548+ sending non-critical messages in which errors can safely be ignored. Other
549+ users need to catch the exception when this function fails.
550+ '''
551 config = get_config()
552 if not config:
553 return 'rabbitmq settings not available.'
554@@ -62,6 +72,8 @@
555 body.properties['delivery_mode'] = 2 # Persistent
556 channel.basic_publish(body, routing_key=queue)
557 except Exception as e:
558+ if raise_errors:
559+ raise
560 logging.exception('unable to queue up message: %s', e)
561 return str(e)
562 finally:
563
564=== modified file 'ci-utils/ci_utils/create_db.py'
565--- ci-utils/ci_utils/create_db.py 2014-01-02 15:26:27 +0000
566+++ ci-utils/ci_utils/create_db.py 2014-05-09 16:00:26 +0000
567@@ -1,5 +1,5 @@
568 # Ubuntu CI Engine
569-# Copyright 2013 Canonical Ltd.
570+# Copyright 2013, 2014 Canonical Ltd.
571
572 # This program is free software: you can redistribute it and/or modify it
573 # under the terms of the GNU Affero General Public License version 3, as
574
575=== modified file 'ci-utils/ci_utils/restish_utils.py'
576--- ci-utils/ci_utils/restish_utils.py 2014-03-10 22:25:00 +0000
577+++ ci-utils/ci_utils/restish_utils.py 2014-05-09 16:00:26 +0000
578@@ -1,5 +1,5 @@
579 # Ubuntu CI Engine
580-# Copyright 2013 Canonical Ltd.
581+# Copyright 2013, 2014 Canonical Ltd.
582
583 # This program is free software: you can redistribute it and/or modify it
584 # under the terms of the GNU Affero General Public License version 3, as
585
586=== modified file 'ci-utils/ci_utils/setup_helper.py'
587--- ci-utils/ci_utils/setup_helper.py 2014-03-10 22:25:00 +0000
588+++ ci-utils/ci_utils/setup_helper.py 2014-05-09 16:00:26 +0000
589@@ -21,7 +21,7 @@
590 from setuptools.package_index import PackageIndex
591
592 # A branch with all the ez-install deps in a single directory, ie:
593-# lp:~canonical-ci-engineering/ubuntu-ci-services-itself/deps
594+# lp:~canonical-ci-engineering/uci-engine/deps
595 PINNED_BRANCH = os.environ.get('CI_DEPS_BRANCH', '')
596 _deps = os.path.join(os.path.dirname(__file__), '../../.deps')
597 PINNED_LOCAL = os.environ.get('CI_DEPS_LOCAL', os.path.abspath(_deps))
598
599=== modified file 'ci-utils/ci_utils/tastypie/resource.py'
600--- ci-utils/ci_utils/tastypie/resource.py 2013-12-31 00:58:32 +0000
601+++ ci-utils/ci_utils/tastypie/resource.py 2014-05-09 16:00:26 +0000
602@@ -1,5 +1,5 @@
603 # Ubuntu CI Engine
604-# Copyright 2012-2013 Canonical Ltd.
605+# Copyright 2013, 2014 Canonical Ltd.
606
607 # This program is free software: you can redistribute it and/or modify it
608 # under the terms of the GNU Affero General Public License version 3, as
609
610=== modified file 'ci-utils/ci_utils/tastypie/test.py'
611--- ci-utils/ci_utils/tastypie/test.py 2014-01-02 15:42:39 +0000
612+++ ci-utils/ci_utils/tastypie/test.py 2014-05-09 16:00:26 +0000
613@@ -1,5 +1,5 @@
614 # Ubuntu CI Engine
615-# Copyright 2013 Canonical Ltd.
616+# Copyright 2013, 2014 Canonical Ltd.
617
618 # This program is free software: you can redistribute it and/or modify it
619 # under the terms of the GNU Affero General Public License version 3, as
620
621=== modified file 'ci-utils/ci_utils/tests/test_amqp.py'
622--- ci-utils/ci_utils/tests/test_amqp.py 2014-03-16 20:45:28 +0000
623+++ ci-utils/ci_utils/tests/test_amqp.py 2014-05-09 16:00:26 +0000
624@@ -1,5 +1,5 @@
625 # Ubuntu CI Engine
626-# Copyright 2013 Canonical Ltd.
627+# Copyright 2013, 2014 Canonical Ltd.
628
629 # This program is free software: you can redistribute it and/or modify it
630 # under the terms of the GNU Affero General Public License version 3, as
631@@ -48,6 +48,13 @@
632 r = amqp_utils.send('fake_queue', 'fake_message')
633 self.assertIsNone(r)
634
635+ connect.side_effect = RuntimeError('testSent')
636+ r = amqp_utils.send('fake_queue', 'fake message')
637+ self.assertIsNotNone(r)
638+
639+ with self.assertRaises(RuntimeError):
640+ amqp_utils.send('fake_queue', 'fake message', True)
641+
642 @mock.patch('ci_utils.amqp_utils._run_forever')
643 @mock.patch('ci_utils.amqp_utils.connection')
644 def testProcessQueue(self, connection, run_forever):
645
646=== modified file 'ci-utils/ci_utils/tests/test_restish.py'
647--- ci-utils/ci_utils/tests/test_restish.py 2014-03-10 22:25:00 +0000
648+++ ci-utils/ci_utils/tests/test_restish.py 2014-05-09 16:00:26 +0000
649@@ -1,5 +1,5 @@
650 # Ubuntu CI Engine
651-# Copyright 2013 Canonical Ltd.
652+# Copyright 2013, 2014 Canonical Ltd.
653
654 # This program is free software: you can redistribute it and/or modify it
655 # under the terms of the GNU Affero General Public License version 3, as
656
657=== modified file 'ci-utils/ci_utils/ticket_states.py'
658--- ci-utils/ci_utils/ticket_states.py 2014-01-28 12:28:13 +0000
659+++ ci-utils/ci_utils/ticket_states.py 2014-05-09 16:00:26 +0000
660@@ -1,5 +1,5 @@
661 # Ubuntu Continuous Integration Engine
662-# Copyright 2013 Canonical Ltd.
663+# Copyright 2014 Canonical Ltd.
664
665 # This program is free software: you can redistribute it and/or modify it
666 # under the terms of the GNU Affero General Public License version 3, as
667
668=== modified file 'cli/ci_libs/__init__.py'
669--- cli/ci_libs/__init__.py 2014-01-13 00:04:18 +0000
670+++ cli/ci_libs/__init__.py 2014-05-09 16:00:26 +0000
671@@ -1,5 +1,5 @@
672 # Ubuntu Continuous Integration Engine
673-# Copyright 2014 Canonical Ltd.
674+# Copyright 2013, 2014 Canonical Ltd.
675
676 # This program is free software: you can redistribute it and/or modify it
677 # under
678
679=== modified file 'cli/tests/test_file_handler.py'
680--- cli/tests/test_file_handler.py 2014-03-10 22:25:00 +0000
681+++ cli/tests/test_file_handler.py 2014-05-09 16:00:26 +0000
682@@ -1,6 +1,6 @@
683 #!/usr/bin/env python
684 # Ubuntu Continuous Integration Engine
685-# Copyright 2014 Canonical Ltd.
686+# Copyright 2013, 2014 Canonical Ltd.
687
688 # This program is free software: you can redistribute it and/or modify it
689 # under
690
691=== modified file 'debian/control'
692--- debian/control 2014-03-14 10:37:28 +0000
693+++ debian/control 2014-05-09 16:00:26 +0000
694@@ -15,7 +15,7 @@
695 python-setuptools
696 Standards-Version: 3.9.4
697 X-Python-Version: >= 2.7
698-Vcs-Bzr: https://code.launchpad.net/~canonical-ci-engineering/ubuntu-ci-services-itself/trunk
699+Vcs-Bzr: https://code.launchpad.net/~canonical-ci-engineering/uci-engine/trunk
700
701 Package: uci-utils
702 Architecture: all
703
704=== modified file 'debian/copyright'
705--- debian/copyright 2014-03-07 00:31:30 +0000
706+++ debian/copyright 2014-05-09 16:00:26 +0000
707@@ -1,6 +1,6 @@
708 Format: http://www.debian.org/doc/packaging-manuals/copyright-format/1.0/
709 Upstream-Name: uci-source
710-Source: http://launchpad.net/ubuntu-ci-services-itself
711+Source: http://launchpad.net/uci-engine
712
713 Files: *
714 Copyright: 2013-2014 Canonical Ltd.
715
716=== modified file 'image-builder/imagebuilder/__init__.py'
717--- image-builder/imagebuilder/__init__.py 2014-03-10 22:25:00 +0000
718+++ image-builder/imagebuilder/__init__.py 2014-05-09 16:00:26 +0000
719@@ -1,5 +1,5 @@
720 # Ubuntu CI Engine
721-# Copyright 2013 Canonical Ltd.
722+# Copyright 2013, 2014 Canonical Ltd.
723
724 # This program is free software: you can redistribute it and/or modify it
725 # under the terms of the GNU Affero General Public License version 3, as
726
727=== modified file 'image-builder/imagebuilder/cloud_image.py'
728--- image-builder/imagebuilder/cloud_image.py 2014-05-07 00:56:11 +0000
729+++ image-builder/imagebuilder/cloud_image.py 2014-05-09 16:00:26 +0000
730@@ -103,6 +103,21 @@
731 time.sleep(timeout)
732
733
734+def _setup_apt_mirror(status_cb, fname):
735+ try:
736+ mirror = ci_utils.unit_config.get('apt_mirror')
737+ except KeyError:
738+ status_cb('no apt mirror configured')
739+ return False
740+ status_cb('updating apt mirror to: {}'.format(mirror))
741+ with open(fname) as f:
742+ lines = f.readlines()
743+ with open(fname, 'w') as f:
744+ for line in lines:
745+ f.write(line.replace('archive.ubuntu.com', mirror))
746+ return True
747+
748+
749 def _setup_ppas(chroot, repos, series, status_cb):
750 status_cb('importing ppa signing keys...')
751 lp = launchpad.lp_login()
752@@ -260,6 +275,8 @@
753
754 _setup_grub(mountpoint, status_cb)
755
756+ _setup_apt_mirror(
757+ status_cb, os.path.join(mountpoint, 'etc/apt/sources.list'))
758 _setup_ppas(mountpoint, repos, series, status_cb)
759
760 _run_chroot_cmds(mountpoint, packages, status_cb)
761@@ -283,10 +300,19 @@
762
763
764 def main():
765+ import mock
766+ ds = mock.Mock()
767 dump_stack.install_stack_dump_signal()
768 args = _get_args()
769- return modify_image(args.image, args.repository, args.series,
770- args.package)
771+
772+ def cb(msg):
773+ print(msg)
774+
775+ try:
776+ return modify_image(args.image, args.repository, args.series,
777+ args.package, ds, cb)
778+ except subprocess.CalledProcessError as e:
779+ print(e.output)
780
781 if __name__ == '__main__':
782 main()
783
784=== modified file 'image-builder/imagebuilder/resources/v1.py'
785--- image-builder/imagebuilder/resources/v1.py 2014-03-14 10:37:28 +0000
786+++ image-builder/imagebuilder/resources/v1.py 2014-05-09 16:00:26 +0000
787@@ -21,7 +21,6 @@
788 from ci_utils import amqp_utils, json_status, restish_utils
789
790 log = logging.getLogger(__name__)
791-WORKER_QUEUE = 'imagebuilder'
792
793
794 def _build_image(base_image, ppa_list, package_list, progress_trigger,
795@@ -41,7 +40,7 @@
796 log.error('Unable to notify progress trigger, aborting build_image')
797 return http.service_unavailable(body=r)
798
799- r = amqp_utils.send(WORKER_QUEUE, json.dumps(params))
800+ r = amqp_utils.send(amqp_utils.IMAGE_BUILDER_QUEUE, json.dumps(params))
801 if r:
802 r = http.service_unavailable(body=r)
803 return r
804@@ -50,7 +49,7 @@
805 def _status():
806 status = json_status.JSONStatus()
807 status.add_rabbit_configured()
808- status.add_rabbit_worker_health(WORKER_QUEUE)
809+ status.add_rabbit_worker_health(amqp_utils.IMAGE_BUILDER_QUEUE)
810 return restish_utils.json_ok(status.results)
811
812
813
814=== modified file 'image-builder/imagebuilder/run_worker.py'
815--- image-builder/imagebuilder/run_worker.py 2014-03-15 02:14:19 +0000
816+++ image-builder/imagebuilder/run_worker.py 2014-05-09 16:00:26 +0000
817@@ -44,4 +44,4 @@
818
819
820 if __name__ == '__main__':
821- ImageBuildWorker().main('imagebuilder')
822+ ImageBuildWorker().main(amqp_utils.IMAGE_BUILDER_QUEUE)
823
824=== modified file 'image-builder/imagebuilder/tests/test_api_v1.py'
825--- image-builder/imagebuilder/tests/test_api_v1.py 2014-03-14 10:37:28 +0000
826+++ image-builder/imagebuilder/tests/test_api_v1.py 2014-05-09 16:00:26 +0000
827@@ -1,5 +1,5 @@
828 # Ubuntu CI Services
829-# Copyright 2013 Canonical Ltd.
830+# Copyright 2013, 2014 Canonical Ltd.
831
832 # This program is free software: you can redistribute it and/or modify it
833 # under the terms of the GNU Affero General Public License version 3, as
834
835=== modified file 'image-builder/imagebuilder/tests/test_modify_cloud_image.py'
836--- image-builder/imagebuilder/tests/test_modify_cloud_image.py 2014-05-07 01:15:41 +0000
837+++ image-builder/imagebuilder/tests/test_modify_cloud_image.py 2014-05-09 16:00:26 +0000
838@@ -1,5 +1,5 @@
839 # Ubuntu CI Engine
840-# Copyright 2013 Canonical Ltd.
841+# Copyright 2014 Canonical Ltd.
842
843 # This program is free software: you can redistribute it and/or modify it
844 # under the terms of the GNU Affero General Public License version 3, as
845@@ -14,6 +14,8 @@
846 # along with this program. If not, see <http://www.gnu.org/licenses/>.
847
848 import subprocess
849+import tempfile
850+import textwrap
851 import unittest
852
853 import mock
854@@ -25,6 +27,35 @@
855 def setUp(self):
856 super(TestCloudImage, self).setUp()
857
858+ @mock.patch('ci_utils.unit_config')
859+ def test_apt_mirror_not_configured(self, unit_config):
860+ unit_config.get.side_effect = KeyError('apt_mirror')
861+ status_cb = mock.Mock()
862+ self.assertFalse(cloud_image._setup_apt_mirror(status_cb, '/ignored'))
863+
864+ @mock.patch('ci_utils.unit_config')
865+ def test_apt_mirror_configured(self, unit_config):
866+ unit_config.get.return_value = 'foo.archive.ubuntu.com'
867+ status_cb = mock.Mock()
868+
869+ orig = textwrap.dedent('''\
870+ deb http://archive.ubuntu.com/ubuntu saucy main restricted universe
871+ deb http://archive.ubuntu.com/ubuntu saucy-updates main restricted
872+ deb http://security.ubuntu.com/ubuntu saucy-security main restricted
873+ ''')
874+ expected = textwrap.dedent('''\
875+ deb http://foo.archive.ubuntu.com/ubuntu saucy main restricted universe
876+ deb http://foo.archive.ubuntu.com/ubuntu saucy-updates main restricted
877+ deb http://security.ubuntu.com/ubuntu saucy-security main restricted
878+ ''')
879+
880+ with tempfile.NamedTemporaryFile() as f:
881+ f.write(orig)
882+ f.flush()
883+ self.assertTrue(cloud_image._setup_apt_mirror(status_cb, f.name))
884+ with open(f.name) as f:
885+ self.assertEqual(expected, f.read())
886+
887 @mock.patch('imagebuilder.cloud_image.launchpad')
888 def test_parse_ppas(self, launchpad):
889 launchpad.get_lp_user_name.return_value = 'foo'
890
891=== modified file 'juju-deployer/branch-source-builder.yaml.tmpl'
892--- juju-deployer/branch-source-builder.yaml.tmpl 2014-03-14 10:37:28 +0000
893+++ juju-deployer/branch-source-builder.yaml.tmpl 2014-05-09 16:00:26 +0000
894@@ -18,7 +18,7 @@
895 - ""
896 bsb-gunicorn:
897 charm: gunicorn
898- branch: lp:charms/precise/gunicorn@28
899+ branch: lp:~canonical-ci-engineering/charms/precise/gunicorn/fix-config-changed-departed@30
900 options:
901 wsgi_wsgi_file: bsbuilder.wsgi:app
902 bsb-worker:
903@@ -36,7 +36,7 @@
904 install_keys: |
905 - ""
906 rabbit:
907- branch: lp:~canonical-ci-engineering/charms/precise/ubuntu-ci-services-itself/rabbitmq-server@46
908+ branch: lp:~canonical-ci-engineering/charms/precise/uci-engine/rabbitmq-server@46
909 charm: rabbitmq
910 options:
911 management_plugin: true
912
913=== modified file 'juju-deployer/configs/unit_config.yaml.tmpl'
914--- juju-deployer/configs/unit_config.yaml.tmpl 2014-05-07 01:21:22 +0000
915+++ juju-deployer/configs/unit_config.yaml.tmpl 2014-05-09 16:00:26 +0000
916@@ -33,3 +33,5 @@
917 tr_flavors:
918 - m1.medium
919 - standard.medium
920+
921+apt_mirror: nova.clouds.archive.ubuntu.com
922
923=== modified file 'juju-deployer/deploy.py'
924--- juju-deployer/deploy.py 2014-05-07 01:25:30 +0000
925+++ juju-deployer/deploy.py 2014-05-09 16:00:26 +0000
926@@ -1,6 +1,7 @@
927 #!/usr/bin/python
928
929 import argparse
930+import base64
931 import hashlib
932 import os
933 import sys
934@@ -10,6 +11,7 @@
935 import shutil
936 import tempfile
937 import textwrap
938+import yaml
939 from distutils import spawn
940 from itertools import chain
941
942@@ -27,7 +29,7 @@
943 deployer_dir = os.path.abspath(os.path.dirname(sys.argv[0]))
944 # ci-utils probably isn't in our pythonpath
945 sys.path.append(os.path.join(os.path.dirname(__file__), '../ci-utils'))
946-from ci_utils import dump_stack
947+from ci_utils import dump_stack, data_store
948
949
950 # Initialize bzrlib and its plugins
951@@ -61,7 +63,7 @@
952 cheetah_vars = {
953 'OS_USERNAME': None,
954 'CI_CODE_SOURCE': 'branch',
955- 'CI_BRANCH': 'lp:ubuntu-ci-services-itself',
956+ 'CI_BRANCH': 'lp:uci-engine',
957 'CI_PAYLOAD_URL': 'n/a',
958 'CI_LAUNCHPAD_USER': None,
959 'CI_LAUNCHPAD_PPA_OWNER': None,
960@@ -210,6 +212,7 @@
961 # For an obscure reason tag:0.3.1 is not in lp:juju-deployer
962 get_or_pull_branch('lp:juju-deployer', dest, revision='revno:103')
963 prepend_path('PYTHONPATH', dest)
964+ sys.path.insert(0, dest)
965 # FIXME: This duplicates setup.py and shouldn't be needed. This specific
966 # piece (juju-deployer) is a weird beast, it installs from source (not pip
967 # nor package), it requires amulet which itself requires urllib3 (among
968@@ -299,13 +302,8 @@
969 Also set the relevant environment variables used to populate the deployer
970 scripts and charm configurations.
971 """
972- base = os.path.join(os.path.dirname(__file__), '..')
973 print('Preparing local branch upload...')
974
975- ciutils = os.path.join(base, 'ci-utils')
976- sys.path.append(ciutils)
977- from ci_utils import data_store
978-
979 config = {
980 'auth_url': os.environ['OS_AUTH_URL'],
981 'auth_user': os.environ['OS_USERNAME'],
982@@ -313,7 +311,11 @@
983 'auth_tenant_name': os.environ['OS_TENANT_NAME'],
984 'auth_region': os.environ['OS_REGION_NAME']
985 }
986- ds = data_store.DataStore('ci-payload', config, public=True)
987+ try:
988+ ds = data_store.DataStore('ci-payload', config, public=True)
989+ except data_store.DataStoreException as exc:
990+ error_msg = ("ERROR: data-store: {}")
991+ sys.exit(error_msg.format(exc))
992
993 with tempfile.NamedTemporaryFile() as ball:
994 fingerprint = build_payload(ball.name)
995@@ -333,6 +335,67 @@
996 os.environ['CI_BRANCH'] = payload_url
997
998
999+def _install(deployer_dir, generated):
1000+ args = ['juju-deployer', '-v'] + generated + ['ci-airline-staging']
1001+ print 'Running {}'.format(' '.join(args))
1002+ try:
1003+ subprocess.check_call(args,
1004+ cwd=os.path.join(deployer_dir, '../charms'))
1005+ except Exception as exc:
1006+ print 'Problem deploying "ci-airline-staging": {}'.format(str(exc))
1007+
1008+
1009+def _get_deployer(configs):
1010+ import deployer.config # lazy load since this is added by this script
1011+ import deployer.utils
1012+
1013+ deployer.utils.setup_logging()
1014+ dep = deployer.config.ConfigStack(configs).get('ci-airline-staging')
1015+
1016+ # ensure we pull down the charms we need for upgrade
1017+ args = []
1018+ for c in configs:
1019+ args.extend(['-c', c])
1020+ args.append('ci-airline-staging')
1021+ subprocess.check_call(['juju-deployer', '-b'] + args)
1022+
1023+ return dep
1024+
1025+
1026+def _deployer_options(service, deployer_dir):
1027+ keyvals = []
1028+ for key, val in service.get('options', {}).iteritems():
1029+ include_token = 'include-base64://'
1030+ if type(val) is str and val.startswith(include_token):
1031+ val = val.replace(include_token, '')
1032+ p = os.path.join(deployer_dir, val)
1033+ with open(p) as f:
1034+ val = base64.b64encode(f.read())
1035+ keyvals.append('%s=%s' % (key, val))
1036+ return keyvals
1037+
1038+
1039+def _upgrade(deployer_dir, configs):
1040+ print 'Discovering services to upgrade'
1041+ deployment = _get_deployer(configs)
1042+ status = subprocess.check_output(['juju', 'status'])
1043+ services = yaml.safe_load(status)['services']
1044+ for svc in services:
1045+ dsvc = deployment.data['services'][svc]
1046+ unit_config = dsvc.get('options', {}).get('unit-config')
1047+ if unit_config:
1048+ print 'Upgrading unit-config for {}'.format(svc)
1049+ keyvals = _deployer_options(dsvc, deployer_dir)
1050+ if keyvals:
1051+ print 'Upgrading configs for {}'.format(svc)
1052+ subprocess.check_call(['juju', 'set', svc] + keyvals)
1053+ print 'Upgrading charm for {}'.format(svc)
1054+ subprocess.check_call(
1055+ ['juju', 'upgrade-charm', '--force', svc], cwd=deployer_dir)
1056+ print 'Upgrade requests have been issued.'
1057+ print 'Check back with "juju status" to track completion'
1058+
1059+
1060 def _get_args(args=None):
1061 parser = argparse.ArgumentParser(
1062 description='use juju-deployer to deploy services')
1063@@ -340,6 +403,7 @@
1064 help='''Use the current tip of this payload branch for
1065 the code-base rather than a tarball of the current
1066 directory.''')
1067+ # Andy said he won't use it again -- vila 2014-03-17
1068 parser.add_argument('--reuse-payload', action='store_true',
1069 help='''Use the already uploaded payload for
1070 the code-base rather than a tarball of the current
1071@@ -347,6 +411,8 @@
1072 parser.add_argument('--private-ppas-only', action='store_true',
1073 help='''Sets the environment to use only private PPAs
1074 when building packages.''')
1075+ parser.add_argument('--upgrade', action='store_true',
1076+ help='Upgrade an existing deployment.')
1077 parser.add_argument('specific', metavar='<config.yaml>', nargs='*',
1078 help='Use a specific list of juju-deployer configs.')
1079 parser.add_argument('--private-launchpad', action='store_true',
1080@@ -405,13 +471,13 @@
1081
1082 build_charms()
1083
1084- args = ['juju-deployer', '-v'] + generated + ['ci-airline-staging']
1085- print 'Running {}'.format(' '.join(args))
1086- try:
1087- subprocess.check_call(args,
1088- cwd=os.path.join(deployer_dir, '../charms'))
1089- except Exception as exc:
1090- print 'Problem deploying "ci-airline-staging": {}'.format(str(exc))
1091+ if args.upgrade:
1092+ for x in generated:
1093+ configs = [x for x in generated if x != '-c']
1094+ _upgrade(working_dir, configs)
1095+ else:
1096+ _install(deployer_dir, generated)
1097+
1098
1099 if __name__ == '__main__':
1100 dump_stack.install_stack_dump_signal()
1101
1102=== modified file 'juju-deployer/image-builder.yaml.tmpl'
1103--- juju-deployer/image-builder.yaml.tmpl 2014-03-14 10:37:28 +0000
1104+++ juju-deployer/image-builder.yaml.tmpl 2014-05-09 16:00:26 +0000
1105@@ -17,7 +17,7 @@
1106 - ""
1107 imagebuild-gunicorn:
1108 charm: gunicorn
1109- branch: lp:charms/precise/gunicorn@28
1110+ branch: lp:~canonical-ci-engineering/charms/precise/gunicorn/fix-config-changed-departed@30
1111 options:
1112 wsgi_wsgi_file: imagebuilder.wsgi:app
1113 imagebuild-worker:
1114@@ -36,7 +36,7 @@
1115 install_keys: |
1116 - ""
1117 rabbit:
1118- branch: lp:~canonical-ci-engineering/charms/precise/ubuntu-ci-services-itself/rabbitmq-server@46
1119+ branch: lp:~canonical-ci-engineering/charms/precise/uci-engine/rabbitmq-server@46
1120 charm: rabbitmq
1121 options:
1122 management_plugin: true
1123
1124=== modified file 'juju-deployer/lander.yaml.tmpl'
1125--- juju-deployer/lander.yaml.tmpl 2014-03-14 10:37:28 +0000
1126+++ juju-deployer/lander.yaml.tmpl 2014-05-09 16:00:26 +0000
1127@@ -16,12 +16,12 @@
1128 - ""
1129 lander-gunicorn:
1130 charm: gunicorn
1131- branch: lp:charms/precise/gunicorn@28
1132+ branch: lp:~canonical-ci-engineering/charms/precise/gunicorn/fix-config-changed-departed@30
1133 options:
1134 wsgi_wsgi_file: lander.wsgi:app
1135 lander-jenkins:
1136 charm: jenkins
1137- branch: lp:~ci-engineering-private/charms/precise/jenkins/ci-train@60
1138+ branch: lp:~ci-engineering-private/charms/precise/jenkins/ci-train@62
1139 constraints: "cpu-cores=2 mem=8 root-disk=8192M"
1140 exposed: true
1141 options:
1142@@ -39,11 +39,11 @@
1143 bot_password: lander-bot
1144 sources: ${CI_PPA}
1145 packages: "python-amqplib python-swiftclient lazr.enum"
1146- plugins: "parameterized-trigger:2.14"
1147+ plugins: "parameterized-trigger"
1148 unit-config: include-base64://configs/unit_config.yaml
1149 main: ./run-python ./lander/lander/run_worker.py --service-name ts-django --service-port 8080 --delay 10
1150 rabbit:
1151- branch: lp:~canonical-ci-engineering/charms/precise/ubuntu-ci-services-itself/rabbitmq-server@46
1152+ branch: lp:~canonical-ci-engineering/charms/precise/uci-engine/rabbitmq-server@46
1153 charm: rabbitmq
1154 relations:
1155 - ["lander-jenkins:extension", "lander-jenkins-sub:extension"]
1156
1157=== modified file 'juju-deployer/ppa-assigner.yaml.tmpl'
1158--- juju-deployer/ppa-assigner.yaml.tmpl 2014-03-10 22:25:00 +0000
1159+++ juju-deployer/ppa-assigner.yaml.tmpl 2014-05-09 16:00:26 +0000
1160@@ -19,7 +19,7 @@
1161 branch: lp:charms/precise/postgresql@84
1162 charm: postgresql
1163 ppa-gunicorn:
1164- branch: lp:charms/precise/gunicorn@28
1165+ branch: lp:~canonical-ci-engineering/charms/precise/gunicorn/fix-config-changed-departed@30
1166 charm: gunicorn
1167 options:
1168 python_path: /srv/ppa_django/ci-utils:/srv/ppa_django/ppa-assigner
1169
1170=== modified file 'juju-deployer/production-only.yaml'
1171--- juju-deployer/production-only.yaml 2014-03-10 22:25:00 +0000
1172+++ juju-deployer/production-only.yaml 2014-05-09 16:00:26 +0000
1173@@ -8,7 +8,7 @@
1174 # like tests/ppa_assigner/test.py which would trigger cleaning of a
1175 # ppa when its not really necessary
1176 ppa-cleaner:
1177- branch: lp:~canonical-ci-engineering/charms/precise/ubuntu-ci-services-itself/upstart@6
1178+ branch: lp:~canonical-ci-engineering/charms/precise/uci-engine/upstart@7
1179 charm: upstart
1180 options:
1181 service-name: ppa-cleaner
1182
1183=== modified file 'juju-deployer/test-runner.yaml.tmpl'
1184--- juju-deployer/test-runner.yaml.tmpl 2014-03-10 22:25:00 +0000
1185+++ juju-deployer/test-runner.yaml.tmpl 2014-05-09 16:00:26 +0000
1186@@ -3,7 +3,7 @@
1187 services:
1188 tr-gunicorn:
1189 charm: gunicorn
1190- branch: lp:charms/precise/gunicorn@28
1191+ branch: lp:~canonical-ci-engineering/charms/precise/gunicorn/fix-config-changed-departed@30
1192 options:
1193 wsgi_wsgi_file: tstrun.wsgi:appl
1194 tr-restish:
1195@@ -30,7 +30,6 @@
1196 tr-rabbit-worker:
1197 charm: rabbitmq-worker
1198 options:
1199- # for tests branch: lp:~vila/ubuntu-ci-services-itself/testy
1200 main: ./run-python ./test_runner/tstrun/run_worker.py
1201 vcs: ${CI_CODE_SOURCE}
1202 branch: ${CI_BRANCH}
1203@@ -43,7 +42,7 @@
1204 install_keys: |
1205 - ""
1206 rabbit:
1207- branch: lp:~canonical-ci-engineering/charms/precise/ubuntu-ci-services-itself/rabbitmq-server@46
1208+ branch: lp:~canonical-ci-engineering/charms/precise/uci-engine/rabbitmq-server@46
1209 charm: rabbitmq
1210 options:
1211 management_plugin: true
1212
1213=== modified file 'juju-deployer/test_deploy.py'
1214--- juju-deployer/test_deploy.py 2014-03-14 23:53:04 +0000
1215+++ juju-deployer/test_deploy.py 2014-05-09 16:00:26 +0000
1216@@ -150,5 +150,6 @@
1217 self.assertTrue(os.path.exists(os.path.join(charm2, 'make_called')))
1218 self.assertItemsEqual(['charm1', 'charm2', 'webui'], os.listdir(p))
1219
1220+
1221 if __name__ == '__main__':
1222 unittest.main()
1223
1224=== modified file 'juju-deployer/ticket-system.yaml.tmpl'
1225--- juju-deployer/ticket-system.yaml.tmpl 2014-02-25 14:14:43 +0000
1226+++ juju-deployer/ticket-system.yaml.tmpl 2014-05-09 16:00:26 +0000
1227@@ -30,7 +30,7 @@
1228 branch: lp:charms/precise/postgresql@84
1229 charm: postgresql
1230 ts-gunicorn:
1231- branch: lp:charms/precise/gunicorn@28
1232+ branch: lp:~canonical-ci-engineering/charms/precise/gunicorn/fix-config-changed-departed@30
1233 charm: gunicorn
1234 options:
1235 python_path: /srv/ts_django/ci-utils:/srv/ts_django/ticket_system
1236
1237=== modified file 'lander/bin/lander_merge_parameters.py'
1238--- lander/bin/lander_merge_parameters.py 2014-01-02 21:42:38 +0000
1239+++ lander/bin/lander_merge_parameters.py 2014-05-09 16:00:26 +0000
1240@@ -1,6 +1,6 @@
1241 #!/usr/bin/env python
1242 # Ubuntu CI Engine
1243-# Copyright 2013 Canonical Ltd.
1244+# Copyright 2013, 2014 Canonical Ltd.
1245
1246 # This program is free software: you can redistribute it and/or modify it
1247 # under the terms of the GNU Affero General Public License version 3, as
1248
1249=== modified file 'lander/bin/lander_service_wrapper.py'
1250--- lander/bin/lander_service_wrapper.py 2014-03-17 03:49:03 +0000
1251+++ lander/bin/lander_service_wrapper.py 2014-05-09 16:00:26 +0000
1252@@ -188,7 +188,6 @@
1253 def _handle_bsbuilder(args):
1254 config = _get_config(args)
1255 ticket = _ticket_api(config)
1256- url = '%s/api/v1/build_source' % config['master']['bsb_url']
1257
1258 queue = '%s-bsbuilder' % config['master']['progress_trigger']
1259
1260@@ -205,8 +204,7 @@
1261
1262 ticket.set_pkg_waiting()
1263 try:
1264- _post(url, params)
1265-
1266+ amqp_utils.send(amqp_utils.BSBUILDER_QUEUE, json.dumps(params), True)
1267 logger.info('starting progress handler...')
1268 return _drain_progress(args.amqp_config, queue, ticket,
1269 ticket.set_pkg_inprogress,
1270@@ -254,16 +252,10 @@
1271 config = _get_config(args)
1272 ticket = _ticket_api(config)
1273
1274- url = '%s/api/v1/build_image' % config['master']['imgbuilder_url']
1275-
1276 queue = '%s-imgbuilder' % config['master']['progress_trigger']
1277
1278 # Extract the image parameters from the master request parameters
1279 request_parameters = ticket.get_full_ticket()
1280- # TODO Remove the hardcoded cloud, do same as series and base_image
1281- image = {'image_type': 'cloud',
1282- 'series': request_parameters.get('series', ''),
1283- 'url_list': [request_parameters.get('base_image'), '']}
1284 ppa_list = [
1285 config['bsbuilder']['ppa'],
1286 request_parameters['master_ppa'],
1287@@ -271,14 +263,16 @@
1288 params = {
1289 'ticket_id': config['master']['request_id'],
1290 'cancel_url': os.environ['BUILD_URL'] + 'api/json',
1291- 'base_image': image,
1292+ 'image': request_parameters['base_image'],
1293+ 'series': request_parameters['series'],
1294 'package_list': _get_binary_packages(ticket, request_parameters),
1295 'ppa_list': ppa_list,
1296 'progress_trigger': queue,
1297 }
1298 ticket.set_image_waiting()
1299 try:
1300- _post(url, params)
1301+ amqp_utils.send(
1302+ amqp_utils.IMAGE_BUILDER_QUEUE, json.dumps(params), True)
1303 logger.info('starting progress handler...')
1304 return _drain_progress(args.amqp_config, queue, ticket,
1305 ticket.set_image_inprogress,
1306@@ -348,8 +342,6 @@
1307 config = _get_config(args)
1308 ticket = _ticket_api(config)
1309
1310- url = '%s/api/v1/test_image' % config['master']['tr_url']
1311-
1312 # The queue should be unique
1313 queue = '%s-testrunner' % config['master']['progress_trigger']
1314 request_parameters = ticket.get_full_ticket()
1315@@ -367,9 +359,10 @@
1316 'package_list': packages,
1317 'progress_trigger': queue,
1318 }
1319+
1320 ticket.set_testing_waiting()
1321 try:
1322- _post(url, params)
1323+ amqp_utils.send(amqp_utils.TEST_RUNNER_QUEUE, json.dumps(params), True)
1324 logger.info('starting progress handler...')
1325 return _drain_progress(args.amqp_config, queue, ticket,
1326 ticket.set_testing_inprogress,
1327
1328=== modified file 'lander/bin/ticket_api.py'
1329--- lander/bin/ticket_api.py 2014-03-14 10:37:28 +0000
1330+++ lander/bin/ticket_api.py 2014-05-09 16:00:26 +0000
1331@@ -1,5 +1,5 @@
1332 # Ubuntu CI Engine
1333-# Copyright 2013 Canonical Ltd.
1334+# Copyright 2014 Canonical Ltd.
1335
1336 # This program is free software: you can redistribute it and/or modify it
1337 # under the terms of the GNU Affero General Public License version 3, as
1338
1339=== modified file 'lander/lander/__init__.py'
1340--- lander/lander/__init__.py 2014-03-10 22:25:00 +0000
1341+++ lander/lander/__init__.py 2014-05-09 16:00:26 +0000
1342@@ -1,5 +1,5 @@
1343 # Ubuntu CI Engine
1344-# Copyright 2013 Canonical Ltd.
1345+# Copyright 2013, 2014 Canonical Ltd.
1346
1347 # This program is free software: you can redistribute it and/or modify it
1348 # under the terms of the GNU Affero General Public License version 3, as
1349
1350=== modified file 'lander/lander/resources/v1.py'
1351--- lander/lander/resources/v1.py 2014-01-10 22:51:44 +0000
1352+++ lander/lander/resources/v1.py 2014-05-09 16:00:26 +0000
1353@@ -1,5 +1,5 @@
1354 # Ubuntu CI Engine
1355-# Copyright 2013 Canonical Ltd.
1356+# Copyright 2013, 2014 Canonical Ltd.
1357
1358 # This program is free software: you can redistribute it and/or modify it
1359 # under the terms of the GNU Affero General Public License version 3, as
1360
1361=== modified file 'lander/lander/tests/test_service_wrapper.py'
1362--- lander/lander/tests/test_service_wrapper.py 2014-03-14 10:37:28 +0000
1363+++ lander/lander/tests/test_service_wrapper.py 2014-05-09 16:00:26 +0000
1364@@ -222,11 +222,12 @@
1365 self.assertEqual(ppa, data['ppa'])
1366
1367 @mock.patch('lander_service_wrapper.TicketApi')
1368- @mock.patch('lander_service_wrapper._post')
1369- def testBSBuilderUnexpected(self, post, api):
1370+ @mock.patch('ci_utils.amqp_utils.send')
1371+ def testBSBuilderUnexpected(self, send, api):
1372 '''Ensure unexpected errors still generate a response'''
1373 err_msg = 'foo bar'
1374- post.side_effect = RuntimeError(err_msg)
1375+ send.side_effect = RuntimeError(err_msg)
1376+ api().get_full_ticket.return_value = {'subticket': 1}
1377
1378 args = self.base_args + ['--service', 'bsbuilder']
1379 args = lander_service_wrapper._get_parser().parse_args(args)
1380@@ -241,14 +242,15 @@
1381 self.assertEqual(err_msg, data['error'])
1382
1383 @mock.patch('lander_service_wrapper.TicketApi')
1384- @mock.patch('lander_service_wrapper._post')
1385+ @mock.patch('ci_utils.amqp_utils.send')
1386 @mock.patch('ci_utils.amqp_utils.connection')
1387- def testBSBuilderSucceeds(self, connection, post, api):
1388+ def testBSBuilderSucceeds(self, connection, send, api):
1389 '''Ensure we generate the proper response if things succeed.'''
1390
1391 # mock the run-forever logic to ensure we return what we expect
1392 state = {'state': 'COMPLETED', 'result': 'PASSED',
1393 'foo': 'bar', 'exit': True}
1394+ api().get_full_ticket.return_value = {'subticket': 1}
1395
1396 def amqp():
1397 def run_forever(channel, queue, callback):
1398@@ -274,11 +276,13 @@
1399 self.assertEqual(state, data)
1400
1401 @mock.patch('lander_service_wrapper.TicketApi')
1402- @mock.patch('lander_service_wrapper._post')
1403- def testImageBuilderUnexpected(self, post, api):
1404+ @mock.patch('ci_utils.amqp_utils.send')
1405+ def testImageBuilderUnexpected(self, send, api):
1406 '''Ensure unexpected errors still generate a response'''
1407 err_msg = 'foo bar'
1408- post.side_effect = RuntimeError(err_msg)
1409+ send.side_effect = RuntimeError(err_msg)
1410+ api().get_full_ticket.return_value = {
1411+ 'master_ppa': 1, 'series': 1, 'base_image': 1}
1412
1413 args = self.base_args + ['--service', 'image_builder']
1414 args = lander_service_wrapper._get_parser().parse_args(args)
1415@@ -293,14 +297,15 @@
1416 self.assertEqual(err_msg, data['error'])
1417
1418 @mock.patch('lander_service_wrapper.TicketApi')
1419- @mock.patch('lander_service_wrapper._post')
1420+ @mock.patch('ci_utils.amqp_utils.send')
1421 @mock.patch('ci_utils.amqp_utils.connection')
1422- def testImageBuilderSucceeds(self, connection, post, api):
1423+ def testImageBuilderSucceeds(self, connection, send, api):
1424 '''Ensure we generate the proper response if things succeed.
1425
1426 While technically "succeeding" its checking for a successful FAILURE
1427 '''
1428-
1429+ api().get_full_ticket.return_value = {
1430+ 'master_ppa': 1, 'series': 1, 'base_image': 1}
1431 # mock the run-forever logic to ensure we return what we expect
1432 state = {'state': 'COMPLETED', 'result': 'FAILED',
1433 'foo': 'bar', 'exit': True}
1434@@ -329,11 +334,11 @@
1435 self.assertEqual(state, data)
1436
1437 @mock.patch('lander_service_wrapper.TicketApi')
1438- @mock.patch('lander_service_wrapper._post')
1439+ @mock.patch('ci_utils.amqp_utils.send')
1440 @mock.patch('ci_utils.amqp_utils.connection')
1441- def testRunnerSucceeds(self, connection, post, api):
1442+ def testRunnerSucceeds(self, connection, send, api):
1443 '''Ensure we generate the proper response if things succeed.'''
1444-
1445+ api().get_full_ticket.return_value = {}
1446 # mock the run-forever logic to ensure we return what we expect
1447 state = {'state': 'COMPLETED', 'result': 'PASSED',
1448 'foo': 'bar', 'exit': True}
1449@@ -362,11 +367,12 @@
1450 self.assertEqual(state, data)
1451
1452 @mock.patch('lander_service_wrapper.TicketApi')
1453- @mock.patch('lander_service_wrapper._post')
1454- def testRunnerUnexpected(self, post, api):
1455+ @mock.patch('ci_utils.amqp_utils.send')
1456+ def testRunnerUnexpected(self, send, api):
1457 '''Ensure unexpected errors still generate a response'''
1458 err_msg = 'foo bar'
1459- post.side_effect = RuntimeError(err_msg)
1460+ send.side_effect = RuntimeError(err_msg)
1461+ api().get_full_ticket.return_value = {}
1462
1463 args = self.base_args + ['--service', 'test_runner']
1464 args = lander_service_wrapper._get_parser().parse_args(args)
1465
1466=== modified file 'lander/lander/tests/test_v1.py'
1467--- lander/lander/tests/test_v1.py 2014-01-02 21:42:38 +0000
1468+++ lander/lander/tests/test_v1.py 2014-05-09 16:00:26 +0000
1469@@ -1,5 +1,5 @@
1470 # Ubuntu CI Engine
1471-# Copyright 2013 Canonical Ltd.
1472+# Copyright 2014 Canonical Ltd.
1473
1474 # This program is free software: you can redistribute it and/or modify it
1475 # under the terms of the GNU Affero General Public License version 3, as
1476
1477=== modified file 'ppa-assigner/local_settings.py.example'
1478--- ppa-assigner/local_settings.py.example 2014-01-29 11:21:16 +0000
1479+++ ppa-assigner/local_settings.py.example 2014-05-09 16:00:26 +0000
1480@@ -1,5 +1,5 @@
1481 # Ubuntu CI Engine
1482-# Copyright 2013 Canonical Ltd.
1483+# Copyright 2013, 2014 Canonical Ltd.
1484
1485 # This program is free software: you can redistribute it and/or modify it
1486 # under the terms of the GNU Affero General Public License version 3, as
1487
1488=== modified file 'ppa-assigner/ppa_assigner/__init__.py'
1489--- ppa-assigner/ppa_assigner/__init__.py 2014-03-10 22:25:00 +0000
1490+++ ppa-assigner/ppa_assigner/__init__.py 2014-05-09 16:00:26 +0000
1491@@ -1,5 +1,5 @@
1492 # Ubuntu CI Engine
1493-# Copyright 2013 Canonical Ltd.
1494+# Copyright 2013, 2014 Canonical Ltd.
1495
1496 # This program is free software: you can redistribute it and/or modify it
1497 # under the terms of the GNU Affero General Public License version 3, as
1498
1499=== modified file 'ppa-assigner/ppa_assigner/launchpad.py'
1500--- ppa-assigner/ppa_assigner/launchpad.py 2014-05-07 01:13:00 +0000
1501+++ ppa-assigner/ppa_assigner/launchpad.py 2014-05-09 16:00:26 +0000
1502@@ -1,5 +1,5 @@
1503 # Ubuntu CI Engine
1504-# Copyright 2012-2013 Canonical Ltd.
1505+# Copyright 2013, 2014 Canonical Ltd.
1506
1507 # This program is free software: you can redistribute it and/or modify it
1508 # under the terms of the GNU Affero General Public License version 3, as
1509
1510=== modified file 'run-tests'
1511--- run-tests 2014-03-14 10:37:28 +0000
1512+++ run-tests 2014-05-09 16:00:26 +0000
1513@@ -283,7 +283,7 @@
1514 result = results.TextResult(stdout, verbosity=2)
1515 else:
1516 result = subunit.TestProtocolClient(stdout)
1517- # We'll run tests, start the run
1518+ # We'll run tests, setup the result accordingly
1519 result.startTestRun()
1520 # Load the tests, keeping only the ones required by the user
1521 suite = load_regular_component_tests(options.include_regexps,
1522
1523=== modified file 'tarmac.sh'
1524--- tarmac.sh 2014-03-10 22:25:00 +0000
1525+++ tarmac.sh 2014-05-09 16:00:26 +0000
1526@@ -16,7 +16,7 @@
1527
1528 # REQUIRES: python-pip python-virtualenv python-setuptools python-dev
1529
1530-export CI_DEPS_BRANCH="lp:~canonical-ci-engineering/ubuntu-ci-services-itself/deps"
1531+export CI_DEPS_BRANCH="lp:~canonical-ci-engineering/uci-engine/deps"
1532
1533 set -e
1534 if [ -z $1 ] ; then
1535
1536=== modified file 'test_runner/run_test.py'
1537--- test_runner/run_test.py 2014-03-14 10:37:28 +0000
1538+++ test_runner/run_test.py 2014-05-09 16:00:26 +0000
1539@@ -48,15 +48,13 @@
1540 os.mkdir('results')
1541
1542
1543-def get_dsc_path(package, cwd):
1544- """Returns the '.dsc' path found in the 'apt-get source' files."""
1545- # FIXME: It would be better to just use the files from the ticket but: 1)
1546- # it requires changing several components API, 2) It's not guaranteed to be
1547- # enough. apt-get source *is* guaranteed to be enough. -- vila 2014-03-11
1548+def get_src_tree_path(pacakge, cwd):
1549+ """Returns the path where 'apt-get source' did the extraction."""
1550 files = os.listdir(cwd)
1551 for f in files:
1552- if f.endswith('.dsc'):
1553- return os.path.join(cwd, f)
1554+ path = os.path.join(cwd, f)
1555+ if os.path.isdir(path):
1556+ return path
1557
1558
1559 def parse_summary_line(line):
1560@@ -137,25 +135,33 @@
1561
1562
1563 def run_test(package):
1564- output = open('result.subunit', 'wb')
1565+ subunit_path = '{}.subunit'.format(package)
1566+ output = open(subunit_path, 'wb')
1567 result = subunit.TestProtocolClient(output)
1568 result.startTestRun()
1569 os.mkdir(package)
1570 # MISSINGTEST: Unable to locate package (wrong name or whatever)
1571 # MISSINGTEST: no deb-src in source.list
1572 run('apt-get', 'source', package, cwd=package)
1573- # MISSINGTEST: dsc file not found
1574- dsc_path = get_dsc_path(package, cwd=package)
1575+ # MISSINGTEST: no dir found
1576+ src_tree_path = get_src_tree_path(package, cwd=package)
1577 try:
1578- cmd = ['sudo', 'adt-run', dsc_path,
1579- '--summary', 'summary.log',
1580+ summary_path = '{}-summary.log'.format(package)
1581+ log_path = '{}.log'.format(package)
1582+ cmd = ['sudo', 'adt-run',
1583+ # Restrictions: build-needed will override --no-built-binaries
1584+ # so it's safe to specify it.
1585+ '--no-built-binaries',
1586+ '--unbuilt-tree', src_tree_path,
1587+ '--summary', summary_path,
1588+ '--log-file', log_path,
1589 # Required to get files produced in 'results'
1590 '--paths-testbed',
1591 '--output-dir', 'results',
1592 '---', 'adt-virt-null']
1593 proc, out, err = run(*cmd, out=None, err=None, check_rc=False)
1594 rc = proc.returncode
1595- for name, status in parse_summary('summary.log'):
1596+ for name, status in parse_summary(summary_path):
1597 process_test_result(package, result, name, status)
1598 finally:
1599 result.stopTestRun()
1600
1601=== modified file 'test_runner/tstrun/__init__.py'
1602--- test_runner/tstrun/__init__.py 2014-03-10 22:25:00 +0000
1603+++ test_runner/tstrun/__init__.py 2014-05-09 16:00:26 +0000
1604@@ -19,12 +19,6 @@
1605
1606 HERE = os.path.abspath(os.path.dirname(__file__))
1607
1608-# FIXME: Everything below sounds like config material, may be they should just
1609-# be moved into their own config.py file. -- vila 2014-02-05
1610-
1611-# The test runner uses a rabbit queue for test requests.
1612-internal_rabbit_queue = 'test_runner'
1613-
1614
1615 # Creating the nova instance for the testbed and storing test results in the
1616 # data store require nova credentials.
1617
1618=== modified file 'test_runner/tstrun/resources/root.py'
1619--- test_runner/tstrun/resources/root.py 2014-01-07 18:07:34 +0000
1620+++ test_runner/tstrun/resources/root.py 2014-05-09 16:00:26 +0000
1621@@ -1,5 +1,5 @@
1622 # Ubuntu CI Engine
1623-# Copyright 2013 Canonical Ltd.
1624+# Copyright 2014 Canonical Ltd.
1625
1626 # This program is free software: you can redistribute it and/or modify it
1627 # under the terms of the GNU Affero General Public License version 3, as
1628
1629=== modified file 'test_runner/tstrun/resources/v1.py'
1630--- test_runner/tstrun/resources/v1.py 2014-03-14 10:37:28 +0000
1631+++ test_runner/tstrun/resources/v1.py 2014-05-09 16:00:26 +0000
1632@@ -26,15 +26,13 @@
1633 restish_utils,
1634 )
1635
1636-import tstrun
1637-
1638 log = logging.getLogger(__name__)
1639
1640
1641 def _status():
1642 status = json_status.JSONStatus()
1643 status.add_rabbit_configured()
1644- status.add_rabbit_worker_health(tstrun.internal_rabbit_queue)
1645+ status.add_rabbit_worker_health(amqp_utils.TEST_RUNNER_QUEUE)
1646 return restish_utils.json_ok(status.results)
1647
1648
1649@@ -49,7 +47,7 @@
1650 if res:
1651 log.error('Unable to notify progress queue, aborting test run')
1652 return http.service_unavailable(body=res)
1653- res = amqp_utils.send(tstrun.internal_rabbit_queue, json.dumps(params))
1654+ res = amqp_utils.send(amqp_utils.TEST_RUNNER_QUEUE, json.dumps(params))
1655 if res:
1656 res = http.service_unavailable(body=res)
1657 return res
1658
1659=== modified file 'test_runner/tstrun/run_worker.py'
1660--- test_runner/tstrun/run_worker.py 2014-03-14 21:54:37 +0000
1661+++ test_runner/tstrun/run_worker.py 2014-05-09 16:00:26 +0000
1662@@ -28,39 +28,60 @@
1663 super(TestRunnerWorker, self).__init__('image_test')
1664 self.data_store = None
1665
1666- def save_subunit(self, log, package, stream, results):
1667- # make this exception-safe since we can already report pass/fail
1668- # with or without this file
1669- try:
1670- log.info('Saving subunit results for {}'.format(package))
1671- name = 'test-runner.{}-subunit-stream'.format(package)
1672- url = self.data_store.put_file(name, stream, 'text/plain')
1673- results.setdefault('artifacts', []).append({
1674- 'name': name,
1675- 'reference': url,
1676- 'type': 'RESULTS',
1677- })
1678- except:
1679- log.exception(
1680- 'Unable to upload subunit result for {}'.format(package))
1681-
1682- def save_testbed_console(self, log, test_bed, results):
1683- # make this exception-safe since we can already report pass/fail
1684- # with or without this file
1685- try:
1686- log.info('Saving testbed console')
1687- name = 'test-runner.testbed-cloud-init.log'
1688- console = test_bed.get_cloud_init_console()
1689- url = self.data_store.put_file(name, console, 'text/plain')
1690- results.setdefault('artifacts', []).append({
1691- 'name': name,
1692- 'reference': url,
1693- 'type': 'LOGS',
1694- })
1695- except:
1696- log.exception('unable to upload testbed console')
1697-
1698- def handle_request(self, log, params):
1699+ def save_artifact(self, logger, results, name, value, description=None,
1700+ kind='LOGS'):
1701+ """Save an artifact catching and reporting exceptions.
1702+
1703+ This should only be used to upload artifacts after the pass/fail status
1704+ has been established to it's safe to fail the upload.
1705+
1706+ :param logger: To report execution.
1707+
1708+ :param results: The dict holding the 'artifacts' attribute.
1709+
1710+ :param name: The artifact name.
1711+
1712+ :param value: The artifact value as a string.
1713+
1714+ :param description: For reporting purposes.
1715+
1716+ :param kind: The kind of artifact (defaults to 'LOGS').
1717+ """
1718+ if description is None:
1719+ description = name
1720+ try:
1721+ logger.info('Saving {}'.format(description))
1722+ url = self.data_store.put_file(name, value, 'text/plain')
1723+ self._create_artifact('{}.{}'.format(self.logger_name, name),
1724+ url, results, kind)
1725+ except:
1726+ logger.exception('Unable to upload {}'.format(description))
1727+
1728+ def save_testbed_console(self, logger, results, test_bed):
1729+ self.save_artifact(logger, results,
1730+ 'testbed-cloud-init.log',
1731+ test_bed.get_cloud_init_console(),
1732+ 'testbed console')
1733+
1734+ def save_testbed_artifacts(self, logger, results, test_bed, package):
1735+ subunit_path = '{}.subunit'.format(package)
1736+ self.save_artifact(
1737+ logger, results,
1738+ subunit_path, test_bed.get_remote_content(subunit_path),
1739+ 'subunit results for {}'.format(package),
1740+ 'RESULTS')
1741+ package_log_path = '{}.log'.format(package)
1742+ self.save_artifact(
1743+ logger, results,
1744+ package_log_path, test_bed.get_remote_content(package_log_path),
1745+ 'adt-run log for {}'.format(package))
1746+ summary_log_path = '{}-summary.log'.format(package)
1747+ self.save_artifact(
1748+ logger, results,
1749+ summary_log_path, test_bed.get_remote_content(summary_log_path),
1750+ 'adt-run summary for {}'.format(package))
1751+
1752+ def handle_request(self, logger, params):
1753 ticket_id = params['ticket_id']
1754 progress_queue = params['progress_trigger']
1755 image_id = params['image_id']
1756@@ -69,7 +90,7 @@
1757 results = {}
1758
1759 def status_cb(msg):
1760- log.info(msg)
1761+ logger.info(msg)
1762 amqp_utils.progress_update(progress_queue, {'message': msg})
1763
1764 self.data_store = self._create_data_store(ticket_id)
1765@@ -79,41 +100,42 @@
1766 test_bed = testbed.TestBed('testbed-{}'.format(progress_queue),
1767 flavors, image_id, status_cb)
1768 except:
1769- log.exception(
1770+ logger.exception(
1771 'The testbed creation for {} failed'.format(ticket_id))
1772 return amqp_utils.progress_failed, results
1773
1774 try:
1775 test_bed.setup()
1776 except:
1777- log.exception(
1778+ logger.exception(
1779 'The testbed setup for {} failed'.format(ticket_id))
1780 if test_bed.instance is not None:
1781- self.save_testbed_console(log, test_bed, results)
1782+ self.save_testbed_console(logger, results, test_bed)
1783 test_bed.teardown()
1784 return amqp_utils.progress_failed, results
1785
1786+ self.save_testbed_console(logger, results, test_bed)
1787 status_cb('The test bed is ready')
1788 # The tests will succeed unless they fail ;)
1789 notify = amqp_utils.progress_completed
1790 try:
1791 for package in package_list:
1792 if params.get('cancelled'):
1793- log.error('The request for {} has been cancelled,'
1794- ' exiting'.format(ticket_id))
1795+ logger.error('The request for {} has been cancelled,'
1796+ ' exiting'.format(ticket_id))
1797 return amqp_utils.progress_failed, results
1798 status_cb('Testing {}'.format(package))
1799 # uci-vms shell adt-run ... --- adt-virt-null
1800- return_code, subunit_stream = test_bed.run_test(package)
1801+ return_code = test_bed.run_test(package)
1802 # 0 is success, 8 is skipped and considered a success
1803 if return_code not in (0, 8):
1804 # At least one test failed
1805 notify = amqp_utils.progress_failed
1806- self.save_subunit(log, package, subunit_stream, results)
1807+ self.save_testbed_artifacts(logger, results, test_bed, package)
1808 status_cb('Test completed for ticket {}'.format(ticket_id))
1809 return notify, results
1810 except:
1811- log.exception(
1812+ logger.exception(
1813 'Exception while handling ticket {}'.format(ticket_id))
1814 raise
1815 finally:
1816@@ -121,4 +143,4 @@
1817
1818
1819 if __name__ == '__main__':
1820- TestRunnerWorker().main(tstrun.internal_rabbit_queue)
1821+ TestRunnerWorker().main(amqp_utils.TEST_RUNNER_QUEUE)
1822
1823=== modified file 'test_runner/tstrun/testbed.py'
1824--- test_runner/tstrun/testbed.py 2014-03-14 17:57:46 +0000
1825+++ test_runner/tstrun/testbed.py 2014-05-09 16:00:26 +0000
1826@@ -31,7 +31,7 @@
1827 log = logging.getLogger(__name__)
1828
1829
1830-class NovaClientException(Exception):
1831+class TestBedException(Exception):
1832 pass
1833
1834
1835@@ -179,7 +179,7 @@
1836 pass
1837 # MISSINGTEST: some cloud doesn't provide one of our expected flavors
1838 # -- vila 2014-01-06
1839- raise NovaClientException(
1840+ raise TestBedException(
1841 'None of {} can be found'.format(self.flavors))
1842
1843 def setup(self):
1844@@ -235,9 +235,9 @@
1845 time.sleep(5)
1846 # MISSINGTEST: What if the instance doesn't come up after max_tries ?
1847 if status == 'BUILD':
1848- raise NovaClientException('Instance never came up')
1849+ raise TestBedException('Instance never came up')
1850 else:
1851- raise NovaClientException('Instance never provided an IP')
1852+ raise TestBedException('Instance never provided an IP')
1853
1854 def get_cloud_init_console(self, length=None):
1855 return self.instance.get_console_output(length)
1856@@ -245,13 +245,22 @@
1857 def wait_for_cloud_init(self):
1858 # FIXME: cloud_init_timeout should be a config option (related to
1859 # get_ip_timeout and probably the two can be merged) -- vila 2014-01-30
1860- cloud_init_timeout = 300 # in seconds so 5 minutes
1861+ cloud_init_timeout = 600 # in seconds so 10 minutes
1862 timeout_limit = time.time() + cloud_init_timeout
1863 while time.time() < timeout_limit:
1864 # A relatively cheap way to catch cloud-init completion is to watch
1865 # the console for the specific message we specified in user-data).
1866 try:
1867 console = self.get_cloud_init_console(10)
1868+ except exceptions.OverLimit:
1869+ # This happens rarely but breaks badly if not caught. elmo
1870+ # recommended a 30 seconds nap in that case.
1871+ nap_time = 30
1872+ msg = ('Rate limit while acquiring nova console for {},'
1873+ ' will sleep for {} seconds')
1874+ log.exception(msg.format(self.instance.id, nap_time))
1875+ time.sleep(nap_time)
1876+ continue
1877 except requests.ConnectionError:
1878 # The reported status is the one known before the last attempt.
1879 log.warn('Received connection error for {},'
1880@@ -262,11 +271,9 @@
1881 # We're good to go
1882 log.info(
1883 'cloud-init completed for {}'.format(self.instance.id))
1884- # FIXME: Right place to report how long it
1885- # took for cloud-init to finish. -- vila 2014-01-30
1886 return
1887 time.sleep(5)
1888- raise NovaClientException('Instance never completed cloud-init')
1889+ raise TestBedException('Instance never completed cloud-init')
1890
1891 def teardown(self):
1892 self.nova.servers.delete(self.instance.id)
1893@@ -295,12 +302,18 @@
1894 out, err = proc.communicate()
1895 return proc, out, err
1896
1897+ def get_remote_content(self, path):
1898+ proc, content, err = self.ssh('cat', path, out=subprocess.PIPE,
1899+ err=subprocess.PIPE)
1900+ if proc.returncode:
1901+ # We didn't get a proper content, report it instead
1902+ content = ("{} couldn't be copied from testbed {}:\n"
1903+ "error: {}\n".format(path, self.ip, err))
1904+ return content
1905+
1906 def run_test(self, package):
1907 proc, _, _ = self.ssh('/tmp/run_test.py', package)
1908- # FIXME: Does that look like a pipe ? -- vila 2014-02-06
1909- _, subunit_stream, _ = self.ssh('cat', 'result.subunit',
1910- out=subprocess.PIPE)
1911- return proc.returncode, subunit_stream
1912+ return proc.returncode
1913
1914
1915 # The following are helpers for local manual tests, they should disappear once
1916@@ -345,6 +358,10 @@
1917 from ci_utils import dump_stack
1918 dump_stack.install_stack_dump_signal()
1919 test_bed = test_print_ip()
1920- print test_bed.run_test('libpng') # Known to run no tests
1921- print test_bed.run_test('juju-core')
1922+ # libpng is known to run no tests
1923+ for package in ('libpng',):
1924+ print test_bed.run_test(package)
1925+# print test_bed.get_remote_content('{}.log'.format(package))
1926+ print test_bed.get_remote_content('{}-summary.log'.format(package))
1927+ print test_bed.get_remote_content('{}.subunit'.format(package))
1928 test_bed.teardown()
1929
1930=== modified file 'test_runner/tstrun/wsgi.py'
1931--- test_runner/tstrun/wsgi.py 2014-01-07 18:07:34 +0000
1932+++ test_runner/tstrun/wsgi.py 2014-05-09 16:00:26 +0000
1933@@ -1,6 +1,6 @@
1934 #!/usr/bin/env python
1935 # Ubuntu CI Engine
1936-# Copyright 2013 Canonical Ltd.
1937+# Copyright 2014 Canonical Ltd.
1938
1939 # This program is free software: you can redistribute it and/or modify it
1940 # under the terms of the GNU Affero General Public License version 3, as
1941
1942=== modified file 'tests/image_builder/test.py'
1943--- tests/image_builder/test.py 2014-02-19 23:49:36 +0000
1944+++ tests/image_builder/test.py 2014-05-09 16:00:26 +0000
1945@@ -43,7 +43,9 @@
1946 },
1947 'ppa_list': ['ppa:fakeproject/test'],
1948 'package_list': ['fakepackage'],
1949- 'progress_trigger': 'triggername'
1950+ 'progress_trigger': 'triggername',
1951+ 'ticket_id': 'ticket-1',
1952+ 'cancel_url': 'http://bar.baz/cancel'
1953 }
1954 resp, content = self.post_build_image(body)
1955 self.assertEqual('204', resp['status'])
1956@@ -59,11 +61,16 @@
1957 resp, content = self.get_status()
1958 self.assertEqual('200', resp['status'])
1959 data = json.loads(content)
1960- self.assertEqual(True, data['rabbit_configured'])
1961+ self.assertEqual([{'label': 'rabbit configured', 'status': 'okay',
1962+ 'value': True},
1963+ {'label': 'workers-online', 'status': 'okay',
1964+ 'value': 1}],
1965+ data)
1966
1967 def test_worker_running(self):
1968 '''ensure the rabbit worker is deployed and running'''
1969 self.assert_job_running('imagebuild_worker', 'imagebuild-worker')
1970
1971+
1972 if __name__ == "__main__":
1973 unittest.main()
1974
1975=== modified file 'tests/run.py'
1976--- tests/run.py 2014-03-10 22:25:00 +0000
1977+++ tests/run.py 2014-05-09 16:00:26 +0000
1978@@ -236,15 +236,20 @@
1979
1980
1981 def needs_tunnel():
1982- """Check to see if juju's Swift bucket exists, and the IP it refers to is
1983- pointed at by a sshuttle instance."""
1984+ """Check to see if a tunnel is required and up.
1985
1986- log.info("Checking to see if a tunnel is needed.")
1987+ When required, it depends on juju's Swift bucket existing, and the IP it
1988+ refers to is pointed at by a sshuttle instance.
1989+ """
1990+ log.info("Checking to see if a tunnel needs to be setup.")
1991
1992 instance = get_bootstrap_nova_instance()
1993 ip_addr = get_bootstrap_ip(instance, 1)
1994- sshuttle_pid = _get_pid_of_sshuttle(ip_addr)
1995- return sshuttle_pid < 0
1996+ if private_IP(ip_addr):
1997+ sshuttle_pid = _get_pid_of_sshuttle(ip_addr)
1998+ return sshuttle_pid < 0
1999+ else:
2000+ return False
2001
2002
2003 def _connect_to_swift():
2004@@ -322,11 +327,34 @@
2005 return state['state-instances'][0]
2006
2007
2008+def private_IP(addr):
2009+ """Check whether an IP address can be routed.
2010+ :param addr: The IP address as a string.
2011+
2012+ :returns: True if the address is private (can't be routed). False
2013+ otherwise.
2014+
2015+ """
2016+ if addr.startswith('10.'): # 10.0.0.0 - 10.255.255.255
2017+ return True
2018+ elif addr.startswith('192.168'): # 192.168.0.0 - 192.168.255.255
2019+ return True
2020+ elif addr.startswith('172.'): # 172.16.0.0 - 172.31.255.255
2021+ a, b, c, d = addr.split('.')
2022+ if 16 <= int(b) <= 31:
2023+ return True
2024+ return False
2025+
2026+
2027 def get_bootstrap_ip(instance, max_tries=120):
2028- """Wait for the juju bootstrap node to boot and obtain an IP address so we
2029- can tunnel through it.
2030-
2031- Raises NotFound if the instance has not been spawned."""
2032+ """Wait for the juju bootstrap node to boot and obtain an IP address.
2033+
2034+ This is a pre-requisite to setup a tunnel.
2035+
2036+ :raises: NotFound if the instance has not been spawned.
2037+
2038+ :raises: Exception if no IP can be found after max_tries*3 seconds.
2039+ """
2040
2041 log.info("Getting the bootstrap IP address.")
2042
2043@@ -336,24 +364,25 @@
2044 'service_type': 'compute'}
2045 nt = client.Client(*args, **kwargs)
2046
2047- net = ''
2048 tries = 0
2049 while tries < max_tries:
2050 # Raises NotFound if the instance does not exist.
2051 server = nt.servers.get(instance)
2052- net = server.networks.get('canonistack')
2053- if net:
2054- net = net[0]
2055- # Don't sleep anymore. We have our IP; break out of the loop.
2056- break
2057+ networks = server.networks.values()
2058+ if networks:
2059+ # We use a single network and don't care how it's named
2060+ net = networks[0]
2061+ if net:
2062+ # Either we have a single address or the last one has been
2063+ # added to make the instance reachable (floating or public).
2064+ net = net[-1]
2065+ # Don't sleep anymore. We have our IP, let's move on
2066+ return net
2067
2068 tries += 1
2069 if tries != max_tries:
2070 time.sleep(3)
2071- else:
2072- raise Exception('Timed out waiting for bootstrap IP.')
2073-
2074- return net
2075+ raise Exception('Timed out waiting for bootstrap IP.')
2076
2077
2078 def destroy_tunnel():
2079@@ -440,13 +469,16 @@
2080
2081
2082 def check_nova_env():
2083- """Ensure we have the right keys set in the environment for talking to
2084- Canonistack."""
2085+ """Ensure we have the right keys set in the environment.
2086
2087- if os.environ.get('OS_USERNAME'):
2088- return True
2089- else:
2090- return False
2091+ We assume that the targeted cloud is openstack based.
2092+ """
2093+ for key in ('OS_USERNAME', 'OS_PASSWORD', 'OS_AUTH_URL'):
2094+ if os.environ.get(key, None) is None:
2095+ msg = ('Please source a novarc file before running this harness.'
2096+ ' {} is not currently set'.format(key))
2097+ log.error(msg)
2098+ sys.exit(1)
2099
2100
2101 def check_juju_version():
2102@@ -473,22 +505,20 @@
2103
2104
2105 def main():
2106+ check_nova_env()
2107+
2108 test_dir = os.path.dirname(sys.argv[0])
2109 deploy.check_environment()
2110 tests = get_tests(test_dir)
2111 if not tests:
2112- sys.exit(0)
2113+ # Running no tests is an error
2114+ sys.exit(1)
2115
2116 if not check_juju_version():
2117 msg = 'Please install a more recent version of juju (1.16.5 or later).'
2118 log.error(msg)
2119 sys.exit(1)
2120
2121- if not check_nova_env():
2122- msg = 'Please source a novarc file before running this harness.'
2123- log.error(msg)
2124- sys.exit(1)
2125-
2126 if not check_sudoers():
2127 msg = 'Please run:\necho "$USER %s" > %s' % (SUDOERS, SUDOERS_FILE)
2128 log.error(msg)
2129@@ -508,8 +538,7 @@
2130
2131 if needs_bootstrap():
2132 bootstrap()
2133- wait_then_tunnel()
2134- elif needs_tunnel():
2135+ if needs_tunnel():
2136 wait_then_tunnel()
2137
2138 destroy_services()
2139@@ -524,7 +553,9 @@
2140 try:
2141 subprocess.check_call(['python', test])
2142 except subprocess.CalledProcessError as e:
2143- returncode = e.returncode
2144+ # Accumulate all errors as any test failure should fail the whole
2145+ # run
2146+ returncode = returncode + e.returncode
2147
2148 # Let's see if there are some failed instances that we can gather
2149 # logs from.
2150
2151=== modified file 'tests/test_run.py'
2152--- tests/test_run.py 2014-03-10 22:25:00 +0000
2153+++ tests/test_run.py 2014-05-09 16:00:26 +0000
2154@@ -16,6 +16,7 @@
2155 from ucitests import fixtures
2156
2157
2158+# Disable all (almost) log output from the tests and the code they run
2159 logging.disable(logging.CRITICAL)
2160
2161
2162@@ -137,5 +138,25 @@
2163 result = [x for x in run.find_failed_units()]
2164 self.assertEqual(result, ['ci-juju-gui/0'])
2165
2166+
2167+class TestPrivateIP(unittest.TestCase):
2168+
2169+ def test_public(self):
2170+ self.assertFalse(run.private_IP('2.2.2.2'))
2171+
2172+ def test_class_A_private(self):
2173+ self.assertTrue(run.private_IP('10.0.4.5'))
2174+
2175+ def test_class_B_public(self):
2176+ self.assertFalse(run.private_IP('172.15.2.245'))
2177+
2178+ def test_class_B_private(self):
2179+ self.assertTrue(run.private_IP('172.24.12.36'))
2180+
2181+ def test_class_C_private(self):
2182+ self.assertTrue(run.private_IP('192.168.0.16'))
2183+
2184+
2185+
2186 if __name__ == '__main__':
2187 unittest.main()
2188
2189=== modified file 'tests/test_runner/test.py'
2190--- tests/test_runner/test.py 2014-03-14 10:37:28 +0000
2191+++ tests/test_runner/test.py 2014-05-09 16:00:26 +0000
2192@@ -26,6 +26,7 @@
2193 'image_id': image_id,
2194 'package_list': package_list,
2195 'progress_trigger': progress_trigger,
2196+ 'cancel_url': 'http://bar.baz/cancel',
2197 }),
2198 )
2199 return resp, content
2200
2201=== modified file 'tests/ticket_system/test.py'
2202--- tests/ticket_system/test.py 2014-01-29 17:27:07 +0000
2203+++ tests/ticket_system/test.py 2014-05-09 16:00:26 +0000
2204@@ -97,5 +97,6 @@
2205 self.assertEqual('200', resp['status'])
2206 self.assertEqual('before_change@example.com', content['owner'])
2207
2208+
2209 if __name__ == "__main__":
2210 unittest.main()
2211
2212=== modified file 'ticket_system/project/admin.py'
2213--- ticket_system/project/admin.py 2014-01-21 22:37:56 +0000
2214+++ ticket_system/project/admin.py 2014-05-09 16:00:26 +0000
2215@@ -1,5 +1,5 @@
2216 # Ubuntu Continuous Integration Engine
2217-# Copyright 2013 Canonical Ltd.
2218+# Copyright 2013, 2014 Canonical Ltd.
2219
2220 # This program is free software: you can redistribute it and/or modify it
2221 # under the terms of the GNU Affero General Public License version 3, as
2222
2223=== modified file 'ticket_system/project/api.py'
2224--- ticket_system/project/api.py 2014-01-21 22:37:56 +0000
2225+++ ticket_system/project/api.py 2014-05-09 16:00:26 +0000
2226@@ -1,5 +1,5 @@
2227 # Ubuntu Continuous Integration Engine
2228-# Copyright 2013 Canonical Ltd.
2229+# Copyright 2013, 2014 Canonical Ltd.
2230
2231 # This program is free software: you can redistribute it and/or modify it
2232 # under the terms of the GNU Affero General Public License version 3, as
2233
2234=== modified file 'ticket_system/project/models.py'
2235--- ticket_system/project/models.py 2014-01-23 20:39:24 +0000
2236+++ ticket_system/project/models.py 2014-05-09 16:00:26 +0000
2237@@ -1,5 +1,5 @@
2238 # Ubuntu Continuous Integration Engine
2239-# Copyright 2013 Canonical Ltd.
2240+# Copyright 2013, 2014 Canonical Ltd.
2241
2242 # This program is free software: you can redistribute it and/or modify it
2243 # under the terms of the GNU Affero General Public License version 3, as
2244
2245=== modified file 'ticket_system/ticket/admin.py'
2246--- ticket_system/ticket/admin.py 2014-01-24 16:39:35 +0000
2247+++ ticket_system/ticket/admin.py 2014-05-09 16:00:26 +0000
2248@@ -1,5 +1,5 @@
2249 # Ubuntu Continuous Integration Engine
2250-# Copyright 2013 Canonical Ltd.
2251+# Copyright 2013, 2014 Canonical Ltd.
2252
2253 # This program is free software: you can redistribute it and/or modify it
2254 # under the terms of the GNU Affero General Public License version 3, as
2255
2256=== modified file 'ticket_system/ticket/tests/test_write_api.py'
2257--- ticket_system/ticket/tests/test_write_api.py 2014-03-14 10:37:28 +0000
2258+++ ticket_system/ticket/tests/test_write_api.py 2014-05-09 16:00:26 +0000
2259@@ -1,5 +1,5 @@
2260 # Ubuntu Continuous Integration Engine
2261-# Copyright 2013 Canonical Ltd.
2262+# Copyright 2013, 2014 Canonical Ltd.
2263
2264 # This program is free software: you can redistribute it and/or modify it
2265 # under the terms of the GNU Affero General Public License version 3, as
2266
2267=== modified file 'ticket_system/ticket_system/__init__.py'
2268--- ticket_system/ticket_system/__init__.py 2014-03-10 22:25:00 +0000
2269+++ ticket_system/ticket_system/__init__.py 2014-05-09 16:00:26 +0000
2270@@ -1,5 +1,5 @@
2271 # Ubuntu Continuous Integration Engine
2272-# Copyright 2013 Canonical Ltd.
2273+# Copyright 2013, 2014 Canonical Ltd.
2274
2275 # This program is free software: you can redistribute it and/or modify it
2276 # under the terms of the GNU Affero General Public License version 3, as

Subscribers

People subscribed via source and target branches