Merge lp:~psivaa/uci-engine/image-watcher into lp:uci-engine

Proposed by Para Siva on 2015-01-15
Status: Merged
Approved by: Para Siva on 2015-01-17
Approved revision: 931
Merged at revision: 923
Proposed branch: lp:~psivaa/uci-engine/image-watcher
Merge into: lp:uci-engine
Diff against target: 335 lines (+302/-0)
6 files modified
image-watcher/imagewatcher/__init__.py (+15/-0)
image-watcher/imagewatcher/queues.py (+34/-0)
image-watcher/imagewatcher/tests/test_trigger_process.py (+43/-0)
image-watcher/imagewatcher/tests/test_watch_image.py (+58/-0)
image-watcher/imagewatcher/trigger_process.py (+29/-0)
image-watcher/imagewatcher/watch_image.py (+123/-0)
To merge this branch: bzr merge lp:~psivaa/uci-engine/image-watcher
Reviewer Review Type Date Requested Status
Paul Larson Approve on 2015-01-16
Celso Providelo (community) 2015-01-15 Approve on 2015-01-15
Review via email: mp+246552@code.launchpad.net

Commit message

Image watcher to trigger the workflow for when a new image available in http://cloud-images.ubuntu.com/

Description of the change

Image watcher to trigger when there is a new image present in http://cloud-images.ubuntu.com/ubuntu-core/devel. The script checks the content of http://cloud-images.ubuntu.com/ubuntu-core/devel/core/current/build-info.txt to find if there is a new version in the server.

In terms of message passing to a queue, once we find a new image in the server we provide the version, channel and the test branch in the snappy.images queue for the next service to consume.

To post a comment you must log in.
lp:~psivaa/uci-engine/image-watcher updated on 2015-01-15
927. By Para Siva on 2015-01-15

Revert the unneeded amqp_utils change

928. By Para Siva on 2015-01-15

add a test to make sure the process triggers when started afresh

Celso Providelo (cprov) wrote :

Psivaa,

Thanks for working on this script.

I have nothing against monitoring build-info.txt instead of using simplestreams (python-simplestreams), as long as it works, as it seems to be. We can invest time later to rewrite the whole service to benefit of simplestreams goodies (supports the whole procedure from stream to glance).

There are some questions and remarks inline, specially regarding the topology of the new CLIs and how the charm is supposed to deploy them and its implication on configuration.

review: Needs Information
lp:~psivaa/uci-engine/image-watcher updated on 2015-01-15
929. By Para Siva on 2015-01-15

Review comment fixes, do not use specific config, and always use run-python

Para Siva (psivaa) wrote :

Thanks cprov for the comments.

I have removed using a special config file and relying on the amqp_utils.get_config() for the config information. For this as you pointed out, amqp-relation hook needs to be implemented in the cron-cmd charm.

Thanks again.

Celso Providelo (cprov) wrote :

Psivaa,

Thanks for the cleanup, I have just a minor suggestion about allowing image_watch.main() to be properly tested, but it's not a blocker.

Don't forget to coordinate with Joe for supporting amqp-relation in cron-cmd charm.

review: Approve
Paul Larson (pwlars) wrote :

> Psivaa,
>
> Thanks for working on this script.
>
> I have nothing against monitoring build-info.txt instead of using
> simplestreams (python-simplestreams), as long as it works, as it seems to be.
> We can invest time later to rewrite the whole service to benefit of
> simplestreams goodies (supports the whole procedure from stream to glance).
This is actually possible now, we discussed yesterday and I think we all agreed *not* to do that though right?

Paul Larson (pwlars) wrote :

Celso mentioned (if I understood correctly) in a MP for me that we should take all of these services for ubuntucore/snappy and organize them under a single directory rather than having an individual directory for each one off the root of our tree. I think this probably makes sense because they are all related to ubuntucore/snappy, rather than, for instance, a generic image watcher that may have to monitor some other type of source and handle things differently.

Some additional comments below

review: Needs Fixing
Celso Providelo (cprov) wrote :

Psivaa, Paul,

I retracted my concerns about having one directory per service (CLIs + worker python module), it the right direction.

With that tree organisation we run CLIs locally or via with `run-python.py` and properly configure rabbit-worker. To me, it looks perfectly fine and acceptable and if it needs rework we can deal with it later.

[]

review: Approve
lp:~psivaa/uci-engine/image-watcher updated on 2015-01-16
930. By Para Siva on 2015-01-16

Review comment fix, account for empty version number in the server

931. By Para Siva on 2015-01-16

Remove pep8 and pyflake testing locally

Para Siva (psivaa) wrote :

Thanks,
@plars: I have addressed your comments inline. and fixed where appropriate. About checking the latest against the glance image-list, we need to have a uniquely corresponding image in glance to the image in cloud-image. Pending that, I believe we only could check against a local cache. Please let me know what you think. Thanks

Paul Larson (pwlars) wrote :

I'm fine if we sort out that detail later. I believe we can just call the glance image whatever we want though. The image we download (file) will be something like: devel-core-amd64-disk1.img so maybe we could call it something like devel-core-amd64-${imageid} (ex. devel-core-amd64-20150115.1)

In light of the shift of direction though, I think I'd rather see this get merged and we can sort out those details later. It's easy enough to change things once they are in place, and you've done a great job of getting the ball rolling with this. Thanks for working on this!

review: Approve

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1=== added directory 'image-watcher'
2=== added directory 'image-watcher/imagewatcher'
3=== added file 'image-watcher/imagewatcher/__init__.py'
4--- image-watcher/imagewatcher/__init__.py 1970-01-01 00:00:00 +0000
5+++ image-watcher/imagewatcher/__init__.py 2015-01-16 11:11:59 +0000
6@@ -0,0 +1,15 @@
7+#!/usr/bin/env python
8+# Ubuntu CI Engine
9+# Copyright 2015 Canonical Ltd.
10+
11+# This program is free software: you can redistribute it and/or modify it
12+# under the terms of the GNU Affero General Public License version 3, as
13+# published by the Free Software Foundation.
14+
15+# This program is distributed in the hope that it will be useful, but
16+# WITHOUT ANY WARRANTY; without even the implied warranties of
17+# MERCHANTABILITY, SATISFACTORY QUALITY, or FITNESS FOR A PARTICULAR
18+# PURPOSE. See the GNU Affero General Public License for more details.
19+
20+# You should have received a copy of the GNU Affero General Public License
21+# along with this program. If not, see <http://www.gnu.org/licenses/>.
22
23=== added file 'image-watcher/imagewatcher/queues.py'
24--- image-watcher/imagewatcher/queues.py 1970-01-01 00:00:00 +0000
25+++ image-watcher/imagewatcher/queues.py 2015-01-16 11:11:59 +0000
26@@ -0,0 +1,34 @@
27+# Ubuntu CI Engine
28+# Copyright 2014 Canonical Ltd.
29+
30+# This program is free software: you can redistribute it and/or modify it
31+# under the terms of the GNU Affero General Public License version 3, as
32+# published by the Free Software Foundation.
33+
34+# This program is distributed in the hope that it will be useful, but
35+# WITHOUT ANY WARRANTY; without even the implied warranties of
36+# MERCHANTABILITY, SATISFACTORY QUALITY, or FITNESS FOR A PARTICULAR
37+# PURPOSE. See the GNU Affero General Public License for more details.
38+
39+# You should have received a copy of the GNU Affero General Public License
40+# along with this program. If not, see <http://www.gnu.org/licenses/>.
41+
42+import json
43+
44+
45+from ci_utils import amqp_utils
46+
47+
48+class PublisherQueue(object):
49+ """A proxy to the queue library.
50+
51+ This implementation targets ci_utils/amqp_utils but can be replaced for
52+ tests.
53+ """
54+
55+ def __init__(self, queue_name):
56+ self.queue_name = queue_name
57+
58+ def publish(self, msg):
59+ amqp_utils.send(self.queue_name, json.dumps(msg),
60+ raise_errors=True)
61
62=== added directory 'image-watcher/imagewatcher/tests'
63=== added file 'image-watcher/imagewatcher/tests/__init__.py'
64=== added file 'image-watcher/imagewatcher/tests/test_trigger_process.py'
65--- image-watcher/imagewatcher/tests/test_trigger_process.py 1970-01-01 00:00:00 +0000
66+++ image-watcher/imagewatcher/tests/test_trigger_process.py 2015-01-16 11:11:59 +0000
67@@ -0,0 +1,43 @@
68+# Copyright 2015 Canonical Ltd.
69+
70+# This program is free software: you can redistribute it and/or modify it
71+# under the terms of the GNU Affero General Public License version 3, as
72+# published by the Free Software Foundation.
73+
74+# This program is distributed in the hope that it will be useful, but
75+# WITHOUT ANY WARRANTY; without even the implied warranties of
76+# MERCHANTABILITY, SATISFACTORY QUALITY, or FITNESS FOR A PARTICULAR
77+# PURPOSE. See the GNU Affero General Public License for more details.
78+
79+# You should have received a copy of the GNU Affero General Public License
80+# along with this program. If not, see <http://www.gnu.org/licenses/>.
81+
82+import unittest
83+
84+
85+from ucitests import assertions
86+
87+
88+from imagewatcher import trigger_process
89+from ci_utils.testing import fixtures
90+
91+
92+class TestPost(unittest.TestCase):
93+
94+ def test_trigger_simple(self):
95+ queue = fixtures.FakeTaskQueue('that_queue')
96+ trigger_process.post(queue, 111, 'ubuntu-core/devel', 'lp:fake')
97+ assertions.assertLength(self, 1, queue.msgs)
98+ self.assertEqual(dict(channel='ubuntu-core/devel', revision=111,
99+ test_branch='lp:fake'),
100+ queue.msgs[0])
101+
102+ def test_trigger_with_output(self):
103+ queue = fixtures.FakeTaskQueue('that_queue')
104+ trigger_process.post(queue, 111, 'ubuntu-core/devel', 'lp:fake',
105+ 'this_queue')
106+ assertions.assertLength(self, 1, queue.msgs)
107+ self.assertEqual(dict(channel='ubuntu-core/devel', revision=111,
108+ test_branch='lp:fake',
109+ output_queue='this_queue'),
110+ queue.msgs[0])
111
112=== added file 'image-watcher/imagewatcher/tests/test_watch_image.py'
113--- image-watcher/imagewatcher/tests/test_watch_image.py 1970-01-01 00:00:00 +0000
114+++ image-watcher/imagewatcher/tests/test_watch_image.py 2015-01-16 11:11:59 +0000
115@@ -0,0 +1,58 @@
116+# Ubuntu CI Engine
117+# Copyright 2015 Canonical Ltd.
118+
119+# This program is free software: you can redistribute it and/or modify it
120+# under the terms of the GNU Affero General Public License version 3, as
121+# published by the Free Software Foundation.
122+
123+# This program is distributed in the hope that it will be useful, but
124+# WITHOUT ANY WARRANTY; without even the implied warranties of
125+# MERCHANTABILITY, SATISFACTORY QUALITY, or FITNESS FOR A PARTICULAR
126+# PURPOSE. See the GNU Affero General Public License for more details.
127+
128+# You should have received a copy of the GNU Affero General Public License
129+# along with this program. If not, see <http://www.gnu.org/licenses/>.
130+
131+import unittest
132+import os
133+
134+from imagewatcher import watch_image
135+
136+
137+class TestWatchImage(unittest.TestCase):
138+ def setUp(self):
139+ super(TestWatchImage, self).setUp()
140+ self.fake_resp = [u'VERSION=112',
141+ u'BUILD_TYPE=ubuntu-core/devel',
142+ u'BUILD_DATE=20150109',
143+ u'IMG_FORMAT=qcow2',
144+ u'IMG_COMPAT=0.10',
145+ u'DEVICE=generic_amd64',
146+ u'SERIAL=20150109.2',
147+ u'BUILD_JOB_ID=35',
148+ u'']
149+
150+ def test_record_and_return_latest_version(self):
151+ with open('/tmp/fake', 'w') as f:
152+ f.write('111')
153+ ret_val = watch_image.record_and_return('/tmp/fake', self.fake_resp)
154+ os.remove('/tmp/fake')
155+ self.assertEquals(ret_val, ('112', u'ubuntu-core/devel'))
156+
157+ def test_do_not_record_and_return_non_latest(self):
158+ with open('/tmp/fake2', 'w') as f:
159+ f.write('113')
160+ ret_val = watch_image.record_and_return('/tmp/fake2', self.fake_resp)
161+ os.remove('/tmp/fake2')
162+ self.assertEquals(ret_val, None)
163+
164+ def test_freshly_record_and_return_latest_version(self):
165+ if os.path.exists('/tmp/fake3'):
166+ os.remove('/tmp/fake3')
167+ ret_val = watch_image.record_and_return('/tmp/fake3', self.fake_resp)
168+ os.remove('/tmp/fake3')
169+ self.assertEquals(ret_val, ('112', u'ubuntu-core/devel'))
170+
171+ def test_get_latest_from_server_failed(self):
172+ with self.assertRaises(watch_image.ImageWatcherError):
173+ watch_image._get_latest_from_server('http://fake', retry=1)
174
175=== added file 'image-watcher/imagewatcher/trigger_process.py'
176--- image-watcher/imagewatcher/trigger_process.py 1970-01-01 00:00:00 +0000
177+++ image-watcher/imagewatcher/trigger_process.py 2015-01-16 11:11:59 +0000
178@@ -0,0 +1,29 @@
179+#!/usr/bin/env python
180+# Ubuntu CI Engine
181+# Copyright 2015 Canonical Ltd.
182+
183+# This program is free software: you can redistribute it and/or modify it
184+# under the terms of the GNU Affero General Public License version 3, as
185+# published by the Free Software Foundation.
186+
187+# This program is distributed in the hope that it will be useful, but
188+# WITHOUT ANY WARRANTY; without even the implied warranties of
189+# MERCHANTABILITY, SATISFACTORY QUALITY, or FITNESS FOR A PARTICULAR
190+# PURPOSE. See the GNU Affero General Public License for more details.
191+
192+# You should have received a copy of the GNU Affero General Public License
193+# along with this program. If not, see <http://www.gnu.org/licenses/>.
194+
195+from imagewatcher import queues
196+
197+
198+def post(queue, revision, channel, test_branch, output=None):
199+ msg = dict(revision=revision, channel=channel, test_branch=test_branch)
200+ if output is not None:
201+ msg['output_queue'] = output
202+ queue.publish(msg)
203+
204+
205+def trigger(revision, channel, test_branch, output_queue):
206+ queue = queues.PublisherQueue('snappy.images')
207+ post(queue, revision, channel, test_branch, output_queue)
208
209=== added file 'image-watcher/imagewatcher/watch_image.py'
210--- image-watcher/imagewatcher/watch_image.py 1970-01-01 00:00:00 +0000
211+++ image-watcher/imagewatcher/watch_image.py 2015-01-16 11:11:59 +0000
212@@ -0,0 +1,123 @@
213+#!/usr/bin/env python
214+# Ubuntu CI Engine
215+# Copyright 2015 Canonical Ltd.
216+
217+# This program is free software: you can redistribute it and/or modify it
218+# under the terms of the GNU Affero General Public License version 3, as
219+# published by the Free Software Foundation.
220+
221+# This program is distributed in the hope that it will be useful, but
222+# WITHOUT ANY WARRANTY; without even the implied warranties of
223+# MERCHANTABILITY, SATISFACTORY QUALITY, or FITNESS FOR A PARTICULAR
224+# PURPOSE. See the GNU Affero General Public License for more details.
225+
226+# You should have received a copy of the GNU Affero General Public License
227+# along with this program. If not, see <http://www.gnu.org/licenses/>.
228+
229+import argparse
230+import logging
231+import requests
232+import time
233+import os
234+import sys
235+
236+from uciconfig import errors
237+
238+from imagewatcher import (
239+ trigger_process,
240+)
241+
242+log = logging.getLogger('ImageWatcher')
243+
244+
245+class ImageWatcherError(Exception):
246+ """Raised if there was an error watching the new image."""
247+
248+
249+def _get_args(args=None):
250+ parser = argparse.ArgumentParser(description='Check if a new core-image is\
251+ available at a given url')
252+ parser.add_argument(
253+ "-u", "--url",
254+ default="http://cloud-images.ubuntu.com/ubuntu-core/devel/"
255+ "core/current/build-info.txt",
256+ help="URL to check if a new core image available"),
257+ parser.add_argument(
258+ "-l", "--location",
259+ default="/tmp/latest-core-image-version",
260+ help="The file containing latest image version")
261+ parser.add_argument(
262+ "-t", "--test-branch",
263+ default="lp:snappy-tests",
264+ help="Branch where the dep8 tests live")
265+ parser.add_argument(
266+ "-o", "--output-queue",
267+ default=None,
268+ help='Send progress/results from the image watcher to the queue'
269+ ' named output_queue.')
270+ return parser.parse_args(args)
271+
272+
273+def _get_latest_from_server(url, retry=3):
274+ try:
275+ response = requests.get(url)
276+ response.raise_for_status()
277+ except (requests.ConnectionError, requests.HTTPError) as e:
278+ if retry == 1:
279+ raise ImageWatcherError(
280+ 'Error downloading the metadata {}'.format(str(e)))
281+ log.warn('Page download failed, retrying in 30 sec')
282+ time.sleep(30)
283+ return _get_latest_from_server(url, retry - 1)
284+ return response.text.split('\n')
285+
286+
287+def record_and_return(location, response):
288+ content = {}
289+ for line in response:
290+ if '=' in line:
291+ k, v = line.strip().split('=')
292+ content[k.strip()] = v.strip()
293+ if content['VERSION'] == '':
294+ log.error('The version number in the cloud-image server'
295+ ' is not an integer but {}'.format(content['VERSION']))
296+ sys.exit(1)
297+
298+ version = content['VERSION']
299+ channel = content['BUILD_TYPE']
300+ try:
301+ if os.path.exists(location):
302+ with open(location, 'r') as cache:
303+ if (cache.read() < version):
304+ with open(location, 'w') as f:
305+ f.write(version)
306+ return version, channel
307+ else:
308+ with open(location, 'w') as f:
309+ f.write(version)
310+ return version, channel
311+ except IOError as e:
312+ log.error('Writing the latest image info failed')
313+ raise ImageWatcherError(
314+ 'Error writing the latest version with {}'.format(str(e)))
315+
316+
317+def main(args=None):
318+ args = _get_args(args=args)
319+ try:
320+ ret = record_and_return(
321+ args.location,
322+ _get_latest_from_server(args.url))
323+ if ret:
324+ trigger_process.trigger(ret[0],
325+ ret[1],
326+ args.test_branch,
327+ args.output_queue)
328+ else:
329+ log.info('No new image found at {}'.format(time.asctime()))
330+ except errors.ConfigError as e:
331+ log.error('Triggering the workflow failed with {}'.format(e))
332+ sys.exit(1)
333+
334+if __name__ == '__main__':
335+ main()

Subscribers

People subscribed via source and target branches