Merge lp:~psivaa/uci-engine/image-watcher into lp:uci-engine
- image-watcher
- Merge into trunk
Status: | Merged |
---|---|
Approved by: | Para Siva |
Approved revision: | 931 |
Merged at revision: | 923 |
Proposed branch: | lp:~psivaa/uci-engine/image-watcher |
Merge into: | lp:uci-engine |
Diff against target: |
335 lines (+302/-0) 6 files modified
image-watcher/imagewatcher/__init__.py (+15/-0) image-watcher/imagewatcher/queues.py (+34/-0) image-watcher/imagewatcher/tests/test_trigger_process.py (+43/-0) image-watcher/imagewatcher/tests/test_watch_image.py (+58/-0) image-watcher/imagewatcher/trigger_process.py (+29/-0) image-watcher/imagewatcher/watch_image.py (+123/-0) |
To merge this branch: | bzr merge lp:~psivaa/uci-engine/image-watcher |
Related bugs: |
Reviewer | Review Type | Date Requested | Status |
---|---|---|---|
Paul Larson | Approve | ||
Celso Providelo (community) | Approve | ||
Review via email: mp+246552@code.launchpad.net |
Commit message
Image watcher to trigger the workflow for when a new image available in http://
Description of the change
Image watcher to trigger when there is a new image present in http://
In terms of message passing to a queue, once we find a new image in the server we provide the version, channel and the test branch in the snappy.images queue for the next service to consume.
Para Siva (psivaa) wrote : | # |
Thanks cprov for the comments.
I have removed using a special config file and relying on the amqp_utils.
Thanks again.
Celso Providelo (cprov) wrote : | # |
Psivaa,
Thanks for the cleanup, I have just a minor suggestion about allowing image_watch.main() to be properly tested, but it's not a blocker.
Don't forget to coordinate with Joe for supporting amqp-relation in cron-cmd charm.
Paul Larson (pwlars) wrote : | # |
> Psivaa,
>
> Thanks for working on this script.
>
> I have nothing against monitoring build-info.txt instead of using
> simplestreams (python-
> We can invest time later to rewrite the whole service to benefit of
> simplestreams goodies (supports the whole procedure from stream to glance).
This is actually possible now, we discussed yesterday and I think we all agreed *not* to do that though right?
Paul Larson (pwlars) wrote : | # |
Celso mentioned (if I understood correctly) in a MP for me that we should take all of these services for ubuntucore/snappy and organize them under a single directory rather than having an individual directory for each one off the root of our tree. I think this probably makes sense because they are all related to ubuntucore/snappy, rather than, for instance, a generic image watcher that may have to monitor some other type of source and handle things differently.
Some additional comments below
Celso Providelo (cprov) wrote : | # |
Psivaa, Paul,
I retracted my concerns about having one directory per service (CLIs + worker python module), it the right direction.
With that tree organisation we run CLIs locally or via with `run-python.py` and properly configure rabbit-worker. To me, it looks perfectly fine and acceptable and if it needs rework we can deal with it later.
[]
Para Siva (psivaa) wrote : | # |
Thanks,
@plars: I have addressed your comments inline. and fixed where appropriate. About checking the latest against the glance image-list, we need to have a uniquely corresponding image in glance to the image in cloud-image. Pending that, I believe we only could check against a local cache. Please let me know what you think. Thanks
Paul Larson (pwlars) wrote : | # |
I'm fine if we sort out that detail later. I believe we can just call the glance image whatever we want though. The image we download (file) will be something like: devel-core-
In light of the shift of direction though, I think I'd rather see this get merged and we can sort out those details later. It's easy enough to change things once they are in place, and you've done a great job of getting the ball rolling with this. Thanks for working on this!
Preview Diff
1 | === added directory 'image-watcher' |
2 | === added directory 'image-watcher/imagewatcher' |
3 | === added file 'image-watcher/imagewatcher/__init__.py' |
4 | --- image-watcher/imagewatcher/__init__.py 1970-01-01 00:00:00 +0000 |
5 | +++ image-watcher/imagewatcher/__init__.py 2015-01-16 11:11:59 +0000 |
6 | @@ -0,0 +1,15 @@ |
7 | +#!/usr/bin/env python |
8 | +# Ubuntu CI Engine |
9 | +# Copyright 2015 Canonical Ltd. |
10 | + |
11 | +# This program is free software: you can redistribute it and/or modify it |
12 | +# under the terms of the GNU Affero General Public License version 3, as |
13 | +# published by the Free Software Foundation. |
14 | + |
15 | +# This program is distributed in the hope that it will be useful, but |
16 | +# WITHOUT ANY WARRANTY; without even the implied warranties of |
17 | +# MERCHANTABILITY, SATISFACTORY QUALITY, or FITNESS FOR A PARTICULAR |
18 | +# PURPOSE. See the GNU Affero General Public License for more details. |
19 | + |
20 | +# You should have received a copy of the GNU Affero General Public License |
21 | +# along with this program. If not, see <http://www.gnu.org/licenses/>. |
22 | |
23 | === added file 'image-watcher/imagewatcher/queues.py' |
24 | --- image-watcher/imagewatcher/queues.py 1970-01-01 00:00:00 +0000 |
25 | +++ image-watcher/imagewatcher/queues.py 2015-01-16 11:11:59 +0000 |
26 | @@ -0,0 +1,34 @@ |
27 | +# Ubuntu CI Engine |
28 | +# Copyright 2014 Canonical Ltd. |
29 | + |
30 | +# This program is free software: you can redistribute it and/or modify it |
31 | +# under the terms of the GNU Affero General Public License version 3, as |
32 | +# published by the Free Software Foundation. |
33 | + |
34 | +# This program is distributed in the hope that it will be useful, but |
35 | +# WITHOUT ANY WARRANTY; without even the implied warranties of |
36 | +# MERCHANTABILITY, SATISFACTORY QUALITY, or FITNESS FOR A PARTICULAR |
37 | +# PURPOSE. See the GNU Affero General Public License for more details. |
38 | + |
39 | +# You should have received a copy of the GNU Affero General Public License |
40 | +# along with this program. If not, see <http://www.gnu.org/licenses/>. |
41 | + |
42 | +import json |
43 | + |
44 | + |
45 | +from ci_utils import amqp_utils |
46 | + |
47 | + |
48 | +class PublisherQueue(object): |
49 | + """A proxy to the queue library. |
50 | + |
51 | + This implementation targets ci_utils/amqp_utils but can be replaced for |
52 | + tests. |
53 | + """ |
54 | + |
55 | + def __init__(self, queue_name): |
56 | + self.queue_name = queue_name |
57 | + |
58 | + def publish(self, msg): |
59 | + amqp_utils.send(self.queue_name, json.dumps(msg), |
60 | + raise_errors=True) |
61 | |
62 | === added directory 'image-watcher/imagewatcher/tests' |
63 | === added file 'image-watcher/imagewatcher/tests/__init__.py' |
64 | === added file 'image-watcher/imagewatcher/tests/test_trigger_process.py' |
65 | --- image-watcher/imagewatcher/tests/test_trigger_process.py 1970-01-01 00:00:00 +0000 |
66 | +++ image-watcher/imagewatcher/tests/test_trigger_process.py 2015-01-16 11:11:59 +0000 |
67 | @@ -0,0 +1,43 @@ |
68 | +# Copyright 2015 Canonical Ltd. |
69 | + |
70 | +# This program is free software: you can redistribute it and/or modify it |
71 | +# under the terms of the GNU Affero General Public License version 3, as |
72 | +# published by the Free Software Foundation. |
73 | + |
74 | +# This program is distributed in the hope that it will be useful, but |
75 | +# WITHOUT ANY WARRANTY; without even the implied warranties of |
76 | +# MERCHANTABILITY, SATISFACTORY QUALITY, or FITNESS FOR A PARTICULAR |
77 | +# PURPOSE. See the GNU Affero General Public License for more details. |
78 | + |
79 | +# You should have received a copy of the GNU Affero General Public License |
80 | +# along with this program. If not, see <http://www.gnu.org/licenses/>. |
81 | + |
82 | +import unittest |
83 | + |
84 | + |
85 | +from ucitests import assertions |
86 | + |
87 | + |
88 | +from imagewatcher import trigger_process |
89 | +from ci_utils.testing import fixtures |
90 | + |
91 | + |
92 | +class TestPost(unittest.TestCase): |
93 | + |
94 | + def test_trigger_simple(self): |
95 | + queue = fixtures.FakeTaskQueue('that_queue') |
96 | + trigger_process.post(queue, 111, 'ubuntu-core/devel', 'lp:fake') |
97 | + assertions.assertLength(self, 1, queue.msgs) |
98 | + self.assertEqual(dict(channel='ubuntu-core/devel', revision=111, |
99 | + test_branch='lp:fake'), |
100 | + queue.msgs[0]) |
101 | + |
102 | + def test_trigger_with_output(self): |
103 | + queue = fixtures.FakeTaskQueue('that_queue') |
104 | + trigger_process.post(queue, 111, 'ubuntu-core/devel', 'lp:fake', |
105 | + 'this_queue') |
106 | + assertions.assertLength(self, 1, queue.msgs) |
107 | + self.assertEqual(dict(channel='ubuntu-core/devel', revision=111, |
108 | + test_branch='lp:fake', |
109 | + output_queue='this_queue'), |
110 | + queue.msgs[0]) |
111 | |
112 | === added file 'image-watcher/imagewatcher/tests/test_watch_image.py' |
113 | --- image-watcher/imagewatcher/tests/test_watch_image.py 1970-01-01 00:00:00 +0000 |
114 | +++ image-watcher/imagewatcher/tests/test_watch_image.py 2015-01-16 11:11:59 +0000 |
115 | @@ -0,0 +1,58 @@ |
116 | +# Ubuntu CI Engine |
117 | +# Copyright 2015 Canonical Ltd. |
118 | + |
119 | +# This program is free software: you can redistribute it and/or modify it |
120 | +# under the terms of the GNU Affero General Public License version 3, as |
121 | +# published by the Free Software Foundation. |
122 | + |
123 | +# This program is distributed in the hope that it will be useful, but |
124 | +# WITHOUT ANY WARRANTY; without even the implied warranties of |
125 | +# MERCHANTABILITY, SATISFACTORY QUALITY, or FITNESS FOR A PARTICULAR |
126 | +# PURPOSE. See the GNU Affero General Public License for more details. |
127 | + |
128 | +# You should have received a copy of the GNU Affero General Public License |
129 | +# along with this program. If not, see <http://www.gnu.org/licenses/>. |
130 | + |
131 | +import unittest |
132 | +import os |
133 | + |
134 | +from imagewatcher import watch_image |
135 | + |
136 | + |
137 | +class TestWatchImage(unittest.TestCase): |
138 | + def setUp(self): |
139 | + super(TestWatchImage, self).setUp() |
140 | + self.fake_resp = [u'VERSION=112', |
141 | + u'BUILD_TYPE=ubuntu-core/devel', |
142 | + u'BUILD_DATE=20150109', |
143 | + u'IMG_FORMAT=qcow2', |
144 | + u'IMG_COMPAT=0.10', |
145 | + u'DEVICE=generic_amd64', |
146 | + u'SERIAL=20150109.2', |
147 | + u'BUILD_JOB_ID=35', |
148 | + u''] |
149 | + |
150 | + def test_record_and_return_latest_version(self): |
151 | + with open('/tmp/fake', 'w') as f: |
152 | + f.write('111') |
153 | + ret_val = watch_image.record_and_return('/tmp/fake', self.fake_resp) |
154 | + os.remove('/tmp/fake') |
155 | + self.assertEquals(ret_val, ('112', u'ubuntu-core/devel')) |
156 | + |
157 | + def test_do_not_record_and_return_non_latest(self): |
158 | + with open('/tmp/fake2', 'w') as f: |
159 | + f.write('113') |
160 | + ret_val = watch_image.record_and_return('/tmp/fake2', self.fake_resp) |
161 | + os.remove('/tmp/fake2') |
162 | + self.assertEquals(ret_val, None) |
163 | + |
164 | + def test_freshly_record_and_return_latest_version(self): |
165 | + if os.path.exists('/tmp/fake3'): |
166 | + os.remove('/tmp/fake3') |
167 | + ret_val = watch_image.record_and_return('/tmp/fake3', self.fake_resp) |
168 | + os.remove('/tmp/fake3') |
169 | + self.assertEquals(ret_val, ('112', u'ubuntu-core/devel')) |
170 | + |
171 | + def test_get_latest_from_server_failed(self): |
172 | + with self.assertRaises(watch_image.ImageWatcherError): |
173 | + watch_image._get_latest_from_server('http://fake', retry=1) |
174 | |
175 | === added file 'image-watcher/imagewatcher/trigger_process.py' |
176 | --- image-watcher/imagewatcher/trigger_process.py 1970-01-01 00:00:00 +0000 |
177 | +++ image-watcher/imagewatcher/trigger_process.py 2015-01-16 11:11:59 +0000 |
178 | @@ -0,0 +1,29 @@ |
179 | +#!/usr/bin/env python |
180 | +# Ubuntu CI Engine |
181 | +# Copyright 2015 Canonical Ltd. |
182 | + |
183 | +# This program is free software: you can redistribute it and/or modify it |
184 | +# under the terms of the GNU Affero General Public License version 3, as |
185 | +# published by the Free Software Foundation. |
186 | + |
187 | +# This program is distributed in the hope that it will be useful, but |
188 | +# WITHOUT ANY WARRANTY; without even the implied warranties of |
189 | +# MERCHANTABILITY, SATISFACTORY QUALITY, or FITNESS FOR A PARTICULAR |
190 | +# PURPOSE. See the GNU Affero General Public License for more details. |
191 | + |
192 | +# You should have received a copy of the GNU Affero General Public License |
193 | +# along with this program. If not, see <http://www.gnu.org/licenses/>. |
194 | + |
195 | +from imagewatcher import queues |
196 | + |
197 | + |
198 | +def post(queue, revision, channel, test_branch, output=None): |
199 | + msg = dict(revision=revision, channel=channel, test_branch=test_branch) |
200 | + if output is not None: |
201 | + msg['output_queue'] = output |
202 | + queue.publish(msg) |
203 | + |
204 | + |
205 | +def trigger(revision, channel, test_branch, output_queue): |
206 | + queue = queues.PublisherQueue('snappy.images') |
207 | + post(queue, revision, channel, test_branch, output_queue) |
208 | |
209 | === added file 'image-watcher/imagewatcher/watch_image.py' |
210 | --- image-watcher/imagewatcher/watch_image.py 1970-01-01 00:00:00 +0000 |
211 | +++ image-watcher/imagewatcher/watch_image.py 2015-01-16 11:11:59 +0000 |
212 | @@ -0,0 +1,123 @@ |
213 | +#!/usr/bin/env python |
214 | +# Ubuntu CI Engine |
215 | +# Copyright 2015 Canonical Ltd. |
216 | + |
217 | +# This program is free software: you can redistribute it and/or modify it |
218 | +# under the terms of the GNU Affero General Public License version 3, as |
219 | +# published by the Free Software Foundation. |
220 | + |
221 | +# This program is distributed in the hope that it will be useful, but |
222 | +# WITHOUT ANY WARRANTY; without even the implied warranties of |
223 | +# MERCHANTABILITY, SATISFACTORY QUALITY, or FITNESS FOR A PARTICULAR |
224 | +# PURPOSE. See the GNU Affero General Public License for more details. |
225 | + |
226 | +# You should have received a copy of the GNU Affero General Public License |
227 | +# along with this program. If not, see <http://www.gnu.org/licenses/>. |
228 | + |
229 | +import argparse |
230 | +import logging |
231 | +import requests |
232 | +import time |
233 | +import os |
234 | +import sys |
235 | + |
236 | +from uciconfig import errors |
237 | + |
238 | +from imagewatcher import ( |
239 | + trigger_process, |
240 | +) |
241 | + |
242 | +log = logging.getLogger('ImageWatcher') |
243 | + |
244 | + |
245 | +class ImageWatcherError(Exception): |
246 | + """Raised if there was an error watching the new image.""" |
247 | + |
248 | + |
249 | +def _get_args(args=None): |
250 | + parser = argparse.ArgumentParser(description='Check if a new core-image is\ |
251 | + available at a given url') |
252 | + parser.add_argument( |
253 | + "-u", "--url", |
254 | + default="http://cloud-images.ubuntu.com/ubuntu-core/devel/" |
255 | + "core/current/build-info.txt", |
256 | + help="URL to check if a new core image available"), |
257 | + parser.add_argument( |
258 | + "-l", "--location", |
259 | + default="/tmp/latest-core-image-version", |
260 | + help="The file containing latest image version") |
261 | + parser.add_argument( |
262 | + "-t", "--test-branch", |
263 | + default="lp:snappy-tests", |
264 | + help="Branch where the dep8 tests live") |
265 | + parser.add_argument( |
266 | + "-o", "--output-queue", |
267 | + default=None, |
268 | + help='Send progress/results from the image watcher to the queue' |
269 | + ' named output_queue.') |
270 | + return parser.parse_args(args) |
271 | + |
272 | + |
273 | +def _get_latest_from_server(url, retry=3): |
274 | + try: |
275 | + response = requests.get(url) |
276 | + response.raise_for_status() |
277 | + except (requests.ConnectionError, requests.HTTPError) as e: |
278 | + if retry == 1: |
279 | + raise ImageWatcherError( |
280 | + 'Error downloading the metadata {}'.format(str(e))) |
281 | + log.warn('Page download failed, retrying in 30 sec') |
282 | + time.sleep(30) |
283 | + return _get_latest_from_server(url, retry - 1) |
284 | + return response.text.split('\n') |
285 | + |
286 | + |
287 | +def record_and_return(location, response): |
288 | + content = {} |
289 | + for line in response: |
290 | + if '=' in line: |
291 | + k, v = line.strip().split('=') |
292 | + content[k.strip()] = v.strip() |
293 | + if content['VERSION'] == '': |
294 | + log.error('The version number in the cloud-image server' |
295 | + ' is not an integer but {}'.format(content['VERSION'])) |
296 | + sys.exit(1) |
297 | + |
298 | + version = content['VERSION'] |
299 | + channel = content['BUILD_TYPE'] |
300 | + try: |
301 | + if os.path.exists(location): |
302 | + with open(location, 'r') as cache: |
303 | + if (cache.read() < version): |
304 | + with open(location, 'w') as f: |
305 | + f.write(version) |
306 | + return version, channel |
307 | + else: |
308 | + with open(location, 'w') as f: |
309 | + f.write(version) |
310 | + return version, channel |
311 | + except IOError as e: |
312 | + log.error('Writing the latest image info failed') |
313 | + raise ImageWatcherError( |
314 | + 'Error writing the latest version with {}'.format(str(e))) |
315 | + |
316 | + |
317 | +def main(args=None): |
318 | + args = _get_args(args=args) |
319 | + try: |
320 | + ret = record_and_return( |
321 | + args.location, |
322 | + _get_latest_from_server(args.url)) |
323 | + if ret: |
324 | + trigger_process.trigger(ret[0], |
325 | + ret[1], |
326 | + args.test_branch, |
327 | + args.output_queue) |
328 | + else: |
329 | + log.info('No new image found at {}'.format(time.asctime())) |
330 | + except errors.ConfigError as e: |
331 | + log.error('Triggering the workflow failed with {}'.format(e)) |
332 | + sys.exit(1) |
333 | + |
334 | +if __name__ == '__main__': |
335 | + main() |
Psivaa,
Thanks for working on this script.
I have nothing against monitoring build-info.txt instead of using simplestreams (python- simplestreams) , as long as it works, as it seems to be. We can invest time later to rewrite the whole service to benefit of simplestreams goodies (supports the whole procedure from stream to glance).
There are some questions and remarks inline, specially regarding the topology of the new CLIs and how the charm is supposed to deploy them and its implication on configuration.