Merge lp:~psivaa/uci-engine/image-watcher into lp:uci-engine

[Para Siva]

Image watcher to trigger the workflow for when a new image available in http://cloud-images.ubuntu.com/

Image watcher to trigger when there is a new image present in http://cloud-images.ubuntu.com/ubuntu-core/devel. The script checks the content of http://cloud-images.ubuntu.com/ubuntu-core/devel/core/current/build-info.txt to find if there is a new version in the server.

In terms of message passing to a queue, once we find a new image in the server we provide the version, channel and the test branch in the snappy.images queue for the next service to consume.

[lp:~psivaa/uci-engine/image-watcher]

927. By Para Siva on 2015-01-15

Revert the unneeded amqp_utils change

928. By Para Siva on 2015-01-15

add a test to make sure the process triggers when started afresh

[Celso Providelo (cprov)]

Psivaa,

Thanks for working on this script.

I have nothing against monitoring build-info.txt instead of using simplestreams (python-simplestreams), as long as it works, as it seems to be. We can invest time later to rewrite the whole service to benefit of simplestreams goodies (supports the whole procedure from stream to glance).

There are some questions and remarks inline, specially regarding the topology of the new CLIs and how the charm is supposed to deploy them and its implication on configuration.

review: Needs Information

[lp:~psivaa/uci-engine/image-watcher]

929. By Para Siva on 2015-01-15

Review comment fixes, do not use specific config, and always use run-python

[Para Siva (psivaa)]

Thanks cprov for the comments.

I have removed using a special config file and relying on the amqp_utils.get_config() for the config information. For this as you pointed out, amqp-relation hook needs to be implemented in the cron-cmd charm.

Thanks again.

[Celso Providelo (cprov)]

Psivaa,

Thanks for the cleanup, I have just a minor suggestion about allowing image_watch.main() to be properly tested, but it's not a blocker.

Don't forget to coordinate with Joe for supporting amqp-relation in cron-cmd charm.

review: Approve

[Paul Larson (pwlars)]

> Psivaa,
>
> Thanks for working on this script.
>
> I have nothing against monitoring build-info.txt instead of using
> simplestreams (python-simplestreams), as long as it works, as it seems to be.
> We can invest time later to rewrite the whole service to benefit of
> simplestreams goodies (supports the whole procedure from stream to glance).
This is actually possible now, we discussed yesterday and I think we all agreed *not* to do that though right?

[Paul Larson (pwlars)]

Celso mentioned (if I understood correctly) in a MP for me that we should take all of these services for ubuntucore/snappy and organize them under a single directory rather than having an individual directory for each one off the root of our tree. I think this probably makes sense because they are all related to ubuntucore/snappy, rather than, for instance, a generic image watcher that may have to monitor some other type of source and handle things differently.

Some additional comments below

review: Needs Fixing

[Celso Providelo (cprov)]

Psivaa, Paul,

I retracted my concerns about having one directory per service (CLIs + worker python module), it the right direction.

With that tree organisation we run CLIs locally or via with `run-python.py` and properly configure rabbit-worker. To me, it looks perfectly fine and acceptable and if it needs rework we can deal with it later.

[]

review: Approve

[lp:~psivaa/uci-engine/image-watcher]

930. By Para Siva on 2015-01-16

Review comment fix, account for empty version number in the server

931. By Para Siva on 2015-01-16

Remove pep8 and pyflake testing locally

[Para Siva (psivaa)]

Thanks,
@plars: I have addressed your comments inline. and fixed where appropriate. About checking the latest against the glance image-list, we need to have a uniquely corresponding image in glance to the image in cloud-image. Pending that, I believe we only could check against a local cache. Please let me know what you think. Thanks

[Paul Larson (pwlars)]

I'm fine if we sort out that detail later. I believe we can just call the glance image whatever we want though. The image we download (file) will be something like: devel-core-amd64-disk1.img so maybe we could call it something like devel-core-amd64-${imageid} (ex. devel-core-amd64-20150115.1)

In light of the shift of direction though, I think I'd rather see this get merged and we can sort out those details later. It's easy enough to change things once they are in place, and you've done a great job of getting the ball rolling with this. Thanks for working on this!

review: Approve

=== added directory 'image-watcher'
=== added directory 'image-watcher/imagewatcher'
=== added file 'image-watcher/imagewatcher/__init__.py'
--- image-watcher/imagewatcher/__init__.py	1970-01-01 00:00:00 +0000
+++ image-watcher/imagewatcher/__init__.py	2015-01-16 11:11:59 +0000
@@ -0,0 +1,15 @@
+#!/usr/bin/env python
+# Ubuntu CI Engine
+# Copyright 2015 Canonical Ltd.
+
+# This program is free software: you can redistribute it and/or modify it
+# under the terms of the GNU Affero General Public License version 3, as
+# published by the Free Software Foundation.
+
+# This program is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranties of
+# MERCHANTABILITY, SATISFACTORY QUALITY, or FITNESS FOR A PARTICULAR
+# PURPOSE.  See the GNU Affero General Public License for more details.
+
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
 
=== added file 'image-watcher/imagewatcher/queues.py'
--- image-watcher/imagewatcher/queues.py	1970-01-01 00:00:00 +0000
+++ image-watcher/imagewatcher/queues.py	2015-01-16 11:11:59 +0000
@@ -0,0 +1,34 @@
+# Ubuntu CI Engine
+# Copyright 2014 Canonical Ltd.
+
+# This program is free software: you can redistribute it and/or modify it
+# under the terms of the GNU Affero General Public License version 3, as
+# published by the Free Software Foundation.
+
+# This program is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranties of
+# MERCHANTABILITY, SATISFACTORY QUALITY, or FITNESS FOR A PARTICULAR
+# PURPOSE.  See the GNU Affero General Public License for more details.
+
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+import json
+
+
+from ci_utils import amqp_utils
+
+
+class PublisherQueue(object):
+    """A proxy to the queue library.
+
+    This implementation targets ci_utils/amqp_utils but can be replaced for
+    tests.
+    """
+
+    def __init__(self, queue_name):
+        self.queue_name = queue_name
+
+    def publish(self, msg):
+        amqp_utils.send(self.queue_name, json.dumps(msg),
+                        raise_errors=True)
 
=== added directory 'image-watcher/imagewatcher/tests'
=== added file 'image-watcher/imagewatcher/tests/__init__.py'
=== added file 'image-watcher/imagewatcher/tests/test_trigger_process.py'
--- image-watcher/imagewatcher/tests/test_trigger_process.py	1970-01-01 00:00:00 +0000
+++ image-watcher/imagewatcher/tests/test_trigger_process.py	2015-01-16 11:11:59 +0000
@@ -0,0 +1,43 @@
+# Copyright 2015 Canonical Ltd.
+
+# This program is free software: you can redistribute it and/or modify it
+# under the terms of the GNU Affero General Public License version 3, as
+# published by the Free Software Foundation.
+
+# This program is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranties of
+# MERCHANTABILITY, SATISFACTORY QUALITY, or FITNESS FOR A PARTICULAR
+# PURPOSE.  See the GNU Affero General Public License for more details.
+
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+import unittest
+
+
+from ucitests import assertions
+
+
+from imagewatcher import trigger_process
+from ci_utils.testing import fixtures
+
+
+class TestPost(unittest.TestCase):
+
+    def test_trigger_simple(self):
+        queue = fixtures.FakeTaskQueue('that_queue')
+        trigger_process.post(queue, 111, 'ubuntu-core/devel', 'lp:fake')
+        assertions.assertLength(self, 1, queue.msgs)
+        self.assertEqual(dict(channel='ubuntu-core/devel', revision=111,
+                              test_branch='lp:fake'),
+                         queue.msgs[0])
+
+    def test_trigger_with_output(self):
+        queue = fixtures.FakeTaskQueue('that_queue')
+        trigger_process.post(queue,  111, 'ubuntu-core/devel', 'lp:fake',
+                             'this_queue')
+        assertions.assertLength(self, 1, queue.msgs)
+        self.assertEqual(dict(channel='ubuntu-core/devel', revision=111,
+                              test_branch='lp:fake',
+                              output_queue='this_queue'),
+                         queue.msgs[0])
 
=== added file 'image-watcher/imagewatcher/tests/test_watch_image.py'
--- image-watcher/imagewatcher/tests/test_watch_image.py	1970-01-01 00:00:00 +0000
+++ image-watcher/imagewatcher/tests/test_watch_image.py	2015-01-16 11:11:59 +0000
@@ -0,0 +1,58 @@
+# Ubuntu CI Engine
+# Copyright 2015 Canonical Ltd.
+
+# This program is free software: you can redistribute it and/or modify it
+# under the terms of the GNU Affero General Public License version 3, as
+# published by the Free Software Foundation.
+
+# This program is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranties of
+# MERCHANTABILITY, SATISFACTORY QUALITY, or FITNESS FOR A PARTICULAR
+# PURPOSE.  See the GNU Affero General Public License for more details.
+
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+import unittest
+import os
+
+from imagewatcher import watch_image
+
+
+class TestWatchImage(unittest.TestCase):
+    def setUp(self):
+        super(TestWatchImage, self).setUp()
+        self.fake_resp = [u'VERSION=112',
+                          u'BUILD_TYPE=ubuntu-core/devel',
+                          u'BUILD_DATE=20150109',
+                          u'IMG_FORMAT=qcow2',
+                          u'IMG_COMPAT=0.10',
+                          u'DEVICE=generic_amd64',
+                          u'SERIAL=20150109.2',
+                          u'BUILD_JOB_ID=35',
+                          u'']
+
+    def test_record_and_return_latest_version(self):
+        with open('/tmp/fake', 'w') as f:
+            f.write('111')
+        ret_val = watch_image.record_and_return('/tmp/fake', self.fake_resp)
+        os.remove('/tmp/fake')
+        self.assertEquals(ret_val, ('112', u'ubuntu-core/devel'))
+
+    def test_do_not_record_and_return_non_latest(self):
+        with open('/tmp/fake2', 'w') as f:
+            f.write('113')
+        ret_val = watch_image.record_and_return('/tmp/fake2', self.fake_resp)
+        os.remove('/tmp/fake2')
+        self.assertEquals(ret_val, None)
+
+    def test_freshly_record_and_return_latest_version(self):
+        if os.path.exists('/tmp/fake3'):
+            os.remove('/tmp/fake3')
+        ret_val = watch_image.record_and_return('/tmp/fake3', self.fake_resp)
+        os.remove('/tmp/fake3')
+        self.assertEquals(ret_val, ('112', u'ubuntu-core/devel'))
+
+    def test_get_latest_from_server_failed(self):
+        with self.assertRaises(watch_image.ImageWatcherError):
+            watch_image._get_latest_from_server('http://fake', retry=1)
 
=== added file 'image-watcher/imagewatcher/trigger_process.py'
--- image-watcher/imagewatcher/trigger_process.py	1970-01-01 00:00:00 +0000
+++ image-watcher/imagewatcher/trigger_process.py	2015-01-16 11:11:59 +0000
@@ -0,0 +1,29 @@
+#!/usr/bin/env python
+# Ubuntu CI Engine
+# Copyright 2015 Canonical Ltd.
+
+# This program is free software: you can redistribute it and/or modify it
+# under the terms of the GNU Affero General Public License version 3, as
+# published by the Free Software Foundation.
+
+# This program is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranties of
+# MERCHANTABILITY, SATISFACTORY QUALITY, or FITNESS FOR A PARTICULAR
+# PURPOSE.  See the GNU Affero General Public License for more details.
+
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+from imagewatcher import queues
+
+
+def post(queue, revision, channel, test_branch, output=None):
+    msg = dict(revision=revision, channel=channel, test_branch=test_branch)
+    if output is not None:
+        msg['output_queue'] = output
+    queue.publish(msg)
+
+
+def trigger(revision, channel, test_branch, output_queue):
+    queue = queues.PublisherQueue('snappy.images')
+    post(queue, revision, channel, test_branch, output_queue)
 
=== added file 'image-watcher/imagewatcher/watch_image.py'
--- image-watcher/imagewatcher/watch_image.py	1970-01-01 00:00:00 +0000
+++ image-watcher/imagewatcher/watch_image.py	2015-01-16 11:11:59 +0000
@@ -0,0 +1,123 @@
+#!/usr/bin/env python
+# Ubuntu CI Engine
+# Copyright 2015 Canonical Ltd.
+
+# This program is free software: you can redistribute it and/or modify it
+# under the terms of the GNU Affero General Public License version 3, as
+# published by the Free Software Foundation.
+
+# This program is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranties of
+# MERCHANTABILITY, SATISFACTORY QUALITY, or FITNESS FOR A PARTICULAR
+# PURPOSE.  See the GNU Affero General Public License for more details.
+
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+import argparse
+import logging
+import requests
+import time
+import os
+import sys
+
+from uciconfig import errors
+
+from imagewatcher import (
+    trigger_process,
+)
+
+log = logging.getLogger('ImageWatcher')
+
+
+class ImageWatcherError(Exception):
+    """Raised if there was an error watching the new image."""
+
+
+def _get_args(args=None):
+    parser = argparse.ArgumentParser(description='Check if a new core-image is\
+                                     available at a given url')
+    parser.add_argument(
+        "-u", "--url",
+        default="http://cloud-images.ubuntu.com/ubuntu-core/devel/"
+                "core/current/build-info.txt",
+        help="URL to check if a new core image available"),
+    parser.add_argument(
+        "-l", "--location",
+        default="/tmp/latest-core-image-version",
+        help="The file containing latest image version")
+    parser.add_argument(
+        "-t", "--test-branch",
+        default="lp:snappy-tests",
+        help="Branch where the dep8 tests live")
+    parser.add_argument(
+        "-o", "--output-queue",
+        default=None,
+        help='Send progress/results from the image watcher to the queue'
+        ' named output_queue.')
+    return parser.parse_args(args)
+
+
+def _get_latest_from_server(url, retry=3):
+    try:
+        response = requests.get(url)
+        response.raise_for_status()
+    except (requests.ConnectionError, requests.HTTPError) as e:
+        if retry == 1:
+            raise ImageWatcherError(
+                'Error downloading the metadata {}'.format(str(e)))
+        log.warn('Page download failed, retrying in 30 sec')
+        time.sleep(30)
+        return _get_latest_from_server(url, retry - 1)
+    return response.text.split('\n')
+
+
+def record_and_return(location, response):
+    content = {}
+    for line in response:
+        if '=' in line:
+            k, v = line.strip().split('=')
+            content[k.strip()] = v.strip()
+    if content['VERSION'] == '':
+        log.error('The version number in the cloud-image server'
+                  ' is not an integer but {}'.format(content['VERSION']))
+        sys.exit(1)
+
+    version = content['VERSION']
+    channel = content['BUILD_TYPE']
+    try:
+        if os.path.exists(location):
+            with open(location, 'r') as cache:
+                if (cache.read() < version):
+                    with open(location, 'w') as f:
+                        f.write(version)
+                        return version, channel
+        else:
+            with open(location, 'w') as f:
+                f.write(version)
+                return version, channel
+    except IOError as e:
+        log.error('Writing the latest image info failed')
+        raise ImageWatcherError(
+            'Error writing the latest version with {}'.format(str(e)))
+
+
+def main(args=None):
+    args = _get_args(args=args)
+    try:
+        ret = record_and_return(
+            args.location,
+            _get_latest_from_server(args.url))
+        if ret:
+            trigger_process.trigger(ret[0],
+                                    ret[1],
+                                    args.test_branch,
+                                    args.output_queue)
+        else:
+            log.info('No new image found at {}'.format(time.asctime()))
+    except errors.ConfigError as e:
+        log.error('Triggering the workflow failed with {}'.format(e))
+        sys.exit(1)
+
+if __name__ == '__main__':
+    main()