Merge lp:~psivaa/core-image-watcher/image-watcher-udf into lp:core-image-watcher

[Para Siva]

Image watcher component to publish messages to a rabbit queue when there is a new image present in the core image server.

Image watcher component to publish messages to a rabbit queue when there is a new image present in the core image server.

The message does not include the test branch details since it could be picked up at a later stage in the workflow. But if that has also to be taken from the same config as its used here, we could include here.

[Joe Talbott (joetalbott)]

A few in-line questions.

review: Needs Information

[Thomi Richards (thomir-deactivatedaccount)]

A few things need fixing - please don't hesitate to ping me if anything doesn't make sense here.

Overall this is looking really nice - thank you!

review: Needs Fixing

[Celso Providelo (cprov)]

Psivaa,

Quick adjustment before review, the configuration file on the charms is called "core-service.conf".

review: Needs Fixing

[lp:~psivaa/core-image-watcher/image-watcher-udf]

9. By Para Siva on 2015-03-25

Review comment fixes

[Para Siva (psivaa)]

Thanks Thomi, Cprov and Joe for the comments. I've addressed as much as possible. Would you be able to take another look please?

Thanks

[Thomi Richards (thomir-deactivatedaccount)]

Hi Siva,

I made a new review, I'm afraid there are still several things to change. Let me know if you have any questions.

Cheers,

review: Needs Fixing

[lp:~psivaa/core-image-watcher/image-watcher-udf]

10. By Para Siva on 2015-03-26

Fixes for thomi's phase 2 review comments

[Para Siva (psivaa)]

Thanks again Thomi for the review. I need to mention that I really enjoyed fixing these comments, especially when you relate them to the fundamental principles.

I have addressed all your comments. Would be great for another look. Thanks

[lp:~psivaa/core-image-watcher/image-watcher-udf]

11. By Para Siva on 2015-03-26

Sleep after finding the latest version

12. By Para Siva on 2015-03-26

testtools for test requirements

13. By Para Siva on 2015-03-26

Use uniform naming test variable

14. By Para Siva on 2015-03-26

Pin testtool version to 1.7.1

[Thomi Richards (thomir-deactivatedaccount)]

Looks great - thanks!

review: Approve

[Celso Providelo (cprov)]

Psivaa,

There is only one remaining point about the kombu connection context, it should have a more restricted and coherent lifetime (inside enqueue())

It might impact badly in tests, lets see how it goes.

review: Needs Fixing

[lp:~psivaa/core-image-watcher/image-watcher-udf]

15. By Para Siva on 2015-03-26

Create connection to rabbit only when there is a new image

16. By Para Siva on 2015-03-26

Add unittest to check_for_new_image

[Para Siva (psivaa)]

Cprove, Thomi,

Thanks a lot again for the reviews and help. Moved kombu connection bit inside enqueue and got rid of the CoreImageWatcher class.
Added a couple of unittests for _check_for_new_image. Thanks

[Celso Providelo (cprov)]

Nice one! I liked the procedural version. Looking forward to see it in production.

review: Approve

=== modified file '.bzrignore'
--- .bzrignore	2015-03-25 03:47:26 +0000
+++ .bzrignore	2015-03-26 20:44:41 +0000
@@ -1,1 +1,2 @@
 core_image_watcher.egg-info
+ve

=== modified file 'README.rst'
--- README.rst	2015-03-24 22:58:34 +0000
+++ README.rst	2015-03-26 20:44:41 +0000
@@ -2,3 +2,67 @@
 ##################
 
 A micro-service that watches for new Ubuntu Core images.
+
+Get the Source
+==============
+
+Branch the code::
+
+  $ bzr branch lp:core-image-watcher
+
+Install the Service
+========================
+
+Install system dependencies::
+
+  $ sudo apt-get install python3-dev
+
+Build and activate a virtualenv with python3::
+
+  $ virtualenv -p python3 --system-site-packages ve
+  $ . ve/bin/activate
+
+Install dependencies from pypi::
+
+  $ pip install -r requirements.txt
+
+...and install some dependencies from phablet-tools PPA::
+
+  $ sudo add-apt-repository ppa:phablet-team/tools
+  $ sudo apt-get update
+  $ sudo apt-get install ubuntu-device-flash
+
+Install the service itself::
+
+  $ python setup.py install
+
+...you may want to install it in 'development mode', which symlinks files, 
+so you can edit/re-run without having to re-install the service. In that 
+case, run::
+
+  $ python setup.py develop
+
+Run the tests!
+==============
+
+Install dependencies::
+
+  $ pip install -r test_requirements.txt
+
+Run those tests - with vigour!::
+
+  $ python setup.py test
+
+The config file
+===============
+
+The sample configuration file in 'core-service.conf'::
+
+   [amqp]
+   uris = amqp://guest:guest@localhost:5672//
+
+   [image]
+   channel = devel-proposed
+   device = generic_amd64
+   location = /tmp/latest-core-image-version
+   poll_period = 60

=== modified file 'core-image-watcher.py'
--- core-image-watcher.py	2015-03-25 02:36:54 +0000
+++ core-image-watcher.py	2015-03-26 20:44:41 +0000
@@ -1,3 +1,21 @@
+#!/usr/bin/env python3
+
+# core-image-watcher
+# Copyright (C) 2015 Canonical
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
 
 from core_image_watcher import main
 

=== added file 'core-service.conf'
--- core-service.conf	1970-01-01 00:00:00 +0000
+++ core-service.conf	2015-03-26 20:44:41 +0000
@@ -0,0 +1,10 @@
+# `core-image-watcher` configuration file.
+
+[amqp]
+uris = amqp://guest:guest@localhost:5672//
+
+[image]
+channel = devel-proposed
+device = generic_amd64
+location = /tmp/latest-core-image-version
+poll_period = 60

=== modified file 'core_image_watcher/__init__.py'
--- core_image_watcher/__init__.py	2015-03-25 02:16:17 +0000
+++ core_image_watcher/__init__.py	2015-03-26 20:44:41 +0000
@@ -1,8 +1,174 @@
-
-
-import select
+# core-image-watcher
+# Copyright (C) 2015 Canonical
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+
+"""Core Image Watcher functional code."""
+
+import argparse
+import configparser
+import logging
+import os
+import subprocess
+import time
+
+import kombu
+
+logger = logging.getLogger(__name__)
+
+API_VERSION = "v1"
+
+
+def _enqueue(body, amqp_uris):
+    """Enqueues a message of `body` to the rabbit queue"""
+    with kombu.Connection(amqp_uris) as connection:
+        logger.info('Triggering request: %s', body, extra=body)
+        try:
+            queue = connection.SimpleQueue(
+                'core.image.{}'.format(API_VERSION))
+            queue.put(body)
+            queue.close()
+        except Exception as exc:
+            logger.error(exc, exc_info=True)
+        logger.info('Done!', extra=body)
+
+
+def _get_version_string_output(channel, device):
+    """Obtains a bytestring of the images info from the core image server"""
+    cmd = ['ubuntu-device-flash',
+           'query ',
+           '--list-images',
+           '--channel',
+           'ubuntu-core/{}'.format(channel),
+           '--device={}'.format(device)]
+    images = None
+    try:
+        images = subprocess.check_output(cmd)
+    except subprocess.CalledProcessError as e:
+        logger.error(e, exc_info=True)
+    finally:
+        return images
+
+
+def _get_latest_image_version(channel,
+                              device,
+                              get_output=_get_version_string_output):
+    """Returns largest image version"""
+    images = get_output(channel, device).split()
+    if (len(images) >= 2):
+        return images[-2].decode('utf-8').replace(':', '')
+    else:
+        logger.info('Could not get the image list by u-d-f')
+        return None
+
+
+def _cache_version_to_disk(location, latest_version):
+    """Caches the version of the image if it's larger than a cached one"""
+    try:
+        if os.path.exists(location):
+            with open(location, 'r') as cache:
+                if (int(cache.read()) < int(latest_version)):
+                    with open(location, 'w') as f:
+                        f.write(latest_version)
+                else:
+                    latest_version = None
+        else:
+            with open(location, 'w') as f:
+                f.write(latest_version)
+    except IOError as e:
+        latest_version = None
+        logger.error('Writing the latest image info failed: %s', (str(e)))
+    finally:
+        return latest_version
+
+
+def _check_for_new_image(location,
+                         channel,
+                         device,
+                         latest_image_version=_get_latest_image_version,
+                         cached_version=_cache_version_to_disk):
+    """Check if a new image is present in the core image server"""
+    latest_version = latest_image_version(channel, device)
+    try:
+        ret = cached_version(location, latest_version)
+        if not ret:
+            # There is no new image
+            # Do not progress
+            return None
+    except Exception as e:
+        logger.error(e, exc_info=True)
+        return None
+    body = {}
+    body['image_name'] = latest_version
+    body['channel'] = channel
+    body['device'] = device
+    return body
+
+
+def configure_logging(config):
+    root_logger = logging.getLogger()
+    root_logger.setLevel(logging.INFO)
+
+    requests_logger = logging.getLogger('requests')
+    requests_logger.setLevel(logging.WARNING)
+
+    # If there is no ./logs directory, fallback to stderr.
+    log_path = os.path.abspath(
+        os.path.join(__file__, '../../logs/core-image-watcher.log'))
+    log_dir = os.path.dirname(log_path)
+    if os.path.exists(log_dir):
+        handler = logging.FileHandler(log_path)
+    else:
+        print("'logs' directory '{}' does not exist, using stderr "
+              "for app log.".format(log_dir))
+        handler = logging.StreamHandler()
+
+    handler.setFormatter(
+        logging.Formatter(
+            '%(asctime)s  %(name)s %(levelname)s: %(message)s'
+        )
+    )
+    root_logger.addHandler(handler)
 
 
 def main():
-    # for now, do nothing at all.
-    select.select([], [], [])
+    parser = argparse.ArgumentParser(
+        description='Core image watcher ...')
+    parser.add_argument('-c', '--conf', default='core-service.conf',
+                        help='Configuration file path')
+    args = parser.parse_args()
+
+    # Load configuration options.
+    config = configparser.ConfigParser()
+    config.read(args.conf)
+
+    configure_logging(config)
+
+    amqp_uris = config.get('amqp', 'uris').split()
+    location = config.get('image', 'location')
+    channel = config.get('image', 'channel')
+    device = config.get('image', 'device')
+    poll_period = float(config.get('image', 'poll_period'))
+
+    try:
+        while True:
+            message_body = _check_for_new_image(location,
+                                                channel,
+                                                device)
+            if message_body:
+                _enqueue(message_body, amqp_uris)
+            time.sleep(poll_period)
+    except KeyboardInterrupt:
+        print('Bye!')

=== added directory 'core_image_watcher/tests'
=== added file 'core_image_watcher/tests/__init__.py'
=== added file 'core_image_watcher/tests/test_image_watcher.py'
--- core_image_watcher/tests/test_image_watcher.py	1970-01-01 00:00:00 +0000
+++ core_image_watcher/tests/test_image_watcher.py	2015-03-26 20:44:41 +0000
@@ -0,0 +1,110 @@
+# Ubuntu CI Engine
+# Copyright 2015 Canonical Ltd.
+
+# This program is free software: you can redistribute it and/or modify it
+# under the terms of the GNU Affero General Public License version 3, as
+# published by the Free Software Foundation.
+
+# This program is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranties of
+# MERCHANTABILITY, SATISFACTORY QUALITY, or FITNESS FOR A PARTICULAR
+# PURPOSE.  See the GNU Affero General Public License for more details.
+
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+import os
+import tempfile
+import testtools
+
+from core_image_watcher import (
+    _cache_version_to_disk,
+    _get_latest_image_version,
+    _check_for_new_image,
+)
+
+
+class TestWatchImage(testtools.TestCase):
+
+    def write_to_tempfile(self, version):
+        temp_file = tempfile.NamedTemporaryFile('w+', delete=False)
+        with temp_file as f:
+            f.write(version)
+        self.addCleanup(os.remove, temp_file.name)
+        return temp_file.name
+
+    def test_cache_version_to_disk(self):
+        temp_file_name = self.write_to_tempfile('111')
+        return_value = _cache_version_to_disk(temp_file_name, '112')
+        self.assertEqual(return_value, '112')
+
+    def test_cache_version_with_different_digits_to_disk(self):
+        temp_file_name = self.write_to_tempfile('23')
+        return_value = _cache_version_to_disk(temp_file_name, '112')
+        self.assertEqual(return_value, '112')
+
+    def test_do_not_cache_the_same_version_to_disk(self):
+        temp_file_name = self.write_to_tempfile('112')
+        return_value = _cache_version_to_disk(temp_file_name, '112')
+        self.assertEqual(return_value, None)
+
+    def test_do_not_proceed(self):
+        temp_file_name = self.write_to_tempfile('113')
+        return_value = _cache_version_to_disk(temp_file_name, '112')
+        self.assertEqual(return_value, None)
+
+    def test_freshly_cache_version_to_disk(self):
+        temp_file = tempfile.NamedTemporaryFile('w+', delete=False)
+        return_value = _cache_version_to_disk(temp_file.name, '112')
+        self.addCleanup(os.remove, temp_file.name)
+        self.assertEqual(return_value, '112')
+
+    def test_get_latest_version(self):
+        def get_output(channel, device):
+            return b"295: description='fake295'\n\
+                   296: description='fake296'\n\
+                   297: description='fake297'\n\
+                   298: description='fake338'\n"
+        observed = _get_latest_image_version('fakechannel',
+                                             'fakedevice',
+                                             get_output)
+        self.assertEqual(observed, '298')
+
+    def test_dont_get_latest_version(self):
+        def get_output(channel, device):
+            return b"100: description='fake100'\n"
+        observed = _get_latest_image_version('fakechannel',
+                                             'fakedevice',
+                                             get_output)
+        self.assertEqual(observed, '100')
+
+    def test_check_for_new_image(self):
+        def _get_latest_image_version(channel, device):
+            return '100'
+
+        def _cache_version_to_disk(location, latest_version):
+            return '99'
+
+        body = _check_for_new_image('fakelocation',
+                                    'fakechannel',
+                                    'fakedevice',
+                                    _get_latest_image_version,
+                                    _cache_version_to_disk)
+        self.assertEqual(body,
+                         {'device': 'fakedevice',
+                          'image_name': '100',
+                          'channel': 'fakechannel'})
+
+    def test_check_for_no_new_image(self):
+        def _get_latest_image_version(channel, device):
+            return '100'
+
+        def _cache_version_to_disk(location, latest_version):
+            return None
+
+        body = _check_for_new_image('fakelocation',
+                                    'fakechannel',
+                                    'fakedevice',
+                                    _get_latest_image_version,
+                                    _cache_version_to_disk)
+        self.assertEqual(body, None)

=== modified file 'requirements.txt'
--- requirements.txt	2015-03-25 01:34:08 +0000
+++ requirements.txt	2015-03-26 20:44:41 +0000
@@ -0,0 +1,1 @@
+kombu==3.0.24

=== modified file 'setup.py'
--- setup.py	2015-03-25 03:47:26 +0000
+++ setup.py	2015-03-26 20:44:41 +0000
@@ -36,5 +36,6 @@
     url='https://launchpad.net/core-image-watcher',
     license='GPLv3',
     packages=find_packages(),
+    test_suite='core_image_watcher.tests',  
     scripts=['core-image-watcher.py']
 )

=== modified file 'test_requirements.txt'
--- test_requirements.txt	2015-03-25 01:34:08 +0000
+++ test_requirements.txt	2015-03-26 20:44:41 +0000
@@ -0,0 +1,1 @@
+testtools==1.7.1