Merge lp:~pwlars/uci-engine/snappy-imageimport into lp:uci-engine

[Paul Larson]

Add a cli/api to download a specified image and put it in glance.

This is just the simple tool to take an image, download it, and import it into glance.

[Celso Providelo (cprov)]

Paul,

Thanks for working on this, the CLI does what is expected.

I am unsure about its location as an separated module. At least, we should s/snappy/ubuntucore/, but I don't like much the idea of creating a new module/directory for each CLI we planned, it is unnecessarily confusing and hard to find.

It needs ci_utils, so if we are going to fork/subprocess to it, we need `run-python` and the scripts could live in an agnostic common path, let's say bin/ or ubuntucore/bin/, because for now we deploy the whole uci-engine tree on nodes.

If we intend to import and use publish_image() inside workers, it should be a module (as the other ci-airline modules) and all CLIs could live in a module, let's say 'ubuntucore', which will eventually provide worker implementations for the services we need.

Any thoughts ?
[]

review: Needs Information

[Paul Larson (pwlars)]

> Paul,
>
> Thanks for working on this, the CLI does what is expected.
>
> I am unsure about its location as an separated module. At least, we should
> s/snappy/ubuntucore/, but I don't like much the idea of creating a new
> module/directory for each CLI we planned, it is unnecessarily confusing and
> hard to find.
Yes, I couldn't think of a better place to put it, but it's not just going to be the CLI. There's another task to create a worker that uses it, so I thought it might be more appropriate to give it it's own directory as we've done with every other service so far. Putting all of these related to snappy under a single directory would be fine I think too, but don't you think we should still logically separate them somehow since they are separate microservices?

Why should it be ubuntucore instead of snappy? I'm not 100% positive of this, but it seems that "snappy" is the new term for what we are testing, thought 99% of it was once called ubuntu core.

> It needs ci_utils, so if we are going to fork/subprocess to it, we need `run-
> python` and the scripts could live in an agnostic common path, let's say bin/
> or ubuntucore/bin/, because for now we deploy the whole uci-engine tree on
> nodes.
I figured I'd let the worker just import it and sort all that out. I'm open to suggestions, but I don't think we have any intention of running this from the command line in production. For testing purposes though, it's very easy to setup a venv, install ci-utils, and run this.

> If we intend to import and use publish_image() inside workers, it should be a
> module (as the other ci-airline modules) and all CLIs could live in a module,
> let's say 'ubuntucore', which will eventually provide worker implementations
> for the services we need.
Why wouldn't we put the cli/modules with the individual service rather than all together? Are you saying they should all live together, or just all under one root like I was talking about above?

review: Needs Information

[Celso Providelo (cprov)]

Paul,

I remember we agreeing that 'ubuntucore' was the right term to use for this story, since it's wider (and less fun) than 'snappy'. But my memory might be failing me.

Considering the imminent switch, I am retracting my concerns about having "image-importer" directory for hosting the CLI and TBD image_importer module for a rabbit worker. It is a standalone service and it matches the tree organisation.

review: Approve

[Evan (ev)]

On 15 January 2015 at 18:26, Celso Providelo
<email address hidden> wrote:
> I remember we agreeing that 'ubuntucore' was the right term to use for this story, since it's wider (and less fun) than 'snappy'. But my memory might be failing me.

The images we're testing are Ubuntu Core. The tests check that snappy
still functions.

[lp:~pwlars/uci-engine/snappy-imageimport]

923. By Paul Larson on 2015-01-16

s/snappy/ubuntucore

[Paul Larson (pwlars)]

Ok, in that case, I've changed the name to ubuntucore right now to take out the reference to snappy. It's easy enough to move around later to if we want to discuss changes to the layout. Going to go ahead and merge with that so that it's here for after we get done with the new sprint priorities.

[Paul Larson (pwlars)]

review: Approve

=== added directory 'ubuntucore-importimage'
=== added file 'ubuntucore-importimage/glanceimage.py'
--- ubuntucore-importimage/glanceimage.py	1970-01-01 00:00:00 +0000
+++ ubuntucore-importimage/glanceimage.py	2015-01-16 16:51:05 +0000
@@ -0,0 +1,86 @@
+#!/usr/bin/env python
+# Ubuntu CI Engine
+# Copyright 2015 Canonical Ltd.
+
+# This program is free software: you can redistribute it and/or modify it
+# under the terms of the GNU Affero General Public License version 3, as
+# published by the Free Software Foundation.
+
+# This program is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranties of
+# MERCHANTABILITY, SATISFACTORY QUALITY, or FITNESS FOR A PARTICULAR
+# PURPOSE.  See the GNU Affero General Public License for more details.
+
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+import argparse
+import logging
+import os
+import requests
+import sys
+import time
+import urlparse
+
+import ci_utils
+from ci_utils import unit_config
+from ci_utils.image_store import ImageStore
+
+
+def _download(url, path, retry=3):
+    ''' Download from a url, retry up to 3 times by default '''
+    urlpath = urlparse.urlsplit(url).path
+    filename = os.path.basename(urlpath)
+    filename = os.path.join(path, filename)
+    try:
+        response = requests.get(url, stream=True)
+        response.raise_for_status()
+        with open(filename, 'wb') as f:
+            for chunk in response.iter_content(chunk_size=64 * 1024):
+                # Ignore keep-alives
+                if not chunk:
+                    continue
+                f.write(chunk)
+                f.flush()
+    except (requests.ConnectionError, requests.HTTPError) as e:
+        if retry == 1:
+            raise(e)
+        logging.warn('Download failed, retrying in 30 sec')
+        time.sleep(30)
+        return _download(url, path, retry - 1)
+    return filename
+
+
+def publish_image(image_url, glance_filename):
+    glance = ImageStore(unit_config.get_auth_config())
+    image_list = [x.name for x in glance.glance.images.list()]
+    if glance_filename in image_list:
+        logging.info(
+            '{} is already in glance, exiting'.format(glance_filename))
+        sys.exit(0)
+    with ci_utils.TmpDir() as tmpdir:
+        logging.info('Downloading {}'.format(image_url))
+        local_copy = _download(image_url, tmpdir)
+        glance.image_create(glance_filename, local_copy)
+
+
+def _get_args():
+    parser = argparse.ArgumentParser(
+        description='Download an image and push it to glance')
+    parser.add_argument('-u', '--url', required=True,
+                        help='URL of the image to download')
+    parser.add_argument('-n', '--name', required=True,
+                        help='Name to store the image as in glance')
+    args = parser.parse_args()
+    return args
+
+
+def main():
+    logging.basicConfig(level=logging.INFO,
+                        format="%(asctime)s %(levelname)s %(message)s")
+    args = _get_args()
+    publish_image(args.url, args.name)
+
+
+if __name__ == '__main__':
+    main()

=== added directory 'ubuntucore-importimage/tests'
=== added file 'ubuntucore-importimage/tests/__init__.py'
=== added file 'ubuntucore-importimage/tests/test_glanceimage.py'
--- ubuntucore-importimage/tests/test_glanceimage.py	1970-01-01 00:00:00 +0000
+++ ubuntucore-importimage/tests/test_glanceimage.py	2015-01-16 16:51:05 +0000
@@ -0,0 +1,41 @@
+import mock
+import unittest
+
+import glanceimage
+
+
+class TestDownload(unittest.TestCase):
+    def setUp(self):
+        super(TestDownload, self).setUp()
+        self.fake_url = 'http://somewhere.fake/snappy/disk1.img'
+
+    @mock.patch('glanceimage.ImageStore')
+    @mock.patch('glanceimage.requests')
+    @mock.patch('__builtin__.open')
+    def test_download(self, mock_open, mock_requests, mock_ImageStore):
+        '''Test that downloading a fake image url works properly'''
+        glance_filename = glanceimage._download(self.fake_url, 'imagename')
+        self.assertEqual('imagename/disk1.img', glance_filename)
+
+    @mock.patch('time.sleep')
+    @mock.patch('glanceimage.ImageStore')
+    @mock.patch('__builtin__.open')
+    def test_download_failed(self, mock_open, mock_ImageStore, mock_sleep):
+        '''Test that a faked url that isn't mocked returns ConnectionError'''
+        with self.assertRaises(glanceimage.requests.ConnectionError):
+            glanceimage._download(self.fake_url, 'imagename')
+
+class TestPublishImage(unittest.TestCase):
+    @mock.patch('glanceclient.v1.images.ImageManager.list')
+    @mock.patch('glanceimage._download')
+    def test_image_already_exists(self, mock_download, mock_list):
+        '''Make sure we don't try to get the image if it already exists'''
+        from collections import namedtuple
+        image = namedtuple('image', ['name'])
+        imagelist = image('fakename')
+        mock_list.return_value = [imagelist]
+
+        with self.assertRaises(SystemExit) as e:
+            glanceimage.publish_image('fakeurl', 'fakename')
+        self.assertEqual(e.exception.code, 0)
+        self.assertFalse(mock_download.called)