Merge lp:~thomir-deactivatedaccount/core-image-publisher/build-cloud-image into lp:core-image-publisher

[Thomi Richards]

Work in progress to add support for building and converting the cloud image

Work in progress to add support for building and converting the cloud image

[Thomi Richards (thomir-deactivatedaccount)]

Happy to land once you fix the one note I made in the diff.

[Celso Providelo (cprov)]

As we discussed on IRC, this code will not work in production until the charm gets extended to grant sudo NOPASSWD powers to the worker user ('core-worker', IRC).

I don't necessarily want to block code changes due to production missing features, but I will only feel comfortable approving this changes if someone is already assigned to do the charm work. Otherwise it will get lost ...

review: Needs Information

[Para Siva (psivaa)]

One question inline. But that could be postponed for another MP.

[Celso Providelo (cprov)]

review: Approve

[Celso Providelo (cprov)]

Actually, u-d-f call needs a tweak (shell=True) and better check if qemu-img call is in the same situation.

review: Needs Fixing

[Celso Providelo (cprov)]

Thomi,

Ignore my comments about u-d-f calls requiring shell, but double check kombu docs about reject vs. requeue behaviour. I think psivaa is right, according to docs, we should call requeue() not reject().

review: Needs Fixing

[Thomi Richards (thomir-deactivatedaccount)]

Yeah, I agree - changed them all to requeue() calls instead.

[Kit Randel (blr)]

Looks fine, although I may be lacking some context. :)

review: Approve (code)

[Ubuntu CI Bot (uci-bot)]

No proposals found for merge of https://code.launchpad.net/~pwlars/core-image-publisher/fix-logger into https://code.launchpad.net/~canonical-ci-engineering/core-image-publisher/trunk.

[Thomi Richards (thomir-deactivatedaccount)]

review: Approve

[Ubuntu CI Bot (uci-bot)]

The attempt to merge lp:~thomir/core-image-publisher/build-cloud-image into lp:core-image-publisher failed. Below is the output from the failed tests.

Using base prefix '/usr'
New python executable in /tmp/venv-core-image-publisherjgiynnkc/bin/python3
Also creating executable in /tmp/venv-core-image-publisherjgiynnkc/bin/python
Installing setuptools, pip...done.
Running virtualenv with interpreter /usr/bin/python3
Ignoring indexes: https://pypi.python.org/simple/
Downloading/unpacking kombu==3.0.24 (from -r requirements.txt (line 1))
Downloading/unpacking python-logstash==0.4.2 (from -r requirements.txt (line 2))
  Running setup.py (path:/tmp/venv-core-image-publisherjgiynnkc/build/python-logstash/setup.py) egg_info for package python-logstash

Downloading/unpacking anyjson>=0.3.3 (from kombu==3.0.24->-r requirements.txt (line 1))
  Running setup.py (path:/tmp/venv-core-image-publisherjgiynnkc/build/anyjson/setup.py) egg_info for package anyjson

Downloading/unpacking amqp>=1.4.5,<2.0 (from kombu==3.0.24->-r requirements.txt (line 1))
  Running setup.py (path:/tmp/venv-core-image-publisherjgiynnkc/build/amqp/setup.py) egg_info for package amqp

Installing collected packages: kombu, python-logstash, anyjson, amqp
  Running setup.py install for python-logstash

  Running setup.py install for anyjson
    Fixing build/lib/anyjson/__init__.py
    Skipping implicit fixer: buffer
    Skipping implicit fixer: idioms
    Skipping implicit fixer: set_literal
    Skipping implicit fixer: ws_comma
    Fixing build/lib/anyjson/__init__.py
    Skipping implicit fixer: buffer
    Skipping implicit fixer: idioms
    Skipping implicit fixer: set_literal
    Skipping implicit fixer: ws_comma

  Running setup.py install for amqp

Successfully installed kombu python-logstash anyjson amqp
Cleaning up...
Downloading/unpacking testtools==1.7.1 (from -r test_requirements.txt (line 1))
Downloading/unpacking python-mimeparse (from testtools==1.7.1->-r test_requirements.txt (line 1))
  Downloading python-mimeparse-0.1.4.tar.gz
  Running setup.py (path:/tmp/venv-core-image-publisherjgiynnkc/build/python-mimeparse/setup.py) egg_info for package python-mimeparse

Downloading/unpacking traceback2 (from testtools==1.7.1->-r test_requirements.txt (line 1))
  Downloading traceback2-1.4.0-py2.py3-none-any.whl
Downloading/unpacking unittest2>=1.0.0 (from testtools==1.7.1->-r test_requirements.txt (line 1))
Downloading/unpacking extras (from testtools==1.7.1->-r test_requirements.txt (line 1))
  Downloading extras-0.0.3.tar.gz
  Running setup.py (path:/tmp/venv-core-image-publisherjgiynnkc/build/extras/setup.py) egg_info for package extras

Downloading/unpacking linecache2 (from traceback2->testtools==1.7.1->-r test_requirements.txt (line 1))
  Downloading linecache2-1.0.0-py2.py3-none-any.whl
Downloading/unpacking six>=1.4 (from unittest2>=1.0.0->testtools==1.7.1->-r test_requirements.txt (line 1))
  Downloading six-1.9.0-py2.py3-none-any.whl
Downloading/unpacking argparse (from unittest2>=1.0.0->testtools==1.7.1->-r test_requirements.txt (line 1))
  Downloading argparse-1.3.0-py2.py3-none-any.whl
Installing collected packages: testtools, python...

[lp:~thomir-deactivatedaccount/core-image-publisher/build-cloud-image]

14. By Thomi Richards on 2015-03-29

Fix typo.

=== modified file 'core_image_publisher/worker.py'
--- core_image_publisher/worker.py	2015-03-27 04:47:12 +0000
+++ core_image_publisher/worker.py	2015-03-29 21:10:35 +0000
@@ -18,6 +18,9 @@
 """Business logic or the service lives here."""
 
 import logging
+import os
+import subprocess
+import tempfile
 from core_image_publisher.queue import enqueue_message
 
 logger = logging.getLogger(__name__)
@@ -36,25 +39,63 @@
             e,
             extra=payload
         )
-        message.reject()
+        message.requeue()
         return
 
-    image_path = download_image(image_name, channel, device)
-    nova_image_path = convert_nova_image(image_path)
-    glance_image_name = upload_image_to_glance(nova_image_path)
+    with tempfile.TemporaryDirectory as tmpdir:
+        try:
+            image_path = download_image(image_name, channel, device, tmpdir)
+        except subprocess.CalledProcessError as e:
+            logger.error("Unable to download core image: %s", e, extra=payload)
+            message.requeue()
+            return
+        try:
+            nova_image_path = convert_nova_image(image_path)
+        except subprocess.CalledProcessError as e:
+            logger.error("Unable to convert core image to qcow2 image: %s", e,
+                extra=payload)
+            message.requeue()
+            return
+        glance_image_name = upload_image_to_glance(nova_image_path)
 
     payload['nova_image'] = glance_image_name
     enqueue_message(payload)
     message.ack()
 
 
-def download_image(name, channel, device):
+def download_image(name, channel, device, tmpdir):
     """Download the ubuntu code image, return a path to it on disk."""
+    image_path = os.path.join(tmpdir, 'core-{}.img'.format(name))
+    cmd = ['sudo',
+           'ubuntu-device-flash',
+           '--revision', name,
+           'core',
+           '--device', generic_amd64,
+           '--channel', channel,
+           '--size', '3',
+           '-o', image_path,
+           '--developer-mode',
+           '--cloud']
+    subprocess.check_call(cmd)
+    return image_path
 
 
 def convert_nova_image(image_path):
-    """Convert a core image to a nova image, return path to the converted image."""
+    """
+    Convert a core image to a nova image, return path to the converted image.
+    """
+    converted_image_path = '{}-cloud.img'.format(image_path.strip('.img'))
+    cmd = ['qemu-img',
+           'convert',
+           '-f', 'raw',
+           '-O', 'qcow2',
+           image_path,
+           converted_image_path]
+    subprocess.check_call(cmd)
+    return converted_image_path
 
 
 def upload_image_to_glance(nova_image_path):
-    """Upload the nova image to glance, returning the name of the image in glance."""
+    """
+    Upload the nova image to glance, returning the name of the image in glance.
+    """