Merge lp:~joetalbott/core-image-builder/add_build_logic into lp:core-image-builder

[Joe Talbott]

Add first pass at image building logic.

Add first pass at image building logic.

[Para Siva (psivaa)]

Could not find any issues except a question inline.

[Francis Ginther (fginther)]

I've logged several comments, but nothing that should block this current MP. In fact, I would feel better landing this and then iterating on the comments in other MPs.

Also, it is my opinion that the channel and release input needs to come from the core-tester-agent. This is to solve the problem of needing to test both vivid and wily packages that land into two different core channels. Perhaps there is another solution that I've overlooked.

review: Approve

[Ubuntu CI Bot (uci-bot)]

The attempt to merge lp:~joetalbott/core-image-builder/add_build_logic into lp:core-image-builder failed. Below is the output from the failed tests.

Using base prefix '/usr'
New python executable in /tmp/venv-core-image-builder7hpvuyab/bin/python3
Also creating executable in /tmp/venv-core-image-builder7hpvuyab/bin/python
Installing setuptools, pip...done.
Running virtualenv with interpreter /usr/bin/python3
Ignoring indexes: https://pypi.python.org/simple/
Downloading/unpacking cffi (from -r requirements.txt (line 5))
  Running setup.py (path:/tmp/venv-core-image-builder7hpvuyab/build/cffi/setup.py) egg_info for package cffi

Downloading/unpacking pyasn1 (from -r requirements.txt (line 6))
  Running setup.py (path:/tmp/venv-core-image-builder7hpvuyab/build/pyasn1/setup.py) egg_info for package pyasn1

Downloading/unpacking kombu==3.0.24 (from -r requirements.txt (line 7))
Downloading/unpacking python-logstash==0.4.2 (from -r requirements.txt (line 8))
  Running setup.py (path:/tmp/venv-core-image-builder7hpvuyab/build/python-logstash/setup.py) egg_info for package python-logstash

Downloading/unpacking python-glanceclient==0.17.0 (from -r requirements.txt (line 9))
Downloading/unpacking uservice-utils==1.0.3 (from -r requirements.txt (line 10))
Downloading/unpacking traceback2==1.4.0 (from -r requirements.txt (line 11))
Downloading/unpacking pycparser (from cffi->-r requirements.txt (line 5))
  Running setup.py (path:/tmp/venv-core-image-builder7hpvuyab/build/pycparser/setup.py) egg_info for package pycparser

    warning: no previously-included files matching 'yacctab.*' found under directory 'tests'
    warning: no previously-included files matching 'lextab.*' found under directory 'tests'
    warning: no previously-included files matching 'yacctab.*' found under directory 'examples'
    warning: no previously-included files matching 'lextab.*' found under directory 'examples'
Downloading/unpacking anyjson>=0.3.3 (from kombu==3.0.24->-r requirements.txt (line 7))
  Running setup.py (path:/tmp/venv-core-image-builder7hpvuyab/build/anyjson/setup.py) egg_info for package anyjson

Downloading/unpacking amqp>=1.4.5,<2.0 (from kombu==3.0.24->-r requirements.txt (line 7))
  Running setup.py (path:/tmp/venv-core-image-builder7hpvuyab/build/amqp/setup.py) egg_info for package amqp

Downloading/unpacking argparse (from python-glanceclient==0.17.0->-r requirements.txt (line 9))
Downloading/unpacking python-keystoneclient>=1.0.0 (from python-glanceclient==0.17.0->-r requirements.txt (line 9))
Downloading/unpacking warlock>=1.0.1,<2 (from python-glanceclient==0.17.0->-r requirements.txt (line 9))
  Running setup.py (path:/tmp/venv-core-image-builder7hpvuyab/build/warlock/setup.py) egg_info for package warlock

Downloading/unpacking Babel>=1.3 (from python-glanceclient==0.17.0->-r requirements.txt (line 9))
  Running setup.py (path:/tmp/venv-core-image-builder7hpvuyab/build/Babel/setup.py) egg_info for package Babel

    warning: no previously-included files matching '*' found under directory 'docs/_build'
    warning: no previously-included files matching '*.pyc' found under directory 'tests'
    warning: no previously-...

[lp:~joetalbott/core-image-builder/add_build_logic]

5. By Joe Talbott on 2015-05-26

Fix flake8 warning

[Joe Talbott (joetalbott)]

One more commit to fix a silly flake8 warning. I'm self-approving since it's so minor.

review: Approve

=== modified file 'core_image_builder/__init__.py'
--- core_image_builder/__init__.py	2015-05-19 21:37:40 +0000
+++ core_image_builder/__init__.py	2015-05-26 18:29:22 +0000
@@ -28,7 +28,7 @@
 
 from core_image_builder.queue import (
     create_connection_from_config,
-    CoreImageResultBuilder,
+    CoreImageBuilder,
 )
 from core_image_builder import (
     constants,
@@ -67,14 +67,14 @@
     connection = create_connection_from_config(config)
     try:
         with connection:
-            builder = CoreImageResultBuilder(connection)
+            builder = CoreImageBuilder(connection)
             retry_policy = DefaultRetryPolicy(
                 max_retries=3,
                 dead_queue='core.deadletters.{}'.format(constants.API_VERSION)
             )
             monitor = SimpleRabbitQueueWorker(
                 connection,
-                'core.image.{}'.format(constants.API_VERSION),
+                'core.package.{}'.format(constants.API_VERSION),
                 worker.ImageBuilderWorker(config, builder),
                 retry_policy,
             )

=== modified file 'core_image_builder/queue.py'
--- core_image_builder/queue.py	2015-05-19 21:37:40 +0000
+++ core_image_builder/queue.py	2015-05-26 18:29:22 +0000
@@ -24,7 +24,7 @@
 logger = logging.getLogger(__name__)
 
 __all__ = [
-    'CoreImageResultBuilder',
+    'CoreImageBuilder',
     'create_connection_from_config',
 ]
 
@@ -39,7 +39,7 @@
     return kombu.Connection(amqp_uris)
 
 
-class CoreImageResultBuilder(object):
+class CoreImageBuilder(object):
 
     """A callable that knows how to publish results from this service.
 

=== modified file 'core_image_builder/worker.py'
--- core_image_builder/worker.py	2015-05-19 21:37:40 +0000
+++ core_image_builder/worker.py	2015-05-26 18:29:22 +0000
@@ -17,6 +17,7 @@
 
 """Business logic or the service lives here."""
 
+import glob
 import logging
 import os
 import subprocess
@@ -26,7 +27,10 @@
 
 from core_image_builder.constants import LOGGING_EXTRA
 from core_image_builder.cloud import get_glance_client
-from core_image_builder.utils import check_call
+from core_image_builder.utils import (
+    check_call,
+    BetterCalledProcessError,
+)
 
 
 logger = logging.getLogger(__name__)
@@ -47,6 +51,7 @@
             image_name = payload['image_name']
             channel = payload['channel']
             device = payload['device']
+            package_name = payload['package_name']
         except KeyError as e:
             logger.error(
                 "Unable to deserialize message payload - "
@@ -57,9 +62,9 @@
             return MessageActions.Retry
 
         with tempfile.TemporaryDirectory() as tmpdir:
-            logger.info("Beginning image download.", extra=extra)
+            logger.info("Beginning rootfs download.", extra=extra)
             try:
-                image_path = download_image(
+                rootfs_path = download_rootfs(
                     image_name,
                     channel,
                     device,
@@ -72,7 +77,61 @@
                     extra=extra
                 )
                 return MessageActions.Retry
-            logger.info("Ubuntu Core image downloaded OK.", extra=extra)
+
+            logger.info("Adding packages from proposed")
+            MAX_RETRIES = 3
+            count = MAX_RETRIES
+            while count > 0:
+                new_rootfs_path = add_proposed_packages(rootfs_path,
+                                                        package_name)
+                if new_rootfs_path is not None:
+                    break
+
+                # try removing the rootfs as it sometimes is corrupted.
+                # XXX: find out why it's getting corrupted.
+                logger.info("Retrying rootfs download.", extra=extra)
+                os.remove(rootfs_path)
+                try:
+                    rootfs_path = download_rootfs(
+                        image_name,
+                        channel,
+                        device,
+                        tmpdir
+                    )
+                except subprocess.CalledProcessError as e:
+                    logger.error(
+                        "Unable to download core image: %s",
+                        e,
+                        extra=extra
+                    )
+                    return MessageActions.Retry
+
+                count -= 1
+            if count == 0:
+                logger.error(
+                    "Unable to add packages after %d tries: %s",
+                    MAX_RETRIES,
+                    package_name,
+                )
+                return MessageActions.Retry
+
+            try:
+                logger.info("Building the image")
+                image_path = build_image(
+                    image_name,
+                    new_rootfs_path,
+                    channel,
+                    device,
+                    tmpdir
+                )
+            except subprocess.CalledProcessError as e:
+                logger.error(
+                    "Unable to build the image: %s",
+                    e,
+                    extra=extra
+                )
+                return MessageActions.Retry
+            logger.info("Ubuntu Core image build OK.", extra=extra)
 
             try:
                 nova_image_path = convert_nova_image(image_path)
@@ -106,34 +165,94 @@
         return MessageActions.Acknowledge
 
 
-def download_image(name, channel, device, tmpdir):
-    """Download the ubuntu code image, return a path to it on disk."""
-    image_path = os.path.join(tmpdir, 'core-{}.img'.format(name))
-    udf_env = {
-        # u-d-f fills '~/.cache/ubuntuimages/' on downloads,
-        # not particulary useful to us, unless we only have one
-        # worker and perform testing retries.
-        'HOME': tmpdir,
-    }
-    # The folling uses hardcoded parameters for '--channel' and 'release'.
-    # It is ignoring the channel value passed in.
-    cmd = ['sudo',
-           'ubuntu-device-flash',
-           '--revision', name,
-           'core',
-           '--device', device,
-           '--channel', 'edge',
-           '--size', '3',
-           '-o', image_path,
-           '--developer-mode',
-           '--cloud',
-           'rolling']
-    check_call(cmd, env=udf_env)
-    cmd = ['sudo',
-           'rm',
-           '-rf',
-           os.path.join(tmpdir, '.gnupg')]
-    check_call(cmd, env=udf_env)
+def download_rootfs(name, channel, device, tmpdir):
+    """Download the ubuntu code image, return a path to it on disk."""
+    image_path = os.path.join(tmpdir, 'core-{}.img'.format(name))
+
+    # The following uses hardcoded parameters for '--channel' and 'release'.
+    # It is ignoring the channel value passed in.
+    cmd = ['sudo',
+           './ubuntu-device-flash',
+           '--revision', name,
+           '--download-only',
+           'core',
+           '--device', device,
+           '--channel', 'edge',
+           '--size', '3',
+           '-o', image_path,
+           '--developer-mode',
+           '--cloud',
+           'rolling']
+    check_call(cmd)
+
+    cmd = ['sudo',
+           'rm',
+           '-rf',
+           os.path.join(tmpdir, '.gnupg')]
+    check_call(cmd)
+    output = glob.glob('ubuntu-*.tar.xz')
+    return output[0]
+
+
+def add_proposed_packages(rootfs_path, package_name):
+
+    new_rootfs_path = rootfs_path.replace('.tar.xz', '-modified.tar.xz')
+
+    ROOTFS_BASE_DIR = 'system'
+
+    # untar the rootfs
+    cmd = ['sudo', 'tar', 'xJf', rootfs_path]
+    try:
+        check_call(cmd)
+    except BetterCalledProcessError as e:
+        logger.error(e)
+        return None
+
+    # sanity check that the base directory exists
+    if not os.path.exists('system'):
+        logger.error("Failed to untar the rootfs: %s", rootfs_path)
+        return None
+
+    # chroot into the rootfs and install packages
+    cmd = ['sudo',  'chroot', os.path.join(os.getcwd(), ROOTFS_BASE_DIR),
+           'apt-get', ' update']
+    check_call(cmd)
+    cmd = ['sudo',  'chroot', os.path.join(os.getcwd(), ROOTFS_BASE_DIR),
+           'apt-get', ' install ', package_name]
+    check_call(cmd)
+
+    # tar the rootfs back up.
+    cmd = ['sudo', 'tar', 'cJf', new_rootfs_path, ROOTFS_BASE_DIR]
+    check_call(cmd)
+
+    return new_rootfs_path
+
+
+def build_image(name, rootfs_path, channel, device, tmpdir):
+    """Download the ubuntu code image, return a path to it on disk."""
+    image_path = os.path.join(tmpdir, 'core-{}.img'.format(name))
+    # The following uses hardcoded parameters for '--channel' and 'release'.
+    # It is ignoring the channel value passed in.
+    cmd = ['sudo',
+           './ubuntu-device-flash',
+           '--revision', name,
+           'core',
+           '--device', device,
+           '--channel', 'edge',
+           '--size', '3',
+           '--image-part', rootfs_path,
+           '-o', image_path,
+           '--developer-mode',
+           '--cloud',
+           'rolling']
+    check_call(cmd)
+
+    cmd = ['sudo',
+           'rm',
+           '-rf',
+           os.path.join(tmpdir, '.gnupg')]
+    check_call(cmd)
+
     return image_path
 
 

=== added file 'requirements.txt'
--- requirements.txt	1970-01-01 00:00:00 +0000
+++ requirements.txt	2015-05-26 18:29:22 +0000
@@ -0,0 +1,11 @@
+# XXX cprov 20150330: cffi and pyasn1 are cryptography (for glanceclient)
+# dependencies and if not available it uses easy_install for installing
+# and it doesn't respect local pip-cache setup, it ends up hitting pypi
+# on the internet.
+cffi
+pyasn1
+kombu==3.0.24
+python-logstash==0.4.2
+python-glanceclient==0.17.0
+uservice-utils==1.0.3
+traceback2==1.4.0

=== added file 'test_requirements.txt'
--- test_requirements.txt	1970-01-01 00:00:00 +0000
+++ test_requirements.txt	2015-05-26 18:29:22 +0000
@@ -0,0 +1,2 @@
+testtools==1.7.1
+flake8==2.4.0