Merge lp:~fginther/adt-image-mapper/called-by-tarmac into lp:adt-image-mapper

[Francis Ginther]

Add 'called-by-tarmac.py' to execute unit tests automatically by tarmac.

Add 'called-by-tarmac.py' to execute unit tests automatically by tarmac.

[Celso Providelo (cprov)]

review: Approve

[Ubuntu CI Bot (uci-bot)]

The attempt to merge lp:~fginther/adt-image-mapper/called-by-tarmac into lp:adt-image-mapper failed. Below is the output from the failed tests.

Using base prefix '/usr'
New python executable in /tmp/venv-adt-image-mapperbsiv4kvj/bin/python3
Also creating executable in /tmp/venv-adt-image-mapperbsiv4kvj/bin/python
Installing setuptools, pip...done.
Running virtualenv with interpreter /usr/bin/python3
Ignoring indexes: https://pypi.python.org/simple/
Downloading/unpacking cffi (from -r requirements.txt (line 5))
  Running setup.py (path:/tmp/venv-adt-image-mapperbsiv4kvj/build/cffi/setup.py) egg_info for package cffi

Downloading/unpacking pyasn1 (from -r requirements.txt (line 6))
  Running setup.py (path:/tmp/venv-adt-image-mapperbsiv4kvj/build/pyasn1/setup.py) egg_info for package pyasn1

Downloading/unpacking flask==0.10.1 (from -r requirements.txt (line 7))
  Running setup.py (path:/tmp/venv-adt-image-mapperbsiv4kvj/build/flask/setup.py) egg_info for package flask

    warning: no files found matching '*' under directory 'tests'
    warning: no previously-included files matching '*.pyc' found under directory 'docs'
    warning: no previously-included files matching '*.pyo' found under directory 'docs'
    warning: no previously-included files matching '*.pyc' found under directory 'tests'
    warning: no previously-included files matching '*.pyo' found under directory 'tests'
    warning: no previously-included files matching '*.pyc' found under directory 'examples'
    warning: no previously-included files matching '*.pyo' found under directory 'examples'
    no previously-included directories found matching 'docs/_build'
    no previously-included directories found matching 'docs/_themes/.git'
Downloading/unpacking gunicorn==19.3.0 (from -r requirements.txt (line 8))
Downloading/unpacking python-glanceclient==0.14.0 (from -r requirements.txt (line 9))
Downloading/unpacking python-keystoneclient==1.2.0 (from -r requirements.txt (line 10))
Downloading/unpacking pycparser (from cffi->-r requirements.txt (line 5))
  Running setup.py (path:/tmp/venv-adt-image-mapperbsiv4kvj/build/pycparser/setup.py) egg_info for package pycparser

Downloading/unpacking Werkzeug>=0.7 (from flask==0.10.1->-r requirements.txt (line 7))
  Running setup.py (path:/tmp/venv-adt-image-mapperbsiv4kvj/build/Werkzeug/setup.py) egg_info for package Werkzeug

    warning: no previously-included files matching '*.pyc' found under directory 'docs'
    warning: no previously-included files matching '*.pyo' found under directory 'docs'
    warning: no previously-included files matching '*.pyc' found under directory 'tests'
    warning: no previously-included files matching '*.pyo' found under directory 'tests'
    warning: no previously-included files matching '*.pyc' found under directory 'examples'
    warning: no previously-included files matching '*.pyo' found under directory 'examples'
    no previously-included directories found matching 'docs/_build'
    no previously-included directories found matching 'docs/_themes'
Downloading/unpacking Jinja2>=2.4 (from flask==0.10.1->-r requirements.txt (line 7))
  Running setup.py (path:/tmp/venv-adt-image-ma...

[Paul Larson (pwlars)]

This runs locally for me as long as I have python3-dev installed. I checked tarmac and it seems to have it installed already though. Could it be something in the way tarmac is calling it?

[Francis Ginther (fginther)]

plars, I had the exact same thought and just barely beat you to it. I just installed python3-dev on the tarmac system (testing this branch before and seeing the same failure). Now it's passing so re-approving.

[Celso Providelo (cprov)]

We might say that is the expected setup environment documented in our charms:

{{{
REQUIRED_PACKAGES = ['python-virtualenv', 'python3-dev']
}}}

And we got lucky that tarmac already had venv support (from the uci-e times).

We should update our tarmac charm/config for supporting py3 projects.

=== added file 'called-by-tarmac.py'
--- called-by-tarmac.py	1970-01-01 00:00:00 +0000
+++ called-by-tarmac.py	2015-03-17 15:07:45 +0000
@@ -0,0 +1,102 @@
+#!/usr/bin/env python3
+#
+# adt-image-mapper
+# Copyright (C) 2015 Canonical
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+# called-by-tarmac.py is intended to be called by the tarmac merge engine
+#   to perform automatic unit testing on branch landing.
+#   It replaces the manual commands:
+#       bzr branch $PIP_CACHE_BRANCH pip_cache
+#       virtualenv -p python3 ve && \
+#           ./ve/bin/pip install --no-index --find-links=pip-cache \
+#              -r requirements.txt && \
+#           ./ve/bin/pip install -r test_requirements.txt && \
+#           ./ve/bin/python3 setup.py test && \
+#           rm -rf ve
+#
+
+import os
+import subprocess
+import sys
+import tempfile
+
+SERVICE_NAME = 'adt-image-mapper'
+VENV_DIR = 'venv-{}'.format(SERVICE_NAME)
+PIP_DIR = 'pip-cache-'.format(SERVICE_NAME)
+PIP_CACHE_BRANCH = 'lp:~canonical-ci-engineering/{}/' \
+    'pip-cache'.format(SERVICE_NAME)
+
+
+def _run_command(cmd):
+    '''Run the command and return the return code or 0 on success.'''
+
+    print('INFO: Executing: {}'.format(' '.join(cmd)))
+    try:
+        subprocess.check_call(cmd, stderr=subprocess.STDOUT)
+    except subprocess.CalledProcessError as e:
+        print('ERROR: Calling `{}` failed (with code {})'.format(
+            e.cmd, e.returncode))
+        return e.returncode
+    return 0
+
+
+def main():
+    with tempfile.TemporaryDirectory(PIP_DIR) as pip_dir, \
+            tempfile.TemporaryDirectory(prefix=VENV_DIR) as ve_dir:
+        pip_cache = os.path.join(pip_dir, 'cache')
+        all_cmds = (
+            [
+                'bzr',
+                'branch',
+                PIP_CACHE_BRANCH,
+                pip_cache,
+            ],
+            [
+                'virtualenv',
+                '-p',
+                'python3',
+                ve_dir,
+            ],
+            [
+                os.path.join(ve_dir, 'bin', 'pip'),
+                'install',
+                '--no-index',
+                '--find-links={}'.format(pip_cache),
+                '-r',
+                'requirements.txt',
+            ],
+            [
+                os.path.join(ve_dir, 'bin', 'pip'),
+                'install',
+                '-r',
+                'test_requirements.txt',
+            ],
+            [
+                os.path.join(ve_dir, 'bin', 'python3'),
+                'setup.py',
+                'test',
+            ],
+        )
+        for cmd in all_cmds:
+            ret = _run_command(cmd)
+            if ret:
+                # A cmd failed, bubble up the return code to the caller
+                return ret
+    return 0
+
+
+if __name__ == '__main__':
+    sys.exit(main())