Merge lp:~psivaa/uci-engine/install-cu2d-from-tarball into lp:uci-engine

[Para Siva]

Install cupstream2distro from the uci-engine tarball and remove the eatmydata LD_PRELOAD errors seen during deployment.

To install cupstream2distro from the uci-engine tarball rather than from lp:lp:cupstream2distro/uci-airline and to remove the eatmydata LD_PRELOAD errors seen during deployment.

[PS Jenkins bot (ps-jenkins)]

PASSED: Continuous integration, rev:600
http://s-jenkins.ubuntu-ci:8080/job/uci-engine-ci/910/
Executed test runs:

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/uci-engine-ci/910/rebuild

review: Approve (continuous-integration)

[Evan (ev)]

Comments inline.

review: Needs Fixing

[lp:~psivaa/uci-engine/install-cu2d-from-tarball]

600. By Celso Providelo on 2014-06-23

[r=Andy Doan, PS Jenkins bot] Series of assorted small fixes related to the task-force for staging deployment and acceptance tests. from Celso Providelo

[Francis Ginther (fginther)]

There is possibly another way to solve this problem, although I haven't figured out yet if it will work. The /cupstream2distro path is already being installed under /srv/${service_name}/.../cupstream2distro by the rabbit-worker charm that installs the parent. The problem I ran into was getting that path to the .pbuilerrc file. AFAICT, in order to communicate the full path, we'd have to create a specific relationship (something-like 'rabbit-worker-extension') that passes the path via a relationship variable.

The solution proposed here actually works :-), and I think is the better solution for now. I have no further comments, so once ev's are addressed, we can merge this.

[lp:~psivaa/uci-engine/install-cu2d-from-tarball]

601. By Joe Talbott on 2014-06-23

[r=Andy Doan, PS Jenkins bot] ticket-system - Add openid and oauth2 support.

  * This is essentially a NOP at this point and shouldn't affect deployments as the
    API does not yet require authentication.
  from Joe Talbott

602. By Andy Doan on 2014-06-23

[r=Evan Dandrea, PS Jenkins bot] rabbit-worker charm: update logs directory

This makes the upstart logging location for our rabbit workers consistent
to what i/s requires. The log directories will become:

 /srv/ci-airline-imagebuild-worker/logs/upstart.log
 /srv/ci-airline-bsb-worker/logs/upstart.log

 etc.

This had a side-effect of changing the upstart service names, which
I fixed the integration test cases for (as well as added one for
bsbuilder since one didn't exist)
  from Andy Doan

603. By Vincent Ladeuil on 2014-06-24

[r=PS Jenkins bot, Evan Dandrea] Ensures setup.sh fails when any setup.py component fails. 1333590 from Vincent Ladeuil

604. By Vincent Ladeuil on 2014-06-24

[r=Francis Ginther, PS Jenkins bot] Fallback to current dir (/srv/<service>) if HOME is not set for the test runner worker. 1331989 from Vincent Ladeuil

605. By Vincent Ladeuil on 2014-06-24

[r=PS Jenkins bot, Evan Dandrea] Add pep8/flake tests for juju-deployer and tests test files and fix found issues. from Vincent Ladeuil

606. By Andy Doan on 2014-06-24

[r=PS Jenkins bot, Evan Dandrea] lander charm: fix logging

i/s confirmed logging for this componet should go to:

 /srv/ci-airline-lander/logs

This required a few changes:

1) change the "service_dir" to use dashes and not underscores
This made a new upstart service name. This broke that json_status cgi
script *again*. So I tried to change this in a way that it can be
updated in the future without breakage.

2) update the "lander logs" location
We were logging results from processing tickets under "logs". I chaged
this to "lander-logs" so it won't collide with our upstart logs.

3) make the charm / upstart script smarter
This gets us logging in the correct place, but it also lets the upstart
job ensure the proper directories exist before the job is launched. This
makes things safer in the event someone deletes the logs directory and
restarts the service
  from Andy Doan

607. By Ursula Junque on 2014-06-24

[r=PS Jenkins bot, Celso Providelo] Fixes glitches on publisher from Ursula Junque

608. By Celso Providelo on 2014-06-24

[r=Andy Doan, PS Jenkins bot] Trivial fixes for ci-airline staging. from Celso Providelo

609. By Celso Providelo on 2014-06-24

[r=PS Jenkins bot, Andy Doan] Extend the current Webui to link to ticket artifacts using tempurls. from Celso Providelo

610. By Francis Ginther on 2014-06-24

[r=Andy Doan, PS Jenkins bot] Bump version of the upstart charm in production-only.yaml to account for the revision bump in the charm itself. from Francis Ginther

611. By Vincent Ladeuil on 2014-06-24

[r=Francis Ginther, PS Jenkins bot] Upload the run_test.py script explicitly without relying on cloud-init. 1333323 from Vincent Ladeuil

612. By Andy Doan on 2014-06-24

[r=Parameswaran Sivatharman, PS Jenkins bot] restish charm: fix to work with i/s requirement

As discussed with i/s this will get us going with:

 /srv/ci-airline-gatekeeper-restish/logs/access.log
 /srv/ci-airline-gatekeeper-restish/logs/gunicorn.log from Andy Doan

613. By Vincent Ladeuil on 2014-06-25

[r=Ursula Junque, PS Jenkins bot] Don't attempt to create ~/.ssh if it already exists. 1334146 from Vincent Ladeuil

614. By Para Siva on 2014-06-25

[r=Celso Providelo, PS Jenkins bot] This removes the creation and usage of ~/.ubuntu-ci. Consequently ubuntu-ci-autoconfig was removed. Requires -D and -S for development and staging deployments from Parameswaran Sivatharman

615. By Vincent Ladeuil on 2014-06-25

[r=Evan Dandrea, PS Jenkins bot, Andy Doan] Touch a stamp file to ensure the dependency is updated. from Vincent Ladeuil

616. By Andy Doan on 2014-06-25

[r=Paul Larson, PS Jenkins bot] django charm: remove need for hard-coded python paths

This lets us specify relative paths, so that our deployer files don't
require updates when we change the charm or service name from Andy Doan

617. By Andy Doan on 2014-06-25

[r=PS Jenkins bot, Francis Ginther, Evan Dandrea] lander: don't crash when no juju-relations exist

We are seeing something in production where the lander is starting
before the juju-relations directory exist. I'm actually surprised
this code ever worked well. This does a check so that we'll wait
properly.

I've also added a small engine-health-check so the exact error will be
visible in our webui.
  from Andy Doan

618. By Celso Providelo on 2014-06-25

[r=PS Jenkins bot, Francis Ginther, Evan Dandrea] Fixing TempUrl generation in canonistack/prodstack. from Celso Providelo

[Para Siva (psivaa)]

Thanks for the comments. Addressed where possible. charmhelpers dont seem to be much of help here. Just replied inline. Could you pls double check

[lp:~psivaa/uci-engine/install-cu2d-from-tarball]

619. By Ursula Junque on 2014-06-25

[r=Francis Ginther, Celso Providelo] Disables chroot-builder install during deployments, will be switched back on when fixing bug 1312271 from Ursula Junque

[PS Jenkins bot (ps-jenkins)]

PASSED: Continuous integration, rev:602
http://s-jenkins.ubuntu-ci:8080/job/uci-engine-ci/938/
Executed test runs:

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/uci-engine-ci/938/rebuild

review: Approve (continuous-integration)

[lp:~psivaa/uci-engine/install-cu2d-from-tarball]

620. By Andy Doan on 2014-06-25

[r=Paul Larson, PS Jenkins bot] restish charm: provide a way to run cron tasks from Andy Doan

[Evan (ev)]

Some small comments inline, but this is basically there. Resolve these and consider it an implicit +1 from me.

review: Approve

[PS Jenkins bot (ps-jenkins)]

FAILED: Continuous integration, rev:603
http://s-jenkins.ubuntu-ci:8080/job/uci-engine-ci/941/
Executed test runs:

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/uci-engine-ci/941/rebuild

review: Needs Fixing (continuous-integration)

[lp:~psivaa/uci-engine/install-cu2d-from-tarball]

621. By Para Siva on 2014-06-26

Conflict resolving

[PS Jenkins bot (ps-jenkins)]

PASSED: Continuous integration, rev:621
http://s-jenkins.ubuntu-ci:8080/job/uci-engine-ci/942/
Executed test runs:

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/uci-engine-ci/942/rebuild

review: Approve (continuous-integration)

[Para Siva (psivaa)]

Thanks for the comments again. Fixed them too. Holding it for https://bugs.launchpad.net/uci-engine/+bug/1312271 to be complete and as per http://bazaar.launchpad.net/~canonical-ci-engineering/uci-engine/trunk/revision/619

[Para Siva (psivaa)]

Review comments addressed

Unmerged revisions

621. By Para Siva on 2014-06-26

Conflict resolving

=== modified file 'charms/precise/chroot-builder/config.yaml'
--- charms/precise/chroot-builder/config.yaml	2014-05-27 06:04:10 +0000
+++ charms/precise/chroot-builder/config.yaml	2014-06-26 14:33:53 +0000
@@ -20,3 +20,20 @@
             If specified, this will be used as the base url for downloading
             pre-built chroots. The format for the individual files is:
               {chroot_base_url}/{series}-{architecture}.tgz
+  install_root:
+    type: string
+    description: "The root directory the service will be installed in"
+    default: "/srv/"
+  tarball:
+    type: string
+    description: A URL to a tarball of the code
+  vcs:
+    default: "branch"
+    type: string
+    description: "Grab code from a bzr 'branch' or a URL to a 'tarball'"
+  branch:
+    type: string
+    description: "BZR branch the service lives in"
+  revno:
+    type: string
+    description: "Revision or tag to branch from"
 
=== modified file 'charms/precise/chroot-builder/hooks/hooks.py'
--- charms/precise/chroot-builder/hooks/hooks.py	2014-06-03 18:34:17 +0000
+++ charms/precise/chroot-builder/hooks/hooks.py	2014-06-26 14:33:53 +0000
@@ -29,6 +29,8 @@
 
 CHROOT_USER_HOME = '/root'
 CHROOT_ROOT_PATH = '/var/cache/pbuilder'
+EAT_MY_DATA_PRELOAD_LINE = "export LD_PRELOAD=\"${LD_PRELOAD:+$LD_PRELOAD:}\
+/usr/lib/libeatmydata/libeatmydata.so\"\n"
 
 
 def juju_info(msg, level='INFO'):
@@ -143,34 +145,93 @@
     return 0
 
 
+def _service_name(config):
+    '''Returns the service name in a form useful
+    for using as a directory component, with
+    characters that would need to be escaped replaced with _.'''
+    unit = core.hookenv.service_name()
+    for x in (':', '/', '"', "'"):
+        unit = unit.replace(x, '_')
+    return unit
+
+
+def _service_dir(config):
+    return os.path.join(config['install_root'], _service_name(config))
+
+
 def setup_dependencies(config):
-    required_packages = ['bzr', 'cowbuilder']
+    required_packages = ['bzr', 'cowbuilder', 'eatmydata']
     fetch.apt_update()
     juju_info('installing apt packages...')
     fetch.apt_install(required_packages, fatal=True)
 
 
 def setup_pbuilderrc(config):
-    cu2d_dir = '/srv/cu2d'
-    if os.path.exists(cu2d_dir):
-        subprocess.check_call(['bzr', 'update', cu2d_dir])
+    cu2d_dir = _service_dir(config)
+    if config.get('vcs') == 'tarball':
+        _install_from_tarball(config, os.path.exists(cu2d_dir))
     else:
-        try:
-            subprocess.check_call(['bzr', 'branch',
-                                   'lp:cupstream2distro/uci-airline',
-                                   cu2d_dir])
-        except subprocess.CalledProcessError as e:
-            raise EnvironmentError(
-                'Could not get cu2d (return: {})'.format(e.returncode))
+        _install_from_bzr(config, os.path.exists(cu2d_dir))
 
-    shutil.copy(os.path.join(cu2d_dir, 'chroot-tools', '.pbuilderrc'),
+    shutil.copy(os.path.join(cu2d_dir, 'cupstream2distro', 'chroot-tools',
+                             '.pbuilderrc'),
                 CHROOT_USER_HOME)
 
 
+def _install_from_tarball(config, upgrade):
+    tarball_url = config['tarball']
+    juju_info('grabbing service from tarball {}...'.format(tarball_url))
+    cu2d_dir = _service_dir(config)
+    if not upgrade:
+        juju_info(
+            'Creating service directory: {}'.format(cu2d_dir))
+        os.mkdir(cu2d_dir)
+    cmd = 'curl %s | tar -xzC %s' % (tarball_url, cu2d_dir)
+    subprocess.check_call(cmd, shell=True)
+
+
+def _install_from_bzr(config, upgrade):
+    juju_info('grabbing service from bzr...')
+
+    cu2d_dir = _service_dir(config)
+    rev = config.get('revno', '')
+
+    if upgrade:
+        args = ['bzr', 'pull']
+        if rev:
+            args.extend(['-r', rev])
+        subprocess.check_call(args, cwd=cu2d_dir)
+        return
+    else:
+        juju_info('deleting pre-existing service directory: %s' % cu2d_dir)
+        shutil.rmtree(cu2d_dir)
+
+    args = ['bzr', 'branch']
+    if rev:
+        args.extend(['-r', rev])
+    args.append(config['branch'])
+    args.append(cu2d_dir)
+    subprocess.check_call(args)
+
+
+def update_pbuilderrc(config):
+    pbuilderrc = os.path.join(CHROOT_USER_HOME, '.pbuilderrc')
+    if EAT_MY_DATA_PRELOAD_LINE not in open(pbuilderrc, 'rb').readlines()[-1]:
+        try:
+            fh = open(pbuilderrc, 'a')
+            fh.write(EAT_MY_DATA_PRELOAD_LINE)
+        except IOError:
+            print "Can't update pbuiderrc"
+        else:
+            print 'pbuilderrc updated for chroot'
+        fh.close()
+
+
 def config_changed(config):
     setup_dependencies(config)
     setup_pbuilderrc(config)
     setup_chroots(config)
+    update_pbuilderrc(config)
 
 
 def main():
 
=== modified file 'cupstream2distro/chroot-tools/.pbuilderrc'
--- cupstream2distro/chroot-tools/.pbuilderrc	2014-06-10 20:42:46 +0000
+++ cupstream2distro/chroot-tools/.pbuilderrc	2014-06-26 14:33:53 +0000
@@ -30,4 +30,3 @@
 # EXTRAPACKAGES is not obeyed by 'cowbuilder --create' but running
 # 'cowbuilder--update' will happily fix that.
 EXTRAPACKAGES="eatmydata pbuilder bzr bzr-builddeb"
-export LD_PRELOAD="${LD_PRELOAD:+$LD_PRELOAD:}/usr/lib/libeatmydata/libeatmydata.so"
 
=== modified file 'juju-deployer/branch-source-builder.yaml.tmpl'
--- juju-deployer/branch-source-builder.yaml.tmpl	2014-06-25 18:01:51 +0000
+++ juju-deployer/branch-source-builder.yaml.tmpl	2014-06-26 14:33:53 +0000
@@ -15,6 +15,14 @@
                     - ${CI_PPA}
                 install_keys: |
                     - ""
+        ci-airline-bsb-chroot-builder:
+            charm: chroot-builder
+            options:
+                distros: trusty
+                architectures: amd64
+                vcs: ${CI_CODE_SOURCE}
+                branch: ${CI_BRANCH}
+                tarball: ${CI_PAYLOAD_URL}
         ci-airline-rabbit:
             branch: lp:~canonical-ci-engineering/charms/precise/uci-engine/rabbitmq-server@47
             charm: rabbitmq
@@ -22,3 +30,4 @@
                 management_plugin: true
     relations:
         - [ci-airline-bsb-worker, ci-airline-rabbit]
+        - [ci-airline-bsb-worker, ci-airline-bsb-chroot-builder]