Merge lp:~thomir-deactivatedaccount/adt-continuous-deployer/trunk-config-on-disk into lp:adt-continuous-deployer

[Thomi Richards]

Add code to read config from disk.

This branch changes the way we handle service config.

Instead of generating it from command line arguments, we read from a bzr branch on disk.

This is now ready to be merged, and has been tested on the bootstack jumphost for the snappy-proposed-results-checker service. It is backwards compatible - if no config is found in the configs directory, then the fallback option of generating config is used instead.

[lp:~thomir-deactivatedaccount/adt-continuous-deployer/trunk-config-on-disk]

67. By Thomi Richards on 2015-06-04

Don't count non-committed files when checking for config on disk.

[Paul Larson (pwlars)]

A couple of small suggestions, take a look and let me know what you think.

review: Needs Fixing

[lp:~thomir-deactivatedaccount/adt-continuous-deployer/trunk-config-on-disk]

68. By Thomi Richards on 2015-06-04

Updated README.

[Thomi Richards (thomir-deactivatedaccount)]

I replied to your inline comments...

[Paul Larson (pwlars)]

Nice! Thanks for updating that, this will be really useful in the future.

review: Approve

=== modified file 'README'
--- README	2015-04-15 17:26:25 +0000
+++ README	2015-06-04 21:46:36 +0000
@@ -8,7 +8,7 @@
 code is supposed to run in, we try to avoid additional dependencies.
 
 Although for supporting remote logging via Logstash, we need a small package
-that can/should be installed locally::  
+that can/should be installed locally::
 
   $ wget https://pypi.python.org/packages/source/p/python-logstash/\
   python-logstash-0.4.2.tar.gz#md5=30350b384a006c1d1adea6f7df90bf47
@@ -19,8 +19,61 @@
 environment variable, you do not need to install anything.
 
 
-Usage
-=====
+Background
+===========
+
+Mojo Project Pools
+------------------
+
+mojo.py accepts a '--project' argument - this tells it which mojo project
+should be used. If you plan on calling mojo.py directly, you will need to
+specify the mojo project name.
+
+However, cd.py contains a feature where mojo projects are stored in a pool.
+This prevents the need to have a 1:1 relationship between services and
+mojo projects, and allows us to use the mojo projects we have in the best
+possible way. To use the mojo project pool feature:
+
+1. Create the pool directory::
+
+  mkdir ~/mojo_project_pool
+
+2. Create the pool lock directory::
+
+  mkdir ~/mojo_project_pool/locks
+
+3. For each mojo project that you want to use in the pool, create an empty
+   file in the project pool directory. For example, if you have a mojo
+   project called 'ci-0'::
+
+  touch ~/mojo_project_pool/ci-0
+
+4. When calling ci.py, pass the `--use-project-pool` argument, and *don't*
+   pass the `--project` option.
+
+Service Configuration
+---------------------
+
+Most mojo specs require configuration and other files to be present in the
+`/srv/mojo/LOCAL/<mojo_project>/<mojo_stage>/` directory. cd.py will expect
+all these files to be in a bzr repository. To set up service config:
+
+1. Create the config repository::
+
+  bzr init ~/service_configs
+
+2. Within that repository, create a directory that matches the full mojo
+   stage you're deploying. For example::
+
+  cd ~/service_configs
+  mkdir -p ue/mojo-ue-snappy-proposed-selftest-agent/devel
+
+3. Add whatever config files, data files, or other secrets you need to that
+   directory. Commit your changes, and cd.py will ensure those files are
+   present when mojo copies them to the spec workspace.
+
+Creating new Mojo Projects
+--------------------------
 
 If working locally, you will have to create the shared mojo project called
 "mojo-stg-ue-ci-engineering" deployments ::
@@ -31,7 +84,17 @@
 mojo-project are managed by IS. You can choose a different project name,
 but that will force you to pass it via '--project' to `mojo.py`.
 
-Deploy "mojo-ue-ci-rabbit"::
+
+Usage
+=====
+
+Deploying services with mojo.py
+-------------------------------
+
+The mojo.py script handles actually deploying services using mojo. On a
+jumphost, it's rarely called directly, but rather called from cd.py.
+
+To deploy the "mojo-ue-ci-rabbit" service::
 
   $ ./mojo.py \
   --stage ue/mojo-ue-ci-rabbitmq/devel \
@@ -40,42 +103,60 @@
   --base ~/juju-environments \
   # '--devel' only if you're deploying outside PS4. for e.g. bootstack
 
-
-Get the IP address of the rabbit server using its dedicated environment
-information::
-
-  # NOTE: The SHA1_HASH below will vary according to the deployment contents
-  $ JUJU_HOME=~/juju-environments/mojo-ue-ci-rabbit-{SHA1_HASH} \
-  juju status | grep public-address
-
-Deploy the "adt-cloud-worker"::
-
-  $ ./mojo.py \
-  --stage ue/mojo-ue-adt-cloud-worker/devel \
-  --branch lp:~canonical-ci-engineering/canonical-mojo-specs/trunk \
-  --network 415a0839-eb05-4e7a-907c-413c657f4bf5 \
-  --base ~/juju-environments \
-  --amqp-uris "amqp://guest:guest@${AMQP_IP}:5672//"
-  # '--devel' only if you're deploying outside PS4. for e.g. bootstack
-
-Check if "adt-cloud-worker" is correctly configured to access the
-previously deployed "ci-rabbit"::
-
-  $ JUJU_HOME=/srv/juju-environments/mojo-ue-adt-cloud-worker-{SHA1_HASH}/ \
-  juju run --unit adt-cloud-worker/0 \
-  "tail /srv/adt-cloud-worker/logs/adt-cloud-worker.log"
-  Start from server ...
-  ...
-  Open OK!
-  Connected to amqp://guest:**@<<< GIVEN RABBIT IP ADDR >>>:5672//
-  using channel_id: 1
-  Channel open
+The `--stage` environment variable specifies both the mojo spec and the mojo
+stage. It reads the mojo spec from whatever is specified in the `--branch`
+argument.
+
+The `--network` option is used for auto-generated config, and is being
+deprecated.
+
+Inspecting services with list.py and juju
+-----------------------------------------
+
+The list.py script gives us a list of everything that's been deployed. An
+example output is shown below::
+
+  $ ./adt-continuous-deployer/list.py
+  * mojo-ue-snappy-proposed-image-builder:
+    27c224e5: Thu Jun  4 21:14:15 2015 (2 units)
+    705f3930: Thu Jun  4 18:28:16 2015 (2 units)
+    c9ab5b8c: Thu Jun  4 18:14:15 2015 (2 units)
+    deeb15d9: Wed Jun  3 19:35:17 2015 (1 units)
+    a45035f8: Wed Jun  3 19:05:15 2015 (2 units)
+  * mojo-ue-snappy-proposed-image-tester:
+    58dd9477: Wed Jun  3 18:30:47 2015 (2 units)
+  * mojo-ue-snappy-proposed-result-checker:
+    d021d997: Thu Jun  4 01:04:01 2015 (2 units)
+  * mojo-ue-snappy-proposed-selftest-agent:
+    35edcc15: Thu Jun  4 20:40:32 2015 (2 units)
+    8b84a34a: Wed Jun  3 22:20:32 2015 (2 units)
+
+The first level of output is the service name. The second level is the
+deployment hash. The hash is calculated from all the bzr revnos of all the
+branches involved in the deployment. Here we can see several parallel
+deployments of the 'mojo-ue-snappy-proposed-image-builder' service.
+
+We can combine the service name and a deployment hash to get the juju home
+directory. This allows us to run arbitrary juju commands::
+
+  JUJU_HOME=~/juju-environments/mojo-ue-snappy-proposed-image-builder-27c224e5 \
+    juju stat
+
+This is useful for several tasks:
+
+ * Inspecting why a service didn't deploy properly.
+ * Finding the public IP address of a server.
+ * Logging in to a deployed service with `juju ssh`
+ * Running arbitrary commands on the deployed service with `juju run`
+
+Continuous Delivery with cd.py
+------------------------------
 
 Continuous and isolated deployments can be done with `cd.py` which can
-periodically (cron) inspect a mojo spec and has been updated, it call
-`mojo.py` to do a new (and parallel) deployment::
+periodically (via cron, for example) determine if a mojo spec, or any of the
+branches it deploys has been updated, and call mojo.py to deploy a the new
+revision. An example crontab setup is shown below::
 
-  stg-ue-ci-engineering@wendigo:~$ crontab -l
   NET_ID="79126fa3-b675-4f68-b7ec-c5f4be5dbe0e"
   CI_SPECS_BRANCH="lp:~canonical-ci-engineering/canonical-mojo-specs/trunk"
 
@@ -94,7 +175,7 @@
 part of that deployment::
 
   # TAB-separated lines as: <branch>\t<revno>\n
-  $ cat ~/ci-cd-identifiers/4c29c6aa 
+  $ cat ~/ci-cd-identifiers/4c29c6aa
   lp:charms/trusty/haproxy        90
   lp:charms/trusty/apache2        65
   lp:~canonical-ci-engineering/adt-cloud-service/pip-cache        4
@@ -106,17 +187,29 @@
   lp:~canonical-sysadmins/basenode/trunk  80
   lp:~canonical-sysadmins/basenode/trunk  80
 
+This information is also written to the nova instance(s) metadata field.
 
-So, for figuring out what was deployed for a particular juju environment-name
+To figure out what was deployed for a particular juju environment-name
  / machine-name we have to lookup for it's hash file and inspect its contents.
 
-`monitor.py` is to monitor the number of deployments to not go out of hands.
-The default number per each service deployment is now 2 and any more deployments
-will be destroyed. 
+
+Monitoring deployed services with monitor.py
+--------------------------------------------
+
+The `monitor.py` script is used to monitor the number of deployments. Eventually
+we will autmoatically destroy old deployments, but right now this is not
+enabled. The policy in monitor.py is to allow two deployments (a current, and an
+old revision, and to destroy the rest).
 
 A cron job in wendigo employing `monitor.py` looks like the folloging::
+
    8-59/20 * * * * . $HOME/.novarc; export CI_LOGSTASH_HOST=$INTERNAL_LOGSTASH_HOST; ~/adt-continuous-deployer/monitor.py mojo-ue-core-result-checker >> ~/core-result-checker.log 2>&1
 
-`reaper.py` is to destroy an deployment that was deployed using `cd.py` along
-with its environment by using its SHA1 hash value::
+Destroying environments with reaper.py
+--------------------------------------
+
+The `reaper.py` script is used to destroy an deployment that was deployed using
+`cd.py`. The only thing you need to specify is the deployment hash, as shown
+from `list.py`::
+
   ./reaper.py 4c29c6aa

=== added file 'ci_automation/config.py'
--- ci_automation/config.py	1970-01-01 00:00:00 +0000
+++ ci_automation/config.py	2015-06-04 21:46:36 +0000
@@ -0,0 +1,171 @@
+#
+# Copyright (C) 2015 Canonical
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+
+"""CI Automation config helpers.
+
+We store config files on disk as fully realised files (no templating nonsense)
+and copy them to the mojo "secrets" folder before calling mojo to deploy the
+service in question.
+
+This gives us several advantages:
+
+1. It's super-simple to see what config a service will get deployed with -
+   just look in the config directory and examine the last bzr commit contents.
+
+2. We get a log of all changes, who made them, and *why*, since we only get
+   the last bzr commit - unstaged changes are never deployed.
+
+3. We can re-deploy a service by changing the config file, since the config
+   folder's bzr revno is considered to be part of the deployment hash.
+
+All configs must be stored in `CONFIG_DIR`. By default this is set to
+~/service_configs. All functions read this variable without caching, so tests
+can be written by setting this variable to something else for the lifetime of
+the test.
+
+Beneath the config_dir, the folder structure should mimic the
+canonical-mojo-specs project. For example::
+
+    service_configs
+     - ue
+      - mojo-ue-adt-cloud-service
+       - devel
+       - production
+      - mojo-ue-adt-cloud-worker
+       - devel
+       - production
+
+...continued for every mojo spec we want to deploy. Within every stage
+directory ('devel' / 'production' etc), any files present will be present in
+the mojo secrets dir when deployment begins.
+
+"""
+
+import os.path
+import re
+import shutil
+import subprocess
+
+
+__all__ = [
+    'check_config_dir_exists',
+    'deployment_has_config',
+    'export_latest_config_for_deployment',
+    'get_revno_for_deployment',
+]
+
+
+CONFIG_DIR = os.path.expanduser("~/service_configs")
+
+
+def check_config_dir_exists():
+    """Return True if the config directory exists, and is a bzr repository."""
+    global CONFIG_DIR
+    if not os.path.exists(CONFIG_DIR):
+        return False
+    return subprocess.call(
+        ['bzr', 'info'],
+        cwd=CONFIG_DIR,
+        stdout=subprocess.DEVNULL,
+        stderr=subprocess.DEVNULL,
+    ) == 0
+
+
+def deployment_has_config(mojo_stage):
+    """Return true if a mojo spec stage has configuration data on disk.
+
+    'mojo_stage' should be the name of a mojo stage to deploy, like
+    'ue/mojo-ue-adt-cloud-service/devel' or
+    'ue/mojo-ue-adt-cloud-service/production'.
+
+    """
+    global CONFIG_DIR
+    full_path = os.path.join(
+        CONFIG_DIR,
+        mojo_stage,
+    )
+    return (
+        os.path.exists(full_path) and
+        os.path.isdir(full_path) and
+        get_revno_for_deployment(mojo_stage) is not None
+    )
+
+
+def get_revno_for_deployment(mojo_stage):
+    """Get the bzr revno for the given mojo spec & stage.
+
+    'mojo_stage' should be the name of a mojo stage to deploy, like
+    'ue/mojo-ue-adt-cloud-service/devel' or
+    'ue/mojo-ue-adt-cloud-service/production'.
+
+    """
+    global CONFIG_DIR
+    try:
+        output = subprocess.check_output(
+            ['bzr', 'log', '-l', '1', '-S', mojo_stage],
+            cwd=CONFIG_DIR,
+            stderr=subprocess.DEVNULL,
+        ).decode()
+    except subprocess.CalledProcessError as e:
+        # returncode 3 is 'unknown path'
+        if e.returncode != 3:
+            raise
+        return None
+    match = re.search(' *([0-9]+)', output)
+    if match:
+        return match.group(1)
+    return None
+
+
+def export_latest_config_for_deployment(mojo_stage, target_dir):
+    """Export configuration for a mojo spec to 'target_dir'.
+
+    'mojo_stage' should be the name of a mojo stage to deploy, like
+    'ue/mojo-ue-adt-cloud-service/devel' or
+    'ue/mojo-ue-adt-cloud-service/production'.
+
+    'target_dir' is the target location to export to. This will probably be
+    set to '/srv/mojo/LOCAL' in production.
+
+    Directories for the spec and stage will be created in the target directory.
+
+    :raises: ValueError if the given mojo spec & stage don't have configuration
+        in the config dir.
+
+    """
+    global CONFIG_DIR
+    if not deployment_has_config(mojo_stage):
+        raise ValueError(
+            "Mojo stage '{}' have no config on disk.".format(mojo_stage)
+        )
+    target_dir = os.path.join(target_dir, mojo_stage)
+    config_dir = os.path.join(CONFIG_DIR, mojo_stage)
+
+    if not os.path.exists(target_dir):
+        os.makedirs(target_dir)
+    else:
+        for entry in os.listdir(target_dir):
+            full_path = os.path.join(target_dir, entry)
+            if os.path.isdir(full_path):
+                shutil.rmtree(full_path)
+            else:
+                os.unlink(full_path)
+    subprocess.check_call(
+        ['bzr', 'export', target_dir, config_dir],
+        stdout=subprocess.DEVNULL,
+        stderr=subprocess.DEVNULL,
+    )

=== added file 'ci_automation/tests/test_config.py'
--- ci_automation/tests/test_config.py	1970-01-01 00:00:00 +0000
+++ ci_automation/tests/test_config.py	2015-06-04 21:46:36 +0000
@@ -0,0 +1,272 @@
+#!/usr/bin/env python3
+#
+# Copyright (C) 2015 Canonical
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+
+"""CI Automation tests for config helpers."""
+
+import tempfile
+import unittest
+import shutil
+import functools
+import contextlib
+import subprocess
+import os.path
+
+from ci_automation import config
+
+
+class TestConfig(unittest.TestCase):
+
+    def test_config_dir_check_returns_false_if_dir_does_not_exist(self):
+        with tempfile.TemporaryDirectory() as workdir:
+            config_dir = os.path.join(workdir, "doesnotexist")
+            with patch_config_dir(config_dir):
+                self.assertEqual(False, config.check_config_dir_exists())
+
+    def test_config_dir_check_returns_false_if_not_a_bzr_repo(self):
+        with tempfile.TemporaryDirectory() as config_dir:
+            with patch_config_dir(config_dir):
+                self.assertEqual(False, config.check_config_dir_exists())
+
+    def test_config_dir_returns_true_on_empty_repo(self):
+        with BzrConfigDir() as cm:
+            self.assertEqual(True, config.check_config_dir_exists())
+
+    def test_deployment_has_config_returns_false_on_nonexisting_project(self):
+        with BzrConfigDir() as cm:
+            self.assertEqual(
+                False,
+                config.deployment_has_config('ue/nonexist/devel'),
+            )
+
+    def test_deployment_has_config_returns_false_on_nonexisting_stage(self):
+        with BzrConfigDir() as cm:
+            os.makedirs(os.path.join(cm.config_dir, 'ue/test'))
+            self.assertEqual(
+                False,
+                config.deployment_has_config('ue/test/devel'),
+            )
+
+    def test_deployment_has_config_returns_true_on_existing_stage(self):
+        with BzrConfigDir() as cm:
+            cm.add_mojo_stage('ue/my-test-project/devel')
+            self.assertEqual(
+                True,
+                config.deployment_has_config('ue/my-test-project/devel'),
+            )
+
+    def test_get_revno_for_deployment_returns_none_without_bzr_control(self):
+        with BzrConfigDir() as cm:
+            os.makedirs(os.path.join(cm.config_dir, 'ue/project/devel'))
+            self.assertIsNone(
+                config.get_revno_for_deployment('ue/project/devel')
+            )
+
+    def test_get_revno_for_deployment_returns_revno(self):
+        with BzrConfigDir() as cm:
+            cm.add_mojo_stage('ue/project/devel')
+            self.assertEqual(
+                '1',
+                config.get_revno_for_deployment('ue/project/devel'),
+            )
+
+    def test_adding_file_increases_bzr_revno(self):
+        with BzrConfigDir() as cm:
+            project = cm.add_mojo_stage('ue/project/production')
+            with project.write_new_file('config.ini') as f:
+                f.write('[section]\n')
+                f.write('key = value\n')
+            self.assertEqual(
+                '2',
+                config.get_revno_for_deployment('ue/project/production'),
+            )
+
+    def test_projects_dont_increase_each_others_revnos(self):
+        with BzrConfigDir() as cm:
+            project1 = cm.add_mojo_stage('ue/project1/production')
+            project2 = cm.add_mojo_stage('ue/project2/production')
+            self.assertEqual(
+                '1',
+                config.get_revno_for_deployment('ue/project1/production'),
+            )
+            self.assertEqual(
+                '2',
+                config.get_revno_for_deployment('ue/project2/production'),
+            )
+            with project1.write_new_file('README') as f:
+                f.write("Some text")
+            self.assertEqual(
+                '3',
+                config.get_revno_for_deployment('ue/project1/production'),
+            )
+            self.assertEqual(
+                '2',
+                config.get_revno_for_deployment('ue/project2/production'),
+            )
+
+    def test_export_raises_ValueError_for_unconfigured_spec(self):
+        with BzrConfigDir() as cm:
+            with self.assertRaises(ValueError):
+                config.export_latest_config_for_deployment(
+                    'project/devel',
+                    'target'
+                )
+
+    def test_exports_empty_config_dir(self):
+        with BzrConfigDir() as cm, tempfile.TemporaryDirectory() as target_dir:
+            cm.add_mojo_stage('ue/project/production')
+            target_project_path = os.path.join(
+                target_dir,
+                'ue/project/production'
+            )
+            config.export_latest_config_for_deployment(
+                'ue/project/production',
+                target_dir
+            )
+            self.assertTrue(os.path.exists(target_project_path))
+
+    def test_exports_simple_file(self):
+        with BzrConfigDir() as cm, tempfile.TemporaryDirectory() as target_dir:
+            project = cm.add_mojo_stage('ue/project/production')
+            with project.write_new_file('config.ini') as f:
+                f.write("Test config")
+            config.export_latest_config_for_deployment(
+                'ue/project/production',
+                target_dir
+            )
+            target_config_path = os.path.join(
+                target_dir,
+                'ue/project/production/config.ini'
+            )
+            self.assertTrue(os.path.exists(target_config_path))
+            with open(target_config_path) as f:
+                self.assertEqual(f.read(), "Test config")
+
+    def test_exports_directory_heirarchy(self):
+        with BzrConfigDir() as cm, tempfile.TemporaryDirectory() as target_dir:
+            project = cm.add_mojo_stage('ue/project/production')
+            project.create_directory('foo/bar/baz')
+            with project.write_new_file('foo/bar/baz/config.ini') as f:
+                f.write("Test config")
+            config.export_latest_config_for_deployment(
+                'ue/project/production',
+                target_dir
+            )
+            target_config_path = os.path.join(
+                target_dir,
+                'ue/project/production/foo/bar/baz/config.ini'
+            )
+            self.assertTrue(os.path.exists(target_config_path))
+            with open(target_config_path) as f:
+                self.assertEqual(f.read(), "Test config")
+
+
[e-mail]
+def patch_config_dir(new_dir):
+    """A context manager that patches config.CONFIG_DIR."""
+    old_dir = config.CONFIG_DIR
+    config.CONFIG_DIR = new_dir
+    try:
+        yield
+    except:
+        config.CONFIG_DIR = old_dir
+        raise
+
+
+class BzrConfigDir(contextlib.ExitStack):
+    """A context manager that sets up a simple config bzr repo."""
+
+    def __enter__(self):
+        super().__enter__()
+        self.config_dir = self.enter_context(tempfile.TemporaryDirectory())
+        self.enter_context(patch_config_dir(self.config_dir))
+
+        self._create_config_repo()
+        return self
+
+    def _create_config_repo(self):
+        subprocess.check_call(
+            ['bzr', 'init'],
+            stdout=subprocess.DEVNULL,
+            stderr=subprocess.DEVNULL,
+            cwd=self.config_dir,
+        )
+
+    def add_mojo_stage(self, spec_path):
+        full_path = os.path.join(self.config_dir, spec_path)
+        os.makedirs(full_path)
+        subprocess.check_call(
+            ['bzr', 'add', spec_path],
+            stdout=subprocess.DEVNULL,
+            stderr=subprocess.DEVNULL,
+            cwd=self.config_dir,
+        )
+        subprocess.check_call(
+            ['bzr', 'commit', '-m', 'adding {}'.format(spec_path)],
+            stdout=subprocess.DEVNULL,
+            stderr=subprocess.DEVNULL,
+            cwd=self.config_dir,
+        )
+        return ProjectConfigManager(full_path)
+
+
+class ProjectConfigManager(object):
+
+    """A simple interface to make it easy to write data to a test config
+    dir for a mojo project.
+    """
+
+    def __init__(self, path):
+        self.path = path
+
+    @contextlib.contextmanager
+    def write_new_file(self, path):
+        full_path = os.path.join(self.path, path)
+        with open(full_path, 'wt') as f:
+            yield f
+        subprocess.check_call(
+            ['bzr', 'add', path],
+            cwd=self.path,
+            stdout=subprocess.DEVNULL,
+            stderr=subprocess.DEVNULL,
+        )
+        subprocess.check_call(
+            ['bzr', 'commit', '-m', 'adding {}'.format(path)],
+            stdout=subprocess.DEVNULL,
+            stderr=subprocess.DEVNULL,
+            cwd=self.path,
+        )
+
+    def create_directory(self, dir_path):
+        full_path = os.path.join(self.path, dir_path)
+        os.makedirs(full_path)
+        subprocess.check_call(
+            ['bzr', 'add', dir_path],
+            cwd=self.path,
+            stdout=subprocess.DEVNULL,
+            stderr=subprocess.DEVNULL,
+        )
+        subprocess.check_call(
+            ['bzr', 'commit', '-m', 'adding {}'.format(dir_path)],
+            stdout=subprocess.DEVNULL,
+            stderr=subprocess.DEVNULL,
+            cwd=self.path,
+        )
+
+
+if __name__ == '__main__':
+    unittest.main()

=== modified file 'mojo.py'
--- mojo.py	2015-05-27 16:00:59 +0000
+++ mojo.py	2015-06-04 21:46:36 +0000
@@ -27,6 +27,7 @@
 
 from ci_automation import (
     branch,
+    config,
     juju,
     mojo,
     utils,
@@ -157,48 +158,83 @@
     if args.devel:
         fix_permissions(args.project)
 
+    # Thomi - we're moving to config files being stored on disk. As a
+    # transitional stage, we use the new system if it has config on disk,
+    # otherwise we fall back to the old system.
+    if config.deployment_has_config(args.stage):
+        # If config is on disk, warn the user if they're also specifying
+        # config on the command line, as they might get confused otherwise.
+        deprecated_args = (
+            'network',
+            'amqp-uris',
+            'logstash-host',
+            'swift-public-url',
+            'image-mapper-url',
+            'os-username',
+            'os-tenant-name',
+            'os-password',
+            'os-auth-url',
+            'os-region-name',
+            'adt-os-username',
+            'adt-os-tenant-name',
+            'adt-os-password',
+            'adt-os-auth-url',
+            'adt-os-region-name',
+            'adt-extra-args',
+            'adt-http-proxy',
+        )
+        used_deprepcated = [a for a in deprecated_args if a in args]
+        if used_deprepcated:
+            print("Warning: This mojo spec uses config on disk, but you have")
+            print("specified one or more configuration options on the command")
+            print("line. The following options will be ignored:")
+            print(', '.join(used_deprepcated))
+        config_target = os.path.join('/srv/mojo/LOCAL/', args.project)
+        config.export_latest_config_for_deployment(args.stage, config_target)
+    else:
+        # Generate config from command line options:
+        computed_adt_extra_args = ' '.join([
+            args.adt_extra_args,
+            '--net-id={}'.format(args.network) if args.devel else '',
+        ]).strip()
+        computed_adt_image_flavor = (
+            'cpu2-ram2-disk50' if not args.devel else 'm1.small')
+        computed_adt_archive_mirror = (
+            'http://ftpmaster.internal/ubuntu/' if not args.devel else
+            'http://nova.clouds.archive.ubuntu.com/ubuntu/')
+        computed_adt_http_proxy = (
+            'http://squid.internal:3128/' if not args.devel else
+            args.adt_http_proxy
+        )
+
+        create_service_conf(
+            args.project,
+            args.stage,
+            network=args.network,
+            amqp_uris=args.amqp_uris,
+            logstash_host=args.logstash_host,
+            swift_public_url=args.swift_public_url,
+            image_mapper_url=args.image_mapper_url,
+            os_username=args.os_username,
+            os_tenant_name=args.os_tenant_name,
+            os_password=args.os_password,
+            os_auth_url=args.os_auth_url,
+            os_region_name=args.os_region_name,
+            adt_os_username=args.adt_os_username,
+            adt_os_tenant_name=args.adt_os_tenant_name,
+            adt_os_password=args.adt_os_password,
+            adt_os_auth_url=args.adt_os_auth_url,
+            adt_os_region_name=args.adt_os_region_name,
+            adt_extra_args=computed_adt_extra_args,
+            adt_image_flavor=computed_adt_image_flavor,
+            adt_archive_mirror=computed_adt_archive_mirror,
+            adt_http_proxy=computed_adt_http_proxy,
+        )
+
     # Relies on the canonical-mojo-specs layout, e.g.:
     # ue/mojo-ue-adt-cloud-worker/devel
     spec_name = args.stage.split('/')[1]
 
-    computed_adt_extra_args = ' '.join([
-        args.adt_extra_args,
-        '--net-id={}'.format(args.network) if args.devel else '',
-    ]).strip()
-    computed_adt_image_flavor = (
-        'cpu2-ram2-disk50' if not args.devel else 'm1.small')
-    computed_adt_archive_mirror = (
-        'http://ftpmaster.internal/ubuntu/' if not args.devel else
-        'http://nova.clouds.archive.ubuntu.com/ubuntu/')
-    computed_adt_http_proxy = (
-        'http://squid.internal:3128/' if not args.devel else
-        args.adt_http_proxy
-    )
-
-    create_service_conf(
-        args.project,
-        args.stage,
-        network=args.network,
-        amqp_uris=args.amqp_uris,
-        logstash_host=args.logstash_host,
-        swift_public_url=args.swift_public_url,
-        image_mapper_url=args.image_mapper_url,
-        os_username=args.os_username,
-        os_tenant_name=args.os_tenant_name,
-        os_password=args.os_password,
-        os_auth_url=args.os_auth_url,
-        os_region_name=args.os_region_name,
-        adt_os_username=args.adt_os_username,
-        adt_os_tenant_name=args.adt_os_tenant_name,
-        adt_os_password=args.adt_os_password,
-        adt_os_auth_url=args.adt_os_auth_url,
-        adt_os_region_name=args.adt_os_region_name,
-        adt_extra_args=computed_adt_extra_args,
-        adt_image_flavor=computed_adt_image_flavor,
-        adt_archive_mirror=computed_adt_archive_mirror,
-        adt_http_proxy=computed_adt_http_proxy,
-    )
-
     # Juju environment & mojo workspaces have identical names. Names
     # are unique by appending a spec content identifier including
     # all collected branch URLs and their current revision numbers, e.g.: