Merge lp:~benji/charms/precise/juju-gui/use-npm-cache into lp:~juju-gui/charms/precise/juju-gui/trunk

[Benji York]

Use the npm cache to speed installs from a branch

Creating the cache is handled by the GUI Makefile

[Benji York (benji)]

Please take a look.

[Gary Poster (gary)]

Thanks for the docs and comments! They are great to have.

29 + # If the given installable thing ("backend") requires one or more debs
30 + # that are not yet installed, install them.
31 missing = backend.check_packages(*backend.debs)

Yeah, check_packages might be better named install_missing_packages or something like that.

39 + # Inject NPM packages into the cache for faster building.
40 + #prime_npm_cache(get_npm_cache_archive_url())

Uh? I hope this wasn't commented out for your tests. :-)

44 + # XXX Why do we only set up the GUI if the "juju-gui-source"
45 + # configuration is non-default?

It's not non-default: it is not-what-it-was-before--or at least that's the intent. I think that's the behavior too, or we'd be in trouble. :-) At the start, what-it-was-before is some empty value, I believe.

124 + if e.errno != 17: # File exists.

Maybe use errno.EEXIST instead?

126 + uncompress = command('tar', '-x', '-z', '-C', npm_cache_dir, '-f')
127 + cmd_log(uncompress(npm_cache_archive))

That reads very nicely. Good idea.

164 + get_npm_cache_archive_url,

You look like you were on your way to a test? That would have been nice. :-) If not, remove the import. But adding an import of prime_npm_cache and testing them both would be nicer. :-)

Thanks!

Gary

review: Approve

[Benji York (benji)]

On Mon, Apr 29, 2013 at 3:31 PM, Gary Poster <email address hidden> wrote:
> Review: Approve
>
> Thanks for the docs and comments! They are great to have.
>
> 29 + # If the given installable thing ("backend") requires one or more debs
> 30 + # that are not yet installed, install them.
> 31 missing = backend.check_packages(*backend.debs)
>
> Yeah, check_packages might be better named install_missing_packages or
> something like that.

Yep (I was only commenting things at the time, but this looks like a
local-enough change that I'll go ahead and do it).

> 39 + # Inject NPM packages into the cache for faster building.
> 40 + #prime_npm_cache(get_npm_cache_archive_url())
>
> Uh? I hope this wasn't commented out for your tests. :-)

Heh. Yeah, I was doing some non-cache timings and forgot to put it
back. Fixed.

> 44 + # XXX Why do we only set up the GUI if the "juju-gui-source"
> 45 + # configuration is non-default?
>
> It's not non-default: it is not-what-it-was-before--or at least that's
> the intent. I think that's the behavior too, or we'd be in trouble.
> :-) At the start, what-it-was-before is some empty value, I believe.

Ah! I think you're right. Maybe I can come up with variable/method
names that communicate better; or a comment at least.

> 124 + if e.errno != 17: # File exists.
>
> Maybe use errno.EEXIST instead?

Good call. Will do.

> 126 + uncompress = command('tar', '-x', '-z', '-C', npm_cache_dir, '-f')
> 127 + cmd_log(uncompress(npm_cache_archive))
>
> That reads very nicely. Good idea.

I thought so too. That's why I copied it from another part of the code. ;)

> 164 + get_npm_cache_archive_url,
>
> You look like you were on your way to a test? That would have been
> nice. :-)

Indeed. I thought I had added it. I'll do so.
--
Benji York

[lp:~benji/charms/precise/juju-gui/use-npm-cache]

49. By Benji York on 2013-04-30

review fixes

[Benji York (benji)]

Please take a look.

[Benji York (benji)]

*** Submitted:

Use the npm cache to speed installs from a branch

Creating the cache is handled by the GUI Makefile

=== modified file 'hooks/backend.py'
--- hooks/backend.py	2013-04-30 13:53:20 +0000
+++ hooks/backend.py	2013-04-30 14:57:26 +0000
@@ -29,17 +29,19 @@
     IMPROV,
     JUJU_DIR,
     chain,
-    check_packages,
+    find_missing_packages,
     cmd_log,
     fetch_api,
     fetch_gui,
     get_config,
+    get_npm_cache_archive_url,
     legacy_juju,
     merge,
     overrideable,
+    prime_npm_cache,
     save_or_create_certificates,
+    setup_apache,
     setup_gui,
-    setup_apache,
     start_agent,
     start_gui,
     start_improv,
@@ -53,20 +55,40 @@
 
 
 class InstallMixin(object):
+    """Provide for the GUI and its dependencies to be installed."""
+
     def install(self, backend):
+        """Install the GUI and dependencies."""
         config = backend.config
-        missing = backend.check_packages(*backend.debs)
+        # If the given installable thing ("backend") requires one or more debs
+        # that are not yet installed, install them.
+        missing = backend.find_missing_packages(*backend.debs)
         if missing:
             cmd_log(backend.install_extra_repositories(*backend.repositories))
             cmd_log(apt_get_install(*backend.debs))
 
+        # If we are not using a pre-built release of the GUI (i.e., we are
+        # using a branch) then we need to build a release archive to use.
         if backend.different('juju-gui-source'):
-            release_tarball = fetch_gui(
-                config['juju-gui-source'], config['command-log-file'])
-            setup_gui(release_tarball)
+            # Inject NPM packages into the cache for faster building.
+            self._prime_npm_cache()
+            # Build a release from the branch.
+            self._build_and_install_from_branch()
+
+    def _prime_npm_cache(self):
+        # This is a separate method so it can be easily overridden for testing.
+        prime_npm_cache(get_npm_cache_archive_url())
+
+    def _build_and_install_from_branch(self):
+        # This is a separate method so it can be easily overridden for testing.
+        release_tarball = fetch_gui(
+            config['juju-gui-source'], config['command-log-file'])
+        setup_gui(release_tarball)
 
 
 class UpstartMixin(object):
+    """Manage (install, start, stop, etc.) some service via Upstart."""
+
     upstart_scripts = ('haproxy.conf', )
     debs = ('curl', 'openssl', 'haproxy', 'apache2')
 
@@ -152,16 +174,14 @@
 
 
 class Backend(object):
-    """Compose methods and policy needed to interact
-    with a Juju backend. Given a config dict (which typically
-    comes from the JSON de-serialization of config.json in JujuGUI).
+    """Compose methods and policy needed to interact with a Juju backend.
+
+    "config" is a config dict (which typically comes from the JSON
+    de-serialization of config.json in JujuGUI).
     """
 
     def __init__(self, config=None, prev_config=None, **overrides):
-        """
-        Backends function through composition. __init__ becomes the
-        factory method to generate a selection of strategy classes
-        to use together to implement the backend proper.
+        """Generate a selection of strategy classes that implement the backend.
         """
         # Ingest the config and build out the ordered list of
         # backend elements to include
@@ -171,7 +191,7 @@
         self.prev_config = prev_config
         self.overrides = overrides
 
-        # We always use upstart.
+        # We always install the GUI.
         backends = [InstallMixin, ]
 
         api = "python" if legacy_juju() else "go"
@@ -194,7 +214,8 @@
                     "Unable to use sandbox with {} backend".format(api))
             backends.append(GoBackend)
 
-        # All backends can manage the gui.
+        # All backends need to install, start, and stop the services that
+        # provide the GUI.
         backends.append(GuiMixin)
         backends.append(UpstartMixin)
 
@@ -213,8 +234,8 @@
             raise
 
     @overrideable
-    def check_packages(self, *packages):
-        return check_packages(*packages)
+    def find_missing_packages(self, *packages):
+        return find_missing_packages(*packages)
 
     @overrideable
     def service_control(self, service, action):

=== modified file 'hooks/utils.py'
--- hooks/utils.py	2013-04-30 13:53:20 +0000
+++ hooks/utils.py	2013-04-30 14:57:26 +0000
@@ -404,6 +404,32 @@
     render_to_file('apache-site.template', context, JUJU_GUI_SITE)
 
 
+def get_npm_cache_archive_url(Launchpad=Launchpad):
+    """Figure out the URL of the most recent NPM cache archive on Launchpad."""
+    launchpad = Launchpad.login_anonymously('Juju GUI charm', 'production')
+    project = launchpad.projects['juju-gui']
+    # Find the URL of the most recently created NPM cache archive.
+    npm_cache_url = get_release_file_url(project, 'npm-cache', None)
+    return npm_cache_url
+
+
+def prime_npm_cache(npm_cache_url):
+    """Download NPM cache archive and prime the NPM cache with it."""
+    # Download the cache archive and then uncompress it into the NPM cache.
+    npm_cache_archive = os.path.join(CURRENT_DIR, 'npm-cache.tgz')
+    cmd_log(run('curl', '-L', '-o', npm_cache_archive, npm_cache_url))
+    npm_cache_dir = os.path.expanduser('~/.npm')
+    # The NPM cache directory probably does not exist, so make it if not.
+    try:
+        os.mkdir(npm_cache_dir)
+    except OSError, e:
+        # If the directory already exists then ignore the error.
+        if e.errno != errno.EEXIST: # File exists.
+            raise
+    uncompress = command('tar', '-x', '-z', '-C', npm_cache_dir, '-f')
+    cmd_log(uncompress(npm_cache_archive))
+
+
 def fetch_gui(juju_gui_source, logpath):
     """Retrieve the Juju GUI release/branch."""
     # Retrieve a Juju GUI release.
@@ -520,7 +546,7 @@
         shutil.copyfileobj(open(crt_path), pem_file)
 
 
-def check_packages(*packages):
+def find_missing_packages(*packages):
     """Given a list of packages, return the packages which are not installed.
     """
     cache = apt.Cache()

=== modified file 'revision'
--- revision	2013-04-30 13:53:20 +0000
+++ revision	2013-04-30 14:57:26 +0000
@@ -1,1 +1,5 @@
+<<<<<<< TREE
 41
+=======
+44
+>>>>>>> MERGE-SOURCE

=== modified symlink 'tests/backend.py'
=== target was u'../hooks/backend.py'
--- tests/backend.py	1970-01-01 00:00:00 +0000
+++ tests/backend.py	2013-04-30 14:57:26 +0000
@@ -0,0 +1,291 @@
+"""
+A composition system for creating backend object.
+
+Backends implement start(), stop() and install() methods. A backend is composed
+of many mixins and each mixin will implement any/all of those methods and all
+will be called. Backends additionally provide for collecting property values
+from each mixin into a single final property on the backend. There is also a
+feature for determining if configuration values have changed between old and
+new configurations so we can selectively take action.
+"""
+
+from charmhelpers import (
+    RESTART,
+    STOP,
+    log,
+    open_port,
+    service_control,
+)
+from shelltoolbox import (
+    apt_get_install,
+    command,
+    install_extra_repositories,
+    su,
+)
+from utils import (
+    AGENT,
+    APACHE,
+    HAPROXY,
+    IMPROV,
+    JUJU_DIR,
+    chain,
+    find_missing_packages,
+    cmd_log,
+    fetch_api,
+    fetch_gui,
+    get_config,
+    get_npm_cache_archive_url,
+    legacy_juju,
+    merge,
+    overrideable,
+    prime_npm_cache,
+    save_or_create_certificates,
+    setup_apache,
+    setup_gui,
+    start_agent,
+    start_gui,
+    start_improv,
+)
+
+import os
+import shutil
+
+
+apt_get = command('apt-get')
+
+
+class InstallMixin(object):
+    """Provide for the GUI and its dependencies to be installed."""
+
+    def install(self, backend):
+        """Install the GUI and dependencies."""
+        config = backend.config
+        # If the given installable thing ("backend") requires one or more debs
+        # that are not yet installed, install them.
+        missing = backend.find_missing_packages(*backend.debs)
+        if missing:
+            cmd_log(backend.install_extra_repositories(*backend.repositories))
+            cmd_log(apt_get_install(*backend.debs))
+
+        # If we are not using a pre-built release of the GUI (i.e., we are
+        # using a branch) then we need to build a release archive to use.
+        if backend.different('juju-gui-source'):
+            # Inject NPM packages into the cache for faster building.
+            self._prime_npm_cache()
+            # Build a release from the branch.
+            self._build_and_install_from_branch()
+
+    def _prime_npm_cache(self):
+        # This is a separate method so it can be easily overridden for testing.
+        prime_npm_cache(get_npm_cache_archive_url())
+
+    def _build_and_install_from_branch(self):
+        # This is a separate method so it can be easily overridden for testing.
+        release_tarball = fetch_gui(
+            config['juju-gui-source'], config['command-log-file'])
+        setup_gui(release_tarball)
+
+
+class UpstartMixin(object):
+    """Manage (install, start, stop, etc.) some service via Upstart."""
+
+    upstart_scripts = ('haproxy.conf', )
+    debs = ('curl', 'openssl', 'haproxy', 'apache2')
+
+    def install(self, backend):
+        """Set up haproxy and nginx upstart configuration files."""
+        setup_apache()
+        backend.log('Setting up haproxy and nginx start up scripts.')
+        config = backend.config
+        if backend.different(
+                'ssl-cert-path', 'ssl-cert-contents', 'ssl-key-contents'):
+            save_or_create_certificates(
+                config['ssl-cert-path'], config.get('ssl-cert-contents'),
+                config.get('ssl-key-contents'))
+
+        source_dir = os.path.join(os.path.dirname(__file__),  '..', 'config')
+        for config_file in backend.upstart_scripts:
+            shutil.copy(os.path.join(source_dir, config_file), '/etc/init/')
+
+    def start(self, backend):
+        with su('root'):
+            backend.service_control(APACHE, RESTART)
+            backend.service_control(HAPROXY, RESTART)
+
+    def stop(self, backend):
+        with su('root'):
+            backend.service_control(HAPROXY, STOP)
+            backend.service_control(APACHE, STOP)
+
+
+class GuiMixin(object):
+    gui_properties = set([
+        'juju-gui-console-enabled', 'login-help', 'read-only',
+        'serve-tests', 'secure'])
+
+    repositories = ('ppa:juju-gui/ppa',)
+
+    def start(self, config):
+        start_gui(
+            config['juju-gui-console-enabled'], config['login-help'],
+            config['read-only'], config['staging'], config['ssl-cert-path'],
+            config['charmworld-url'], config['serve-tests'],
+            secure=config['secure'], sandbox=config['sandbox'])
+        open_port(80)
+        open_port(443)
+
+
+class SandboxBackend(object):
+    pass
+
+
+class PythonBackend(object):
+
+    def install(self, config):
+        if (not os.path.exists(JUJU_DIR) or
+                config.different('staging', 'juju-api-branch')):
+            fetch_api(config['juju-api-branch'])
+
+    def start(self, backend):
+        backend.start_agent(backend['ssl-cert-path'])
+
+    def stop(self, backend):
+        backend.service_control(AGENT, STOP)
+
+
+class ImprovBackend(object):
+    debs = ('zookeeper', )
+
+    def install(self, config):
+        if (not os.path.exists(JUJU_DIR) or
+                config.different('staging', 'juju-api-branch')):
+            fetch_api(config['juju-api-branch'])
+
+    def start(self, backend):
+        backend.start_improv(
+            backend['staging-environment'], backend['ssl-cert-path'])
+
+    def stop(self, backend):
+        backend.service_control(IMPROV, STOP)
+
+
+class GoBackend(object):
+    debs = ('python-yaml', )
+
+
+class Backend(object):
+    """Compose methods and policy needed to interact with a Juju backend.
+
+    "config" is a config dict (which typically comes from the JSON
+    de-serialization of config.json in JujuGUI).
+    """
+
+    def __init__(self, config=None, prev_config=None, **overrides):
+        """Generate a selection of strategy classes that implement the backend.
+        """
+        # Ingest the config and build out the ordered list of
+        # backend elements to include
+        if config is None:
+            config = get_config()
+        self.config = config
+        self.prev_config = prev_config
+        self.overrides = overrides
+
+        # We always install the GUI.
+        backends = [InstallMixin, ]
+
+        api = "python" if legacy_juju() else "go"
+        sandbox = config.get('sandbox', False)
+        staging = config.get('staging', False)
+
+        if api == 'python':
+            if staging:
+                backends.append(ImprovBackend)
+            elif sandbox:
+                backends.append(SandboxBackend)
+            else:
+                backends.append(PythonBackend)
+        else:
+            if staging:
+                raise ValueError(
+                    "Unable to use staging with {} backend".format(api))
+            if sandbox:
+                raise ValueError(
+                    "Unable to use sandbox with {} backend".format(api))
+            backends.append(GoBackend)
+
+        # All backends need to install, start, and stop the services that
+        # provide the GUI.
+        backends.append(GuiMixin)
+        backends.append(UpstartMixin)
+
+        # record our choice mapping classes to instances
+        for i, b in enumerate(backends):
+            if callable(b):
+                backends[i] = b()
+        self.backends = backends
+
+    def __getitem__(self, key):
+        try:
+            return self.config[key]
+        except KeyError:
+            print("Unable to extract config key '%s' from %s" %
+                (key, self.config))
+            raise
+
+    @overrideable
+    def find_missing_packages(self, *packages):
+        return find_missing_packages(*packages)
+
+    @overrideable
+    def service_control(self, service, action):
+        service_control(service, action)
+
+    @overrideable
+    def start_agent(self, cert_path):
+        start_agent(cert_path)
+
+    @overrideable
+    def start_improv(self, stage_env, cert_path):
+        start_improv(stage_env, cert_path)
+
+    @overrideable
+    def log(self, msg, *args):
+        log(msg, *args)
+
+    @overrideable
+    def install_extra_repositories(self, *packages):
+        if self.config.get('allow-additional-deb-repositories', True):
+            install_extra_repositories(*packages)
+        else:
+            apt_get('update')
+
+    def different(self, *keys):
+        """Return a boolean indicating if the current config
+        value differs from the config value passed in prev_config
+        with respect to any of the passed in string keys.
+        """
+        if self.prev_config is None:
+            return True
+
+        for key in keys:
+            current = self.config.get(key)
+            prev = self.prev_config.get(key)
+            r = current != prev
+            if r:
+                return True
+        return False
+
+    ## Composed Methods
+    install = chain('install')
+    start = chain('start')
+    stop = chain('stop')
+
+    ## Merged Properties
+    dependencies = merge('dependencies')
+    build_dependencies = merge('build_dependencies')
+    staging_dependencies = merge('staging_dependencies')
+
+    repositories = merge('repositories')
+    debs = merge('debs')
+    upstart_scripts = merge('upstart_scripts')

=== modified file 'tests/test_utils.py'
--- tests/test_utils.py	2013-04-28 00:59:29 +0000
+++ tests/test_utils.py	2013-04-30 14:57:26 +0000
@@ -12,25 +12,27 @@
 import charmhelpers
 import yaml
 
+from backend import InstallMixin
 from utils import (
+    API_PORT,
+    JUJU_GUI_DIR,
+    JUJU_PEM,
+    WEB_PORT,
     _get_by_attr,
-    API_PORT,
     cmd_log,
     first_path_in_dir,
     get_api_address,
     get_release_file_url,
     get_zookeeper_address,
-    JUJU_GUI_DIR,
-    JUJU_PEM,
     legacy_juju,
     log_hook,
     parse_source,
+    get_npm_cache_archive_url,
     render_to_file,
     save_or_create_certificates,
     start_agent,
     start_gui,
     start_improv,
-    WEB_PORT,
 )
 # Import the whole utils package for monkey patching.
 import utils
@@ -603,5 +605,86 @@
         self.assertNotIn('redirect scheme https', haproxy_conf)
 
 
+class TestNpmCache(unittest.TestCase):
+    """To speed building from a branch we prepopulate the NPM cache."""
+
+    def test_retrieving_cache_url(self):
+        # The URL for the latest cache file can be retrieved from Launchpad.
+        class FauxLaunchpadFactory(object):
+            @staticmethod
+            def login_anonymously(agent, site):
+                # We download the cache from the production site.
+                self.assertEqual(site, 'production')
+                return FauxLaunchpad
+
+        class CacheFile(object):
+            file_link = 'http://launchpad.example/path/to/cache/file'
+
+            def __str__(self):
+                return 'cache-file-123.tgz'
+
+        class NpmRelease(object):
+            files = [CacheFile()]
+
+        class NpmSeries(object):
+            name = 'npm-cache'
+            releases = [NpmRelease]
+
+        class FauxProject(object):
+            series = [NpmSeries]
+
+        class FauxLaunchpad(object):
+            projects = {'juju-gui': FauxProject()}
+
+        url = get_npm_cache_archive_url(Launchpad=FauxLaunchpadFactory())
+        self.assertEqual(url, 'http://launchpad.example/path/to/cache/file')
+
+
+    def test_InstallMixin_primes_npm_cache(self):
+        # The InstallMixin.install() method primes the NPM cache before
+        # building (and installing) the GUI from a branch.
+        assertFalse = self.assertFalse
+        assertTrue = self.assertTrue
+
+        class TestableInstallMixin(InstallMixin):
+            """An InstallMixin that records actions instead of taking them."""
+            cache_primed = False
+            branch_installed = False
+
+            def _prime_npm_cache(self):
+                # The cache is primed before the branch is installed.
+                assertFalse(self.branch_installed)
+                self.cache_primed = True
+
+            def _build_and_install_from_branch(self):
+                # The cache is primed before the branch is installed.
+                assertTrue(self.cache_primed)
+                self.branch_installed = True
+
+        class FauxBackend(object):
+            """A test backend."""
+            config = None
+            debs = ('DEBS',)
+
+            @classmethod
+            def find_missing_packages(cls, *debs):
+                self.assertEqual(cls.debs, debs)
+                return False
+
+            @classmethod
+            def different(cls, key):
+                self.assertEqual(key, 'juju-gui-source')
+                return True
+
+        mixin = TestableInstallMixin()
+        # Prior to "install" the NPM cache has not been primed.
+        self.assertFalse(mixin.cache_primed)
+        self.assertFalse(mixin.branch_installed)
+        mixin.install(FauxBackend())
+        # After "install" the NPM cache has been primed.
+        self.assertTrue(mixin.cache_primed)
+        self.assertTrue(mixin.branch_installed)
+
+
 if __name__ == '__main__':
     unittest.main(verbosity=2)

=== modified symlink 'tests/utils.py'
=== target was u'../hooks/utils.py'
--- tests/utils.py	1970-01-01 00:00:00 +0000
+++ tests/utils.py	2013-04-30 14:57:26 +0000
@@ -0,0 +1,629 @@
+"""Juju GUI charm utilities."""
+
+__all__ = [
+    'AGENT',
+    'APACHE',
+    'API_PORT',
+    'CURRENT_DIR',
+    'HAPROXY',
+    'IMPROV',
+    'JUJU_DIR',
+    'JUJU_GUI_DIR',
+    'JUJU_GUI_SITE',
+    'JUJU_PEM',
+    'StopChain',
+    'WEB_PORT',
+    'bzr_checkout',
+    'chain',
+    'cmd_log',
+    'fetch_api',
+    'fetch_gui',
+    'first_path_in_dir',
+    'get_api_address',
+    'get_release_file_url',
+    'get_staging_dependencies',
+    'get_zookeeper_address',
+    'legacy_juju',
+    'log_hook',
+    'merge',
+    'overrideable',
+    'parse_source',
+    'render_to_file',
+    'save_or_create_certificates',
+    'setup_apache',
+    'setup_gui',
+    'start_agent',
+    'start_gui',
+    'start_improv',
+    'write_apache_config',
+]
+
+from contextlib import contextmanager
+import json
+import os
+import logging
+import shutil
+from subprocess import CalledProcessError
+import tempfile
+from urlparse import urlparse
+
+from launchpadlib.launchpad import Launchpad
+from shelltoolbox import (
+    Serializer,
+    apt_get_install,
+    command,
+    environ,
+    install_extra_repositories,
+    run,
+    script_name,
+    search_file,
+    su,
+)
+from charmhelpers import (
+    START,
+    get_config,
+    log,
+    service_control,
+    unit_get,
+)
+
+import apt
+import tempita
+
+
+AGENT = 'juju-api-agent'
+APACHE = 'apache2'
+IMPROV = 'juju-api-improv'
+HAPROXY = 'haproxy'
+
+API_PORT = 8080
+WEB_PORT = 8000
+
+CURRENT_DIR = os.getcwd()
+JUJU_DIR = os.path.join(CURRENT_DIR, 'juju')
+JUJU_GUI_DIR = os.path.join(CURRENT_DIR, 'juju-gui')
+JUJU_GUI_SITE = '/etc/apache2/sites-available/juju-gui'
+JUJU_GUI_PORTS = '/etc/apache2/ports.conf'
+JUJU_PEM = 'juju.includes-private-key.pem'
+BUILD_REPOSITORIES = ('ppa:chris-lea/node.js-legacy',)
+DEB_BUILD_DEPENDENCIES = (
+    'bzr', 'imagemagick', 'make',  'nodejs', 'npm',
+)
+DEB_STAGE_DEPENDENCIES = (
+    'zookeeper',
+)
+
+
+# Store the configuration from on invocation to the next.
+config_json = Serializer('/tmp/config.json')
+# Bazaar checkout command.
+bzr_checkout = command('bzr', 'co', '--lightweight')
+# Whether or not the charm is deployed using juju-core.
+# If juju-core has been used to deploy the charm, an agent.conf file must
+# be present in the charm parent directory.
+legacy_juju = lambda: not os.path.exists(
+    os.path.join(CURRENT_DIR, '..', 'agent.conf'))
+
+
+def _get_build_dependencies():
+    """Install deb dependencies for building."""
+    log('Installing build dependencies.')
+    cmd_log(install_extra_repositories(*BUILD_REPOSITORIES))
+    cmd_log(apt_get_install(*DEB_BUILD_DEPENDENCIES))
+
+
+def get_api_address(unit_dir):
+    """Return the Juju API address stored in the uniter agent.conf file."""
+    import yaml  # python-yaml is only installed if juju-core is used.
+    # XXX 2013-03-27 frankban bug=1161443:
+        # currently the uniter agent.conf file does not include the API
+        # address. For now retrieve it from the machine agent file.
+    base_dir = os.path.abspath(os.path.join(unit_dir, '..'))
+    for dirname in os.listdir(base_dir):
+        if dirname.startswith('machine-'):
+            agent_conf = os.path.join(base_dir, dirname, 'agent.conf')
+            break
+    else:
+        raise IOError('Juju agent configuration file not found.')
+    contents = yaml.load(open(agent_conf))
+    return contents['apiinfo']['addrs'][0]
+
+
+def get_staging_dependencies():
+    """Install deb dependencies for the stage (improv) environment."""
+    log('Installing stage dependencies.')
+    cmd_log(apt_get_install(*DEB_STAGE_DEPENDENCIES))
+
+
+def first_path_in_dir(directory):
+    """Return the full path of the first file/dir in *directory*."""
+    return os.path.join(directory, os.listdir(directory)[0])
+
+
+def _get_by_attr(collection, attr, value):
+    """Return the first item in collection having attr == value.
+
+    Return None if the item is not found.
+    """
+    for item in collection:
+        if getattr(item, attr) == value:
+            return item
+
+
+def get_release_file_url(project, series_name, release_version):
+    """Return the URL of the release file hosted in Launchpad.
+
+    The returned URL points to a release file for the given project, series
+    name and release version.
+    The argument *project* is a project object as returned by launchpadlib.
+    The arguments *series_name* and *release_version* are strings. If
+    *release_version* is None, the URL of the latest release will be returned.
+    """
+    series = _get_by_attr(project.series, 'name', series_name)
+    if series is None:
+        raise ValueError('%r: series not found' % series_name)
+    # Releases are returned by Launchpad in reverse date order.
+    releases = list(series.releases)
+    if not releases:
+        raise ValueError('%r: series does not contain releases' % series_name)
+    if release_version is not None:
+        release = _get_by_attr(releases, 'version', release_version)
+        if release is None:
+            raise ValueError('%r: release not found' % release_version)
+        releases = [release]
+    for release in releases:
+        for file_ in release.files:
+            if str(file_).endswith('.tgz'):
+                return file_.file_link
+    raise ValueError('%r: file not found' % release_version)
+
+
+def get_zookeeper_address(agent_file_path):
+    """Retrieve the Zookeeper address contained in the given *agent_file_path*.
+
+    The *agent_file_path* is a path to a file containing a line similar to the
+    following::
+
+        env JUJU_ZOOKEEPER="address"
+    """
+    line = search_file('JUJU_ZOOKEEPER', agent_file_path).strip()
+    return line.split('=')[1].strip('"')
+
+
+@contextmanager
+def log_hook():
+    """Log when a hook starts and stops its execution.
+
+    Also log to stdout possible CalledProcessError exceptions raised executing
+    the hook.
+    """
+    script = script_name()
+    log(">>> Entering {}".format(script))
+    try:
+        yield
+    except CalledProcessError as err:
+        log('Exception caught:')
+        log(err.output)
+        raise
+    finally:
+        log("<<< Exiting {}".format(script))
+
+
+def parse_source(source):
+    """Parse the ``juju-gui-source`` option.
+
+    Return a tuple of two elements representing info on how to deploy Juju GUI.
+    Examples:
+       - ('stable', None): latest stable release;
+       - ('stable', '0.1.0'): stable release v0.1.0;
+       - ('trunk', None): latest trunk release;
+       - ('trunk', '0.1.0+build.1'): trunk release v0.1.0 bzr revision 1;
+       - ('branch', 'lp:juju-gui'): release is made from a branch;
+       - ('url', 'http://example.com/gui'): release from a downloaded file.
+    """
+    if source.startswith('url:'):
+        source = source[4:]
+        # Support file paths, including relative paths.
+        if urlparse(source).scheme == '':
+            if not source.startswith('/'):
+                source = os.path.join(os.path.abspath(CURRENT_DIR), source)
+            source = "file://%s" % source
+        return 'url', source
+    if source in ('stable', 'trunk'):
+        return source, None
+    if source.startswith('lp:') or source.startswith('http://'):
+        return 'branch', source
+    if 'build' in source:
+        return 'trunk', source
+    return 'stable', source
+
+
+def render_to_file(template_name, context, destination):
+    """Render the given *template_name* into *destination* using *context*.
+
+    The tempita template language is used to render contents
+    (see http://pythonpaste.org/tempita/).
+    The argument *template_name* is the name or path of the template file:
+    it may be either a path relative to ``../config`` or an absolute path.
+    The argument *destination* is a file path.
+    The argument *context* is a dict-like object.
+    """
+    template_path = os.path.join(
+        os.path.dirname(__file__), '..', 'config', template_name)
+    template = tempita.Template.from_filename(template_path)
+    with open(destination, 'w') as stream:
+        stream.write(template.substitute(context))
+
+
+results_log = None
+
+
+def _setupLogging():
+    global results_log
+    if results_log is not None:
+        return
+    config = get_config()
+    logging.basicConfig(
+        filename=config['command-log-file'],
+        level=logging.INFO,
+        format="%(asctime)s: %(name)s@%(levelname)s %(message)s")
+    results_log = logging.getLogger('juju-gui')
+
+
+def cmd_log(results):
+    global results_log
+    if not results:
+        return
+    if results_log is None:
+        _setupLogging()
+    # Since 'results' may be multi-line output, start it on a separate line
+    # from the logger timestamp, etc.
+    results_log.info('\n' + results)
+
+
+def start_improv(staging_env, ssl_cert_path,
+                 config_path='/etc/init/juju-api-improv.conf'):
+    """Start a simulated juju environment using ``improv.py``."""
+    log('Setting up staging start up script.')
+    context = {
+        'juju_dir': JUJU_DIR,
+        'keys': ssl_cert_path,
+        'port': API_PORT,
+        'staging_env': staging_env,
+    }
+    render_to_file('juju-api-improv.conf.template', context, config_path)
+    log('Starting the staging backend.')
+    with su('root'):
+        service_control(IMPROV, START)
+
+
+def start_agent(ssl_cert_path, config_path='/etc/init/juju-api-agent.conf'):
+    """Start the Juju agent and connect to the current environment."""
+    # Retrieve the Zookeeper address from the start up script.
+    unit_dir = os.path.realpath(os.path.join(CURRENT_DIR, '..'))
+    agent_file = '/etc/init/juju-{0}.conf'.format(os.path.basename(unit_dir))
+    zookeeper = get_zookeeper_address(agent_file)
+    log('Setting up API agent start up script.')
+    context = {
+        'juju_dir': JUJU_DIR,
+        'keys': ssl_cert_path,
+        'port': API_PORT,
+        'zookeeper': zookeeper,
+    }
+    render_to_file('juju-api-agent.conf.template', context, config_path)
+    log('Starting API agent.')
+    with su('root'):
+        service_control(AGENT, START)
+
+
+def start_gui(
+        console_enabled, login_help, readonly, in_staging, ssl_cert_path,
+        charmworld_url, serve_tests, haproxy_path='/etc/haproxy/haproxy.cfg',
+        config_js_path=None, secure=True, sandbox=False):
+    """Set up and start the Juju GUI server."""
+    with su('root'):
+        run('chown', '-R', 'ubuntu:', JUJU_GUI_DIR)
+    # XXX 2013-02-05 frankban bug=1116320:
+        # External insecure resources are still loaded when testing in the
+        # debug environment. For now, switch to the production environment if
+        # the charm is configured to serve tests.
+    if in_staging and not serve_tests:
+        build_dirname = 'build-debug'
+    else:
+        build_dirname = 'build-prod'
+    build_dir = os.path.join(JUJU_GUI_DIR, build_dirname)
+    log('Generating the Juju GUI configuration file.')
+    is_legacy_juju = legacy_juju()
+    user, password = None, None
+    if is_legacy_juju and in_staging:
+        user, password = 'admin', 'admin'
+    else:
+        user, password = None, None
+
+    api_backend = 'python' if is_legacy_juju else 'go'
+    if secure:
+        protocol = 'wss'
+    else:
+        log('Running in insecure mode! Port 80 will serve unencrypted.')
+        protocol = 'ws'
+
+    context = {
+        'raw_protocol': protocol,
+        'address': unit_get('public-address'),
+        'console_enabled': json.dumps(console_enabled),
+        'login_help': json.dumps(login_help),
+        'password': json.dumps(password),
+        'api_backend': json.dumps(api_backend),
+        'readonly': json.dumps(readonly),
+        'user': json.dumps(user),
+        'protocol': json.dumps(protocol),
+        'sandbox': json.dumps(sandbox),
+        'charmworld_url': json.dumps(charmworld_url),
+    }
+    if config_js_path is None:
+        config_js_path = os.path.join(
+            build_dir, 'juju-ui', 'assets', 'config.js')
+    render_to_file('config.js.template', context, config_js_path)
+
+    write_apache_config(build_dir, serve_tests)
+
+    log('Generating haproxy configuration file.')
+    if is_legacy_juju:
+        # The PyJuju API agent is listening on localhost.
+        api_address = '127.0.0.1:{0}'.format(API_PORT)
+    else:
+        # Retrieve the juju-core API server address.
+        api_address = get_api_address(os.path.join(CURRENT_DIR, '..'))
+    context = {
+        'api_address': api_address,
+        'api_pem': JUJU_PEM,
+        'legacy_juju': is_legacy_juju,
+        'ssl_cert_path': ssl_cert_path,
+        # In PyJuju environments, use the same certificate for both HTTPS and
+        # WebSocket connections. In juju-core the system already has the proper
+        # certificate installed.
+        'web_pem': JUJU_PEM,
+        'web_port': WEB_PORT,
+        'secure': secure
+    }
+    render_to_file('haproxy.cfg.template', context, haproxy_path)
+    log('Starting Juju GUI.')
+
+def write_apache_config(build_dir, serve_tests=False):
+    log('Generating the apache site configuration file.')
+    context = {
+        'port': WEB_PORT,
+        'serve_tests': serve_tests,
+        'server_root': build_dir,
+        'tests_root': os.path.join(JUJU_GUI_DIR, 'test', ''),
+    }
+    render_to_file('apache-ports.template', context, JUJU_GUI_PORTS)
+    render_to_file('apache-site.template', context, JUJU_GUI_SITE)
+
+
+def get_npm_cache_archive_url(Launchpad=Launchpad):
+    """Figure out the URL of the most recent NPM cache archive on Launchpad."""
+    launchpad = Launchpad.login_anonymously('Juju GUI charm', 'production')
+    project = launchpad.projects['juju-gui']
+    # Find the URL of the most recently created NPM cache archive.
+    npm_cache_url = get_release_file_url(project, 'npm-cache', None)
+    return npm_cache_url
+
+
+def prime_npm_cache(npm_cache_url):
+    """Download NPM cache archive and prime the NPM cache with it."""
+    # Download the cache archive and then uncompress it into the NPM cache.
+    npm_cache_archive = os.path.join(CURRENT_DIR, 'npm-cache.tgz')
+    cmd_log(run('curl', '-L', '-o', npm_cache_archive, npm_cache_url))
+    npm_cache_dir = os.path.expanduser('~/.npm')
+    # The NPM cache directory probably does not exist, so make it if not.
+    try:
+        os.mkdir(npm_cache_dir)
+    except OSError, e:
+        # If the directory already exists then ignore the error.
+        if e.errno != 17: # File exists.
+            raise
+    uncompress = command('tar', '-x', '-z', '-C', npm_cache_dir, '-f')
+    cmd_log(uncompress(npm_cache_archive))
+
+
+def fetch_gui(juju_gui_source, logpath):
+    """Retrieve the Juju GUI release/branch."""
+    # Retrieve a Juju GUI release.
+    origin, version_or_branch = parse_source(juju_gui_source)
+    if origin == 'branch':
+        # Make sure we have the dependencies necessary for us to actually make
+        # a build.
+        _get_build_dependencies()
+        # Create a release starting from a branch.
+        juju_gui_source_dir = os.path.join(CURRENT_DIR, 'juju-gui-source')
+        log('Retrieving Juju GUI source checkout from %s.' % version_or_branch)
+        cmd_log(run('rm', '-rf', juju_gui_source_dir))
+        cmd_log(bzr_checkout(version_or_branch, juju_gui_source_dir))
+        log('Preparing a Juju GUI release.')
+        logdir = os.path.dirname(logpath)
+        fd, name = tempfile.mkstemp(prefix='make-distfile-', dir=logdir)
+        log('Output from "make distfile" sent to %s' % name)
+        with environ(NO_BZR='1'):
+            run('make', '-C', juju_gui_source_dir, 'distfile',
+                stdout=fd, stderr=fd)
+        release_tarball = first_path_in_dir(
+            os.path.join(juju_gui_source_dir, 'releases'))
+    else:
+        log('Retrieving Juju GUI release.')
+        if origin == 'url':
+            file_url = version_or_branch
+        else:
+            # Retrieve a release from Launchpad.
+            launchpad = Launchpad.login_anonymously(
+                'Juju GUI charm', 'production')
+            project = launchpad.projects['juju-gui']
+            file_url = get_release_file_url(project, origin, version_or_branch)
+        log('Downloading release file from %s.' % file_url)
+        release_tarball = os.path.join(CURRENT_DIR, 'release.tgz')
+        cmd_log(run('curl', '-L', '-o', release_tarball, file_url))
+    return release_tarball
+
+
+def fetch_api(juju_api_branch):
+    """Retrieve the Juju branch."""
+    # Retrieve Juju API source checkout.
+    log('Retrieving Juju API source checkout.')
+    cmd_log(run('rm', '-rf', JUJU_DIR))
+    cmd_log(bzr_checkout(juju_api_branch, JUJU_DIR))
+
+
+def setup_gui(release_tarball):
+    """Set up Juju GUI."""
+    # Uncompress the release tarball.
+    log('Installing Juju GUI.')
+    release_dir = os.path.join(CURRENT_DIR, 'release')
+    cmd_log(run('rm', '-rf', release_dir))
+    os.mkdir(release_dir)
+    uncompress = command('tar', '-x', '-z', '-C', release_dir, '-f')
+    cmd_log(uncompress(release_tarball))
+    # Link the Juju GUI dir to the contents of the release tarball.
+    cmd_log(run('ln', '-sf', first_path_in_dir(release_dir), JUJU_GUI_DIR))
+
+
+def setup_apache():
+    """Set up apache."""
+    log('Setting up apache.')
+    if not os.path.exists(JUJU_GUI_SITE):
+        cmd_log(run('touch', JUJU_GUI_SITE))
+        cmd_log(run('chown', 'ubuntu:', JUJU_GUI_SITE))
+        cmd_log(
+            run('ln', '-s', JUJU_GUI_SITE,
+                '/etc/apache2/sites-enabled/juju-gui'))
+
+    if not os.path.exists(JUJU_GUI_PORTS):
+        cmd_log(run('touch', JUJU_GUI_PORTS))
+        cmd_log(run('chown', 'ubuntu:', JUJU_GUI_PORTS))
+
+    with su('root'):
+        run('a2dissite', 'default')
+        run('a2ensite', 'juju-gui')
+
+
+def save_or_create_certificates(
+        ssl_cert_path, ssl_cert_contents, ssl_key_contents):
+    """Generate the SSL certificates.
+
+    If both *ssl_cert_contents* and *ssl_key_contents* are provided, use them
+    as certificates; otherwise, generate them.
+
+    Also create a pem file, suitable for use in the haproxy configuration,
+    concatenating the key and the certificate files.
+    """
+    crt_path = os.path.join(ssl_cert_path, 'juju.crt')
+    key_path = os.path.join(ssl_cert_path, 'juju.key')
+    if not os.path.exists(ssl_cert_path):
+        os.makedirs(ssl_cert_path)
+    if ssl_cert_contents and ssl_key_contents:
+        # Save the provided certificates.
+        with open(crt_path, 'w') as cert_file:
+            cert_file.write(ssl_cert_contents)
+        with open(key_path, 'w') as key_file:
+            key_file.write(ssl_key_contents)
+    else:
+        # Generate certificates.
+        # See http://superuser.com/questions/226192/openssl-without-prompt
+        cmd_log(run(
+            'openssl', 'req', '-new', '-newkey', 'rsa:4096',
+            '-days', '365', '-nodes', '-x509', '-subj',
+            # These are arbitrary test values for the certificate.
+            '/C=GB/ST=Juju/L=GUI/O=Ubuntu/CN=juju.ubuntu.com',
+            '-keyout', key_path, '-out', crt_path))
+    # Generate the pem file.
+    pem_path = os.path.join(ssl_cert_path, JUJU_PEM)
+    if os.path.exists(pem_path):
+        os.remove(pem_path)
+    with open(pem_path, 'w') as pem_file:
+        shutil.copyfileobj(open(key_path), pem_file)
+        shutil.copyfileobj(open(crt_path), pem_file)
+
+
+def find_missing_packages(*packages):
+    """Given a list of packages, return the packages which are not installed.
+    """
+    cache = apt.Cache()
+    missing = set()
+    for pkg_name in packages:
+        try:
+            pkg = cache[pkg_name]
+        except KeyError:
+            missing.add(pkg_name)
+            continue
+        if pkg.is_installed:
+            continue
+        missing.add(pkg_name)
+    return missing
+
+
+## Backend support decorators
+class StopChain(Exception):
+    """Stop Processing a chain command without raising
+    another error.
+    """
+
+
+def chain(name, reverse=False):
+    """Helper method to compose a set of strategy objects into
+    a callable.
+
+    Each method is called in the context of its strategy
+    instance (normal OOP) and its argument is the Backend
+    instance.
+    """
+    # chain method calls through all implementing mixins
+    def method(self):
+        workingset = self.backends
+        if reverse:
+            workingset = reversed(workingset)
+        for backend in workingset:
+            call = backend.__class__.__dict__.get(name)
+            if call:
+                try:
+                    call(backend, self)
+                except StopChain:
+                    break
+
+    method.__name__ = name
+    return method
+
+
+def overrideable(f):
+    """Helper to support very limited overrides for use in testing.
+
+    def foo():
+        return True
+    b = Backend(foo=foo)
+    assert b.foo() is True
+    """
+    name = f.__name__
+
+    def overridden(self, *args, **kwargs):
+        if name in self.overrides:
+            return self.overrides[name](*args, **kwargs)
+        else:
+            return f(self, *args, **kwargs)
+
+    overridden.__name__ = name
+    return overridden
+
+
+def merge(name):
+    """Helper to merge a property from a set of strategy objects
+    into a unified set.
+    """
+    # return merged property from every providing backend as a set
+    @property
+    def method(self):
+        result = set()
+        for backend in self.backends:
+            segment = backend.__class__.__dict__.get(name)
+            if segment and isinstance(segment, (list, tuple, set)):
+                result |= set(segment)
+
+        return result
+    return method