Merge lp:~hazmat/juju-jitsu/import-export into lp:juju-jitsu

[Kapil Thangavelu]

jitsu environment import export

Deployment automation for juju. We see lots of shell scripts for creating
environments. Ideally we could just edit a json file or dump an existing
environment to create a new one. These pair of commands (export and import) do
just that.

 you can dump an environment to json with

   jitsu export -e env_name > env.json

 it will capture services, unit counts, constraints, service config, relations, etc.

 That can in turn be imported to another environment with

   jitsu import -e env_name env.json

R=negronil

https://codereview.appspot.com/6350057/

[Kapil Thangavelu (hazmat)]

Reviewers: mp+112819_code.launchpad.net,

Message:
Please take a look.

Description:
jitsu environment import export

Deployment automation for juju. We see lots of shell scripts for
creating
environments. Ideally we could just edit a json file or dump an existing

environment to create a new one. These pair of commands (export and
import) do
just that.

  you can dump an environment to json with

    jitsu export -e env_name > env.json

  it will capture services, unit counts, constraints, service config,
relations, etc.

  That can in turn be imported to another environment with

    jitsu import -e env_name env.json

https://code.launchpad.net/~hazmat/juju-jitsu/import-export/+merge/112819

(do not edit description out of merge proposal)

Please review this at https://codereview.appspot.com/6350057/

Affected files:
   A [revision details]
   A sub-commands/export
   A sub-commands/import

[Juan L. Negron (negronjl)]

Kapil:

Thanks for this. It is very useful.

Some thoughts about both export and import:
- They could both use more verbose help that details how to use the tools.

Some thoughts about import:
- It appears that import needs a local repository ( or I don't know how to tell it to use cs: which could be alleviated by more verbose help )

I tried both export and import. They both work.

Thanks Kapil.

Thanks,

Juan

review: Approve

[lp:~hazmat/juju-jitsu/import-export]

63. By Kapil Thangavelu on 2012-06-30

import allows stdin/pipes, doesn't require local repo when using store exclusively, distinguish file not present vs json errors, doc strings for both import&export

[Kapil Thangavelu (hazmat)]

On Fri, Jun 29, 2012 at 3:06 PM, Juan L. Negron <email address hidden>wrote:

> Review: Approve
>
> Kapil:
>
> Thanks for this. It is very useful.
>
> Some thoughts about both export and import:
> - They could both use more verbose help that details how to use the tools.
>
>
Added more verbose docs

> Some thoughts about import:
> - It appears that import needs a local repository ( or I don't know how to
> tell it to use cs: which could be alleviated by more verbose help )
>

that was an oversight (i had a JUJU_REPOSITORY set during dev), fixed. also
add ability to pipe directly into jitsu import from export.

cheers,

Kapil

[Kapil Thangavelu (hazmat)]

*** Submitted:

jitsu environment import export

Deployment automation for juju. We see lots of shell scripts for
creating
environments. Ideally we could just edit a json file or dump an existing

environment to create a new one. These pair of commands (export and
import) do
just that.

  you can dump an environment to json with

    jitsu export -e env_name > env.json

  it will capture services, unit counts, constraints, service config,
relations, etc.

  That can in turn be imported to another environment with

    jitsu import -e env_name env.json

R=negronil

R=negronjl
CC=
https://codereview.appspot.com/6350057

https://codereview.appspot.com/6350057/

=== added file 'sub-commands/export'
--- sub-commands/export	1970-01-01 00:00:00 +0000
+++ sub-commands/export	2012-06-30 15:56:19 +0000
@@ -0,0 +1,169 @@
+#!/usr/bin/env python
+
+import argparse
+import json
+import logging
+import os
+import zookeeper
+
+from twisted.internet.defer import inlineCallbacks
+from twisted.internet import reactor
+
+from juju.environment.config import EnvironmentsConfig
+
+from juju.state.service import ServiceStateManager
+from juju.state.relation import RelationStateManager
+from juju.state.environment import EnvironmentStateManager
+
+
+log = logging.getLogger("jitsu.export")
+
+
+@inlineCallbacks
+def export(env):
+
+    provider = env.get_machine_provider()
+    client = yield provider.connect()
+
+    data = {'services': [],
+            'relations': []}
+
+    rels = set()
+
+    # Get services
+    services = ServiceStateManager(client)
+    relations = RelationStateManager(client)
+    environment = EnvironmentStateManager(client)
+
+    for s in (yield services.get_all_service_states()):
+        units = yield s.get_unit_names()
+        charm_id = yield s.get_charm_id()
+        constraints = yield s.get_constraints()
+        config = yield s.get_config()
+        exposed = yield s.get_exposed_flag()
+
+        log.info("processing service %s / %s", s.service_name, charm_id)
+
+        # Really!? relation-type not available via state api
+        topology = yield relations._read_topology()
+
+        # More efficient to do this via topology, ie op per rel
+        # instead inspection of both sides by each endpoint, which is
+        # kinda of gross, need some state api support. But trying to
+        # stick to the state api as much as possible.
+        svc_rels = yield relations.get_relations_for_service(s)
+
+
+        for r in svc_rels:
+            relation_type = topology.get_relation_type(r.internal_relation_id)
+            rdata = [
+                (s.service_name,
+                 relation_type,
+                 r.relation_name,
+                 r.relation_role,
+                 r.relation_scope)]
+            if r.relation_role == "peer":
+                rels.add(tuple(rdata))
+                continue
+            rel_services = yield r.get_service_states()
+
+            # Get the endpoint's svc rel state.
+            found = None
+            for ep_svc in rel_services:
+                if ep_svc.service_name == s.service_name:
+                    continue
+                ep_rels = yield relations.get_relations_for_service(ep_svc)
+                for ep_r in ep_rels:
+                    if ep_r.internal_relation_id != r.internal_relation_id:
+                        continue
+                    found = (ep_svc, ep_r)
+                    break
+
+            if not found:
+                log.info("Couldn't decipher rel %s %s",
+                         s.service_name, r)
+                continue
+
+            ep_svc, ep_r = found
+            rdata.append(
+                (ep_svc.service_name,
+                 relation_type,
+                 ep_r.relation_name,
+                 ep_r.relation_role,
+                 ep_r.relation_scope))
+            rdata.sort()
+            rels.add(tuple(rdata))
+
+        data['services'].append(
+            {'name': s.service_name,
+             'charm': charm_id,
+             'exposed': exposed,
+             'unit_count': len(units),
+             'constraints': constraints.data,
+             'config': dict(config.items())})
+
+    data['relations'] = list(rels)
+    data['constraints'] = (yield environment.get_constraints()).data
+    print json.dumps(data, indent=2)
+
+
+@inlineCallbacks
+def run_one(func, *args, **kw):
+    try:
+        yield func(*args, **kw)
+    finally:
+        reactor.stop()
+
+
+def get_environment(env_option):
+    env_config = EnvironmentsConfig()
+    env_config.load_or_write_sample()
+
+    if env_option is None:
+        environment = env_config.get_default()
+    else:
+        environment = env_config.get(env_option)
+
+    return env_config, environment
+
+
+def get_parser():
+    parser = argparse.ArgumentParser(
+        description="Export an environment")
+
+    parser.add_argument(
+        "--environment", "-e", default=os.environ.get("JUJU_ENV"),
+        help="Environment to operate on.")
+
+    return parser
+
+
+def main():
+    """
+    Export a juju environment.
+
+    Prints a json formatted file containing information needed to recreate
+    an environment (services, relations, config).
+
+    Example:
+       jitsu export -e env_name
+
+    Also respects JUJU_ENV for environment to use, defaults to default env.
+    """
+    parser = get_parser()
+    options = parser.parse_args()
+
+    log_options = {
+        "level": logging.DEBUG,
+        "format": "%(asctime)s %(name)s:%(levelname)s %(message)s"}
+    logging.basicConfig(**log_options)
+
+    zookeeper.set_debug_level(0)
+
+    env_config, environment = get_environment(options.environment)
+    reactor.callWhenRunning(run_one, export, environment)
+    reactor.run()
+
+
+if __name__ == '__main__':
+    main()

=== added file 'sub-commands/import'
--- sub-commands/import	1970-01-01 00:00:00 +0000
+++ sub-commands/import	2012-06-30 15:56:19 +0000
@@ -0,0 +1,241 @@
+#!/usr/bin/env python
+
+import argparse
+import json
+import logging
+import os
+import sys
+import textwrap
+import zookeeper
+
+from txzookeeper.retry import RetryClient
+
+from twisted.internet.defer import inlineCallbacks
+from twisted.internet import reactor
+
+from juju.charm.errors import CharmNotFound
+from juju.charm.publisher import CharmPublisher
+from juju.charm.repository import (
+    CharmURL, LocalCharmRepository, RemoteCharmRepository, CS_STORE_URL)
+from juju.control.utils import sync_environment_state
+from juju.environment.config import EnvironmentsConfig
+from juju.errors import JujuError
+
+from juju.state.endpoint import RelationEndpoint
+from juju.state.service import ServiceStateManager
+from juju.state.machine import MachineStateManager
+from juju.state.placement import _unassigned_placement
+from juju.state.relation import RelationStateManager
+
+
+
+log = logging.getLogger("jitsu.import")
+
+
+@inlineCallbacks
+def load(env_configs, env, repo_dir, data):
+    """
+
+    Check for name conflicts
+     - Upload charms
+     - Create services (+ config, constraints)
+     - Add units
+     - Add relations
+    """
+    provider = env.get_machine_provider()
+    client = yield provider.connect()
+    client = RetryClient(client)
+
+    # Make sure the provider is ready to go.
+    yield sync_environment_state(client, env_configs, env.name)
+
+    # Get state managers
+    services = ServiceStateManager(client)
+    relations = RelationStateManager(client)
+    machines = MachineStateManager(client)
+
+    # First detect conflicts
+    existing_services = yield services.get_all_service_states()
+    env_svc_names = [e.service_name for e in existing_services]
+    import_svc_names = [i['name'] for i in data['services']]
+    conflict_svc_names = set(env_svc_names) & set(import_svc_names)
+    if conflict_svc_names:
+        raise JujuError(
+            "Import has name conflicts with existing services %s" % (
+                ", ".join(conflict_svc_names)))
+
+    # Next find all the nesc charms
+    if repo_dir:
+        repository = LocalCharmRepository(repo_dir)
+    else:
+        repository = None
+
+    store = RemoteCharmRepository(CS_STORE_URL)
+
+    charms = []
+    for s in data['services']:
+        curl = CharmURL.infer(s['charm'], env.default_series)
+        if s['charm'].startswith('local:'):
+            if repository is None:
+                raise JujuError("Local charm needed but repository not specified")
+            try:
+                charm = yield repository.find(curl)
+            except CharmNotFound:
+                # retry with less specific version
+                charm = yield repository.find(
+                    CharmURL.infer(
+                        s['charm'][:s['charm'].rfind('-')],
+                        env.default_series))
+            charms.append((s['charm'], charm))
+        else:
+            assert s['charm'].startswith('cs:')
+            charm = yield store.find(curl)
+            charms.append((s['charm'], charm))
+
+    # Upload charms to the environment
+    publisher = CharmPublisher(client, provider.get_file_storage())
+    for cid, c in charms:
+        log.info("Publishing charm %s", cid)
+        yield publisher.add_charm(
+            str(CharmURL.infer(cid, env.default_series)),
+            c)
+    charm_states = yield publisher.publish()
+
+    # Index by url
+    charm_map = dict([(ci[0], cs) for ci, cs in zip(charms, charm_states)])
+    constraint_set = yield provider.get_constraint_set()
+
+    # Create services
+    for s in data['services']:
+        log.info("Creating service %s %s", s['name'], charm_map[s['charm']].id)
+        constraints = constraint_set.load(s['constraints'])
+        svc = yield services.add_service_state(
+            s['name'],
+            charm_map[s['charm']],
+            constraints)
+        config = yield svc.get_config()
+        config.update(s['config'])
+        yield config.write()
+
+        # Create units
+        log.info("Creating units")
+        for i in range(s['unit_count']):
+            unit = yield svc.add_unit_state()
+            yield _unassigned_placement(client, machines, unit)
+
+    # Add relations
+    for r in data['relations']:
+        if len(r) == 1:
+            eps = [RelationEndpoint(*r[0])]
+        else:
+            eps = [RelationEndpoint(*r[0]), RelationEndpoint(*r[1])]
+        log.info("Adding relation %s" %(
+            " ".join(map(
+                lambda r: "%s:%s %s" % (r.service_name, r.relation_name, r.relation_role),
+                eps))))
+        yield relations.add_relation_state(*eps)
+
+
+
+def get_parser():
+    parser = argparse.ArgumentParser(
+        formatter_class=argparse.RawDescriptionHelpFormatter,
+        description=textwrap.dedent(main.__doc__))
+
+    parser.add_argument(
+        "--environment", "-e", default=os.environ.get("JUJU_ENV"),
+        help="Environment to operate on.")
+
+    parser.add_argument(
+        "--repository", "-r", default=os.environ.get("JUJU_REPOSITORY"),
+        help="Reposiory for local charms.")
+
+    parser.add_argument(
+        "export_file", nargs=None,
+        help="Environment export file")
+
+    return parser
+
+
+@inlineCallbacks
+def run_one(func, *args, **kw):
+    try:
+        yield func(*args, **kw)
+    finally:
+        reactor.stop()
+
+
+def get_environment(env_option):
+    env_config = EnvironmentsConfig()
+    env_config.load_or_write_sample()
+
+    if env_option is None:
+        environment = env_config.get_default()
+    else:
+        environment = env_config.get(env_option)
+
+    return env_config, environment
+
+
+
+def main():
+    """
+    Load an environment export into an environment
+
+      jitsu import <environment-export-json-file>
+
+    The target environment must already be bootstrapped. Any conflicts
+    among service names between the export and the target environment
+    are fatal (and detected early).
+
+    if the environment uses local charms then a local repository needs
+    to be specified. For eample
+
+      jitsu import --repository=$HOME/charms env.json
+
+    Both JUJU_REPOSITORY and JUJU_ENV variables are respected.
+
+    Load also supports reading from stdin ie.
+
+      jitsu export -e staging | jitsu import -e production
+    """
+    parser = get_parser()
+    options = parser.parse_args()
+
+    log_options = {
+        "level": logging.DEBUG,
+        "format": "%(asctime)s %(name)s:%(levelname)s %(message)s"}
+    logging.basicConfig(**log_options)
+
+    zookeeper.set_debug_level(0)
+
+    if options.export_file == "-":
+        export_file = sys.stdin
+    else:
+        export_path = os.path.abspath(
+            os.path.expanduser(options.export_file))
+        if not os.path.exists(export_path):
+            raise JujuError("Invalid export file, does not exist %s" % export_path)
+        export_file = open(export_path)
+
+    try:
+        export_data = json.loads(export_file.read())
+    except Exception, e:
+        raise JujuError(
+            "Invalid environment export, malformed json %s %s" % (
+                options.export_file, e))
+
+    env_config, environment = get_environment(options.environment)
+
+    reactor.callWhenRunning(
+        run_one, load,
+        env_config,
+        environment,
+        options.repository,
+        export_data,
+        )
+    reactor.run()
+
+
+if __name__ == '__main__':
+    main()