Merge lp:~bjornt/landscape-charm/bundles-render-local into lp:~landscape/landscape-charm/trunk

[Björn Tillenius]

Allow the branch or charm of landscape-server to be given to
render-bundles.

This allows you to generate bundles that use a local branch or charm.

It's meant to be used by lp:~bjornt/landscape-charm/deploy-local-branch.
You can see the diff for that branch here:

    https://pastebin.canonical.com/158174/

This will allow you to use 'make stage-landscape-charm' to deploy the
landscape-server branch you're working on.

Testing instructions:

In a landscape-server branch:

    CHARM_BRANCH=lp:~bjornt/landscape-charm/deploy-local-branch make stage-landscape-charm
    bzr checkout lp:~bjornt/landscape-charm/bundles-render-local build/landscape-charm/bundles

To deploy with Juju 1.25 and juju-deployer:

    make -C build/landscape-charm deploy-local

To deploy with Juju 2:

    make -C build/landscape-charm bundles-local-charm
    juju-2.0 build/landscape-charm/bundles/build/landscape-scalable/bundle.yaml

[lp:~bjornt/landscape-charm/bundles-render-local]

28. By Björn Tillenius on 2016-06-10

Review comments.

Unmerged revisions

28. By Björn Tillenius on 2016-06-10

Review comments.

27. By Björn Tillenius on 2016-06-07

Allow the landscape-server branch to specified to render-bundles.

26. By Björn Tillenius on 2016-06-07

Allow the landscape-server charm to specified to render-bundles.

25. By David Britton on 2016-05-27

Merge bundles-trunk-xenial [f=1585389] [r=chad.smith,landscape-builder,fginther] [a=David Britton]
- Use Xenial charm for postgres and landscape.
- rework push/diff/publish workflow for new 'charm' tool
- make CHARM_STORE_USER required in Makefile for safety.

24. By David Britton on 2016-05-19

Just do a make render [trivial]

23. By Andreas Hasenack on 2016-05-18

Updated charm revisions

22. By Adam Collard on 2016-04-08

Bump us (again) to 16.03 from 15.11 in the bundle [trivial] [r=ack]

21. By David Britton on 2016-04-07

don't use machines in 'scalable' case.

20. By David Britton on 2016-04-07

check in rendered changes to bundles.

19. By David Britton on 2016-04-07

to: needs to be an array in v4 bundles.

=== added file '.bzrignore'
--- .bzrignore	1970-01-01 00:00:00 +0000
+++ .bzrignore	2016-06-07 14:03:21 +0000
@@ -0,0 +1,1 @@
+build

=== renamed file '.bzrignore' => '.bzrignore.moved'
=== added file 'Makefile'
--- Makefile	1970-01-01 00:00:00 +0000
+++ Makefile	2016-06-07 14:03:21 +0000
@@ -0,0 +1,29 @@
+diff-charm-store: check-env render
+	./charm-store diff $${CHARM_STORE_USER}
+
+push-charm-store: check-env render
+	./charm-store push $${CHARM_STORE_USER}
+
+publish-charm-store: check-env render
+	./charm-store publish-latest $${CHARM_STORE_USER}
+
+check-env:
+ifndef CHARM_STORE_USER
+	$(error CHARM_STORE_USER is undefined, hint: launchpad id.)
+endif
+
+clean:
+	@rm -rf build
+
+render: deps clean
+	@./render-bundles
+	
+update-charm-revisions:
+	@./update-charm-revisions \
+		$(EXTRA_UPDATE_ARGUMENTS) \
+		apache2 postgresql juju-gui haproxy rabbitmq-server nfs
+
+deps:
+	@./ubuntu-deps
+
+.PHONY: clean render commit-charm-store update-charm-revisions compare-charm-store deps

=== renamed file 'Makefile' => 'Makefile.moved'
=== added file 'README.md'
--- README.md	1970-01-01 00:00:00 +0000
+++ README.md	2016-06-07 14:03:21 +0000
@@ -0,0 +1,28 @@
+Overview
+========
+
+This branch contains the necessary config and instructions for using
+juju-deployer/juju-quickstart to deploy the landscape charm.
+
+Deployment should be straightforward.
+
+Dense Deployment - MAAS
+=======================
+For MAAS where LXCs can be addressed externally, you can deploy to a single
+machine while at the same time making the service scalable in the future:
+
+  juju quickstart u/landscape/landscape-dense-maas/
+
+Dense Deployment - Other
+========================
+For other substrates, use the plain dense deployment which allows your
+landscape server to be fully reachable, but will not allow easy scaling
+should load increase.
+
+  juju quickstart u/landscape/landscape-dense/
+
+Scalable Deployment
+===================
+For a truly scalable deployment, the following stanza should be used.
+
+  juju quickstart u/landscape/landscape-scalable

=== renamed file 'README.md' => 'README.md.moved'
=== added file 'charm-store'
--- charm-store	1970-01-01 00:00:00 +0000
+++ charm-store	2016-06-07 14:03:21 +0000
@@ -0,0 +1,70 @@
+#!/bin/bash -e
+#
+# Mostly intended to be called with make targets
+# Needs make clean, make render called first.
+#
+
+ACTION=$1
+STORE_USER=${2:-landscape}
+
+if [ -z "$ACTION" ]; then
+    cat <<EOF
+Usage: $0 ACTION [user]
+   actions:
+        push - push to 'unpublished' channel
+        publish-latest - publish the latest copy
+        diff - diff your branch to the latest stable channel copy
+   user:
+        charm store user name
+EOF
+    exit 1
+fi
+
+if [ "$ACTION" == "push" ]; then
+    echo "# Will be pushing new [unpublished] versions to the charm store"
+    echo "#   as user: $STORE_USER"
+fi
+
+for target in landscape-dense-maas landscape-dense landscape-scalable; do
+
+    if [ "$ACTION" == "diff" ]; then
+        echo "# [$target] Pulling fresh copy from stable channel"
+        charm pull --channel stable "cs:~${STORE_USER}/bundle/${target}" build/${target}-stable
+
+        echo "# [$target] Diff from latest stable"
+        diff -Naur build/${target}-stable build/${target} || /bin/true
+    fi
+
+    if [ "$ACTION" == "push" ]; then
+        echo "# [$target] Pushing to namespace ~${STORE_USER}, unpublished channel"
+        url=$(charm push build/${target} cs:~${STORE_USER}/bundle/${target} | grep ^url | awk '{ print $2 }')
+
+        echo "# [$target] Publishing to development channel"
+        charm publish --channel development $url
+    fi
+
+    if [ "$ACTION" == "publish-latest" ]; then
+        latest=$(charm show --channel unpublished --format=json cs:~${STORE_USER}/bundle/${target} | jq -r '.["revision-info"].Revisions[0]')
+        echo "# [$target] Publishing latest unpublished revision, stable channel"
+        charm publish $latest
+    fi
+
+done
+
+cat <<EOF
+#
+# You can manually examine things if you wish:"
+#   ll build/*"
+#   ll build/*-stable"
+#
+# Show details in the store:
+#   charm show [--channel unpublished] cs:~${STORE_USER}/bundle/landscape-dense-maas
+#
+# Make sure everyone can use your charm if you want:
+#   charm grant cs:~${STORE_USER}/bundle/BUNDLE_NAME-VERSION everyone
+#
+# Publish like this (controls what clients see by default, you can
+# even set it to a previous known good version): 
+#   charm publish cs:~${STORE_USER}/bundle/BUNDLE_NAME-VERSION
+#
+EOF

=== added file 'landscape-template.jinja2'
--- landscape-template.jinja2	1970-01-01 00:00:00 +0000
+++ landscape-template.jinja2	2016-06-07 14:03:21 +0000
@@ -0,0 +1,99 @@
+series: xenial
+services:
+    rabbitmq-server:
+        charm: cs:xenial/rabbitmq-server-0
+        {% if rabbitmq["to"] %}
+        to:
+            - {{ rabbitmq["to"] }}
+        {% endif %}
+        num_units: 1
+        annotations:
+            "gui-x": "600"
+            "gui-y": "370"
+    postgresql:
+        charm: cs:xenial/postgresql-99
+        {% if postgresql["memory"] %}
+        constraints: mem={{ postgresql["memory"] }}
+        {% endif %}
+        {% if postgresql["to"] %}
+        to:
+            - {{ postgresql["to"] }}
+        {% endif %}
+        num_units: 1
+        options:
+            extra_packages: python-apt postgresql-contrib postgresql-.*-debversion postgresql-plpython-.*
+            max_connections: {{ postgresql.max_connections }}
+            max_prepared_transactions: {{ postgresql.max_connections }}
+            {% if postgresql.manual_tuning %}
+            performance_tuning: manual
+            {% endif %}
+            {% if postgresql.shared_buffers %}
+            shared_buffers: {{ postgresql.shared_buffers }}
+            {% endif %}
+            {% if postgresql.checkpoint_segments %}
+            checkpoint_segments: {{ postgresql.checkpoint_segments }}
+            {% endif %}
+            {% if postgresql.maintenance_work_mem %}
+            maintenance_work_mem: {{ postgresql.maintenance_work_mem }}
+            {% endif %}
+            {% if postgresql.work_mem %}
+            work_mem: {{ postgresql.work_mem }}
+            {% endif %}
+            {% if postgresql.effective_cache_size %}
+            effective_cache_size: {{ postgresql.effective_cache_size }}
+            {% endif %}
+        annotations:
+            "gui-x": "600"
+            "gui-y": "120"
+    haproxy:
+        charm: cs:trusty/haproxy-19
+        {% if haproxy["to"] %}
+        to:
+            - {{ haproxy["to"] }}
+        {% endif %}
+        expose: True
+        num_units: 1
+        options:
+            enable_monitoring: True
+            monitoring_allowed_cidr: "0.0.0.0/0"
+            monitoring_password: "haproxy"
+            default_timeouts: "queue 60000, connect 5000, client 120000, server 120000"
+            # Don't deploy default haproxy service on port 80
+            services: ""
+            source: backports
+            ssl_cert: SELFSIGNED
+        annotations:
+            "gui-x": "1200"
+            "gui-y": "120"
+    landscape-server:
+        {% if landscape["branch"] %}
+        branch: {{ landscape["branch"] }}
+        {% else %}
+        charm: {{ landscape["charm"] }}
+        {% endif %}
+        {% if landscape["memory"] %}
+        constraints: mem={{ landscape["memory"] }}
+        {% endif %}
+        {% if landscape["to"] %}
+        to:
+            - {{ landscape["to"] }}
+        {% endif %}
+        num_units: 1
+        options:
+            source: {{ landscape["source"] }}
+            key: {{ landscape["key"] }}
+            {% if landscape["ssl-cert"] %}
+            ssl-cert: {{ landscape["ssl-cert"] }}
+            ssl-key: {{ landscape["ssl-key"] }}
+            {% endif %}
+        annotations:
+            "gui-x": "950"
+            "gui-y": "120"
+{% if landscape["to"] %}
+machines:
+    "0": {}
+{% endif %}
+relations:
+    - [landscape-server, rabbitmq-server]
+    - [landscape-server, haproxy]
+    - ["landscape-server:db", "postgresql:db-admin"]

=== added file 'render-bundles'
--- render-bundles	1970-01-01 00:00:00 +0000
+++ render-bundles	2016-06-07 14:03:21 +0000
@@ -0,0 +1,80 @@
+#!/usr/bin/python3
+"""Render the landscape-server bundles."""
+
+import argparse
+import os
+import shutil
+
+from jinja2 import FileSystemLoader, Environment
+
+DEFAULTS = {
+    "rabbitmq": {},
+    "postgresql": {
+        "max_connections": 500,
+        "max_prepared_transactions": 500,
+        "memory": 2048},
+    "haproxy": {},
+    "landscape": {
+        "charm": "cs:~davidpbritton/xenial/landscape-server",
+        "memory": 2048,
+        "source":
+            "deb http://ppa.launchpad.net/landscape/16.03/ubuntu trusty main",
+        "key": "4652B4E6"}
+}
+
+FLAVORS = [
+    # scalable
+    {"name": "scalable"},
+
+    # dense
+    {"name": "dense",
+     "rabbitmq": {
+         "to": "lxc:0"},
+     "postgresql": {
+         "to": "lxc:0"},
+     "haproxy": {
+         "to": "0"},
+     "landscape": {
+         "to": "lxc:0"}},
+
+    # dense-maas
+    {"name": "dense-maas",
+     "rabbitmq": {
+         "to": "lxc:0"},
+     "postgresql": {
+         "to": "lxc:0"},
+     "haproxy": {
+         "to": "lxc:0"},
+     "landscape": {
+         "to": "lxc:0"}},
+]
+
+if __name__ == "__main__":
+    current_dir = os.path.dirname(__file__)
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument("--landscape-branch", type=str, required=False)
+    parser.add_argument("--landscape-charm", type=str, required=False)
+    args = parser.parse_args()
+    environment = Environment(
+        loader=FileSystemLoader(current_dir),
+        trim_blocks=True, lstrip_blocks=True,
+        keep_trailing_newline=True)
+    template = environment.get_template("landscape-template.jinja2")
+    for flavor in FLAVORS:
+        build_dir = os.path.join(current_dir, "build")
+        path_parts = [build_dir, "landscape-%s" % flavor["name"]]
+        os.makedirs(os.path.join(*path_parts))
+        shutil.copy("README.md", os.path.join(*path_parts))
+
+        path_parts.append("bundle.yaml")
+        with open(os.path.join(*path_parts), "w") as fd:
+            context = DEFAULTS.copy()
+            for key in context.keys():
+                context[key].update(flavor.get(key, {}))
+            context["name"] = flavor["name"]
+            if args.landscape_charm:
+                context["landscape"]["charm"] = args.landscape_charm
+            if args.landscape_branch:
+                context["landscape"]["branch"] = args.landscape_branch
+            fd.write(template.render(context))
+    print("Generated bundles in {}/...".format(build_dir))

=== added file 'ubuntu-deps'
--- ubuntu-deps	1970-01-01 00:00:00 +0000
+++ ubuntu-deps	2016-06-07 14:03:21 +0000
@@ -0,0 +1,53 @@
+#!/bin/bash -u
+
+set -u
+
+INTERACTIVE=0
+tty -s && INTERACTIVE=1
+
+function apt_get() {
+    local skip=0
+
+    # first check if this is necessary - we don't want to ask for sudo
+    # unless we need it, since this is run from make build-devel
+    output=$(apt-get -s $*)
+    if [ $? -ne 0 ]; then
+        echo "Error: Dependency simulation failed: $*"
+        exit 1
+    fi
+
+    # Check for any lines that start with Inst, which signifies
+    # a package will be installed or upgraded
+    echo "$output" | grep '^Inst ' > /dev/null || skip=1
+
+    if [ $skip -eq 1 ]; then
+        echo " * Skipping: $*"
+        return
+    fi
+
+    if [ $INTERACTIVE -eq 0 ]; then
+        DEBIAN_FRONTEND=noninteractive sudo apt-get \
+            -y --force-yes -o Dpkg::Options::='--force-confold' $*
+    else
+        sudo apt-get $*
+    fi
+
+    if [ $? -ne 0 ]; then
+        echo "Error: Package installation failed: $*"
+        exit 1
+    fi
+}
+
+
+function install_deps {
+    apt_get install \
+        python3-jinja2 \
+        jq
+}
+
+
+# If running non-interactively, go ahead and apt-get update
+# before we start, keeps jenkins slaves and the like updated
+[ $INTERACTIVE -eq 0 ] && sudo apt-get update
+
+install_deps

=== added file 'update-charm-revisions'
--- update-charm-revisions	1970-01-01 00:00:00 +0000
+++ update-charm-revisions	2016-06-07 14:03:21 +0000
@@ -0,0 +1,98 @@
+#!/usr/bin/python3
+
+import argparse
+import fileinput
+import re
+import sys
+from urllib.request import urlopen
+from yaml import load
+from glob import glob
+
+
+class Charm(object):
+
+    def __init__(self, charm_name, series):
+        """Create a charm object.
+        @param charm_name: name of charm
+        """
+        reference = "cs:{}/{}".format(series, charm_name)
+        base_url = "https://store.juju.ubuntu.com/charm-info?charms="
+        url = "{}{}".format(base_url, reference)
+        response = urlopen(url)
+        self.charm_data = load(response)[reference]
+        self.name = charm_name
+        if "errors" in self.charm_data:
+            raise IOError("{}: {}".format(
+                charm_name, self.charm_data["errors"]))
+        self.revision = self.charm_data["revision"]
+        self.store_url = "cs:{}/{}-{}".format(series, self.name, self.revision)
+
+
+def _get_charm(charm_name, series):
+    """
+    Small wrapper to try to get the series charm you specify.  Falls back
+    to precise if you specified something else
+    """
+    try:
+        return Charm(charm_name, series)
+    except IOError as e:
+        if series != "precise":
+            try:
+                return Charm(charm_name, series="precise")
+            except IOError as precise_e:
+                raise IOError(
+                    "%s: not found? (even tried precise): {}, {}".format(
+                        charm_name, e, precise_e))
+        raise
+
+
+def replace_in_config(filename, charms, series):
+    """
+    Iterate over each charm, replacing all occurences of old charm store
+    urls with new versions.  Intent is to leave file in a state that can
+    be tested and checked in if successful.
+
+    Look for lines like:
+      charm: cs:trusty/juju-gui-83
+
+    Or:
+      series: trusty
+    """
+    for line in fileinput.input(filename, inplace=1):
+        pattern = "^(\s*)series:\s.*$"
+        line = re.sub(pattern, r"\1series: %s" % series, line)
+        for charm in charms:
+            pattern = "cs:[^/]+/{}-[0-9]+".format(charm.name)
+            line = re.sub(pattern, charm.store_url, line)
+        sys.stdout.write(line)
+
+    print("# Config written, use `bzr diff %s` to see any changes" % filename)
+
+
+def get_options():
+    """Parse and return command line options."""
+    description = "Put new charmstore descriptors into a deployer file."
+    parser = argparse.ArgumentParser(description=description)
+    parser.add_argument("charms", help="List charms to update", nargs='*')
+    parser.add_argument("--series", help="Charm series", default="xenial")
+    return parser.parse_args()
+
+
+def main():
+    options = get_options()
+    charms = []
+    if len(options.charms):
+        print("# Latest upstream versions of charms ({}):".format(
+            options.series))
+    for charm_name in options.charms:
+        charm = _get_charm(charm_name, series=options.series)
+        print("#   {}: {}".format(charm_name, charm.revision))
+        charms.append(charm)
+
+    for filename in glob("*.jinja2"):
+        replace_in_config(filename, charms, series=options.series)
+
+    sys.exit(0)
+
+if __name__ == "__main__":
+    main()