Merge lp:~ahasenack/landscape-charm/new-bundle-render into lp:~landscape/landscape-charm/bundles-stable

[Andreas Hasenack]

Sync bundle generation with what we have in bundles-trunk so that both produce the same set of files and directories. The content is still the same as it was before this branch.

Sync bundle generation with what we have in bundles-trunk so that both produce the same set of files and directories. The bundle content is still the same as it was before this branch.

I also added a small fix to the diff script so that it ignores the bundles.yaml (note the plural) file which is not used and is just an artifact of the charm store.

The diff is a bit big, here is what I did:
- merged with bundles-trunk
- removed the individual non-template yaml files representing each bundle: there are now generated and not part of the branch
- added --exclude bundles.yaml to the diff action in the charm-store diff

Everything else is a result of the merge from bundles-trunk.

To test:
charm login
charm whoami (to verify login). You may have to login on jujucharms.com with a browser before for all this to work (known bug in the charm store)

export CHARM_STORE_USER=landscape
make diff-charm-store

Should show no diff, because the rendered bundles match what is in the charm store currently. The rendered bundles are in the build/ directory.

Another test you can do:
make update-charm-revisions
bzr diff landscape-template.jinja2

Should show charm revision updates, still for the trusty series.

Bump to xenial:
bzr revert (if you ran the test above)
export EXTRA_UPDATE_ARGUMENTS="--series xenial"
make update-charm-revisions
bzr diff landscape-template.jinja2

That should bump all bundles to xenial and latest charm revisions

I just didn't test the actual push and publish commands (makefile targets).

[lp:~ahasenack/landscape-charm/new-bundle-render]

19. By Andreas Hasenack on 2016-07-22

Keep trusty as the default

20. By Andreas Hasenack on 2016-07-22

Added the charm package to the ubuntu dependencies to install.

[🤖 Landscape Builder (landscape-builder)]

review: Abstain (executing tests)

[🤖 Landscape Builder (landscape-builder)]

Command: make render
Result: Success
Revno: 20
Branch: lp:~ahasenack/landscape-charm/new-bundle-render
Jenkins: https://ci.lscape.net/job/latch-test/5506/

review: Approve (test results)

[lp:~ahasenack/landscape-charm/new-bundle-render]

21. By Andreas Hasenack on 2016-07-22

 - add the juju stable ppa if on trusty, because the charm tool isn't available
   on trusty proper.
 - run apt-get update if we decided we need to install something.

[🤖 Landscape Builder (landscape-builder)]

review: Abstain (executing tests)

[🤖 Landscape Builder (landscape-builder)]

Command: make render
Result: Success
Revno: 21
Branch: lp:~ahasenack/landscape-charm/new-bundle-render
Jenkins: https://ci.lscape.net/job/latch-test/5507/

review: Approve (test results)

[lp:~ahasenack/landscape-charm/new-bundle-render]

22. By Andreas Hasenack on 2016-07-22

Added python3-yaml to ubuntu-deps.

23. By Andreas Hasenack on 2016-07-22

Depend on deps in update-charm-revisions makefile target.

24. By Andreas Hasenack on 2016-07-22

We don't have apache2, juju-gui or nfs in our bundle anymore, so remove them from the update list.

[Francis Ginther (fginther)]

Please remove the push-charm-store and publish-charm-store makefile targets as the don't appear to be needed hear and removes the risk of someone accidentally using them to publish a test version of our bundles.

Otherwise looks good.

review: Approve

[Francis Ginther (fginther)]

Ooops, wrong state.

review: Needs Fixing

[🤖 Landscape Builder (landscape-builder)]

review: Abstain (executing tests)

[🤖 Landscape Builder (landscape-builder)]

Command: make render
Result: Success
Revno: 24
Branch: lp:~ahasenack/landscape-charm/new-bundle-render
Jenkins: https://ci.lscape.net/job/latch-test/5509/

review: Approve (test results)

[Ursula Junque (ursinha)]

After all dependencies were fixed it's working fine. It's just odd that when I bump to xenial it changes haproxy charm to a precise one: https://pastebin.canonical.com/161523/. If that's expected and you address fginther's comments then +1.

review: Approve

[lp:~ahasenack/landscape-charm/new-bundle-render]

25. By Andreas Hasenack on 2016-07-22

Fallback to trusty charms, not precise.

[🤖 Landscape Builder (landscape-builder)]

review: Abstain (executing tests)

[🤖 Landscape Builder (landscape-builder)]

Command: make render
Result: Success
Revno: 25
Branch: lp:~ahasenack/landscape-charm/new-bundle-render
Jenkins: https://ci.lscape.net/job/latch-test/5510/

review: Approve (test results)

[Francis Ginther (fginther)]

Disregard my comment. Approve.

review: Approve

=== added file '.bzrignore'
--- .bzrignore	1970-01-01 00:00:00 +0000
+++ .bzrignore	2016-07-22 15:12:19 +0000
@@ -0,0 +1,1 @@
+build

=== modified file 'Makefile'
--- Makefile	2016-02-09 18:14:40 +0000
+++ Makefile	2016-07-22 15:12:19 +0000
@@ -1,18 +1,29 @@
-compare-charm-store:
-	./charm-store
-
-commit-charm-store:
-	CHARM_STORE_COMMIT=1 ./charm-store
+diff-charm-store: check-env render
+	./charm-store diff $${CHARM_STORE_USER}
+
+push-charm-store: check-env render
+	./charm-store push $${CHARM_STORE_USER}
+
+publish-charm-store: check-env render
+	./charm-store publish-latest $${CHARM_STORE_USER}
+
+check-env:
+ifndef CHARM_STORE_USER
+	$(error CHARM_STORE_USER is undefined, hint: launchpad id.)
+endif
 
 clean:
 	@rm -rf build
 
-render:
+render: deps clean
 	@./render-bundles
 	
-update-charm-revisions:
+update-charm-revisions: deps
 	@./update-charm-revisions \
 		$(EXTRA_UPDATE_ARGUMENTS) \
-		apache2 postgresql juju-gui haproxy rabbitmq-server nfs
-
-.PHONY: clean render commit-charm-store update-charm-revisions compare-charm-store
+		postgresql haproxy rabbitmq-server
+
+deps:
+	@./ubuntu-deps
+
+.PHONY: clean render commit-charm-store update-charm-revisions compare-charm-store deps

=== modified file 'charm-store'
--- charm-store	2016-04-07 21:39:46 +0000
+++ charm-store	2016-07-22 15:12:19 +0000
@@ -1,23 +1,70 @@
 #!/bin/bash -e
-
-COMMIT=${CHARM_STORE_COMMIT:-}
-
-for i in *.yaml; do
-    revno=$(bzr revno)
-    target=$(basename $i .yaml)
-    if [ -e build/$target ]; then
-        echo "build/$target exists, please remove before continuing (make clean)"
-        exit 1
-    fi
-    mkdir -p build
-    bzr co lp:~landscape/charms/bundles/$target/bundle build/$target
-    cp $target.yaml build/$target/bundle.yaml
-    cp README.md build/$target/README.md
-    if [ "$COMMIT" == "" ]; then
-        echo "Diff from Charm Store: $target"
-        (cd build/$target && bzr diff) || /bin/true
-    else
-        echo "Committing to Charm Store: $target"
-        (cd build/$target && bzr commit -m "$COMMIT (stable: r$revno)")
-    fi
+#
+# Mostly intended to be called with make targets
+# Needs make clean, make render called first.
+#
+
+ACTION=$1
+STORE_USER=${2:-landscape}
+
+if [ -z "$ACTION" ]; then
+    cat <<EOF
+Usage: $0 ACTION [user]
+   actions:
+        push - push to 'unpublished' channel
+        publish-latest - publish the latest copy
+        diff - diff your branch to the latest stable channel copy
+   user:
+        charm store user name
+EOF
+    exit 1
+fi
+
+if [ "$ACTION" == "push" ]; then
+    echo "# Will be pushing new [unpublished] versions to the charm store"
+    echo "#   as user: $STORE_USER"
+fi
+
+for target in landscape-dense-maas landscape-dense landscape-scalable; do
+
+    if [ "$ACTION" == "diff" ]; then
+        echo "# [$target] Pulling fresh copy from stable channel"
+        charm pull --channel stable "cs:~${STORE_USER}/bundle/${target}" build/${target}-stable
+
+        echo "# [$target] Diff from latest stable"
+        diff -Naur --exclude bundles.yaml build/${target}-stable build/${target} || /bin/true
+    fi
+
+    if [ "$ACTION" == "push" ]; then
+        echo "# [$target] Pushing to namespace ~${STORE_USER}, unpublished channel"
+        url=$(charm push build/${target} cs:~${STORE_USER}/bundle/${target} | grep ^url | awk '{ print $2 }')
+
+        echo "# [$target] Publishing to development channel"
+        charm publish --channel development $url
+    fi
+
+    if [ "$ACTION" == "publish-latest" ]; then
+        latest=$(charm show --channel unpublished --format=json cs:~${STORE_USER}/bundle/${target} | jq -r '.["revision-info"].Revisions[0]')
+        echo "# [$target] Publishing latest unpublished revision, stable channel"
+        charm publish $latest
+    fi
+
 done
+
+cat <<EOF
+#
+# You can manually examine things if you wish:"
+#   ll build/*"
+#   ll build/*-stable"
+#
+# Show details in the store:
+#   charm show [--channel unpublished] cs:~${STORE_USER}/bundle/landscape-dense-maas
+#
+# Make sure everyone can use your charm if you want:
+#   charm grant cs:~${STORE_USER}/bundle/BUNDLE_NAME-VERSION everyone
+#
+# Publish like this (controls what clients see by default, you can
+# even set it to a previous known good version): 
+#   charm publish cs:~${STORE_USER}/bundle/BUNDLE_NAME-VERSION
+#
+EOF

=== removed file 'landscape-dense-maas.yaml'
--- landscape-dense-maas.yaml	2016-07-21 19:53:48 +0000
+++ landscape-dense-maas.yaml	1970-01-01 00:00:00 +0000
@@ -1,59 +0,0 @@
-series: trusty
-services:
-    rabbitmq-server:
-        charm: cs:trusty/rabbitmq-server-43
-        to:
-            - lxc:0
-        num_units: 1
-        annotations:
-            "gui-x": "600"
-            "gui-y": "370"
-    postgresql:
-        charm: cs:trusty/postgresql-40
-        constraints: mem=2048
-        to:
-            - lxc:0
-        num_units: 1
-        options:
-            extra_packages: python-apt postgresql-contrib postgresql-.*-debversion postgresql-plpython-.*
-            max_connections: 500
-            max_prepared_transactions: 500
-        annotations:
-            "gui-x": "600"
-            "gui-y": "120"
-    haproxy:
-        charm: cs:trusty/haproxy-16
-        to:
-            - lxc:0
-        expose: True
-        num_units: 1
-        options:
-            enable_monitoring: True
-            monitoring_allowed_cidr: "0.0.0.0/0"
-            monitoring_password: "haproxy"
-            default_timeouts: "queue 60000, connect 5000, client 120000, server 120000"
-            # Don't deploy default haproxy service on port 80
-            services: ""
-            source: backports
-            ssl_cert: SELFSIGNED
-        annotations:
-            "gui-x": "1200"
-            "gui-y": "120"
-    landscape-server:
-        charm: cs:trusty/landscape-server
-        constraints: mem=2048
-        to:
-            - lxc:0
-        num_units: 1
-        options:
-            source: deb http://ppa.launchpad.net/landscape/16.06/ubuntu trusty main
-            key: 4652B4E6
-        annotations:
-            "gui-x": "950"
-            "gui-y": "120"
-machines:
-    "0": {}
-relations:
-    - [landscape-server, rabbitmq-server]
-    - [landscape-server, haproxy]
-    - ["landscape-server:db", "postgresql:db-admin"]

=== removed file 'landscape-dense.yaml'
--- landscape-dense.yaml	2016-07-21 19:53:48 +0000
+++ landscape-dense.yaml	1970-01-01 00:00:00 +0000
@@ -1,59 +0,0 @@
-series: trusty
-services:
-    rabbitmq-server:
-        charm: cs:trusty/rabbitmq-server-43
-        to:
-            - lxc:0
-        num_units: 1
-        annotations:
-            "gui-x": "600"
-            "gui-y": "370"
-    postgresql:
-        charm: cs:trusty/postgresql-40
-        constraints: mem=2048
-        to:
-            - lxc:0
-        num_units: 1
-        options:
-            extra_packages: python-apt postgresql-contrib postgresql-.*-debversion postgresql-plpython-.*
-            max_connections: 500
-            max_prepared_transactions: 500
-        annotations:
-            "gui-x": "600"
-            "gui-y": "120"
-    haproxy:
-        charm: cs:trusty/haproxy-16
-        to:
-            - 0
-        expose: True
-        num_units: 1
-        options:
-            enable_monitoring: True
-            monitoring_allowed_cidr: "0.0.0.0/0"
-            monitoring_password: "haproxy"
-            default_timeouts: "queue 60000, connect 5000, client 120000, server 120000"
-            # Don't deploy default haproxy service on port 80
-            services: ""
-            source: backports
-            ssl_cert: SELFSIGNED
-        annotations:
-            "gui-x": "1200"
-            "gui-y": "120"
-    landscape-server:
-        charm: cs:trusty/landscape-server
-        constraints: mem=2048
-        to:
-            - lxc:0
-        num_units: 1
-        options:
-            source: deb http://ppa.launchpad.net/landscape/16.06/ubuntu trusty main
-            key: 4652B4E6
-        annotations:
-            "gui-x": "950"
-            "gui-y": "120"
-machines:
-    "0": {}
-relations:
-    - [landscape-server, rabbitmq-server]
-    - [landscape-server, haproxy]
-    - ["landscape-server:db", "postgresql:db-admin"]

=== removed file 'landscape-scalable.yaml'
--- landscape-scalable.yaml	2016-07-21 19:53:48 +0000
+++ landscape-scalable.yaml	1970-01-01 00:00:00 +0000
@@ -1,49 +0,0 @@
-series: trusty
-services:
-    rabbitmq-server:
-        charm: cs:trusty/rabbitmq-server-43
-        num_units: 1
-        annotations:
-            "gui-x": "600"
-            "gui-y": "370"
-    postgresql:
-        charm: cs:trusty/postgresql-40
-        constraints: mem=2048
-        num_units: 1
-        options:
-            extra_packages: python-apt postgresql-contrib postgresql-.*-debversion postgresql-plpython-.*
-            max_connections: 500
-            max_prepared_transactions: 500
-        annotations:
-            "gui-x": "600"
-            "gui-y": "120"
-    haproxy:
-        charm: cs:trusty/haproxy-16
-        expose: True
-        num_units: 1
-        options:
-            enable_monitoring: True
-            monitoring_allowed_cidr: "0.0.0.0/0"
-            monitoring_password: "haproxy"
-            default_timeouts: "queue 60000, connect 5000, client 120000, server 120000"
-            # Don't deploy default haproxy service on port 80
-            services: ""
-            source: backports
-            ssl_cert: SELFSIGNED
-        annotations:
-            "gui-x": "1200"
-            "gui-y": "120"
-    landscape-server:
-        charm: cs:trusty/landscape-server
-        constraints: mem=2048
-        num_units: 1
-        options:
-            source: deb http://ppa.launchpad.net/landscape/16.06/ubuntu trusty main
-            key: 4652B4E6
-        annotations:
-            "gui-x": "950"
-            "gui-y": "120"
-relations:
-    - [landscape-server, rabbitmq-server]
-    - [landscape-server, haproxy]
-    - ["landscape-server:db", "postgresql:db-admin"]

=== modified file 'render-bundles'
--- render-bundles	2016-07-21 19:53:48 +0000
+++ render-bundles	2016-07-22 15:12:19 +0000
@@ -1,6 +1,10 @@
 #!/usr/bin/python3
 
+"""Render the landscape-server bundles."""
+
+import argparse
 import os
+import shutil
 
 from jinja2 import FileSystemLoader, Environment
 
@@ -45,14 +49,31 @@
 ]
 
 if __name__ == "__main__":
+    current_dir = os.path.dirname(__file__)
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument("--landscape-branch", type=str, required=False)
+    parser.add_argument("--landscape-charm", type=str, required=False)
+    args = parser.parse_args()
     environment = Environment(
-        loader=FileSystemLoader("."), trim_blocks=True, lstrip_blocks=True,
+        loader=FileSystemLoader(current_dir),
+        trim_blocks=True, lstrip_blocks=True,
         keep_trailing_newline=True)
     template = environment.get_template("landscape-template.jinja2")
     for flavor in FLAVORS:
-        with open("landscape-%s.yaml" % flavor["name"], "w") as fd:
+        build_dir = os.path.join(current_dir, "build")
+        path_parts = [build_dir, "landscape-%s" % flavor["name"]]
+        os.makedirs(os.path.join(*path_parts))
+        shutil.copy("README.md", os.path.join(*path_parts))
+
+        path_parts.append("bundle.yaml")
+        with open(os.path.join(*path_parts), "w") as fd:
             context = DEFAULTS.copy()
+            for key in context:
+                context[key].update(flavor.get(key, {}))
             context["name"] = flavor["name"]
-            for key in DEFAULTS.keys():
-                context[key].update(flavor.get(key, {}))
+            if args.landscape_charm:
+                context["landscape"]["charm"] = args.landscape_charm
+            if args.landscape_branch:
+                context["landscape"]["branch"] = args.landscape_branch
             fd.write(template.render(context))
+    print("Generated bundles in {}/...".format(build_dir))

=== added file 'ubuntu-deps'
--- ubuntu-deps	1970-01-01 00:00:00 +0000
+++ ubuntu-deps	2016-07-22 15:12:19 +0000
@@ -0,0 +1,62 @@
+#!/bin/bash -u
+
+set -u
+
+INTERACTIVE=0
+tty -s && INTERACTIVE=1
+
+function apt_get() {
+    local skip=0
+
+    # first check if this is necessary - we don't want to ask for sudo
+    # unless we need it, since this is run from make build-devel
+    output=$(apt-get -s $*)
+    if [ $? -ne 0 ]; then
+        echo "Error: Dependency simulation failed: $*"
+        exit 1
+    fi
+
+    # Check for any lines that start with Inst, which signifies
+    # a package will be installed or upgraded
+    echo "$output" | grep '^Inst ' > /dev/null || skip=1
+
+    if [ $skip -eq 1 ]; then
+        echo " * Skipping: $*"
+        return
+    fi
+    
+    release=$(lsb_release -cs)
+    # needed for the "charm" tool
+    if [ "$release" == "trusty" ]; then
+        sudo add-apt-repository -y ppa:juju/stable
+    fi
+    sudo apt-get update
+
+    if [ $INTERACTIVE -eq 0 ]; then
+        DEBIAN_FRONTEND=noninteractive sudo apt-get \
+            -y --force-yes -o Dpkg::Options::='--force-confold' $*
+    else
+        sudo apt-get $*
+    fi
+
+    if [ $? -ne 0 ]; then
+        echo "Error: Package installation failed: $*"
+        exit 1
+    fi
+}
+
+
+function install_deps {
+    apt_get install \
+        python3-jinja2 \
+        python3-yaml \
+        jq \
+        charm
+}
+
+
+# If running non-interactively, go ahead and apt-get update
+# before we start, keeps jenkins slaves and the like updated
+[ $INTERACTIVE -eq 0 ] && sudo apt-get update
+
+install_deps

=== modified file 'update-charm-revisions'
--- update-charm-revisions	2016-02-09 18:13:49 +0000
+++ update-charm-revisions	2016-07-22 15:12:19 +0000
@@ -36,13 +36,13 @@
     try:
         return Charm(charm_name, series)
     except IOError as e:
-        if series != "precise":
+        if series != "trusty":
             try:
-                return Charm(charm_name, series="precise")
-            except IOError as precise_e:
+                return Charm(charm_name, series="trusty")
+            except IOError as trusty_e:
                 raise IOError(
-                    "%s: not found? (even tried precise): {}, {}".format(
-                        charm_name, e, precise_e))
+                    "%s: not found? (even tried trusty): {}, {}".format(
+                        charm_name, e, trusty_e))
         raise