Merge lp:~cjwatson/launchpad-buildd/charm into lp:launchpad-buildd

[Colin Watson]

Add a Juju charm which can be used to deploy local launchpad-buildd instances.

[Colin Watson (cjwatson)]

Superseded by https://code.launchpad.net/~cjwatson/launchpad-buildd/+git/launchpad-buildd/+merge/383062.

Unmerged revisions

213. By Colin Watson on 2016-12-05

Split out push and push-with-packages targets, since we don't need to push (dummy) resources every time.

212. By Colin Watson on 2016-12-05

Improve handling of attaching resources after initial deployment.

211. By Colin Watson on 2016-12-05

Add "push" target.

210. By Colin Watson on 2016-12-05

Unhold packages before installing them.

209. By Colin Watson on 2016-12-05

Set ntphost to the empty string rather than removing it entirely, to avoid confusing the postinst.

208. By Colin Watson on 2016-12-05

Treat zero-sized resources as missing, since the charm store doesn't support truly-optional resources.

207. By Colin Watson on 2016-12-05

Avoid formatting glitch.

206. By Colin Watson on 2016-12-05

Add a Juju charm which can be used to deploy local launchpad-buildd
instances.

=== modified file '.bzrignore'
--- .bzrignore	2015-11-04 14:10:28 +0000
+++ .bzrignore	2016-12-05 14:48:05 +0000
@@ -1,5 +1,7 @@
 *.egg-info
 *.pyc
+charm/dist
+charm/tmp
 dist
 debian/files
 debian/launchpad-buildd

=== added directory 'charm'
=== added file 'charm/Makefile'
--- charm/Makefile	1970-01-01 00:00:00 +0000
+++ charm/Makefile	2016-12-05 14:48:05 +0000
@@ -0,0 +1,77 @@
+NAME ?= launchpad-buildd
+TOPDIR ?= $(CURDIR)/..
+BUILDDIR ?= $(CURDIR)/dist
+TMPDIR ?= $(CURDIR)/tmp
+
+CHARM_SERIES ?= xenial
+CHARM_SRC ?= $(CURDIR)
+JUJU_REPOSITORY = $(BUILDDIR)
+CHARMDIR = $(BUILDDIR)/$(CHARM_SERIES)/$(NAME)
+CHARMREPODIR = $(BUILDDIR)/build
+RESOURCES = $(BUILDDIR)/resources/$(NAME)
+CHARM = $(CHARMDIR)/.done
+LAYER_PATH = $(TMPDIR)/layer
+INTERFACE_PATH = $(TMPDIR)/interface
+EXTRA_CHARM_BUILD_ARGS ?=
+DEPLOY_ENV ?= devel
+CHARM_STORE_URL ?= cs:~launchpad/launchpad-buildd
+
+export INTERFACE_PATH
+export LAYER_PATH
+export JUJU_REPOSITORY
+
+
+build: $(CHARM)
+
+packages: $(RESOURCES)
+
+build-with-packages: build packages
+
+$(BUILDDIR):
+	mkdir -p $@
+
+$(CHARM): $(CHARM_SRC) | $(BUILDDIR)
+	rm -rf $(CHARMDIR)
+	charm build -o $(BUILDDIR) -s $(CHARM_SERIES) -n $(NAME) $(EXTRA_CHARM_BUILD_ARGS)
+	touch $(CHARMDIR)/dummy-launchpad-buildd.deb
+	touch $(CHARMDIR)/dummy-python-lpbuildd.deb
+	touch $@
+
+clean:
+	rm -rf $(BUILDDIR) $(TMPDIR)
+
+$(RESOURCES):
+	rm -rf $(TMPDIR)
+	mkdir -p $(TMPDIR)
+	rsync -a --exclude /charm $(TOPDIR)/ $(TMPDIR)/$(NAME)/
+	cd $(TMPDIR)/$(NAME)/ && debuild -b -uc -us
+	mkdir -p $(RESOURCES)
+	cp -a $(TMPDIR)/launchpad-buildd_*.deb \
+	      $(RESOURCES)/launchpad-buildd.deb
+	cp -a $(TMPDIR)/python-lpbuildd_*.deb $(RESOURCES)/python-lpbuildd.deb
+
+create-privileged-model:
+	juju add-model privileged localhost
+	lxc profile set juju-privileged security.privileged true
+
+deploy:
+	juju deploy $(CHARMDIR)
+
+deploy-with-packages:
+	juju deploy \
+		--resource=launchpad-buildd=$(RESOURCES)/launchpad-buildd.deb \
+		--resource=python-lpbuildd=$(RESOURCES)/python-lpbuildd.deb \
+		$(CHARMDIR)
+
+push:
+	charm push $(CHARMDIR) $(CHARM_STORE_URL)
+
+push-with-packages:
+	charm push \
+		--resource=launchpad-buildd=$(CHARMDIR)/dummy-launchpad-buildd.deb \
+		--resource=python-lpbuildd=$(CHARMDIR)/dummy-python-lpbuildd.deb \
+		$(CHARMDIR) $(CHARM_STORE_URL)
+
+.PHONY: build packages build-with-packages clean
+.PHONY: create-privileged-model deploy deploy-with-packages
+.PHONY: push push-with-packages

=== added file 'charm/README.md'
--- charm/README.md	1970-01-01 00:00:00 +0000
+++ charm/README.md	2016-12-05 14:48:05 +0000
@@ -0,0 +1,59 @@
+# Overview
+
+This charm installs a Launchpad builder, which can build packages in
+response to requests from a Launchpad instance.  It is mainly intended for
+use by Launchpad developers testing changes to builder handling.
+
+# Setup
+
+Builders need to be able to unpack chroots, which involves being able to
+create device nodes.  Unprivileged LXD containers cannot do this.  If you
+want to use this with the LXD provider, you should therefore do this first:
+
+```
+make create-privileged-model
+```
+
+... or, if you need more control, some variation on this:
+
+```
+juju add-model privileged localhost
+lxc profile set juju-privileged security.privileged true
+```
+
+# Deployment
+
+You can either deploy the stock launchpad-buildd package from a PPA, or
+build your own.
+
+Installing from a PPA is the default; just deploy this charm.
+
+If you're building your own package, then you already have the
+launchpad-buildd code checked out.  In the `charm/` subdirectory, build the
+charm and packages together:
+
+```
+make build-with-packages
+```
+
+Then deploy the charm, attaching the packages as resources:
+
+```
+make deploy-with-packages
+```
+
+Either way, this should eventually give you a running builder.  Find out its
+host name (e.g. `juju-XXXXXX-0.lxd`) and [add it to your local Launchpad
+instance](https://launchpad.dev/builders/+new) (e.g.
+`http://juju-XXXXXX-0.lxd:8221/`).
+
+# Notes
+
+This charm gives you a non-virtualized builder, since there is no reset from
+a base image between builds; you'll need to make sure that any archives or
+snaps with builds you intend to dispatch to this builder have the "Require
+virtualized builders" option disabled.
+
+The Launchpad development wiki has [instructions on setting up the rest of
+Launchpad](https://dev.launchpad.net/Soyuz/HowToUseSoyuzLocally).
+You can skip the parts about installing the builder.

=== added file 'charm/config.yaml'
--- charm/config.yaml	1970-01-01 00:00:00 +0000
+++ charm/config.yaml	2016-12-05 14:48:05 +0000
@@ -0,0 +1,3 @@
+options:
+    install_sources:
+        default: ppa:launchpad/buildd-staging

=== added file 'charm/layer.yaml'
--- charm/layer.yaml	1970-01-01 00:00:00 +0000
+++ charm/layer.yaml	2016-12-05 14:48:05 +0000
@@ -0,0 +1,13 @@
+repo: lp:launchpad-buildd
+includes:
+    - layer:basic
+    - layer:apt
+options:
+    apt:
+        packages:
+            - bzr-builder
+            - git-build-recipe
+            - quilt
+ignore:
+    - dist
+    - tmp

=== added file 'charm/metadata.yaml'
--- charm/metadata.yaml	1970-01-01 00:00:00 +0000
+++ charm/metadata.yaml	2016-12-05 14:48:05 +0000
@@ -0,0 +1,21 @@
+name: launchpad-buildd
+summary: Launchpad builder
+description: |
+    A system that can build packages in response to requests from a
+    Launchpad instance.
+tags:
+    - application_development
+maintainer: Colin Watson <[e-mail]>
+subordinate: false
+series:
+    - xenial
+resources:
+    launchpad-buildd:
+        type: file
+        filename: launchpad-buildd.deb
+        description: The main Launchpad builder package.
+    python-lpbuildd:
+        type: file
+        filename: python-lpbuildd.deb
+        description: Python libraries supporting the Launchpad builder.
+min-juju-version: 2.0.0

=== added directory 'charm/reactive'
=== added file 'charm/reactive/launchpad-buildd.py'
--- charm/reactive/launchpad-buildd.py	1970-01-01 00:00:00 +0000
+++ charm/reactive/launchpad-buildd.py	2016-12-05 14:48:05 +0000
@@ -0,0 +1,96 @@
+# Copyright 2016 Canonical Ltd.  This software is licensed under the
+# GNU Affero General Public License version 3 (see the file LICENSE).
+
+from __future__ import print_function
+
+import os.path
+import re
+import shutil
+import subprocess
+
+from charmhelpers import fetch
+from charmhelpers.core import (
+    hookenv,
+    host,
+    )
+from charms.apt import status_set
+from charms.reactive import (
+    hook,
+    only_once,
+    remove_state,
+    set_state,
+    when,
+    when_not,
+    )
+
+
+@only_once
+def install():
+    with open("/etc/default/launchpad-buildd", "w") as default_file:
+        print("RUN_NETWORK_REQUESTS_AS_ROOT=yes", file=default_file)
+    set_state("launchpad-buildd.needs_install")
+
+
+@hook("upgrade-charm", "config-changed")
+def mark_needs_install():
+    set_state("launchpad-buildd.needs_install")
+
+
+@when_not("apt.needs_update")
+@when("launchpad-buildd.needs_install")
+def install_packages():
+    cache_dir = os.path.join(hookenv.charm_dir(), "cache")
+    host.mkdir(cache_dir, perms=0o755)
+    to_install = []
+    packages = ["launchpad-buildd", "python-lpbuildd"]
+    options = ["--option=Dpkg::Options::=--force-confold"]
+    resource_paths = [hookenv.resource_get(package) for package in packages]
+    if all(path and os.path.getsize(path) for path in resource_paths):
+        # Install from resources.
+        changed = False
+        for package, resource_path in zip(packages, resource_paths):
+            local_path = os.path.join(cache_dir, "{}.deb".format(package))
+            to_install.append((local_path, resource_path))
+            if host.file_hash(local_path) != host.file_hash(resource_path):
+                changed = True
+        if not changed:
+            return
+        options.append("--reinstall")
+    else:
+        # We don't have resource-provided packages, so just install from the
+        # PPA.
+        to_install.extend([(None, package) for package in packages])
+    new_paths = [new_path for _, new_path in to_install]
+    try:
+        status_set(None, "Installing {}".format(",".join(packages)))
+        fetch.apt_unhold(packages)
+        fetch.apt_install(new_paths, options=options)
+        fetch.apt_hold(packages)
+    except subprocess.CalledProcessError:
+        status_set(
+            "blocked",
+            "Unable to install packages {}".format(",".join(packages)))
+    else:
+        for local_path, resource_path in to_install:
+            if local_path is not None:
+                shutil.copy2(resource_path, local_path)
+        # ntp.buildd isn't likely to work outside of the Canonical
+        # datacentre, and LXD containers can't set the system time.  Let's
+        # just not worry about NTP.
+        config_path = "/etc/launchpad-buildd/default"
+        with open(config_path) as config_file:
+            config = config_file.read()
+        config = re.sub(r"^ntphost = .*", "ntphost = ", config, flags=re.M)
+        with open(config_path + ".new", "w") as new_config_file:
+            new_config_file.write(config)
+        os.rename(config_path + ".new", config_path)
+        remove_state("launchpad-buildd.needs_install")
+        set_state("launchpad-buildd.installed")
+
+
+@when("apt.installed.bzr-builder")
+@when("apt.installed.git-build-recipe")
+@when("apt.installed.quilt")
+@when("launchpad-buildd.installed")
+def mark_active():
+    status_set("active", "Builder running")

=== modified file 'debian/changelog'
--- debian/changelog	2016-11-30 10:20:32 +0000
+++ debian/changelog	2016-12-05 14:48:05 +0000
@@ -2,6 +2,8 @@
 
   * buildsnap: Grant access to the proxy during the build phase as well as
     during the pull phase (LP: #1642281).
+  * Add a Juju charm which can be used to deploy local launchpad-buildd
+    instances.
 
  -- Colin Watson <[e-mail]>  Fri, 04 Nov 2016 10:47:41 +0000