Merge ~cjwatson/launchpad:charm-buildd-manager into launchpad:master

[Colin Watson]

charm: Add a launchpad-buildd-manager charm

I'm sure this needs some more work, but it should be a reasonable starting point.

b60d785... by Colin Watson on 2023-07-03

Add a basic README

c96fdd5... by Colin Watson on 2023-07-03

Add start-services/stop-services actions

In our legacy production deployment, we stop and start these services
around certain kinds of maintenance actions, mainly network maintenance
affecting the build farm. The preferred way to do this sort of thing in
Juju is using actions, so define a couple of suitable actions; once we
deploy this to production, we'll set up an SSH trigger allowing
developers to run any of an allowlist of actions on production.

[Guruprasad (lgp171188)]

LGTM 👍

review: Approve

db5d9bd... by Colin Watson on 2023-07-06

Explain which services are started/stopped

diff --git a/charm/Makefile b/charm/Makefile
index 857f6b2..2106149 100644
--- a/charm/Makefile
+++ b/charm/Makefile
@@ -17,6 +17,7 @@ CHARMS := \
 	launchpad-admin \
 	launchpad-appserver \
 	launchpad-assets \
+	launchpad-buildd-manager \
 	launchpad-debian-importer \
 	launchpad-librarian
 
diff --git a/charm/launchpad-buildd-manager/README.md b/charm/launchpad-buildd-manager/README.md
new file mode 100644
index 0000000..ad0c783
--- /dev/null
+++ b/charm/launchpad-buildd-manager/README.md
@@ -0,0 +1,19 @@
+# Launchpad build farm manager
+
+This charm runs a service that supervises the Launchpad build farm.
+
+You will need the following relations:
+
+    juju relate launchpad-buildd-manager:db postgresql:db
+    juju relate launchpad-buildd-manager rabbitmq-server
+
+## Maintenance actions
+
+To stop the build farm manager (perhaps in preparation for network
+maintenance affecting the build farm), run:
+
+    juju run-action --wait launchpad-buildd-manager/leader stop-services
+
+To start them again once maintenance is complete:
+
+    juju run-action --wait launchpad-buildd-manager/leader start-services
diff --git a/charm/launchpad-buildd-manager/actions.yaml b/charm/launchpad-buildd-manager/actions.yaml
new file mode 100644
index 0000000..7bfe649
--- /dev/null
+++ b/charm/launchpad-buildd-manager/actions.yaml
@@ -0,0 +1,11 @@
+start-services:
+  description: |
+    Start the launchpad-buildd-manager service.  Usually run after
+    maintenance.
+stop-services:
+  description: |
+    Stop the launchpad-buildd-manager service.  Usually run in preparation
+    for maintenance.  (Note that this does not stop services in a way that
+    will persist across a reboot.  It also doesn't disable cron jobs, since
+    those are handled by the cron-control mechanism instead; see
+    lp.services.scripts.base.cronscript_enabled.)
diff --git a/charm/launchpad-buildd-manager/actions/actions.py b/charm/launchpad-buildd-manager/actions/actions.py
new file mode 100755
index 0000000..a27a67e
--- /dev/null
+++ b/charm/launchpad-buildd-manager/actions/actions.py
@@ -0,0 +1,53 @@
+#! /usr/bin/python3
+# Copyright 2023 Canonical Ltd.  This software is licensed under the
+# GNU Affero General Public License version 3 (see the file LICENSE).
+
+import subprocess
+import sys
+import traceback
+from pathlib import Path
+
+sys.path.append("lib")
+
+from charms.layer import basic  # noqa: E402
+
+basic.bootstrap_charm_deps()
+basic.init_config_states()
+
+from charmhelpers.core import hookenv  # noqa: E402
+
+
+def start_services():
+    service = "launchpad-buildd-manager.service"
+    hookenv.log(f"Starting {service}.")
+    subprocess.run(["systemctl", "start", service], check=True)
+    hookenv.action_set({"result": "Services started"})
+
+
+def stop_services():
+    service = "launchpad-buildd-manager.service"
+    hookenv.log(f"Stopping {service}.")
+    subprocess.run(["systemctl", "stop", service], check=True)
+    hookenv.action_set({"result": "Services stopped"})
+
+
+def main(argv):
+    action = Path(argv[0]).name
+    try:
+        if action == "start-services":
+            start_services()
+        elif action == "stop-services":
+            stop_services()
+        else:
+            hookenv.action_fail(f"Action {action} not implemented.")
+    except Exception:
+        hookenv.action_fail("Unhandled exception")
+        tb = traceback.format_exc()
+        hookenv.action_set(dict(traceback=tb))
+        hookenv.log(f"Unhandled exception in action {action}:")
+        for line in tb.splitlines():
+            hookenv.log(line)
+
+
+if __name__ == "__main__":
+    main(sys.argv)
diff --git a/charm/launchpad-buildd-manager/actions/start-services b/charm/launchpad-buildd-manager/actions/start-services
new file mode 120000
index 0000000..405a394
--- /dev/null
+++ b/charm/launchpad-buildd-manager/actions/start-services
@@ -0,0 +1 @@
+actions.py
\ No newline at end of file
diff --git a/charm/launchpad-buildd-manager/actions/stop-services b/charm/launchpad-buildd-manager/actions/stop-services
new file mode 120000
index 0000000..405a394
--- /dev/null
+++ b/charm/launchpad-buildd-manager/actions/stop-services
@@ -0,0 +1 @@
+actions.py
\ No newline at end of file
diff --git a/charm/launchpad-buildd-manager/charmcraft.yaml b/charm/launchpad-buildd-manager/charmcraft.yaml
new file mode 100644
index 0000000..1e26547
--- /dev/null
+++ b/charm/launchpad-buildd-manager/charmcraft.yaml
@@ -0,0 +1,63 @@
+type: charm
+bases:
+  - build-on:
+    - name: ubuntu
+      channel: "20.04"
+      architectures: [amd64]
+    run-on:
+    - name: ubuntu
+      channel: "20.04"
+      architectures: [amd64]
+parts:
+  charm-wheels:
+    source: https://git.launchpad.net/~ubuntuone-hackers/ols-charm-deps/+git/wheels
+    source-commit: "42c89d9c66dbe137139b047fd54aed49b66d1a5e"
+    source-submodules: []
+    source-type: git
+    plugin: dump
+    organize:
+      "*": charm-wheels/
+    prime:
+      - "-charm-wheels"
+  ols-layers:
+    source: https://git.launchpad.net/ols-charm-deps
+    source-commit: "f63ae0386275bf9089b30c8abae252a0ea523633"
+    source-submodules: []
+    source-type: git
+    plugin: dump
+    organize:
+      "*": layers/
+    stage:
+      - layers
+    prime:
+      - "-layers"
+  launchpad-layers:
+    after:
+      - ols-layers
+    source: https://git.launchpad.net/launchpad-layers
+    source-commit: "42a4b4c4f62936b1d050c775e84f7364dfb5efc0"
+    source-submodules: []
+    source-type: git
+    plugin: dump
+    organize:
+      launchpad-base: layers/layer/launchpad-base
+      launchpad-db: layers/layer/launchpad-db
+      launchpad-payload: layers/layer/launchpad-payload
+    stage:
+      - layers
+    prime:
+      - "-layers"
+  launchpad-buildd-manager:
+    after:
+      - charm-wheels
+      - launchpad-layers
+    source: .
+    plugin: reactive
+    build-snaps: [charm]
+    build-packages: [libpq-dev, python3-dev]
+    build-environment:
+      - CHARM_LAYERS_DIR: $CRAFT_STAGE/layers/layer
+      - CHARM_INTERFACES_DIR: $CRAFT_STAGE/layers/interface
+      - PIP_NO_INDEX: "true"
+      - PIP_FIND_LINKS: $CRAFT_STAGE/charm-wheels
+    reactive-charm-build-arguments: [--binary-wheels-from-source]
diff --git a/charm/launchpad-buildd-manager/config.yaml b/charm/launchpad-buildd-manager/config.yaml
new file mode 100644
index 0000000..ea622d3
--- /dev/null
+++ b/charm/launchpad-buildd-manager/config.yaml
@@ -0,0 +1,60 @@
+options:
+  active:
+    type: boolean
+    description: If true, enable jobs that may change the database.
+    default: true
+  artifactory_read_credentials:
+    type: string
+    description: >
+      Credentials for reading from Artifactory repositories (formatted as
+      "user:token").
+    default: ""
+  builder_proxy_auth_api_admin_secret:
+    type: string
+    description: >
+      Admin secret for requesting tokens from the builder proxy service.
+    default: ""
+  builder_proxy_auth_api_admin_username:
+    type: string
+    description: Admin username for the builder proxy service.
+    default: ""
+  builder_proxy_auth_api_endpoint:
+    type: string
+    description: Endpoint for builder proxy authentication service.
+    default: ""
+  builder_proxy_host:
+    type: string
+    description: Builder HTTP proxy host.
+    default: ""
+  builder_proxy_port:
+    type: int
+    description: Builder HTTP proxy port.
+    default: 3128
+  builder_reset_private_ssh_key:
+    type: string
+    description: >
+      Base64-encoded private SSH key, used to request builder resets.
+    default: ""
+  builder_reset_public_ssh_key:
+    type: string
+    description: >
+      Base64-encoded public SSH key, used to request builder resets.
+    default: ""
+  cibuild_config:
+    type: string
+    description: >
+      YAML-encoded dictionary mapping pillars to dictionaries of
+      configuration items to set for CI builds of those pillars.
+    default: ""
+  socket_timeout:
+    type: int
+    description: >
+      The time in seconds that buildd-manager will wait for a reply from
+      non-virtualized builders.
+    default: 40
+  virtualized_socket_timeout:
+    type: int
+    description: >
+      The time in seconds that buildd-manager will wait for a reply from
+      non-virtualized builders.
+    default: 30
diff --git a/charm/launchpad-buildd-manager/layer.yaml b/charm/launchpad-buildd-manager/layer.yaml
new file mode 100644
index 0000000..a9518a1
--- /dev/null
+++ b/charm/launchpad-buildd-manager/layer.yaml
@@ -0,0 +1,12 @@
+includes:
+  - layer:launchpad-db
+repo: https://git.launchpad.net/launchpad
+options:
+  ols-pg:
+    databases:
+      db:
+        name: launchpad_dev
+        roles:
+          - buildd_manager
+          - process_upload
+          - retry_depwait
diff --git a/charm/launchpad-buildd-manager/metadata.yaml b/charm/launchpad-buildd-manager/metadata.yaml
new file mode 100644
index 0000000..0e28e9f
--- /dev/null
+++ b/charm/launchpad-buildd-manager/metadata.yaml
@@ -0,0 +1,16 @@
+name: launchpad-buildd-manager
+display-name: launchpad-buildd-manager
+summary: Launchpad build farm manager
+maintainer: Colin Watson <[e-mail]>
+description: |
+  Launchpad is an open source suite of tools that help people and teams
+  to work together on software projects.
+
+  This charm runs a service that supervises the Launchpad build farm,
+  dispatching jobs to idle builders and collecting results.
+tags:
+  # https://juju.is/docs/charm-metadata#heading--charm-store-fields
+  - network
+series:
+  - focal
+subordinate: false
diff --git a/charm/launchpad-buildd-manager/reactive/launchpad-buildd-manager.py b/charm/launchpad-buildd-manager/reactive/launchpad-buildd-manager.py
new file mode 100644
index 0000000..708a4a5
--- /dev/null
+++ b/charm/launchpad-buildd-manager/reactive/launchpad-buildd-manager.py
@@ -0,0 +1,148 @@
+# Copyright 2023 Canonical Ltd.  This software is licensed under the
+# GNU Affero General Public License version 3 (see the file LICENSE).
+
+import base64
+import os.path
+import subprocess
+
+import yaml
+from charmhelpers.core import hookenv, host, templating
+from charms.launchpad.base import get_service_config
+from charms.launchpad.db import lazr_config_files
+from charms.launchpad.payload import (
+    config_file_path,
+    configure_cron,
+    configure_lazr,
+    home_dir,
+)
+from charms.reactive import helpers, remove_state, set_state, when, when_not
+from ols import base
+
+
+def base64_decode(value):
+    return base64.b64decode(value.encode("ASCII"))
+
+
+def configure_keys(config):
+    ssh_dir = os.path.join(home_dir(), ".ssh")
+    ssh_private_path = os.path.join(ssh_dir, "builder-reset")
+    ssh_public_path = os.path.join(ssh_dir, "builder-reset.pub")
+    if (
+        config["builder_reset_private_ssh_key"]
+        and config["builder_reset_public_ssh_key"]
+    ):
+        hookenv.log("Writing SSH keys.")
+        if not os.path.exists(ssh_dir):
+            host.mkdir(
+                ssh_dir, owner=base.user(), group=base.user(), perms=0o700
+            )
+        host.write_file(
+            ssh_private_path,
+            base64_decode(config["builder_reset_private_ssh_key"]),
+            owner=base.user(),
+            group=base.user(),
+            perms=0o600,
+        )
+        host.write_file(
+            ssh_public_path,
+            base64_decode(config["builder_reset_public_ssh_key"]),
+            owner=base.user(),
+            group=base.user(),
+            perms=0o644,
+        )
+    else:
+        for path in (ssh_private_path, ssh_public_path):
+            if os.path.exists(path):
+                os.unlink(path)
+
+
+def configure_service(config):
+    hookenv.log("Writing systemd service.")
+    templating.render(
+        "launchpad-buildd-manager.service.j2",
+        "/lib/systemd/system/launchpad-buildd-manager.service",
+        config,
+    )
+    subprocess.run(["systemctl", "daemon-reload"], check=True)
+
+
+def configure_logrotate(config):
+    hookenv.log("Writing logrotate configuration.")
+    templating.render(
+        "logrotate.conf.j2",
+        "/etc/logrotate.d/launchpad-buildd-manager",
+        config,
+        perms=0o644,
+    )
+
+
+def config_files():
+    files = []
+    files.extend(lazr_config_files())
+    files.append(
+        config_file_path("launchpad-buildd-manager/launchpad-lazr.conf")
+    )
+    files.append(
+        config_file_path(
+            "launchpad-buildd-manager-secrets-lazr.conf", secret=True
+        )
+    )
+    return files
+
+
+@when("launchpad.db.configured")
+@when_not("service.configured")
+def configure():
+    config = get_service_config()
+    config["buildd_manager_dir"] = os.path.join(
+        base.base_dir(), "buildd-manager"
+    )
+    config["cibuild_config"] = yaml.safe_load(config["cibuild_config"])
+    host.mkdir(
+        config["buildd_manager_dir"],
+        owner=base.user(),
+        group=base.user(),
+        perms=0o755,
+    )
+    configure_lazr(
+        config,
+        "launchpad-buildd-manager-lazr.conf",
+        "launchpad-buildd-manager/launchpad-lazr.conf",
+    )
+    configure_lazr(
+        config,
+        "launchpad-buildd-manager-secrets-lazr.conf",
+        "launchpad-buildd-manager-secrets-lazr.conf",
+        secret=True,
+    )
+    configure_keys(config)
+    configure_service(config)
+    configure_logrotate(config)
+    configure_cron(config, "crontab.j2")
+
+    if config["active"]:
+        if helpers.any_file_changed(
+            [
+                base.version_info_path(),
+                "/lib/systemd/system/launchpad-buildd-manager.service",
+            ]
+            + config_files()
+        ):
+            hookenv.log("Config files changed; restarting buildd-manager")
+            host.service_restart("launchpad-buildd-manager")
+        host.service_resume("launchpad-buildd-manager")
+    else:
+        host.service_pause("launchpad-buildd-manager")
+
+    set_state("service.configured")
+
+
+@when("service.configured")
+@when_not("launchpad.db.configured")
+def deconfigure():
+    remove_state("service.configured")
+
+
+@when("service.configured")
+def check_is_running():
+    hookenv.status_set("active", "Ready")
diff --git a/charm/launchpad-buildd-manager/templates/crontab.j2 b/charm/launchpad-buildd-manager/templates/crontab.j2
new file mode 100644
index 0000000..435cf43
--- /dev/null
+++ b/charm/launchpad-buildd-manager/templates/crontab.j2
@@ -0,0 +1,30 @@
+TZ=UTC
+MAILTO={{ cron_mailto }}
+LPCONFIG=launchpad-buildd-manager
+
+{% if active -%}
+# Automatically retry builds in the "Dependency wait" state if their
+# dependencies can now be satisfied.
+25 * * * * {{ code_dir }}/cronscripts/buildd-retry-depwait.py -q --log-file=DEBUG2:{{ logs_dir }}/buildd-retry-depwait.log
+
+# Process uploaded builds.
+* * * * * {{ code_dir }}/scripts/process-upload.py -C buildd --builds {{ buildd_manager_dir }}/ -q --log-file=DEBUG:{{ logs_dir }}/process-build-uploads.log
+
+{% endif -%}
+# Clean up the accepted queue every hour, as it's redundant:
+# https://bugs.launchpad.net/launchpad/+bug/361192
+45 * * * *  find {{ buildd_manager_dir }}/accepted/ -maxdepth 1 -type d -execdir rm -rf {} + >/dev/null 2>&1
+
+# Directories older than 1 month can be deleted
+00 00 * * * find {{ buildd_manager_dir }}/rejected/ -maxdepth 1 -type d -mtime +30 -execdir rm -rf {} + >/dev/null 2>&1
+
+# Clean out failed directory: https://portal.admin.canonical.com/C98568
+0 1 * * * find {{ buildd_manager_dir }}/failed/ -maxdepth 1 -type d -mtime +7 -execdir rm -rf {} + >/dev/null 2>&1
+
+# Give up on in-progress downloads from builders after a couple of days
+0 2 * * * find {{ buildd_manager_dir }}/grabbing/ -maxdepth 1 -type d -mtime +2 -execdir rm -rf {} + >/dev/null 2>&1
+
+# Catch up with publishing OOPSes that were temporarily spooled to disk due
+# to RabbitMQ being unavailable.
+*/15 * * * * {{ code_dir }}/bin/datedir2amqp --exchange oopses --host {{ rabbitmq_host }} --username {{ rabbitmq_username }} --password {{ rabbitmq_password }} --vhost {{ rabbitmq_vhost }} --repo {{ oopses_dir }} --key ""
+
diff --git a/charm/launchpad-buildd-manager/templates/launchpad-buildd-manager-lazr.conf b/charm/launchpad-buildd-manager/templates/launchpad-buildd-manager-lazr.conf
new file mode 100644
index 0000000..d09009c
--- /dev/null
+++ b/charm/launchpad-buildd-manager/templates/launchpad-buildd-manager-lazr.conf
@@ -0,0 +1,36 @@
+# Public configuration data.  The contents of this file may be freely shared
+# with developers if needed for debugging.
+
+# A schema's sections, keys, and values are automatically inherited, except
+# for '.optional' sections.  Update this config to override key values.
+# Values are strings, except for numbers that look like ints.  The tokens
+# true, false, and none are treated as True, False, and None.
+
+{% from "macros.j2" import opt -%}
+
+[meta]
+extends: ../launchpad-db-lazr.conf
+
+[builddmaster]
+authentication_endpoint: http://{{ domain_xmlrpc_private }}:{{ port_xmlrpc }}/authserver
+authentication_timeout: 60
+{{- opt("builder_proxy_auth_api_admin_username", builder_proxy_auth_api_admin_username) }}
+{{- opt("builder_proxy_auth_api_endpoint", builder_proxy_auth_api_endpoint) }}
+{{- opt("builder_proxy_host", builder_proxy_host) }}
+{{- opt("builder_proxy_port", builder_proxy_port) }}
+root: {{ buildd_manager_dir }}
+socket_timeout: {{ socket_timeout }}
+virtualized_socket_timeout: {{ virtualized_socket_timeout }}
+{%- if builder_reset_private_ssh_key and builder_reset_public_ssh_key %}
+vm_resume_command: ssh -o StrictHostKeyChecking=no -i /home/{{ user }}/.ssh/builder-reset ppa@%(vm_host)s ppa-reset %(buildd_name)s
+{%- endif %}
+
+{% if cibuild_config -%}
+{% for pillar, config in cibuild_config.items() -%}
+[cibuild.{{ pillar }}]
+{%- for key, value in config.items() %}
+{{ key }}: {{ value }}
+{%- endfor %}
+{%- endfor %}
+{% endif %}
+
diff --git a/charm/launchpad-buildd-manager/templates/launchpad-buildd-manager-secrets-lazr.conf b/charm/launchpad-buildd-manager/templates/launchpad-buildd-manager-secrets-lazr.conf
new file mode 100644
index 0000000..af20066
--- /dev/null
+++ b/charm/launchpad-buildd-manager/templates/launchpad-buildd-manager-secrets-lazr.conf
@@ -0,0 +1,18 @@
+# Secret configuration data.  This is stored in an overlay directory, mainly
+# to avoid accidental information leaks from the public configuration file.
+# Entries in this file should not be shared with developers, although the
+# structure of the file is not secret, only configuration values.
+
+# A schema's sections, keys, and values are automatically inherited, except
+# for '.optional' sections.  Update this config to override key values.
+# Values are strings, except for numbers that look like ints.  The tokens
+# true, false, and none are treated as True, False, and None.
+
+{% from "macros.j2" import opt -%}
+
+[artifactory]
+{{- opt("read_credentials", artifactory_read_credentials) }}
+
+[builddmaster]
+{{- opt("builder_proxy_auth_api_admin_secret", builder_proxy_auth_api_admin_secret) }}
+
diff --git a/charm/launchpad-buildd-manager/templates/launchpad-buildd-manager.service.j2 b/charm/launchpad-buildd-manager/templates/launchpad-buildd-manager.service.j2
new file mode 100644
index 0000000..0e6ae18
--- /dev/null
+++ b/charm/launchpad-buildd-manager/templates/launchpad-buildd-manager.service.j2
@@ -0,0 +1,18 @@
+[Unit]
+Description=Launchpad build farm manager
+After=network.target
+ConditionPathExists=!{{ code_dir }}/maintenance.txt
+
+[Service]
+User={{ user }}
+Group={{ user }}
+WorkingDirectory={{ code_dir }}
+Environment=LPCONFIG=launchpad-buildd-manager
+ExecStart={{ code_dir }}/bin/twistd --python={{ code_dir }}/daemons/buildd-manager.tac --logfile={{ logs_dir }}/buildd-manager.log --pidfile= --nodaemon --umask=0022
+ExecReload=/bin/kill -USR1 $MAINPID
+Restart=on-failure
+LimitNOFILE=65536
+
+[Install]
+WantedBy=multi-user.target
+
diff --git a/charm/launchpad-buildd-manager/templates/logrotate.conf.j2 b/charm/launchpad-buildd-manager/templates/logrotate.conf.j2
new file mode 100644
index 0000000..657e00d
--- /dev/null
+++ b/charm/launchpad-buildd-manager/templates/logrotate.conf.j2
@@ -0,0 +1,16 @@
+{{ logs_dir }}/*.log
+{
+    rotate 90
+    daily
+    dateext
+    delaycompress
+    compress
+    notifempty
+    missingok
+    create 0644 {{ user }} {{ user }}
+    sharedscripts
+    postrotate
+        systemctl reload buildd-manager.service
+    endscript
+}
+