Merge ~cjwatson/launchpad:doc-charm-workflow into launchpad:master

[Colin Watson]

doc: Add explanation of charm development

This is doubtless incomplete, but I thought I'd at least get everything I could immediately think of written down as a starting point for others.

[Simone Pelosi (pelpsi)]

LGTM!

review: Approve

diff --git a/doc/conf.py b/doc/conf.py
index f43f4fd..3f6130a 100644
--- a/doc/conf.py
+++ b/doc/conf.py
@@ -38,7 +38,7 @@ master_doc = "index"
 
 # General information about the project.
 project = "Launchpad"
-copyright = "2004-2022, Canonical Ltd."
+copyright = "2004-2023, Canonical Ltd."
 
 # The version info for the project you're documenting, acts as replacement for
 # |version| and |release|, also used in various other places throughout the
diff --git a/doc/explanation/charms.rst b/doc/explanation/charms.rst
new file mode 100644
index 0000000..6fe52d3
--- /dev/null
+++ b/doc/explanation/charms.rst
@@ -0,0 +1,113 @@
+=================
+Charm development
+=================
+
+The direction of our official deployments is to use `Juju charms
+<https://juju.is/docs/sdk>`_.  (We still have a number of manually-deployed
+systems, so we aren't there yet.)
+
+To get an overview of how this works, you'll need to look in the ``charm/``
+directory of Launchpad itself, as well as `ols-layers
+<https://git.launchpad.net/ols-charm-deps>`_, `launchpad-layers
+<https://git.launchpad.net/launchpad-layers>`_, and `launchpad-mojo-specs
+<https://git.launchpad.net/launchpad-mojo-specs>`_.  Each of the
+subdirectories of ``charm/`` represents a single logical function which can
+be deployed as a Juju `application <https://juju.is/docs/olm/application>`_
+with one or more `units <https://juju.is/docs/olm/unit>`_.  The layers
+provide common code used by multiple charms.  The specs are used with `Mojo
+<https://mojo.canonical.com/>`_ to coordinate whole deployments of multiple
+applications; they contain configuration of individual applications and
+`integrations <https://juju.is/docs/olm/integration>`_ between applications.
+
+Principles
+==========
+
+Wherever possible, charm code should live in the same repository as the code
+it deploys (the payload).  This makes it easier to evolve both in parallel.
+
+Launchpad is open source.  Its charms and its configuration (aside from a
+small number of necessary secrets) should also be open source.  As well as
+being the right thing to do, this also allows using machinery such as
+Launchpad's charm recipes that upload to `Charmhub <https://charmhub.io/>`_.
+When used in combination with Charmhub, Juju can easily be instructed to
+upgrade charms and update configuration using a single `bundle
+<https://juju.is/docs/olm/bundle>`_, allowing the top-level spec to be
+relatively simple.
+
+Each charm should correspond to a deployment of a single top-level payload.
+On the other hand, it's fine for a single payload to have multiple charms
+corresponding to different ways in which it can be deployed: for example,
+Launchpad itself will have charms for the appservers, buildd-manager,
+publishers, and so on.
+
+If a legacy deployment bundled multiple logical functions onto a single
+machine purely for economic convenience, don't be afraid to split those up
+in a way that makes sense.  However, there's no need to go to extremes: if
+multiple cron jobs all have broadly the same system requirements, then we're
+unlikely to want each cron job to be deployed using its own Juju
+application.
+
+It will not always make sense to expose every single configuration option of
+the payload directly in the charm.  Some configuration options may only
+exist for internal testing purposes or for backward-compatibility, and some
+may make sense for the charm to use internally but not expose in ``juju
+config``.  A good rule of thumb is to consider whether a given configuration
+option needs to differ between deployments; if it doesn't, there's probably
+no need to expose it.
+
+`DRY <https://en.wikipedia.org/wiki/Don%27t_repeat_yourself>`_ applies to
+configuration as well as code, and can help to avoid gratuitous differences
+between deployments.  `Jinja <https://jinja.palletsprojects.com/>`_
+templates are widely used in both charms and Mojo specs as part of this.
+
+Keep multi-datacentre operation in mind where possible.  We don't have
+enough experience with this yet to know what we'll need to do, but it's
+likely to involve deploying parts of an application in different datacentres
+from other parts, so loose coupling will help: for example, it may be useful
+to allow configuring connections using explicit configuration as well as or
+instead of Juju integrations.
+
+Workflow
+========
+
+You can run test deployments using `Juju <https://juju.is/docs/olm>`_ and
+`LXD <https://linuxcontainers.org/lxd/introduction/>`_.  If you don't
+already have a suitable testbed, then see the `Juju tutorial
+<https://juju.is/docs/olm/get-started-with-juju>`_ for how to set one up;
+you should use the non-Kubernetes approach here.
+
+Each Mojo spec has a ``README.md`` file explaining how to deploy it, and
+that's usually the easiest way to get started.  You should normally use the
+corresponding ``devel`` stage, as that's intended for local deployments: for
+example, it will normally deploy fewer units, and doesn't assume that parts
+of Canonical's internal infrastructure will be available.
+
+Once you've successfully deployed an environment, you will probably want to
+iterate on it in various ways.  You can build a new charm using ``charmcraft
+pack`` in the appropriate subdirectory, and then use ``juju refresh`` to
+upgrade your local deployment to that.  You can change configuration items
+using ``juju config``.  Alternatively, you can make a local clone of the
+Mojo spec and point ``mojo run`` at that rather than at a repository on
+``git.launchpad.net``, and then you can iterate by changing the spec.
+
+Use ``juju debug-log`` and ``juju status`` liberally to observe what's
+happening as you make changes.  See `How to debug a charm
+<https://juju.is/docs/sdk/debug-a-charm>`_ for more specific advice on that
+topic.
+
+Secrets
+=======
+
+Cryptographic secrets should not be stored in Mojo specs, and nor should
+some other pieces of information (such as configuration relevant to
+preventing spam).  These are instead stored in a secrets file on the
+relevant deployment host (``launchpad-bastion-ps5.internal`` or
+``is-bastion-ps5.internal`` for official deployments), and are updated
+manually.  The ``bundle`` command in the Mojo manifest will have a
+``local=`` parameter pointing to this file, relative to
+``$MOJO_ROOT/LOCAL/$MOJO_PROJECT/$MOJO_STAGE``.
+
+Managing secrets like this is more cumbersome than updating Mojo specs, so
+try to keep it to a minimum.  In some cases there may be automation
+available to help, such as the `autocert charm
+<https://charmhub.io/autocert>`_.
diff --git a/doc/explanation/index.rst b/doc/explanation/index.rst
index 9b66e6d..21299fd 100644
--- a/doc/explanation/index.rst
+++ b/doc/explanation/index.rst
@@ -13,3 +13,4 @@ Explanation
    architecture
    pip
    favicon
+   charms