Merge ~cjwatson/launchpad-buildd:improve-deployment-docs into launchpad-buildd:master

[Colin Watson]

Improve deployment documentation

This is based on https://wiki.canonical.com/InformationInfrastructure/ISO/BuildInfrastructure/BuilddFixing#Upgrading_launchpad-buildd_in_scalingstack, but I've reorganized it a bit to try to make it easier to follow.

[Jürgen Gmach (jugmac00)]

Thank you! The documentation looks good! Looking forward to use it / testdrive it :-)

A few minor things:
- `tox -e docs` gives a warning, easily fixed with that diff
https://pastebin.canonical.com/p/cZSfGfR88T/

explanation:
 https://stackoverflow.com/a/14067756/672833

- Could you please update the build command in tox to turn warnings into errors by adding `-W`?https://www.sphinx-doc.org/en/master/man/sphinx-build.html#cmdoption-sphinx-build-W
While not critical here, this might prevent future issues.

review: Approve

07d7ec9... by Colin Watson on 2022-07-26

Use anonymous references to avoid Sphinx warning

Thanks to Jürgen Gmach.

c4ff8e5... by Colin Watson on 2022-07-26

Turn sphinx-build warnings into errors

[Colin Watson (cjwatson)]

Thanks, I thought I'd checked for warnings but indeed had managed to miss that. I've applied both your suggestions.

diff --git a/debian/changelog b/debian/changelog
index 5f3804e..7824911 100644
--- a/debian/changelog
+++ b/debian/changelog
@@ -1,3 +1,9 @@
+launchpad-buildd (217) UNRELEASED; urgency=medium
+
+  * Improve deployment documentation.
+
+ -- Colin Watson <[e-mail]>  Mon, 18 Jul 2022 15:07:23 +0100
+
 launchpad-buildd (216) focal; urgency=medium
 
   [ Andrey Fedoseev ]
diff --git a/docs/how-to/deployment.rst b/docs/how-to/deployment.rst
index 67473a3..34898a3 100644
--- a/docs/how-to/deployment.rst
+++ b/docs/how-to/deployment.rst
@@ -1,21 +1,60 @@
 How to deploy launchpad-buildd
 ******************************
 
-The following steps need to be performed before `Upgrading the builders
-<https://wiki.canonical.com/InformationInfrastructure/ISO/BuildInfrastructure/BuilddFixing>`_.
+In Canonical's datacentre environments, launchpad-buildd is deployed as a
+``.deb`` package installed in a fleet of VMs.  To upgrade it, we need to
+rebuild the VM images.
 
-#. Ensure everything has been merged to master, that the `recipe
+Each environment uses its own PPA and management environment:
+
++--------------------------------------------------+--------------------------------------------------------------------------------------------------------------------+
+| Environment                                      | PPA and management environment                                                                                     |
++==================================================+====================================================================================================================+
+| `production <https://launchpad.net/builders>`_   | `ppa:launchpad/ubuntu/buildd <https://launchpad.net/~launchpad/+archive/ubuntu/buildd/+packages>`_                 |
+|                                                  | ``prod-launchpad-vbuilders@is-bastion-ps5``                                                                        |
++--------------------------------------------------+--------------------------------------------------------------------------------------------------------------------+
+| `dogfood <https://dogfood.paddev.net/builders>`_ | `ppa:launchpad/ubuntu/buildd-staging <https://launchpad.net/~launchpad/+archive/ubuntu/buildd-staging/+packages>`_ |
+|                                                  | ``stg-vbuilder@launchpad-bastion-ps5``                                                                             |
++--------------------------------------------------+--------------------------------------------------------------------------------------------------------------------+
+
+These instructions use various tools from `ubuntu-archive-tools
+<https://git.launchpad.net/ubuntu-archive-tools>`_ (``copy-package`` and
+``manage-builders``).
+
+Testing on dogfood
+------------------
+
+#. Ensure everything has been merged to master.
+
+#. Check that the `recipe
    <https://code.launchpad.net/~launchpad/+recipe/launchpad-buildd-daily>`_
-   has built successfully, and that the resulting package has been published
-   in the `Launchpad PPA
+   has built successfully (you can start a build manually if required), and
+   that the resulting package has been published in the `Launchpad PPA
    <https://launchpad.net/~launchpad/+archive/ubuntu/ppa/+packages>`_.
 
-#. Upgrade the dogfood builders
-   (you may need someone on the LP team with permissions to help with this;
-   see `documentation <https://wiki.canonical.com/InformationInfrastructure/ISO/BuildInfrastructure/BuilddFixing#Upgrading_launchpad-buildd_in_scalingstack>`_).
+#. Run ``copy-package --from=ppa:launchpad/ubuntu/ppa --suite=focal
+   --to=ppa:launchpad/ubuntu/buildd-staging -b launchpad-buildd`` to copy
+   the current version of launchpad-buildd to the deployment PPA.
+
+#. `Wait for PPA publishing to complete
+   <https://launchpad.net/~launchpad/+archive/ubuntu/buildd-staging/+packages>`__.
+
+#. Run ``mojo run -m manifest-rebuild-images`` in the management environment
+   (``stg-vbuilder@launchpad-bastion-ps5``) to start rebuilding images.
+   After a minute or so, ``juju status glance-simplestreams-sync-\*`` will
+   show "Synchronising images"; once this says "Sync completed", images have
+   been rebuilt.
+
+#. Builders will get the new image after they finish their next build (or
+   are disabled) and go through being reset.  Since dogfood's build farm is
+   typically mostly idle, you can use ``manage-builders -l dogfood --reset``
+   to reset all builders and force them to pick up the new image.
 
 #. Perform QA on dogfood until satisfied.
 
+Releasing to production
+-----------------------
+
 #. Create a new release branch, e.g. ``release-213``, based on master.
 
 #. Run ``DEBEMAIL="<email address>" DEBFULLNAME="<name>" dch -rD focal``.
@@ -26,11 +65,31 @@ The following steps need to be performed before `Upgrading the builders
    ``Release version 213`` for review.
 
 #. Once the release branch has merged to master,
-   tag the release commit (e.g. ``git tag 213 && git push origin --tags``) and
-   check https://code.launchpad.net/~launchpad/+recipe/launchpad-buildd-daily
-   for the recipe build to happen.
-   You can start a build if required.
-
-#. File an upgrade RT (`sample <https://portal.admin.canonical.com/C150737>`_),
-   noting the version number and possibly multiple suites/releases
-   (`IS procedure <https://wiki.canonical.com/InformationInfrastructure/ISO/BuildInfrastructure/BuilddFixing>`_).
+   tag the release commit (e.g. ``git tag 213 && git push origin 213``).
+
+#. Check that the `recipe
+   <https://code.launchpad.net/~launchpad/+recipe/launchpad-buildd-daily>`_
+   has built successfully (you can start a build manually if required), and
+   that the resulting package has been published in the `Launchpad PPA
+   <https://launchpad.net/~launchpad/+archive/ubuntu/ppa/+packages>`_.
+
+#. Run ``copy-package --from=ppa:launchpad/ubuntu/ppa --suite=focal
+   --to=ppa:launchpad/ubuntu/buildd -b launchpad-buildd`` to copy the
+   current version of launchpad-buildd to the deployment PPA.
+
+#. `Wait for PPA publishing to complete
+   <https://launchpad.net/~launchpad/+archive/ubuntu/buildd/+packages>`__.
+
+#. File an RT ticket asking IS to run ``mojo run -m
+   manifest-rebuild-images`` in the management environment
+   (``prod-launchpad-vbuilders@is-bastion-ps5``) to start rebuilding images.
+   (`cRT#151858 <https://portal.admin.canonical.com/C151858>`_ will allow
+   this step to be self-service.)
+
+#. Once image builds complete, builders will get the new image after they
+   finish their next build (or are disabled) and go through being reset.
+   `Build farm administrators
+   <https://launchpad.net/~launchpad-buildd-admins/+members>`_ can use
+   ``manage-builders --virt --idle --reset`` to reset idle builders.
+
+#. Close any bugs fixed by the new release.
diff --git a/tox.ini b/tox.ini
index 4f38868..83137f5 100644
--- a/tox.ini
+++ b/tox.ini
@@ -6,4 +6,4 @@ description = Build documentation via Sphinx.
 basepython = python3
 extras = docs
 commands =
-    sphinx-build -b html -d docs/_build/doctrees docs docs/_build/html
+    sphinx-build -W -b html -d docs/_build/doctrees docs docs/_build/html