Merge ~cjwatson/launchpad-buildd:self-service-rebuild-images into launchpad-buildd:master

[Colin Watson]

Describe waiting for image rebuilds in more detail

In https://code.launchpad.net/~cjwatson/launchpad-buildd/+git/launchpad-buildd/+merge/427478, Jürgen asked "How do you know the image building has finished?" We don't have a straightforward view of this, unfortunately, but here's some more detail.

[Jürgen Gmach (jugmac00)]

thanks for clarifying

review: Approve

diff --git a/docs/how-to/deployment.rst b/docs/how-to/deployment.rst
index 8e9d83e..6952c28 100644
--- a/docs/how-to/deployment.rst
+++ b/docs/how-to/deployment.rst
@@ -85,10 +85,27 @@ Releasing to production
    staging management environment (``stg-vbuilder@launchpad-bastion-ps5``)
    to start rebuilding images.
 
-#. Once image builds complete, builders will get the new image after they
-   finish their next build (or are disabled) and go through being reset.
-   `Build farm administrators
+#. Once the new image is rebuilt, which normally takes on the order of 15-60
+   minutes depending on the architecture, builders will get the new image
+   after they finish their next build (or are disabled) and go through being
+   reset.  As a result, ``manage-builders -v`` should start showing the new
+   version over time.
+
+#. Wait for the new version to appear for at least one builder in each
+   region and architecture (other than ``riscv64``, which is managed
+   separately).  If this doesn't happen after 90 minutes, then ask IS for
+   assistance in investigating; they can start by checking ``juju status``
+   in ``[e-mail]``.
+
+#. Once the updated version is visible for at least one builder in each
+   region and architecture (other than ``riscv64``), `build farm
+   administrators
    <https://launchpad.net/~launchpad-buildd-admins/+members>`_ can use
-   ``manage-builders --virt --idle --reset`` to reset idle builders.
+   ``manage-builders --virt --idle --builder-version=<old-version> --reset``
+   to reset idle builders, thereby causing builders that haven't taken any
+   builds recently to catch up.
+
+#. Ask Colin Watson or William Grant to reflash ``riscv64`` builders to the
+   new version (currently a separate manual process).
 
 #. Close any bugs fixed by the new release.