OCI multiarch upload leads to temporary invalid manifest

This gets more worse when one build worked and already pushed and another build failed (see eg. https://launchpad.net/~cloud-images-release-managers/cloud-images/+oci/ubuntu-base/+recipe/21.04/ for an example and the attached screenshot where s390x failed to upload).

In that case, some architectures are uploaded to the target registry and others are not. That leads to an inconsistent state on the registry where some arch images are up-to-date and others are still old.

We talked about this a bit in our last team meeting, and I promised to write up the conclusions.

Design
======

There seems to have been some disagreement about exactly what we're trying to achieve, e.g. a suggestion that this is by design and that we want to avoid a given tag showing image versions from different build sets across architectures (which would inherently conflict with fixing this bug).

My design statement is that we want the behaviour to be as close to the observed behaviour of the snap store as possible within the semantics of OCI registries: as such, uploading a new image to a tag for one architecture should in general not affect other architectures. If you're following a tag such as "edge" that's used as a direct upload target for builds, then it is expected that that tag may sometimes refer to builds from a different build set across architectures. People who want a consistent view, in the sense of only ever seeing builds from a single build set for the same tag on different architectures, should use some other tag (such as "stable" or "candidate") that's maintained by promoting full build sets from elsewhere.

This corresponds to the behaviour of the snap store (where "edge" is typically a direct upload target for builds and may skew across architectures, but less risky channels such as "stable" are normally supposed to be maintained by promotion of build sets from elsewhere rather than by direct uploads). It also corresponds to the behaviour of the primary Ubuntu archive, where e.g. "impish-proposed" is a direct upload target for builds and may skew across architectures in various ways, but "impish" is kept consistent by CI machinery.

There is one wrinkle: if a Launchpad recipe was previously configured to build for a certain set of architectures and then is later reconfigured to build for a smaller set, then I think that should probably cause the removed architectures to be removed from the corresponding multi-arch manifests in registries (at least when later builds happen). This isn't a firm design requirement in the way that the above is, but it seems likely to be what most people want to happen. So, if there are no objections, I'd suggest that we should preserve only those entries in the multi-arch manifest for a given registry/image/tag that correspond to architectures that are still configured in the recipe.

Implementation
==============

The way the current implementation works is quite confusing, and as a result I spent a while being unable to see why this bug was happening until I cross-referenced it with some logs.

It looks as though `OCIRegistryClient.makeMultiArchManifest` fetches the current manifest from the registry and mutates it, which would achieve roughly the design I outlined above (with the exception of removing architectures when they've been removed from the recipe's configuration). However, what actually happens on each upload is that we first call `OCIRegistryClient.upload`, which pushes a single-architecture manifest to the registry, and then in a second step we call `OCIRegistryClient.uploadManifestList`, which fetches the current manifest from the registry and augments it (which in p...