importer-service-worker: set as default repository
Use --set-as-default-repository when calling "git ubuntu import" unless
--no-set-as-default-repository is specified. This hooks everything up so
that we set the default repository when we push from the official
importer service instance.
Now that LP: #1992500 is resolved, the importer has permission to set
repositories it pushes as defaults for the targets by itself. This is
generally what we want for production git-ubuntu imports.
This adds this capability as a feature enabled by a CLI argument. The
default is not to do so, just like --push.
If rich history has been indicated using the changes file, but the
remote ref specified does not exist, then "git fetch" fails with a
message starting "fatal: couldn't find remote ref".
We already have code that retries in case there's a network outage
making rich history fetch temporarily unavailable.
However, this causes the "ref doesn't exist" case to also result in a
soft failure, so an import of an upload that is erroneous in this way
never succeeds.
Instead, we should treat this as a hard RichHistoryNotFoundError, which
already has a handling path that falls back to a synthesized commit.
Unfortunately there doesn't seem to be any good way for "git fetch" to
differentiate these cases, and we're using "git fetch" instead of the
pygit2 API because of https://github.com/libgit2/pygit2/issues/1060. So
we resort to parsing the output. This requires LC_ALL=C.UTF-8 to avoid
translation string related issues, and we also add a test to ensure that
we'll notice if this assumption about git's behaviour changes.
Ideally I'd also have a test that verifies that LC_ALL=C.UTF-8 does do
what we need, by perhaps expecting a translated string with LC_ALL
changed to something else. But it doesn't look like git has any
translated strings in practice, so it wouldn't be straightforward to
cause it to output something different. I think we'll need to rely on
just specifying it our call.
Now that we are importing all* packages, allowlists (including
"allowlist_team") and phasing are no longer required, so we can remove
this functionality and remove significant sections of code.
Previously we were augmenting the "user-specified" allowlist with an
allowlist generated by reading the apt repository for source packages,
placing them in main and universe buckets, and then applying "phasing"
to them to allow us to include proportional subsets of main and
universe. This made sense at the time, but is no longer necessary.
The downside of the previous approach is that it locked the set of
source packages imported to those present at the time the poller was
last restarted. This would miss new source packages until after the
poller was restarted and a further upload was made to them after the
restart.
Now that we've imported the majority of packages currently published in
any pocket of any release that is not EOL, it's simpler to import any
package that isn't otherwise denylisted. We can maintain the denylist
with the set of edge case imports that still need attention to avoid
unnecessary importer resources.
This requires just a denylist and nothing else.
Since the apt repository parsing was being done solely to determine the
list of source packages in the archive for the purposes of the poller,
the apt repository reading and validation code can also be removed,
together with corresponding tests.
According to the import specification, it's fine to parse Thurs as
Thu, Tues as Tue, and Sept as Sep, as these are all unambiguous. This
fixes imports of the following:
Applications of multiple regular expression replacements are factored
out into _apply_re_substitutions(). This function isn't explicitly
tested; the comprehensive test coverage of its only caller
Changelog._parse_changelog_date() is sufficient.