If rich history has been indicated using the changes file, but the
remote ref specified does not exist, then "git fetch" fails with a
message starting "fatal: couldn't find remote ref".
We already have code that retries in case there's a network outage
making rich history fetch temporarily unavailable.
However, this causes the "ref doesn't exist" case to also result in a
soft failure, so an import of an upload that is erroneous in this way
never succeeds.
Instead, we should treat this as a hard RichHistoryNotFoundError, which
already has a handling path that falls back to a synthesized commit.
Unfortunately there doesn't seem to be any good way for "git fetch" to
differentiate these cases, and we're using "git fetch" instead of the
pygit2 API because of https://github.com/libgit2/pygit2/issues/1060. So
we resort to parsing the output. This requires LC_ALL=C.UTF-8 to avoid
translation string related issues, and we also add a test to ensure that
we'll notice if this assumption about git's behaviour changes.
Ideally I'd also have a test that verifies that LC_ALL=C.UTF-8 does do
what we need, by perhaps expecting a translated string with LC_ALL
changed to something else. But it doesn't look like git has any
translated strings in practice, so it wouldn't be straightforward to
cause it to output something different. I think we'll need to rely on
just specifying it our call.
Now that we are importing all* packages, allowlists (including
"allowlist_team") and phasing are no longer required, so we can remove
this functionality and remove significant sections of code.
Previously we were augmenting the "user-specified" allowlist with an
allowlist generated by reading the apt repository for source packages,
placing them in main and universe buckets, and then applying "phasing"
to them to allow us to include proportional subsets of main and
universe. This made sense at the time, but is no longer necessary.
The downside of the previous approach is that it locked the set of
source packages imported to those present at the time the poller was
last restarted. This would miss new source packages until after the
poller was restarted and a further upload was made to them after the
restart.
Now that we've imported the majority of packages currently published in
any pocket of any release that is not EOL, it's simpler to import any
package that isn't otherwise denylisted. We can maintain the denylist
with the set of edge case imports that still need attention to avoid
unnecessary importer resources.
This requires just a denylist and nothing else.
Since the apt repository parsing was being done solely to determine the
list of source packages in the archive for the purposes of the poller,
the apt repository reading and validation code can also be removed,
together with corresponding tests.
According to the import specification, it's fine to parse Thurs as
Thu, Tues as Tue, and Sept as Sep, as these are all unambiguous. This
fixes imports of the following:
Applications of multiple regular expression replacements are factored
out into _apply_re_substitutions(). This function isn't explicitly
tested; the comprehensive test coverage of its only caller
Changelog._parse_changelog_date() is sufficient.
In importing source packages that exist in non-EOL releases but have
since been deleted, these further twelve packages fail to import and
have never been imported, so add these to the denylist for now.
These package are not yet imported. They will need individual
investigation to fix the edge case issues that cause them to fail. In
the meantime, we won't attempt to import them automatically to avoid
blocking up worker slots.
