Merge ~paelzer/ubuntu-archive-tools:identify-removals-from-rebuilds into ubuntu-archive-tools:main

[Christian Ehrhardt ]

[Christian Ehrhardt  (paelzer)]

The --help output is meant to be rather complete and self-explanatory, but please let me know if you'd need/want to change something.

$ ./identify-removals-from-rebuilds.py --help
usage: identify-removals-from-rebuilds.py [-h] -s SERIES -c CSV_FILE [--min-arch-fails N] [--no-filtering] [--skip check-id [check-id ...]] [--verbose]

Identify removal candidates based on archive rebuild results.

This helper reads the csv files co-created together with html based archive
rebuild reports by
  https://git.launchpad.net/~ubuntu-test-rebuild/lp-ftbfs-report/tree/source/build_status.py.

For each package in that list it checks a set of constraints that help to
decide if a package is maybe better off to be removed from the archive instead
of spending lots of effort to fix the FTBFS.

This is meant to be a countermeasure of too many FTBFS that are leftover in an
active release and helps Archive Admins to identify candidates on which they
can then consider a removal.

options:
  -h, --help show this help message and exit
  -s SERIES, --series SERIES
                        Name of the series to check dependencies against.
  -c CSV_FILE, --csv CSV_FILE
                        Path to CVS file as generated for archive rebuilds.
  --min-arch-fails N Check not_min_arch_fail will ignore cases which have less architectures failing than this value. For example, set this to 5 to only show cases which fail on 5 or
                        more (Default: 1).
  --no-filtering By default the list is filtered by each check usually making the output small and readable as well as execution fast. By setting this option there will be no
                        filtering, instead the report will include all FTBFS and the result of each check will be reported per package (Warning: this will be slow).
  --skip check-id [check-id ...]
                        Skip the checks passed to this argument. Potential values listed below as "Known checks" (default: none).
  --verbose Report for per package that was excluded from the final report, why it was excluded and provides info about internal errors. (default: False)

Known checks:
  - enough_failing_arches => Did enough architectures fail to be a problem?
  - inacceptable_state => Is the fail state really inacceptable?
  - no_dependencies => Does nothing depend on this packages binaries?
  - no_build_dependencies => Does nothing build-depend on this package?
  - is_not_seeded => Is this package not seeded in Ubuntu?

It has options to skip particular checks if you want to have a different (than the check-all default) look at those lists.

[Christian Ehrhardt  (paelzer)]

Example usage against a public archive rebuild report (you can pass it path or URL to csv or html file, in the example I point to the html report which users usually look at manually):

$ ./identify-removals-from-rebuilds.py --series jammy --csv https://people.canonical.com/~ginggs/ftbfs-report/test-rebuild-20211217-jammy-jammy.html
Reading CSV file https://people.canonical.com/~ginggs/ftbfs-report/test-rebuild-20211217-jammy-jammy.html ...
Info: got an html file path, using: https://people.canonical.com/~ginggs/ftbfs-report/test-rebuild-20211217-jammy-jammy.csv instead
Info: http[s]::// path, fetching https://people.canonical.com/~ginggs/ftbfs-report/test-rebuild-20211217-jammy-jammy.csv
1893 cases found
Check "enough_failing_arches": Did enough architectures fail to be a problem?
Check done, 1893 cases left
Check "inacceptable_state": Is the fail state really inacceptable?
Check done, 1851 cases left
Check "no_dependencies": Does nothing depend on this packages binaries?
Check done, 0 cases left
No cases left, exiting ...

As you can see after applying the checks that we have discussed in Prague there is nothing left for a removal :-/
But I hope that this might not always be true (otherwise creating this was rather useless) especially if such rebuild reports are more recently done or of a more recent release (in both cases not that many cases are fixed already).

[Christian Ehrhardt  (paelzer)]

There is also a (slow) non-filtering mode in which you'll get a tabular view of all checked attributes. Here an example of that as well (I'm sure launchpad will slay the layout, but it looks fine in a console):

$ ./identify-removals-from-rebuilds.py --series jammy --csv /home/paelzer/work/lp-ftbfs-report/lp-ftbfs-report/test-rebuild-20211217-jammy-jammy-short.csv --no-filter
Reading CSV file /home/paelzer/work/lp-ftbfs-report/lp-ftbfs-report/test-rebuild-20211217-jammy-jammy-short.csv ...
14 cases found
Check "enough_failing_arches": Did enough architectures fail to be a problem?
Check "inacceptable_state": Is the fail state really inacceptable?
Check "no_dependencies": Does nothing depend on this packages binaries?
Check "no_build_dependencies": Does nothing build-depend on this package?
Check "is_not_seeded": Is this package not seeded in Ubuntu?
14 (100.0%)

## Final report ##
Found 14 relevant cases
Captions:
Fail = Did enough architectures fail to be a problem?
AccS = Is the fail state really inacceptable?
Dep = Does nothing depend on this packages binaries?
Bdep = Does nothing build-depend on this package?
Seed = Is this package not seeded in Ubuntu?

| Source Package | State | Failing Architectures | Fail | AccS | Dep | Bdep | Seed |
| grubzfs-testsuite | Failed to upload | riscv64 armhf | 2 | Yes | No | No | No |
| zsys | Always FTBFS | riscv64 armhf | 2 | No | Yes | Yes | No |
| zsys | Always DepWait | s390x | 1 | No | Yes | Yes | No |
| cross-toolchain-base | Always FTBFS | amd64 | 1 | No | Yes | Yes | No |
| icu | Always FTBFS | amd64 arm64 armhf ppc64el s390x | 5 | No | Yes | Yes | Yes |
| pyicu | Always FTBFS | arm64 armhf ppc64el s390x amd64 | 5 | No | Yes | Yes | Yes |
| glibc | Always FTBFS | armhf s390x | 2 | No | Yes | Yes | Yes |
| strace | Always FTBFS | arm64 armhf ppc64el amd64 s390x | 5 | No | Yes | Yes | Yes |
| servicelog | Always FTBFS | ppc64el | 1 | No | Yes | No | No |
| ppc64-diag | Always FTBFS | ppc64el | 1 | No | Yes | No | No |
| libreoffice | Always FTBFS | arm64 | 1 | No | Yes | Yes | Yes |
| mir | Always FTBFS | riscv64 armhf arm64 s390x ppc64el amd64 | 6 | No | No | No | Yes |
| squid | Always FTBFS | riscv64 armhf arm64 s390x ppc64el amd64 | 6 | No | Yes | No | No |
| libstatgrab | Always FTBFS | armhf arm64 s390x riscv64 amd64 ppc64el i386 | 7 | No | Yes | Yes | Yes |

[Christian Ehrhardt  (paelzer)]

Since it is a new file there will be no conflict no matter how long this MP is stale.
But I'd like to hear if this does fulfill what we discussed, therefore a 2023 ping for a review.

[Christian Ehrhardt  (paelzer)]

Hi and happy 2023,
There was no feedback yet and it might have fallen out of your inboxes "most-recent" cache.

But the recent Lunar Test rebuild is a great opportunity to bring this up again.
I've run the tool against [1] and found candidates to consider.

This test rebuild has just started and it does not mean we should lightly remove all those.
But this is a great chance to think about what other checks we might want to have following those already implemented (if any)?

Nice list pointing to some questionable cases like [2] for example.

The one thing that comes to my mind is that it might be useful to also point to the package on LP (like [2] in addition to the build log) and report the version. But I'll wait what you others think before spending more time, maybe it isn't as useful as we initially thought?

[1]: https://people.canonical.com/~ginggs/ftbfs-report/test-rebuild-20221215-lunar-normal-lunar.html
[2]: https://launchpad.net/ubuntu/+source/audio-convert

[Christian Ehrhardt  (paelzer)]

https://paste.ubuntu.com/p/7Xj3bJkDqW/

The list still is long, hence I'm waiting for suggestions what to further filter on.

[Steve Langasek (vorlon)]

Test run of the script, the very first package listed for me is:

| linux-starfive-5.17 | Failed to build | riscv64 |

which is an Ubuntu-specific package and shouldn't be removed because it FTBFS.

I thought when we discussed this in Prague, one of the criteria for packages going on this list was that it had been removed from Debian testing. Am I misremembering? I definitely don't see that as one of the checks in the implementation. (Checking for the package being present in Debian unstable but absent from Debian testing is also a check that can be done against downloaded Sources files without a lot of separate API calls, so I would expect it to significantly speed up the runtime.)

OTOH, although the first 4 packages reported are Ubuntu-specific packages that should not be removed, the fifth one is a FTBFS Ubuntu-specific package that *should* be pushed towards
removal: bot-sentry had one upload to Ubuntu in 2008, no uploads since. No idea how it's flown under the radar as being FTBFS. However, because it's Ubuntu-specific, I think we have a duty to file bugs in Launchpad first and give a maintainer an opportunity to respond rather than simply removing it as we would for a leaf package synced from Debian that has been removed from testing. Perhaps the output should be partitioned into two categories, for Debian vs Ubuntu removal candidates?

Another category of information that is important for deciding what to do with the package, and takes effort to manually chase down - if the publishing history says the package was originally synced from Debian, but is no longer in testing or unstable, it should definitely be removed - such packages can be removed using process-removals, but were probably missed at the time of the Debian removal because they still had Ubuntu-specific reverse-dependencies.

So looking at the first 10 packages in the report, I have:
- 4 Ubuntu-specific source packages that should not be removed because of a build failure because they are maintained
- 2 Ubuntu-specific package that are candidates for removal, but did not yet have any bugs filed (bot-sentry, add-apt-key)
- 2 Ubuntu-specific packages that had a bug report filed in LP since {2013,2016} about being totally useless, which I've therefore removed directly (foo-plugins, add-apt-key)
- 3 packages that are present in Debian testing and unstable and should therefore be out of scope for this report as they are not low-hanging fruit (glosstex, hotspot, acl2)

That's not a great hit rate at present and I think we need to improve it quite a bit before I would want to spend much more time on the output. Perhaps improved as per some of the above comments, perhaps you have other ideas.

In any case, I would like to see some improvements here before merging.

review: Needs Fixing

[Christian Ehrhardt  (paelzer)]

> I thought when we discussed this in Prague, one of the criteria for packages going on this list was that it had been removed from Debian testing. Am I misremembering?

It might have been mentioned, but it wasn't on the notes we have been taking.
I agree that this check will be quite useful and will consider adding it.

> Perhaps the output should be partitioned into two categories, for Debian vs Ubuntu removal candidates?

I'd not partition it, but add "ubuntu-only" a non filtering check, which would then appear as an extra column. That fits more nicely to the test and report architecture and should be sufficient to allow looking at those two categories separately. It also allows a user to change that

> Another category of information that is important for deciding what to do with the package, and takes effort to manually chase down - if the publishing history says the package was originally synced from Debian, but is no longer in testing or unstable, it should definitely be removed

Yep, that is another good suggestion for a check - again most likely a reported field instead of a filtering one.

> That's not a great hit rate at present and I think we need to improve it quite a bit before I would want to spend much more time on the output.
...
> In any case, I would like to see some improvements here before merging.

No problem, hence I was asking for opinions after I completed what my notes from Prague listed.

I'll also add the ideas I had which mostly help to navigate the result (report version, link to launchpad overview and tracker) to ease follow on activity.

[Christian Ehrhardt  (paelzer)]

I think another non filtering check could be to probe if the package version was untouched since the last release, if so it is more likely to be old cruft that is unmaintained.
I think we could report the oldest release "since when it is on that version".
I need to check how slow such a check would be, in the past I was quite happy with local madison-lite - but that is quite some setup effort.

[Christian Ehrhardt  (paelzer)]

Changes:
- check where a package exists to help decision making
- add links to Ubuntu and Debian overview pages to the output
- check and report since when a package was not touched
- some table style polishing
- speed up the is-seeded check

Example of the new output:
https://paste.ubuntu.com/p/643K6dgjgF/

[Christian Ehrhardt  (paelzer)]

I think this is ready for another look

[Christian Ehrhardt  (paelzer)]

From my look at the latest list it is really showing quite a lot good candidates.
Plenty of the case Steve described as "removed from testing but still in Ubuntu" and given that we report the last update we see many are from many years ago.

The set of Ubuntu-only packages also seems good for cleanup, either filing bugs or pinging the owning team. Or consider removing stuff that was important and ubuntu-only long ago, but not anymore.

I'm happy to learn more exclusion arguments if others look at the list and find another systematic reason to not be in this list we would consider for cleanup. If there are no such arguments, we might consider merging it as it then seems to be as helpful as it can be.

[Steve Langasek (vorlon)]

halting my review for the moment because downloading the Debian Sources.gz for me is so unreliable I'm not able to iterate.

[Christian Ehrhardt  (paelzer)]

> halting my review for the moment because downloading the Debian Sources.gz for me is so unreliable I'm not able to iterate.

Interesting, I didn't face any such when working on it yesterday and in December.
Is it anything we can fix on our side, or just some kind of overload-at-server or bad-connection?
... trying myself ...
Hmm, yeah this is indeed, ... umm, "rather slow" today.

For the sake of helping to overcome this problem in general I have reworked the downloading code:
1. It now uses retry mechanisms of python3-tenacity for any kind of flaky issue
2. Allow to specify alternative archive URLs

The former works without anything needing to be done by the user.
The latter has the usual "give me a prefix" options.
The following worked really well for me today: `--archive-ubuntu http://de.archive.ubuntu.com/ubuntu --archive-ubuntu-ports http://mirror.kumi.systems/ubuntu-ports`
If you have a local archive mirror on your disk anyway you can even use: `--archive-ubuntu /var/apt-mirror/ubuntu --archive-ubuntu-ports /var/apt-mirror/ubuntu` which saves you the download time.

While looking at it I've found that it also has grown complex enough to use proper logging, which I added as well.

Please let me know, if you faced something else I (actually the code) could help with.

[Steve Langasek (vorlon)]

On Fri, Jan 13, 2023 at 08:27:38AM -0000, Christian Ehrhardt  wrote:
> > halting my review for the moment because downloading the Debian Sources.gz for me is so unreliable I'm not able to iterate.

> Interesting, I didn't face any such when working on it yesterday and in
> December. Is it anything we can fix on our side, or just some kind of
> overload-at-server or bad-connection?
> ... trying myself ...
> Hmm, yeah this is indeed, ... umm, "rather slow" today.

I know there were some Canonical networking issues happening yesterday, so
perhaps it's related. It seems better this morning.

[Steve Langasek (vorlon)]

review: Needs Fixing

[Christian Ehrhardt  (paelzer)]

Thanks for your further suggestions and your experience on how the list could be further trimmed.

Just FYI on the delay, since I implement this mostly in breaks in between meetings this takes a while to complete.

So far I've fixed the few trivial changes you asked for and added the filtering of cases which did not have a binary built on the architectures that are now reported as failing.
It shrunk the list from 245 -> 229 cases.
This is fine, but you said "I find the list is dominated by packages that build on amd64 but ftbfs on one or more of our ports" which sounds like there should be more. If you, by any chance, have a list of cases left from your tests that were in this category please let me know so that I could cross check them.

I hope I'm able to find some time to look at the caching you asked for before marking it ready for review again.

[Christian Ehrhardt  (paelzer)]

> "wrong semantics of counting number of failed archs instead of number of successful archs."

I'm not sure I agree with that being the wrong semantic.
In general I'd assume one is generally interested as soon as at least "one fails" which is the default. OTOH it is clear that a package only is in the list to begin with if at least one fail happened. So in the default config I admit that all will pass (only takes about 20 µs which should be no problem; otherwise I'd change it to only run if someone has specified --min-arch-fails > 1).

But the actual use-case is, if some day one wants to look for those cases that are not one-offs and for that wants to look for those failing on more than one architecture. For that one might bump up --min-arch-fails.

Remember that comparison to "former succeeding builds" is done in other stages.
They are later as they need to reach out more and thereby are slower.

This is not what this check is about, therefore counting "successful builds" instead would be odd here as there might be packages which only try to build on 3, others on all, others on just one. If the filter here would be "at least that many successful builds" what would it really check then and what would be the default number of such builds to make them not being reported?

> I would suggest we add cache handling for these ...

In regard to the caching I have added it, but did not (yet) modified any other tool so that we can first agree on the structure of the cache. For now I have gone with a flat structure in the cache directory based on what they fetch - that is rather readable in code and works fine.

Potential further changes (not sure how far we want/need to go):
- Put all caching function in a file shared between the scripts using it (since we usually run from git I'm not keen on making it a full lib in lib path, but more like a local relative include)
- Modify other users of caching to use this shared code as well (currently find-proposed-cluster and retry-autopkgtest-regressions do have their own copy)
- Modify other consumers of Package/Source data to use the cache
- Change the flat structure to a directory tree structure that matches where it fetches from, but then it would come awfully close to be a full apt-mirror which I'd not like

All of these might be overkill - and I've held them back until I hear your preference on those and the basic cache structure to begin with.

[Christian Ehrhardt  (paelzer)]

FYI: I've started to illustrate how this could be used as lib inside the UAT repo.
I've some ideas to make it more useful in general, please allow me another day or two to push these new commits once completed.

[Steve Langasek (vorlon)]

On Wed, Feb 08, 2023 at 08:52:58AM -0000, Christian Ehrhardt  wrote:
> > "wrong semantics of counting number of failed archs instead of number of successful archs."

> I'm not sure I agree with that being the wrong semantic.
> In general I'd assume one is generally interested as soon as at least "one fails" which is the default. OTOH it is clear that a package only is in the list to begin with if at least one fail happened. So in the default config I admit that all will pass (only takes about 20 µs which should be no problem; otherwise I'd change it to only run if someone has specified --min-arch-fails > 1).

> But the actual use-case is, if some day one wants to look for those cases that are not one-offs and for that wants to look for those failing on more than one architecture. For that one might bump up --min-arch-fails.

> Remember that comparison to "former succeeding builds" is done in other stages.
> They are later as they need to reach out more and thereby are slower.

> This is not what this check is about, therefore counting "successful
> builds" instead would be odd here as there might be packages which only
> try to build on 3, others on all, others on just one. If the filter here
> would be "at least that many successful builds" what would it really check
> then and what would be the default number of such builds to make them not
> being reported?

My earlier comment, buried in the code comments on an earlier revision:

"From my POV, a package is an obvious candidate for source removal and should
be on this report only if it fails to build for all architectures for which
it is tried. This is different than having a fixed number of architectures
it fails on, because some packages limit the architectures they try to build
on, or build only Architecture: all packages, so the necessary number of
packages it fails on could be 1.

Packages that FTBFS on only a subset of architectures for which they
previously built are not good candidates for archive admins to do source
removal on. We should prefer either binary removal or fixing the build -
and deciding which of these is the correct approach requires developer time,
so I don't think they should be on this report."

This is what I mean by counting successful builds.

> - Put all caching function in a file shared between the scripts using it
> (since we usually run from git I'm not keen on making it a full lib in
> lib path, but more like a local relative include)

> - Modify other users of caching to use this shared code as well (currently
> find-proposed-cluster and retry-autopkgtest-regressions do have their
> own copy)

These are good ideas I think, but probably not as part of this particular
MP?

> - Change the flat structure to a directory tree structure that matches
> where it fetches from, but then it would come awfully close to be a full
> apt-mirror which I'd not like

I'm not keen on this one

[Christian Ehrhardt  (paelzer)]

> This is what I mean by counting successful builds.

Thanks for explaining!
Since we already have this scenario handled via the "nobinaries" check I think there is no need to change/add things in that regard.

> > - Put all caching function in a file shared between the scripts using it
> > (since we usually run from git I'm not keen on making it a full lib in
> > lib path, but more like a local relative include)
>
> > - Modify other users of caching to use this shared code as well (currently
> > find-proposed-cluster and retry-autopkgtest-regressions do have their
> > own copy)
>
> These are good ideas I think, but probably not as part of this particular
> MP?

Ok, I'll not modify the other users/scripts when landing this MP.
But I'd at least make it easy to be re-used and show how it could be used.
Therefore I've moved it into an includable source to be the first example.

By chance Utkarsh was visiting me last weekend and already had a problem/task which would be able to re-use it :-)

> > - Change the flat structure to a directory tree structure that matches
> > where it fetches from, but then it would come awfully close to be a full
> > apt-mirror which I'd not like
>
> I'm not keen on this one

Good, then it is already two of us :-)

I now pushed the latest changes, let me know if you think we can land it this way or if more is needed.

a70e101... by Christian Ehrhardt  on 2023-02-17

uat_lib/cache.py: update copyright

In the initial form this had some code of Iain which is why I kept
it initially, but in the form it was committed there is actually
nothing left of it, so drop him from the license header.

Signed-off-by: Christian Ehrhardt <email address hidden>

[Christian Ehrhardt  (paelzer)]

By now some of the existing caching was already refactored to be in utils.py
See https://git.launchpad.net/ubuntu-archive-tools/commit/?id=c60760abfefd92838557cf05a413f03ea4af407f

I'll need to revamp my MR to extend and use that, back to WIP until I found the time for that.

[Steve Langasek (vorlon)]

This is looking pretty good now in terms of behavior. I've reviewed 60 packages so far from the list of recommendations; of these:
- 57 have been removed
- 2 are appropriate for the script to recommend, but I have not removed them
- 1, c-evo-dh, looks wrong to me to be included in the list because it has not failed to build on any of the archs for which binaries are currently published, it only failed on armhf, riscv64, and s390x which the script correctly reports.

[Christian Ehrhardt  (paelzer)]

Reminder from MM chat:

[01:11] <vorlon> [05e] one thing that could be a nice improvement would be if it included a link to file a removal bug with a standard template
[01:11] <vorlon> [05f] à la https://bugs.launchpad.net/ubuntu/+source/opendrim-lmp-os/+bug/2015155

[Steve Langasek (vorlon)]

It appears the issues with the quality of the recommendations have been resolved. I've been removing dozens of ftbfs packages from lunar over the past two weeks using identity-removals-from-rebuilds as input.

I have some quibbles with the UX of the output and I know Christian still wants to do some code refactoring, but from my perspective this is ready to land and be more widely consumed by archive admins.

review: Approve

[Christian Ehrhardt  (paelzer)]

Reminder for a debug session:
[16:56] <vorlon> cpaelzer: so a rerun of identify-removals-from-rebuilds is now reporting candidates that it wasn't reporting yesterday, by almost double... hmm. (example: chibi-scheme)
[16:57] <vorlon> cpaelzer: and chibi-scheme shouldn't be a candidate, the only build failure is an arch for which it's not built in lunar... dunno what's failed there when this check worked previously
[17:09] <cpaelzer> I didn't change any code yet
[17:09] <cpaelzer> since it checks various conditions, like versions in Debian/Ubuntu and such maybe any of that changed and makes it a candidate now
[17:10] <cpaelzer> I could download the ftbfs report, reduce it to just chibi-scheme and then via cmdline we can bump up the loglevel to understand more in detail why it is shown now

[Steve Langasek (vorlon)]

This change in behavior was root-caused to a change in the contents of the report, so not a regression in the tool.

[Christian Ehrhardt  (paelzer)]

More reminders for improvement from chat:
... or they are `superseded` failures on the ftbfs report and should not be candidates

TODO: check if we can detect and filter out the superseded cases

Unmerged commits

a70e101... by Christian Ehrhardt  on 2023-02-17

uat_lib/cache.py: update copyright

In the initial form this had some code of Iain which is why I kept
it initially, but in the form it was committed there is actually
nothing left of it, so drop him from the license header.

Signed-off-by: Christian Ehrhardt <email address hidden>

5301c02... by Christian Ehrhardt  on 2023-02-16

make caching usable outside of identify-removals-from-rebuilds

Signed-off-by: Christian Ehrhardt <email address hidden>

fa3869d... by Christian Ehrhardt  on 2023-02-15

make caching usable outside of identify-removals-from-rebuilds

Signed-off-by: Christian Ehrhardt <email address hidden>

f4797f3... by Christian Ehrhardt  on 2023-02-08

identify-removals-from-rebuilds: use UAT caching

Use the common path for caching data in regard to ubuntu
archive tools fro Packages and Sources.

If users point to a local path with a mirror via commandline arguments
no caching is used as the data already is local and that would
just create redundant data on disk.

Suggested by Steve and inspired by as it is used in
retry-autopkgtest-regressions and find-proposed-cluster.

Signed-off-by: Christian Ehrhardt <email address hidden>

7603642... by Christian Ehrhardt  on 2023-02-07

identify-removals-from-rebuilds: remove duplicate Error:

On some logger.error calls there was still another 'Error:' from
the pre-logger code which appears as duplicate in the output.
Remove those.

Signed-off-by: Christian Ehrhardt <email address hidden>

5e78466... by Christian Ehrhardt  on 2023-01-17

identify-removals-from-rebuilds: filter cases that have no binaries

If a package has no binaries on the architecture that is now failing
to build we consider this not a problem, filter those out.

Signed-off-by: Christian Ehrhardt <email address hidden>

9f14293... by Christian Ehrhardt  on 2023-01-17

identify-removals-from-rebuilds: add missing punctuation

Signed-off-by: Christian Ehrhardt <email address hidden>

ea317fa... by Christian Ehrhardt  on 2023-01-17

identify-removals-from-rebuilds: readbility enhancements

Signed-off-by: Christian Ehrhardt <email address hidden>

cd304fe... by Christian Ehrhardt  on 2023-01-17

identify-removals-from-rebuilds: make source processing less noisy

Signed-off-by: Christian Ehrhardt <email address hidden>

1e690d9... by Christian Ehrhardt  on 2023-01-16

identify-removals-from-rebuilds: ignore pkgs that built

The states 'ALWAYSFTBFS' and 'ALWAYSDEPWAIT' have never produced
anything that went into Ubuntu, hence we do not need to include them.
Reducing these false positives by adding them to acceptable-states.

Signed-off-by: Christian Ehrhardt <email address hidden>

diff --git a/identify-removals-from-rebuilds b/identify-removals-from-rebuilds
new file mode 100755
index 0000000..6037fbd
--- /dev/null
+++ b/identify-removals-from-rebuilds
@@ -0,0 +1,677 @@
+#!/usr/bin/python3
+# -*- coding: utf-8 -*-
+# Copyright © 2022 Christian Ehrhardt <[e-mail]>
+# License:
+# GPLv2 (or later), see /usr/share/common-licenses/GPL
+"""Identify removal candidates based on archive rebuild results.
+
+This helper reads the csv files co-created together with html based archive
+rebuild reports by
+  https://git.launchpad.net/~ubuntu-test-rebuild/lp-ftbfs-report/tree/source/build_status.py.
+
+For each package in that list it checks a set of constraints that help to
+decide if a package is maybe better off to be removed from the archive instead
+of spending lots of effort to fix the FTBFS.
+
+This is meant to be a countermeasure of too many FTBFS that are leftover in an
+active release and helps Archive Admins to identify candidates on which they
+can then consider a removal.
+"""
+
+# Requirements (Ubuntu Packages):
+# - distro-info
+# - ubuntu-dev-tools
+# - python3-requests
+# - python3-slugify
+# - python3-tenacity
+
+import argparse
+import json
+import logging
+import re
+import subprocess
+import sys
+import urllib.request
+
+from urllib.parse import unquote
+
+import apt_pkg
+
+from debian.debian_support import Version
+
+import uat_lib.cache as uat_cache
+
+LOGGER = logging.getLogger(__name__)
+
+
+STATE_TO_TEXT = {
+    'FAILEDTOBUILD': 'Failed to build',
+    'ALWAYSFTBFS':   'Always FTBFS',
+    'ALWAYSDEPWAIT': 'Always DepWait',
+    'NOREGRFTBFS':   'NoRegr FTBFS',
+    'NOREGRDEPWAIT': 'NoRegr DepWait',
+    'MANUALDEPWAIT': 'Dependency wait',
+    'CHROOTWAIT':    'Chroot problem',
+    'UPLOADFAIL':    'Failed to upload',
+    'CANCELLED':     'Cancelled build',
+}
+
+STATES_ACCEPTABLE = ['MANUALDEPWAIT',
+                     'CHROOTWAIT',
+                     'UPLOADFAIL',
+                     'CANCELLED',
+                     'ALWAYSFTBFS',
+                     'ALWAYSDEPWAIT']
+
+ARCHES = {
+    "amd64",
+    "arm64",
+    "armhf",
+    "i386",
+    "ppc64el",
+    "riscv64",
+    "s390x",
+}
+
+SEED_URL = 'http://qa.ubuntuwire.org/ubuntu-seeded-packages/seeded.json.gz'
+
+ARCHIVE_DEBIAN = "https://ftp.debian.org/debian"
+ARCHIVE_UBUNTU = "http://archive.ubuntu.com/ubuntu"
+ARCHIVE_UBUNTU_PORTS = "http://ports.ubuntu.com/ubuntu-ports"
+
+
+def parse_csv_file(lines=None):
+    """Parse Data written by build_status.py.
+
+    Sadly this isn't really CSV conform, using inner comma in architecture
+    list without quoting breaks standard parsers. But OTOH it isn't too
+    complex either.
+    """
+    ftbfs = []
+    for row in lines:
+        if isinstance(row, bytes):
+            row = row.decode("utf-8")
+        try:
+            package, buildlog, explain = row.rstrip('\n').split(',', 2)
+        except ValueError:
+            LOGGER.error('row does not match expected csv format: %s',
+                         row)
+            continue
+
+        allarches, delim, state = explain.rpartition(' ')
+        if not allarches and delim and state:
+            LOGGER.error('row does not match compound format: %s',
+                         explain)
+            continue
+
+        arches = [a.strip() for a in allarches.strip('[]').split(',')]
+        ftbfs.append({"pkg": package,
+                      "buildlog": buildlog,
+                      "fail_arches": arches,
+                      "state": state,
+                      })
+    return ftbfs
+
+
+def load_csv_file(csv_file_name=None):
+    """Read the result of an archive rebuild report.
+
+    If using .html suffix replace it with .csv
+    If prefixed with http[s]: fetch it via http
+    """
+    if csv_file_name.endswith(".html"):
+        csv_file_name = re.sub(r'(.*).html', r'\1.csv', csv_file_name)
+        LOGGER.debug('Got an html file path, using: %s instead', csv_file_name)
+
+    if csv_file_name.startswith(('http://', 'https://')):
+        LOGGER.debug('http[s]::// path, fetching %s', csv_file_name)
+        try:
+            with urllib.request.urlopen(csv_file_name) as data:
+                return parse_csv_file(data)
+        except (urllib.error.HTTPError, urllib.error.URLError):
+            LOGGER.error('failed to fetch %s', csv_file_name)
+            sys.exit(1)
+    else:
+        try:
+            with open(csv_file_name, newline='', encoding="utf-8") as file:
+                return parse_csv_file(file)
+        except OSError:
+            LOGGER.error('failed to read %s', csv_file_name)
+            sys.exit(1)
+    return []
+
+
+def load_seeded_sources():
+    '''Download seeding info and map to sources.
+
+    This is slow, but much faster than calling seeded-in-ubuntu each time.
+    Returns tuple with a dict and a list:
+    - Dict:
+      - key: seeded source package
+      - value: list of binaries related to this source
+    - List: contains all source packages that relate to a seeded binary
+    '''
+    seeded_sources = []
+    src_to_bin = {}
+
+    with uat_cache.CachedResource(SEED_URL) as seed:
+        seeded_list = json.load(seed)
+
+    apt_pkg.init()
+    components = get_components("ubuntu")
+    for component in components:
+        for arch in ARCHES:
+            url = get_pkg_url(ARGS.series, component, arch)
+            with uat_cache.CachedResource(url) as sources_file:
+                apt_sources = apt_pkg.TagFile(sources_file)
+                for section in apt_sources:
+                    binary = section["Package"]
+                    try:
+                        source = section["Source"]
+                    except KeyError:
+                        source = binary
+
+                    if source in src_to_bin:
+                        if binary not in src_to_bin[source]:
+                            src_to_bin[source].append(binary)
+                    else:
+                        src_to_bin[source] = [binary]
+
+                    if binary in seeded_list:
+                        seeded_sources.append(source)
+
+    return (src_to_bin, seeded_sources)
+
+
+def check_enough_failing_arches(ftbfs, _pkg):
+    """Check how many architectures failed and filter if not enough."""
+    failing_arch_count = len(ftbfs.get("fail_arches"))
+    if failing_arch_count < ARGS.min_arch_fails:
+        return False
+    return failing_arch_count
+
+
+def check_inacceptable_state(ftbfs, _pkg):
+    """Check if fail state is usually not caused by the PKG (= acceptable)."""
+    return ftbfs.get("state") not in STATES_ACCEPTABLE
+
+
+def get_rmadison_info(rma):
+    '''returns the text output of rmadison as structure.
+
+    Strips common suffixes and epochs if present.
+    Might raise a KeyError if there was nothing found.'''
+    rma_info = []
+    for rma_line in rma.split('\n'):
+        rma_elements = rma_line.split('|')
+        if len(rma_elements) == 4:
+            rma_release = rma_elements[2].strip()
+            for sfx in ['/universe', '/restricted', '/multiverse']:
+                rma_release = rma_release.removesuffix(sfx)
+            for pocket in ["", "-security", "-updates"]:
+                rma_release = rma_release.removesuffix(pocket)
+            # might have an epoch prefix to strip
+            rma_version = re.sub(r'^\d+:', '', rma_elements[1].strip())
+            rma_version = Version(rma_version)
+            rma_arches = rma_elements[3].strip().split(',')
+            rma_info.append({'release': rma_release,
+                             'ver': rma_version,
+                             'arch': rma_arches})
+    return rma_info
+
+
+def check_untouched(ftbfs, pkg):
+    '''Check if the version of this package didn't change.
+
+    Packages passing this check are more likely old cruft that
+    is unmaintained. Returns the first Ubuntu rekease that had this version.'''
+    try:
+        cmd = ['rmadison', '-u', 'ubuntu', '-a', 'source', pkg]
+        rma = subprocess.check_output(cmd,
+                                      stderr=subprocess.STDOUT,
+                                      encoding='utf-8')
+        rma_info = get_rmadison_info(rma)
+    except (subprocess.CalledProcessError, KeyError):
+        LOGGER.debug('Package not found by rmadison: %s', pkg)
+        return "not found"
+    if len(rma_info) == 0:
+        return "not found"
+    sorted_rma_info = sorted(rma_info, key=lambda d: d['ver'])
+
+    # CSV does not contain the version, derive from url
+
+    build_log_url = unquote(ftbfs['buildlog'])
+    versionmatch = re.match(rf'.*{pkg}_(.*)_BUILDING.txt', build_log_url)
+    if versionmatch and versionmatch.group(1) is not None:
+        try:
+            build_version = Version(versionmatch.group(1))
+        except ValueError:
+            return "Invalid version"
+    else:
+        LOGGER.debug('could not find version in %s', ftbfs["buildlog"])
+        return "no version"
+
+    if build_version < sorted_rma_info[0]['ver']:
+        return "too old"
+
+    # return earliest release that had this version
+    for release_info in sorted_rma_info:
+        if release_info['ver'] == build_version:
+            return release_info['release']
+
+    return "not in rmadison"
+
+
+def check_nobinaries(ftbfs, pkg):
+    '''Check if the failing architectures did have binaries built before.
+
+    Such packages are not considered a regression and it is not a problem
+    for later maintenance nor a reason to be removed.'''
+    if pkg not in SRC_TO_BIN:
+        LOGGER.debug('no binaries known for source %s', pkg)
+        return False
+
+    try:
+        cmd = ['rmadison',
+               '-u', 'ubuntu',
+               '-s', ARGS.series,
+               '-a', ','.join(str(x) for x in ftbfs.get("fail_arches")),
+               ','.join(SRC_TO_BIN[pkg])]
+        rma = subprocess.check_output(cmd,
+                                      stderr=subprocess.STDOUT,
+                                      encoding='utf-8')
+        if rma:
+            LOGGER.debug('Got existing builds for src: (%s): "%s"', pkg, rma)
+            rma_info = get_rmadison_info(rma)
+            # we could report all, but that wastes space, report one arch
+            # that we have found plus the number of further arches that
+            # were listed in the ftbfs, but exist in the target suite.
+            arches = []
+            for info in rma_info:
+                arches.extend(info['arch'])
+            arches = list(set(arches))
+            extra_count = len(arches)-1
+            return f'{rma_info[0]["arch"][0]} +{extra_count}'
+
+        LOGGER.debug('Got no binaries in %s for src (%s)', ARGS.series, pkg)
+        return False
+
+    except (subprocess.CalledProcessError, KeyError):
+        LOGGER.error('Package not found by rmadison: %s', pkg)
+        return False
+
+
+def check_existence(_ftbfs, pkg):
+    '''Check existence in Ubuntu, Debian-unstable and Debian-testing.
+       - If it is in unstable, but absent from testing it is a good candidate
+         for removal
+       - If it is only in Ubuntu, keep it as we need to maintain it (but report
+         it as ubuntu only)
+       - otherwise filter it from the list as it is not a low hanging
+         fruit for removal.'''
+    if pkg not in SOURCE_MAP['unstable'] and pkg not in SOURCE_MAP['testing']:
+        return "Ubuntu-only"
+    if pkg not in SOURCE_MAP['ubuntu']:
+        # already removed
+        return False
+    if pkg in SOURCE_MAP['unstable'] and pkg not in SOURCE_MAP['testing']:
+        if 'ubuntu' in SOURCE_MAP["ubuntu"][pkg]:
+            return "Not in testing (delta)"
+        return "Not in testing (sync)"
+    # Ignore all others
+    return False
+
+
+def check_has_no_depends(_ftbfs, pkg):
+    """Check if nothing depends on binaries of this package."""
+    cmd = ['reverse-depends', f'--release={ARGS.series}', f'src:{pkg}']
+    try:
+        dep = subprocess.check_output(cmd, stderr=subprocess.STDOUT)
+    except subprocess.CalledProcessError:
+        LOGGER.debug('Package not found by reverse-depends: %s', pkg)
+        return False
+    return dep == b'No reverse dependencies found\n'
+
+
+def check_has_no_bdepends(_ftbfs, pkg):
+    """Check if nothing build-depends on this package."""
+    cmd = ['reverse-depends', f'--release={ARGS.series}',
+           '--build-depends', f'src:{pkg}']
+    try:
+        bdep = subprocess.check_output(cmd, stderr=subprocess.STDOUT)
+    except subprocess.CalledProcessError:
+        LOGGER.debug('Package not found by reverse-depends: %s', pkg)
+        return False
+    return bdep == b'No reverse dependencies found\n'
+
+
+def check_is_not_seeded(_ftbfs, pkg):
+    """Check if this package is not seeded in Ubuntu."""
+    return pkg not in SEEDED_SOURCES
+
+
+# This list also defines the order in which checks are applied
+# To make this more effective the checks are sorted by speed with those
+# reaching out externally like reverse-depends and seeded-in-ubuntu at the
+# end which will cause them to be rarely executed.
+#
+# Usually the what remains in the list after filtering has the same attributes.
+# Therefore, by default, no check results are individually reported.
+# OTOH if --no-filter is set each checked attribute is reported.
+# Defining it as a per check attribute leaves flexibility for later added
+# checks to be always reported by setting reportworthy => True.
+CHECKS = {
+    "enough_failing_arches": {
+        "enabled": True,
+        "reportworthy": False,
+        "checktxt": "Did enough architectures fail to be a problem?",
+        "fn": check_enough_failing_arches,
+        "short": "Fail",
+        },
+    "inacceptable_state":         {
+        "enabled": True,
+        "reportworthy": False,
+        "checktxt": "Is the fail state really inacceptable?",
+        "fn": check_inacceptable_state,
+        "short": "Acceptable",
+        },
+    "is_not_seeded":                {
+        "enabled": True,
+        "reportworthy": False,
+        "checktxt": "Is this package not seeded in Ubuntu?",
+        "fn": check_is_not_seeded,
+        "short": "Seeded in Ubuntu",
+        },
+    "existence":         {
+        "enabled": True,
+        "reportworthy": True,
+        "checktxt": "Absent from testing or Ubuntu-only?",
+        "fn": check_existence,
+        "short": "Exists in (sync/delta)",
+        },
+    "no_dependencies":         {
+        "enabled": True,
+        "reportworthy": False,
+        "checktxt": "Does nothing depend on this packages binaries?",
+        "fn": check_has_no_depends,
+        "short": "Has Dep",
+        },
+    "no_build_dependencies":   {
+        "enabled": True,
+        "reportworthy": False,
+        "checktxt": "Does nothing build-depend on this package?",
+        "fn": check_has_no_bdepends,
+        "short": "Has Bdep",
+        },
+    "nobinaries":         {
+        "enabled": True,
+        "reportworthy": True,
+        "checktxt": "Did the failing architectures have binaries published?",
+        "fn": check_nobinaries,
+        "short": "Has Binaries in",
+        },
+    "untouched":         {
+        "enabled": True,
+        "reportworthy": True,
+        "checktxt": "Since when was there no update?",
+        "fn": check_untouched,
+        "short": "Untouched since",
+        },
+}
+
+
+def filter_ftbfs_list(ftbfs_l):
+    """Runs all enabled checks on the list of FTBFS and reduces the list.
+
+    Note: Strips the list in each iteration to trigger less tests each round.
+    """
+    ftbfs_remaining = ftbfs_l
+    for check, checkdetails in CHECKS.items():
+        if check == "internal-error":
+            continue
+        if not checkdetails["enabled"]:
+            LOGGER.info('Check %s is disabled', check)
+            continue
+        LOGGER.info('Check "%s"', checkdetails["checktxt"])
+        filtered = []
+        progress = 0
+        remaining = len(ftbfs_remaining)
+        for ftbfs in ftbfs_remaining:
+            pkg = ftbfs.get("pkg")
+            progress += 1
+            print(f'{progress} ({progress/remaining*100:5,.1f}%)',
+                  end='\r')
+            result = checkdetails["fn"](ftbfs, pkg)
+            ftbfs |= {check: result}
+            if not result:
+                LOGGER.debug('PKG "%s" skipped by check "%s".', pkg, check)
+                continue
+            filtered.append(ftbfs)
+
+        if not ARGS.no_filtering:
+            LOGGER.info('Check done: %s cases left', len(filtered))
+            if len(filtered) == 0:
+                LOGGER.info('No cases left, exiting ...')
+                sys.exit(0)
+            ftbfs_remaining = filtered
+
+    return ftbfs_remaining
+
+
+# From https://stackoverflow.com/questions/40419276
+def link(uri, label=None):
+    """Wrap text into URL that is a link in e.g. vtw based terminals"""
+    if label is None:
+        label = uri
+    parameters = ''
+
+    # OSC 8 ; params ; URI ST <name> OSC 8 ;; ST
+    escape_mask = '\033]8;{};{}\033\\{}\033]8;;\033\\'
+
+    return escape_mask.format(parameters, uri, label)
+
+
+def report_ftbfs(ftbfs_l):
+    """Report the ftbfs that are left after filtering."""
+    LOGGER.info('## Final report ##')
+    LOGGER.info('Found %s relevant cases', len(ftbfs_l))
+
+    pkg_width = 1
+    for ftbfs in ftbfs_l:
+        pkg_width = max(pkg_width, len(ftbfs.get("pkg")))
+
+    header = ('| Pkg-Info '
+              f'| {"Source Package":{pkg_width}} '
+              f'| {"State":<18} '
+              f'| {"Failing Architectures":45} ')
+    for _name, details in CHECKS.items():
+        if details["enabled"] and details["reportworthy"]:
+            header += f'| {details["short"]} '
+    header += '|'
+    print(header)
+
+    for ftbfs in ftbfs_l:
+        archstr = ' '.join(str(x) for x in ftbfs.get("fail_arches"))
+        uurl = f'https://launchpad.net/ubuntu/+source/{ftbfs.get("pkg")}'
+        durl = f'https://tracker.debian.org/pkg/{ftbfs.get("pkg")}'
+        txt = ('| ' + link(durl, 'Deb') + '  ' + link(uurl, 'Ubu') + ' '
+               '| ' + link(ftbfs.get("buildlog"),
+                           f'{ftbfs.get("pkg"):{pkg_width}}') + ' '
+               f'| {STATE_TO_TEXT[ftbfs.get("state")]:<18} '
+               f'| {archstr:45} ')
+        for name, details in CHECKS.items():
+            if details["enabled"] and details["reportworthy"]:
+                value = ftbfs.get(name)
+                if value is None:
+                    value = "n/a"
+                # To be able to carry more than true/false (e.g. count) when
+                # a check passes this is unintuitively mapped to the boolean
+                # values, re-translate it back for the user.
+                if isinstance(value, bool):
+                    if value is True:
+                        value = "No"
+                    else:
+                        value = "Yes"
+                clen = len(details["short"])
+                txt += f'| {value:<{clen}s} '
+        txt += '|'
+        print(txt)
+
+
+def get_components(suite):
+    '''Return components for a suite.'''
+    if suite == 'ubuntu':
+        return ("main", "restricted", "universe", "multiverse")
+    return ("main", "contrib", "non-free-firmware", "non-free")
+
+
+def get_src_url(suite, component):
+    '''Return the URL to fetch sources from.'''
+    if suite == 'ubuntu':
+        return (f'{ARGS.archive_ubuntu}/dists/{ARGS.series}/'
+                f'{component}/source/Sources.gz')
+    return (f'{ARGS.archive_debian}/dists/{suite}/'
+            f'{component}/source/Sources.gz')
+
+
+def get_pkg_url(series, component, arch):
+    '''Return the URL to fetch package info from.'''
+    if arch in ['amd64', 'i386']:
+        base_url = ARGS.archive_ubuntu
+    else:
+        base_url = ARGS.archive_ubuntu_ports
+
+    return (f'{base_url}/dists/{series}/'
+            f'{component}/binary-{arch}/Packages.gz')
+
+
+def read_sources():
+    """Read information from the Ubuntu and Debian Sources files.
+
+    Returns a dict containing:
+     * a Ubuntu mapping of source package names to versions
+     * a Debian-unstable mapping of source package names to versions
+     * a Debian-testing mapping of source package names to versions
+    """
+    LOGGER.info('Reading Sources ...')
+    source_map = {'ubuntu': {}, 'testing': {}, 'unstable': {}}
+    apt_pkg.init()
+
+    for suite in source_map:
+        components = get_components(suite)
+        for component in components:
+            url = get_src_url(suite, component)
+            with uat_cache.CachedResource(url) as sources_file:
+                apt_sources = apt_pkg.TagFile(sources_file)
+                for section in apt_sources:
+                    src = section["Package"]
+                    ver = section["Version"]
+                    if (src not in source_map[suite] or
+                            apt_pkg.version_compare(source_map[suite][src], ver) < 0):
+                        source_map[suite][src] = ver
+
+    return source_map
+
+
+def setup_parser():
+    '''Set up the argument parser for this program.'''
+
+    epilog = 'Known checks:'
+    for name, details in CHECKS.items():
+        epilog += f'\n  - {name} => {details["checktxt"]}'
+    formatter = argparse.RawDescriptionHelpFormatter
+
+    parser = argparse.ArgumentParser(epilog=epilog,
+                                     description=__doc__,
+                                     formatter_class=formatter)
+    parser.add_argument(
+        "-s", "--series", required=True,
+        help="Name of the series to check dependencies against.")
+    parser.add_argument(
+        "-c", "--csv", dest="csv_file", required=True,
+        help="Path to CVS file as generated for archive rebuilds.")
+
+    parser.add_argument(
+        "--min-arch-fails",
+        default=1, metavar='N', type=int,
+        help="Check not_min_arch_fail will ignore cases which have less "
+             "architectures failing than this value. For example, set this "
+             "to 5 to only show cases which fail on 5 or more (Default: 1).")
+
+    parser.add_argument(
+        "--no-filtering", action='store_true',
+        help="By default the list is filtered by each check usually making "
+             "the output small and readable as well as execution fast. "
+             "By setting this option there will be no filtering, instead the "
+             "report will include all FTBFS and the result of each check will "
+             "be reported per package (Warning: this will be slow).")
+
+    parser.add_argument(
+        "--skip",
+        type=str,
+        nargs='+',
+        metavar='check-id',
+        help='Skip the checks passed to this argument. Potential values '
+             'listed below as "Known checks" (default: none).')
+
+    parser.add_argument(
+        "--archive-debian",
+        default=ARCHIVE_DEBIAN,
+        help="URL or local path prefix to access the Debian Archive. "
+             f'(Default: {ARCHIVE_DEBIAN})')
+    parser.add_argument(
+        "--archive-ubuntu",
+        default=ARCHIVE_UBUNTU,
+        help="URL or local path prefix to access the Ubuntu Archive. "
+             f'(Default: {ARCHIVE_UBUNTU})')
+    parser.add_argument(
+        "--archive-ubuntu-ports",
+        default=ARCHIVE_UBUNTU_PORTS,
+        help="URL or local path prefix to access the Ubuntu-Ports Archive. "
+             f'(Default: {ARCHIVE_UBUNTU_PORTS})')
+
+    parser.add_argument('-l',
+                        '--loglevel',
+                        default='info',
+                        help='Provide logging level. Example '
+                             '--loglevel debug (Default: info)')
+
+    return parser.parse_args()
+
+
+if __name__ == '__main__':
+    ARGS = setup_parser()
+
+    logging.basicConfig(level=ARGS.loglevel.upper(),
+                        format="%(levelname)s: %(asctime)s - %(message)s")
+
+    if ARGS.series:
+        try:
+            subprocess.check_output(['distro-info', '--series', ARGS.series])
+        except subprocess.CalledProcessError:
+            LOGGER.error('%s is not a valid series.', ARGS.series)
+            sys.exit(1)
+
+    # Without filtering we want the info of all checks reported
+    if ARGS.no_filtering:
+        for _check, checkconfig in CHECKS.items():
+            checkconfig["reportworthy"] = True
+
+    if ARGS.skip:
+        for checkid in ARGS.skip:
+            try:
+                CHECKS[checkid]["enabled"] = False
+            except KeyError:
+                LOGGER.error('%s is not a known check.', checkid)
+                sys.exit(1)
+
+    LOGGER.info('Reading CSV file %s', ARGS.csv_file)
+    ftbfs_list = load_csv_file(csv_file_name=ARGS.csv_file)
+    LOGGER.info('%s cases found', len(ftbfs_list))
+
+    SOURCE_MAP = read_sources()
+    (SRC_TO_BIN, SEEDED_SOURCES) = load_seeded_sources()
+
+    ftbfs_list = filter_ftbfs_list(ftbfs_list)
+
+    report_ftbfs(ftbfs_list)
diff --git a/uat_lib/cache.py b/uat_lib/cache.py
new file mode 100644
index 0000000..5bd1e23
--- /dev/null
+++ b/uat_lib/cache.py
@@ -0,0 +1,135 @@
+# Copyright (C) 2020-2023 Canonical Ltd.
+"""
+Helper to cache commonly used ressources like packages.gz and similar
+in a path common to all ubuntu archive tools. Provides a context manager
+and auto-fetch as well as auto-decompression to allow the programs using
+it to focus on what makes them special and not re-implement fetch&open
+over and over.
+"""
+# Author: Christian Ehrhardt <[e-mail]>
+
+# This library is free software; you can redistribute it and/or
+# modify it under the terms of the GNU Lesser General Public
+# License as published by the Free Software Foundation; either
+# version 2.1 of the License, or (at your option) any later version.
+
+# This library is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+# Lesser General Public License for more details.
+
+# You should have received a copy of the GNU Lesser General Public
+# License along with this library; if not, write to the Free Software
+# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA  02110-1301
+# USA
+
+import gzip
+import logging
+import os
+import tempfile
+
+from contextlib import closing
+from urllib.request import urlopen
+from urllib.error import URLError
+from datetime import datetime, timedelta
+from tenacity import retry, stop_after_delay, stop_after_attempt, wait_fixed
+
+from slugify import slugify
+
+LOGGER = logging.getLogger(__name__)
+
+
+class CachedResource:
+    '''Makes a resource available independent to location and format.'''
+    def __init__(self, resource):
+        '''Open the resource after making it available.
+
+          - If it is an URL fetch it to the cache, otherwise use it directly
+            from the local path
+          - If it is gzip compresses decompresses it
+          - set self.file_obj to the resource as opened file
+        '''
+        self.tmp = False
+
+        if os.path.isfile(resource):
+            LOGGER.debug('Using local file "%s"', resource)
+            fname = resource
+        else:
+            cache_fname = CachedResource.url_to_cache_file(resource)
+            if CachedResource.is_recent_enough(cache_fname):
+                LOGGER.debug('Use "%s" from cache for "%s"',
+                             cache_fname, resource)
+            else:
+                LOGGER.debug('Caching "%s" as "%s"', resource, cache_fname)
+                CachedResource.fetch_resource(resource, cache_fname)
+            fname = cache_fname
+
+        # For now simple detection via file suffix (no magic, no try/except)
+        if fname.endswith('.gz'):
+            fd, self.file_name = tempfile.mkstemp()
+            LOGGER.debug('Tempfile for gzip extraction %s', self.file_name)
+            self.tmp = True
+            try:
+                with closing(gzip.GzipFile(fname)) as gz_file:
+                    with os.fdopen(fd, "wb") as out_file:
+                        out_file.write(gz_file.read())
+            except gzip.BadGzipFile as err:
+                LOGGER.error('Unable to gunzip %s: %s', fname, err)
+                raise err
+        else:
+            self.file_name = fname
+
+        self.file_obj = open(self.file_name, 'r', encoding='utf-8')
+
+    def __enter__(self):
+        '''Return the opened file.'''
+        return self.file_obj
+
+    def __exit__(self, __type, __value, __traceback):
+        '''Close file and remove temporary file if needed.'''
+        self.file_obj.close()
+        if self.tmp:
+            os.remove(self.file_name)
+
+    @staticmethod
+    def get_cache_dir():
+        """Returns the cache base directory for ubuntu archive tools"""
+        fallback = os.path.expanduser(os.path.join('~', '.cache'))
+        cache_dir = os.environ.get('XDG_CACHE_HOME', fallback)
+        uat_cache_path = os.path.join(cache_dir, 'ubuntu-archive-tools')
+        os.makedirs(uat_cache_path, exist_ok=True)
+        return uat_cache_path
+
+    @staticmethod
+    def is_recent_enough(filename, max_age=timedelta(hours=24)):
+        """Checks if a file was last modified within the last max_age_hours."""
+        try:
+            file_mtime = datetime.utcfromtimestamp(os.path.getmtime(filename))
+        except FileNotFoundError:
+            return False
+        cutoff = datetime.utcnow() - max_age
+        return file_mtime > cutoff
+
+    @staticmethod
+    def url_to_cache_file(url):
+        """Derive file name as it will appear in the cache based on the url."""
+        cache_dir = CachedResource.get_cache_dir()
+        # remove (file) incompatible elements of an url
+        return os.path.join(cache_dir, slugify(url, separator='.'))
+
+    @retry(stop=(stop_after_delay(60) | stop_after_attempt(5)),
+           wait=wait_fixed(2))
+    @staticmethod
+    def fetch_resource(url, fname):
+        '''Fetch URL to filename
+
+        Use retry functionality for more reliability over network
+        '''
+
+        try:
+            with closing(urlopen(url)) as url_file:
+                with open(fname, "wb") as comp_file:
+                    comp_file.write(url_file.read())
+        except URLError as err:
+            LOGGER.error('Failed to fetch url %s', url)
+            raise err