Merge ~xnox/britney/+git/britney2-ubuntu:ignore-di into ~ubuntu-release/britney/+git/britney2-ubuntu:master

[Dimitri John Ledkov]

suiteloader: add configuration option whether to load udebs.

udebs do not have symbol versioned shlibdeps, thus it means that when
there is glibc in proposed, any packages built in proposed get a udeb
depends on the strictly higher version of glibc in proposed. This
leads to entaglement of the $world with glibc transition. Similar
story with glib, openssl, etc.

By setting the new configuration option to HAS_UDEBS = no, one can
make suiteloader not look at udebs, and thus use britney even when
installability of udebs is not required.

Redone as a new config option.

[Steve Langasek (vorlon)]

In terms of being able to merge this back to Debian, it would be preferable to make this a configurable option rather than deleting the code.

[Iain Lane (laney)]

Right, this should be configurable please, so that (1) it can be done differently for stable releases, (2) it can be upstreamed - which should ideally be done ASAP if & when we merge something here.

See britney1 make_b2_config for the config file generation. However, for testing, you can ignore b1 and use a static configuration.

Question: would it be better to stop building udebs at all (noudeb profile, I'm thinking like we do for nocheck on riscv64)?

review: Needs Fixing

[Dimitri John Ledkov (xnox)]

@laney

noudeb is very hard, and yes i want to build with noudeb by default, but it is pain to get there. making britney ignore udebs is a very helpful tool to get there.

1) lp doesn't set build-profiles. also launchpad doesn't like when it dispatches a build, and it produces no debs (builds appear as failed, as nothing to build for us, because profiles result in nothing to build)

2) in dpkg, one can make noudeb implicitely declared one, but if one does that nochange rebuilds of all the udeb carrying packages must be uploaded and migrated simultaneously as otherwise britney will not let them migrate

3) there are a lot of dependencies between all the udebs. For example, i'm trying to unwind all the dependencies on *-modules, to be able to stop building kernel udebs. It will take many more package removals and delta in the packaging to eradicate all other udebs.

Thus yes, I hope to patch all the things to support noudeb build profile, and activate it by default. But to get there I would prefer to be able to drop udebs one by one, without stalling migrations and/or making it a mega-transition.

[Iain Lane (laney)]

Cheers. As a first step, let me do a test dry run and we can look at the output.

[Iain Lane (laney)]

Here:

https://people.canonical.com/~ubuntu-archive/laney/proposed-migration/update_excuses.html

Can you take a look and see if this has the desired effect?

It doesn't seem to fix $world. I think that for most packages the tighter relationship is coming from symbols, i.e.:

 __x86_get_cpuid_feature_leaf@GLIBC_2.33 2.33
 fstat64@GLIBC_2.33 2.33
 fstat@GLIBC_2.33 2.33
 fstatat64@GLIBC_2.33 2.33
 fstatat@GLIBC_2.33 2.33
 lstat64@GLIBC_2.33 2.33
 lstat@GLIBC_2.33 2.33
 mallinfo2@GLIBC_2.33 2.33
 mknod@GLIBC_2.33 2.33
 mknodat@GLIBC_2.33 2.33
 stat64@GLIBC_2.33 2.33
 stat@GLIBC_2.33 2.33

or shlibs:

ld-linux-x86-64 2 libc6 (>= 2.33)
libBrokenLocale 1 libc6 (>= 2.33)
libanl 1 libc6 (>= 2.33)
libc 6 libc6 (>= 2.33)
libdl 2 libc6 (>= 2.33)
libm 6 libc6 (>= 2.33)
libmvec 1 libc6 (>= 2.33)
libnsl 1 libc6 (>= 2.33)
libnss_compat 2 libc6 (>= 2.33)
libnss_dns 2 libc6 (>= 2.33)
libnss_files 2 libc6 (>= 2.33)
libnss_hesiod 2 libc6 (>= 2.33)
libpthread 0 libc6 (>= 2.33)
libresolv 2 libc6 (>= 2.33)
librt 1 libc6 (>= 2.33)
libthread_db 1 libc6 (>= 2.33)
libutil 1 libc6 (>= 2.33)

Not denying that this might fix some things (packages which build udebs themselves), but I'm not sure making the claim about "any packages built in proposed" getting a too-tight relationship is right.

[Iain Lane (laney)]

review: Needs Information

[Dimitri John Ledkov (xnox)]

This looks good.

The point is that dpkg in hirsute now stops building udebs, hence things like grub should have less blocking hints.

However in your URL I still see this:

grub2 (2.04-1ubuntu37 to 2.04-1ubuntu40)
Migration status for grub2 (2.04-1ubuntu37 to 2.04-1ubuntu40): BLOCKED: Rejected/violates migration policy/introduces a regression
Issues preventing migration:
grub-emu/amd64 in main cannot depend on libsdl1.2debian in universe
grub-mount-udeb/amd64 in main cannot depend on libfuse2-udeb in universe
grub-mount-udeb/amd64 in main cannot depend on libdevmapper1.02.1-udeb in universe
grub-mount-udeb/amd64 in main cannot depend on libc6-udeb in universe
grub-mount-udeb/arm64 in main cannot depend on libc6-udeb in universe
grub-mount-udeb/arm64 in main cannot depend on libfuse2-udeb in universe
grub-mount-udeb/arm64 in main cannot depend on libdevmapper1.02.1-udeb in universe

How come udebs are being compared and these hints generated?

I expected for udebs to not be assessed or generate excuses like above.

[Dimitri John Ledkov (xnox)]

*not good*

[Dimitri John Ledkov (xnox)]

What was the config for the sample run you did?

[Iain Lane (laney)]

It was run like:

laney@snakefruit:~/proposed-migration$ code/b2/britney.py -v --series hirsute --distribution ubuntu -c code/b2/britney.conf.ubuntu.hirsute 2>&1 | tee ~/tmp-log.txt

and:

laney@snakefruit:~/proposed-migration$ grep HAS_UDEBS code/b2/britney.conf.ubuntu.hirsute
HAS_UDEBS = no

[Iain Lane (laney)]

https://bazaar.launchpad.net/~ubuntu-release/britney/britney1-ubuntu/view/head:/britney#L280

[Dimitri John Ledkov (xnox)]

> https://bazaar.launchpad.net/~ubuntu-
> release/britney/britney1-ubuntu/view/head:/britney#L280

Ok, in that i case i think i need to write a "policy" that looks at package type and ignores them.

cause -proposed is full of uninstallable udebs now.

And i guess britney1 does all that because britney2 does not have partial suites support.

[Dimitri John Ledkov (xnox)]

@Laney added skipping loading udebs in the reading potentially pre-merged packages tagfile.

by guessing. Kind of wish to be able to run britney2 locally to test it out too.

[Dimitri John Ledkov (xnox)]

using doko as a service to rebuild all udeb producing packages; and remove udebs from the release suite.

hence this will be done "properly" without hacks.

review: Disapprove

Unmerged commits

abc3c9d... by Dimitri John Ledkov on 2021-02-24

Also skip udebs, whilst reading pre-merged packages file

d9891a7... by Dimitri John Ledkov on 2021-02-18

suiteloader: add configuration option whether to load udebs.

udebs do not have symbol versioned shlibdeps, thus it means that when
there is glibc in proposed, any packages built in proposed get a udeb
depends on the strictly higher version of glibc in proposed. This
leads to entaglement of the $world with glibc transition. Similar
story with glib, openssl, etc.

By setting the new configuration option to HAS_UDEBS = no, one can
make suiteloader not look at udebs, and thus use britney even when
installability of udebs is not required.

diff --git a/britney.conf b/britney.conf
index 74ecd4b..62403bd 100644
--- a/britney.conf
+++ b/britney.conf
@@ -44,6 +44,9 @@ BREAK_ARCHES      =
 # if you're in this list, you are a new architecture
 NEW_ARCHES        =
 
+# should udebs be looked at?
+HAS_UDEBS         = yes
+
 # priorities and delays
 MINDAYS_LOW       = 0
 MINDAYS_MEDIUM    = 0
diff --git a/britney2/inputs/suiteloader.py b/britney2/inputs/suiteloader.py
index 964f239..14e7462 100644
--- a/britney2/inputs/suiteloader.py
+++ b/britney2/inputs/suiteloader.py
@@ -316,6 +316,10 @@ class DebMirrorLikeSuiteContentLoader(SuiteContentLoader):
             pkg = get_field('Package')
             version = get_field('Version')
             section = get_field('Section')
+            if getattr(self._base_config, 'has_udebs', 'yes') != 'yes':
+                filename = get_field('Filename')
+                if filename.endswith('.udeb'):
+                    continue
 
             # There may be multiple versions of any arch:all packages
             # (in unstable) if some architectures have out-of-date
@@ -489,22 +493,23 @@ class DebMirrorLikeSuiteContentLoader(SuiteContentLoader):
                                             binary_dir,
                                             'Packages')
                     filename = possibly_compressed(filename)
-                    udeb_filename = os.path.join(basedir,
-                                                 component,
-                                                 "debian-installer",
-                                                 binary_dir,
-                                                 "Packages")
-                    # We assume the udeb Packages file is present if the
-                    # regular one is present
-                    udeb_filename = possibly_compressed(udeb_filename)
                     self._read_packages_file(filename,
                                              arch,
                                              suite.sources,
                                              packages)
-                    self._read_packages_file(udeb_filename,
-                                             arch,
-                                             suite.sources,
-                                             packages)
+                    if getattr(self._base_config, 'has_udebs', 'yes') == 'yes':
+                        udeb_filename = os.path.join(basedir,
+                                                     component,
+                                                     "debian-installer",
+                                                     binary_dir,
+                                                     "Packages")
+                        # We assume the udeb Packages file is present if the
+                        # regular one is present
+                        udeb_filename = possibly_compressed(udeb_filename)
+                        self._read_packages_file(udeb_filename,
+                                                 arch,
+                                                 suite.sources,
+                                                 packages)
                 # create provides
                 provides = create_provides_map(packages)
                 binaries[arch] = packages