Merge ~xnox/germinate:foreign-arch-ma-same into germinate:master

[Dimitri John Ledkov]

This branch enables to seed Multi-Arch: same package dependencies from a foreign arch (ie. :i386), making cross arch dependencies, on systems that have executed dpkg --add-architecture.

This does not allow to seed non-Multi-Arch:same packages (ie. application, e.g. wine32). However, this work can be extended to support that (care needs to be taken when expanding non-arch qualified recommends and resolving them from the default architecture).

This branch achieves generating identical list of i386 ship-live packages as hardcoded in the debian-cd branch for eoan release, when executed as

$ ./bin/germinate --arch amd64 --foreign i386 -d eoan -s ubuntu.eoan

[Dimitri John Ledkov (xnox)]

This now passes existing testsuite. It is awkward that testsuite had many packages without an Architecture declared, and then foreign was not set, and like None == None is obviously True!

[Colin Watson (cjwatson)]

For the record, I'm waiting to review this until you've added the foreign arch tests you mentioned and removed the note that the branch is WIP from the description.

[Dimitri John Ledkov (xnox)]

Added a test that loads up both amd64&i386 Packages files, and asserts that behavior does not change with single-arch germinator; and does the right thing with foreign arch option set. Fixed a bug that said testcase surfaced.

I think this is now ready for review.

[Colin Watson (cjwatson)]

review: Needs Fixing

3556b42... by Dimitri John Ledkov on 2019-12-21

Merge remote-tracking branch 'germinate/master' into foreign-arch-ma-same

89bbfe9... by Dimitri John Ledkov on 2019-12-21

Start addressing review feedback

ca08d70... by Dimitri John Ledkov on 2023-04-17

Set default arch to amd64, and allow appending to it

595ccb2... by Dimitri John Ledkov on 2023-04-17

Merge remote-tracking branch 'germinate/master' into foreign-arch-ma-same

a41437a... by Dimitri John Ledkov on 2023-04-17

Update error messages

6214809... by Dimitri John Ledkov on 2023-04-17

revert defaults to be usable

Unmerged commits

6214809... by Dimitri John Ledkov on 2023-04-17

revert defaults to be usable

test:0 (build)

1 → 1 of 1 result

First • Previous • Next • Last

a41437a... by Dimitri John Ledkov on 2023-04-17

Update error messages

test:0 (build)

1 → 1 of 1 result

First • Previous • Next • Last

595ccb2... by Dimitri John Ledkov on 2023-04-17

Merge remote-tracking branch 'germinate/master' into foreign-arch-ma-same

test:0 (build)

1 → 1 of 1 result

First • Previous • Next • Last

ca08d70... by Dimitri John Ledkov on 2023-04-17

Set default arch to amd64, and allow appending to it

89bbfe9... by Dimitri John Ledkov on 2019-12-21

Start addressing review feedback

3556b42... by Dimitri John Ledkov on 2019-12-21

Merge remote-tracking branch 'germinate/master' into foreign-arch-ma-same

b342713... by Dimitri John Ledkov on 2019-10-16

Add foreign arch parsing, to support seeding M-A:same deps

diff --git a/germinate/archive.py b/germinate/archive.py
index efb5e72..f48bbd2 100644
--- a/germinate/archive.py
+++ b/germinate/archive.py
@@ -85,13 +85,15 @@ class Archive:
 class TagFile(Archive):
     """Fetch package lists from a Debian-format archive as apt tag files."""
 
-    def __init__(self, dists, components, arch, mirrors, source_mirrors=None,
+    def __init__(self, dists, components, arches, mirrors, source_mirrors=None,
                  installer_packages=True, cleanup=False):
         """Create a representation of a Debian-format apt archive."""
         if isinstance(dists, six.string_types):
             dists = [dists]
         if isinstance(components, six.string_types):
             components = [components]
+        if isinstance(arches, six.string_types):
+            arches = [arches]
         if isinstance(mirrors, six.string_types):
             mirrors = [mirrors]
         if isinstance(source_mirrors, six.string_types):
@@ -99,7 +101,7 @@ class TagFile(Archive):
 
         self._dists = dists
         self._components = components
-        self._arch = arch
+        self._arches = arches
         self._mirrors = mirrors
         self._installer_packages = installer_packages
         if source_mirrors:
@@ -221,15 +223,16 @@ class TagFile(Archive):
         try:
             for dist in self._dists:
                 for component in self._components:
-                    packages = self._open_tag_files(
-                        self._mirrors, dirname, "Packages", dist, component,
-                        "binary-" + self._arch + "/Packages")
-                    for tag_file in packages:
-                        try:
-                            for section in apt_pkg.TagFile(tag_file):
-                                yield (IndexType.PACKAGES, section)
-                        finally:
-                            tag_file.close()
+                    for arch in self._arches:
+                        packages = self._open_tag_files(
+                            self._mirrors, dirname, arch + "_Packages", dist,
+                            component, "binary-" + arch + "/Packages")
+                        for tag_file in packages:
+                            try:
+                                for section in apt_pkg.TagFile(tag_file):
+                                    yield (IndexType.PACKAGES, section)
+                            finally:
+                                tag_file.close()
 
                     sources = self._open_tag_files(
                         self._source_mirrors, dirname, "Sources", dist,
@@ -247,7 +250,7 @@ class TagFile(Archive):
                             instpackages = self._open_tag_files(
                                 self._mirrors, dirname, "InstallerPackages",
                                 dist, component,
-                                "debian-installer/binary-" + self._arch +
+                                "debian-installer/binary-" + self._arches[0] +
                                 "/Packages")
                         except IOError:
                             # can live without these
diff --git a/germinate/germinator.py b/germinate/germinator.py
index 8817990..379f569 100644
--- a/germinate/germinator.py
+++ b/germinate/germinator.py
@@ -30,6 +30,7 @@ import re
 import sys
 
 import apt_pkg
+import six
 from six.moves.collections_abc import MutableMapping
 
 from germinate.archive import IndexType
@@ -382,15 +383,26 @@ class Germinator(object):
     # Initialisation.
     # ---------------
 
-    def __init__(self, arch):
+    def __init__(self, arches):
         """Create a dependency expander.
 
-        Each instance of this class can only process a single architecture,
-        but can process multiple seed structures against it.
+        Each instance of this class can only process a single main
+        architecture, and an optional single foreign architecture (for
+        Multi-Arch:same only packages), but can process multiple seed
+        structures against it.
+
+        arches can be a string, for main arch only. Or a list, in that
+        case the first element is the main architecture, and the
+        second element is the foreign arch.
 
         """
-        self._arch = arch
-        apt_pkg.config.set("APT::Architecture", self._arch)
+        if isinstance(arches, six.string_types):
+            arches = [arches]
+        self._arches = arches
+
+        apt_pkg.config.set("APT::Architecture", self._arches[0])
+        for arch in self._arches:
+            apt_pkg.config.set("APT::Architectures::", arch)
 
         # Global hints file.
         self._hints = {}
@@ -445,6 +457,17 @@ class Germinator(object):
         pkg = section["Package"]
         ver = section["Version"]
 
+        arch = section.get("Architecture", self._arches[0])
+        if arch not in self._arches + ["all"]:
+            # Skip unknown arches
+            return
+        if arch != self._arches[0]:
+            if section.get("Multi-Arch") == "same":
+                pkg += ":" + arch
+            else:
+                # Skip non-M-A-same foreign packages
+                return
+
         # If we have already seen an equal or newer version of this package,
         # then skip this section.
         if pkg in self._packages:
@@ -465,8 +488,7 @@ class Germinator(object):
 
         self._packages[pkg]["Essential"] = section.get("Essential", "")
 
-        self._packages[pkg]["Architecture"] = \
-            section.get("Architecture", self._arch)
+        self._packages[pkg]["Architecture"] = arch
 
         for field in "Pre-Depends", "Depends", "Recommends", "Built-Using":
             value = section.get(field, "")
@@ -483,7 +505,7 @@ class Germinator(object):
             value = section.get(field, "0")
             self._packages[pkg][field] = int(value)
 
-        src = section.get("Source", pkg)
+        src = section.get("Source", section["Package"])
         idx = src.find("(")
         if idx != -1:
             src = src[:idx].strip()
@@ -494,6 +516,38 @@ class Germinator(object):
 
         self._packages[pkg]["Multi-Arch"] = section.get("Multi-Arch", "none")
 
+        # if Multi-arch allowed/foreign, and main Architecture add
+        # :foreign provides
+        if arch in [self._arches[0], "all"]:
+            if self._packages[pkg]["Multi-Arch"] in ["allowed", "foreign"]:
+                for a in self._arches[1:]:
+                    self._packages[pkg]["Provides"].append(
+                        [("%s:%s" % (pkg, a), '', '')])
+
+        # if Package is from supplementary-arches qualify dependencies
+        # as follows:
+        # no tag, :native are changed to :arch
+        # :any is dropped, meaning use host arch
+        if arch in self._arches[1:]:
+            for field in "Pre-Depends", "Depends", "Recommends", "Provides":
+                new_deps = []
+                for dep in self._packages[pkg][field]:
+                    new_dep = []
+                    for alt in dep:
+                        base = alt[0].split(':', 1)
+                        if len(base) > 1:
+                            tag = base[1]
+                        else:
+                            tag = arch
+                        if tag == "native":
+                            tag = arch
+                        if tag == "any":
+                            tag = ''
+                        new_dep.append(
+                            ('%s:%s' % (alt[0], tag), alt[1], alt[2]))
+                    new_deps.append(new_dep)
+                self._packages[pkg][field] = new_deps
+
         self._packages[pkg]["Kernel-Version"] = section.get(
             "Kernel-Version", "")
 
@@ -843,11 +897,21 @@ class Germinator(object):
                 if startarchspec != -1:
                     archspec = pkg[startarchspec + 1:-1].split()
                     pkg = pkg[:startarchspec - 1]
-                    posarch = [x for x in archspec if not x.startswith('!')]
-                    negarch = [x[1:] for x in archspec if x.startswith('!')]
-                    if self._arch in negarch:
-                        continue
-                    if posarch and self._arch not in posarch:
+                    # include only these arches
+                    posarch = set(
+                        [x for x in archspec if not x.startswith('!')])
+                    # exclude these arches
+                    negarch = set(
+                        [x[1:] for x in archspec if x.startswith('!')])
+                    # all available arches for this germinator
+                    archset = set(self._arches)
+                    # remove blacklisted
+                    archset = archset - negarch
+                    # if there is a whitelist, find common only
+                    if posarch:
+                        archset = archset & posarch
+                    # if we left without any usable arches, skip this
+                    if not archset:
                         continue
 
             pkg = pkg.split()[0]
@@ -962,8 +1026,8 @@ class Germinator(object):
 
             else:
                 # No idea
-                _logger.error("Unknown package %s for seed %s on arch %s",
-                              pkg, seed, self._arch)
+                _logger.error("Unknown package %s for seed %s on arches %s",
+                              pkg, seed, self._arches)
 
         for pkg in seedsnaps:
             seed._snaps.add(pkg)
@@ -1143,7 +1207,7 @@ class Germinator(object):
         within any seed.
 
         """
-        if ":" in depend:
+        if depend.endswith(":any") or depend.endswith(":native"):
             depname, depqual = depend.split(":", 1)
         else:
             depname = depend
@@ -1184,7 +1248,11 @@ class Germinator(object):
                                    seed, build_depend,
                                    always_include_virtual=False):
         """Get the possible candidates for satisfying a dependency."""
-        plain_depname = depname.split(":", 1)[0]
+        depname_split = depname.split(":", 1)
+        plain_depname = depname_split[0]
+        if depname_split[-1] in self._arches[1:]:
+            plain_depname = depname
+
         candidates = []
         if plain_depname in self._packages:
             candidates.append(
@@ -1193,7 +1261,11 @@ class Germinator(object):
             candidates.extend(self._provides[plain_depname].items())
 
         for candpkg, candver in candidates:
+            candpkg_split = candpkg.split(":", 1)
             plain_candpkg = candpkg.split(":", 1)[0]
+            if candpkg_split[-1] in self._arches[1:]:
+                plain_candpkg = candpkg
+
             if plain_candpkg not in self._packages:
                 continue
             allowed = False
@@ -1453,9 +1525,9 @@ class Germinator(object):
                 desc = "recommendation"
             else:
                 desc = "dependency"
-            _logger.error("Unknown %s %s by %s on arch %s", desc,
+            _logger.error("Unknown %s %s by %s on arches %s", desc,
                           self._unparse_dependency(depname, depver, deptype),
-                          pkg, self._arch)
+                          pkg, self._arches)
             return False
 
         if dependlist:
diff --git a/germinate/scripts/germinate_main.py b/germinate/scripts/germinate_main.py
index 329c1e6..3b37b26 100644
--- a/germinate/scripts/germinate_main.py
+++ b/germinate/scripts/germinate_main.py
@@ -82,9 +82,9 @@ def parse_options(argv):
     parser.add_option('-d', '--dist', dest='dist',
                       default=germinate.defaults.dist,
                       help='operate on distribution DIST (default: %default)')
-    parser.add_option('-a', '--arch', dest='arch',
-                      default=germinate.defaults.arch,
-                      help='operate on architecture ARCH (default: %default)')
+    parser.add_option('-a', '--arch', dest='arches',
+                      action='append',
+                      help='operate on architecture ARCH (default: i386)')
     parser.add_option('-c', '--components', dest='components',
                       default='main,restricted', metavar='COMPS',
                       help='operate on components COMPS (default: %default)')
@@ -134,6 +134,9 @@ def parse_options(argv):
     else:
         options.seed_packages = options.seed_packages.split(',')
 
+    if not options.arches:
+        options.arches = [germinate.defaults.arch, ]
+
     return options
 
 
@@ -145,11 +148,11 @@ def main(argv):
     else:
         germinate_logging(logging.INFO)
 
-    g = Germinator(options.arch)
+    g = Germinator(options.arches)
     g._always_follow_build_depends = options.always_follow_build_depends
 
     archive = germinate.archive.TagFile(
-        options.dist, options.components, options.arch,
+        options.dist, options.components, options.arches,
         options.mirrors, source_mirrors=options.source_mirrors,
         installer_packages=options.installer, cleanup=options.cleanup)
     g.parse_archive(archive)
diff --git a/germinate/tests/test_germinator.py b/germinate/tests/test_germinator.py
index aaab106..a848cae 100644
--- a/germinate/tests/test_germinator.py
+++ b/germinate/tests/test_germinator.py
@@ -323,6 +323,71 @@ class TestGerminator(TestCase):
             shutil.rmtree(self.archive_dir)
             shutil.rmtree(self.seeds_dir)
 
+    def test_depends_foreign_multiarch(self):
+        """Test that amd64 packages can recommend multi-arched :i386
+        packages, although it's unspecified in the MultiarchSpec, it
+        is possible with apt and is in use for Nvidia drivers
+
+        """
+        for arches in (["amd64"], ["amd64", "i386"]):
+            self.addSource("focal", "main", "hello", "1.0-1", ["hello"])
+            self.addSource("focal", "main", "glibc", "2.36", ["libc6"])
+            self.addSource("focal", "restricted", "nvidia-graphics", "1.0-1",
+                           ["libnvidia-compute"])
+            self.addPackage("focal", "main", "amd64", "hello", "1.0-1",
+                            fields={
+                                "Architecture": "amd64",
+                                "Depends": "libnvidia-compute",
+                                "Recommends": "libnvidia-compute:i386",
+                            })
+            self.addPackage("focal", "main", "i386", "hello-32bit", "1.0-1",
+                            fields={
+                                "Architecture": "i386",
+                                "Depends": "libc6",
+                            })
+            for arch in "amd64", "i386":
+                self.addPackage("focal", "main", arch, "libc6", "2.36",
+                                fields={
+                                    "Architecture": arch,
+                                    "Multi-Arch": "same"
+                                })
+                self.addPackage("focal", "restricted", arch,
+                                "libnvidia-compute",
+                                "1.0-1", fields={
+                                    "Architecture": arch,
+                                    "Depends": "libc6",
+                                    "Multi-Arch": "same",
+                                })
+
+            branch = "collection.focal"
+            self.addStructureLine(branch, "feature follow-recommends")
+            self.addSeed(branch, "base")
+            self.addSeedPackage(branch, "base", "hello")
+            # This package is seeded, but should be excluded
+            # only M-A:same foreign arch packages are supported atm
+            self.addSeedPackage(branch, "base", "hello-32bit")
+            germinator = Germinator(arches)
+            for arch in "i386", "amd64":
+                for component in "main", "restricted":
+                    archive = TagFile(
+                        "focal", component, arch,
+                        "file://%s" % self.archive_dir)
+                    germinator.parse_archive(archive)
+            structure = self.openSeedStructure(branch)
+            germinator.plant_seeds(structure)
+            germinator.grow(structure)
+            expected = set(["hello",
+                            "libc6",
+                            "libnvidia-compute"])
+            if 'i386' in arches:
+                expected.add("libc6:i386")
+                expected.add("libnvidia-compute:i386")
+            self.assertEqual(
+                expected, set(germinator.get_all(structure)))
+
+            shutil.rmtree(self.archive_dir)
+            shutil.rmtree(self.seeds_dir)
+
     def test_build_depends_all(self):
         """Confirm build-depends of arch: all packages are recursed when
         follow-build-depends-all is set, and not when
@@ -349,7 +414,6 @@ class TestGerminator(TestCase):
             structure = self.openSeedStructure(branch)
             germinator.plant_seeds(structure)
             germinator.grow(structure)
-
             expected = set()
             if sense == "":
                 expected.add("gettext")