Merge ~ximion/germinate:wip/split-arch-all into germinate:master

[Matthias Klumpp]

Hi!

In PureOS we are using the "modern" split-arch:all archive layout, where arch:all packages are not duplicated in each arch:any Packages file, but are downloaded separately. (see https://repo.pureos.net/pureos/dists/ )
This layout is described in https://wiki.debian.org/DebianRepository/Format and may, in the far future, be also used exclusively by Debian.

Anything using APT directory supports it, but unfortunately Germinator did not. This patch changes that.
I hope it is acceptable, the code stripping the PGP signature from the InRelease file does not make me happy, but we need to read that file in order to make sure we get compatibility with split-arch:all and non-split-arch:all repositories.

Thank you for considering this change!
Cheers,
    Matthias

Unmerged commits

b711d73... by Matthias Klumpp on 2023-06-11

Implement support for repositories using split arch:all

This change implements support for modern APT repository layouts that
have arch:all packages split into a separate Packages file, and not
merged with the arch:any Packages file.

This is described in
https://wiki.debian.org/DebianRepository/Format#No-Support-for-Architecture-all

test:0 (build)

1 → 1 of 1 result

First • Previous • Next • Last

diff --git a/germinate/archive.py b/germinate/archive.py
index efb5e72..e45f79c 100644
--- a/germinate/archive.py
+++ b/germinate/archive.py
@@ -108,6 +108,51 @@ class TagFile(Archive):
             self._source_mirrors = mirrors
         self._cleanup = cleanup
 
+    def _uses_split_arch_all(self, dist):
+        """Check if the archive supports & defaults to split arch:all.
+
+        See https://wiki.debian.org/DebianRepository/Format
+        for details."""
+
+        for mirror in self._mirrors:
+            url = (mirror + "dists/" + dist + "/InRelease")
+            req = Request(url)
+
+            if get_request_type(req) == "file":
+                if not os.path.exists(url):
+                    # no Inrelease file found, assume split arch:all
+                    # is not supported.
+                    return False
+
+            with closing(urlopen(req)) as url_f:
+                text_lines = []
+                parse_state = -1
+                for line in url_f:
+                    if parse_state == -1:
+                        if line.startswith(b"-----"):
+                            parse_state = 0
+                            continue
+                        else:
+                            parse_state = 1
+                    if parse_state < 1 and not line.strip():
+                        parse_state = 1
+                        continue
+                    if line.startswith(b"-----"):
+                        break
+                    if parse_state == 1:
+                        text_lines.append(str(line, "utf-8"))
+
+                section = apt_pkg.TagSection(''.join(text_lines))
+                if "all" in section["Architectures"].split(" "):
+                    no_archall_support = section.get(
+                        "No-Support-for-Architecture-all", "").split(" ")
+                    if "Packages" in no_archall_support:
+                        return False
+                    else:
+                        return True
+                    return False
+        return False
+
     def _open_tag_files(self, mirrors, dirname, tagfile_type,
                         dist, component, ftppath):
         def _open_tag_file(mirror, suffix):
@@ -220,10 +265,18 @@ class TagFile(Archive):
 
         try:
             for dist in self._dists:
+                uses_split_arch_all = self._uses_split_arch_all(dist)
+                if uses_split_arch_all:
+                    print("[note] Suite %s uses split arch:all layout." % dist)
+
                 for component in self._components:
                     packages = self._open_tag_files(
                         self._mirrors, dirname, "Packages", dist, component,
                         "binary-" + self._arch + "/Packages")
+                    if uses_split_arch_all:
+                        packages.extend(self._open_tag_files(
+                            self._mirrors, dirname, "Packages_all", dist,
+                            component, "binary-all" + "/Packages"))
                     for tag_file in packages:
                         try:
                             for section in apt_pkg.TagFile(tag_file):
@@ -249,6 +302,12 @@ class TagFile(Archive):
                                 dist, component,
                                 "debian-installer/binary-" + self._arch +
                                 "/Packages")
+                            if uses_split_arch_all:
+                                instpackages.extend(self._open_tag_files(
+                                    self._mirrors, dirname,
+                                    "InstallerPackages_all", dist, component,
+                                    "debian-installer/binary-all" +
+                                    "/Packages"))
                         except IOError:
                             # can live without these
                             _progress("Missing installer Packages file for %s "