Merge ~mwhudson/germinate:apt-config-archive into germinate:master

[Michael Hudson-Doyle]

add a way to use an apt config file to find the indices

This was surprisingly straightforward. The man page changes are a total guess!

[Julian Andres Klode (juliank)]

From the apt side I feel like you'd want to directly pass a sources.list and otherwise initialise stuff with Dir::Etc set to /dev/null, that is create a config on the fly that sets

Dir::Etc::sourceslist "argument";
Dir::Etc "/dev/null";

From the germinate side of things, I'd like to hear more about the use case. Germinate is quite critical and used in very weird places and it's an explicit design decision to not use any apt bits in it, so it's a bit concerning.

review: Needs Information

[Michael Hudson-Doyle (mwhudson)]

So as mentioned out of band, the motivation for this is this ubuntu-cdimage change: https://code.launchpad.net/~mwhudson/ubuntu-cdimage/+git/ubuntu-cdimage-1/+merge/457014.

Passing just the .sources file would be OK, I guess, but would mean redundantly downloading the lists and require a separate configuration of a proxy and so on. Nothing very interesting but not ideal.

I didn't realise germinate had a policy of not relying on apt. I'm not sure having germinate know the layout of apt repositories independently of apt is worth this but well. At least this change should be a noop in the case that the new argument is not used.

[Colin Watson (cjwatson)]

While it's true that germinate tries to be independent of apt, I don't think calling `apt-get indextargets` in an optional code path should be a showstopper. (It would be a problem if it were mandatory, or if it unconditionally imported python-apt, or something like that.)

[Julian Andres Klode (juliank)]

review: Needs Fixing

diff --git a/bin/germinate b/bin/germinate
index 4bc397a..5a6e9ad 100755
--- a/bin/germinate
+++ b/bin/germinate
@@ -1,4 +1,4 @@
-#! /usr/bin/env python
+#! /usr/bin/env python3
 """Expand dependencies in a list of seed packages."""
 
 # Copyright (C) 2012 Canonical Ltd.
diff --git a/debian/changelog b/debian/changelog
index f0ffaec..f928e59 100644
--- a/debian/changelog
+++ b/debian/changelog
@@ -2,6 +2,7 @@ germinate (2.43) UNRELEASED; urgency=medium
 
   [ Michael Hudson-Doyle ]
   * Perform shallow clones of seed repos.
+  * Add a way to use an apt config file to locate the indices.
 
  -- Colin Watson <[e-mail]>  Mon, 20 Nov 2023 09:24:27 +0000
 
diff --git a/germinate/archive.py b/germinate/archive.py
index 222e97e..7532aa2 100644
--- a/germinate/archive.py
+++ b/germinate/archive.py
@@ -21,6 +21,7 @@
 import logging
 import os
 import shutil
+import subprocess
 import tempfile
 from contextlib import closing
 from urllib.parse import quote
@@ -271,3 +272,48 @@ class TagFile(Archive):
         finally:
             if self._cleanup:
                 shutil.rmtree(dirname)
+
+
+class AptArchive(Archive):
+    def __init__(self, config_path):
+        self.config_path = config_path
+
+    def _get_files(self, identifier):
+        env = dict(os.environ, APT_CONFIG=self.config_path)
+        cmd = [
+            "apt-get",
+            "indextargets",
+            "--format",
+            "$(FILENAME)",
+            "Identifier: " + identifier,
+        ]
+        cp = subprocess.run(
+            cmd,
+            env=env,
+            capture_output=True,
+            encoding="utf-8",
+            check=True,
+        )
+        return cp.stdout.splitlines()
+
+    def sections(self):
+        for filename in self._get_files("Packages"):
+            _progress(
+                "processing packages file %s ...",
+                filename,
+            )
+            with open(
+                filename, encoding="UTF-8", errors="replace"
+            ) as file_obj:
+                for section in apt_pkg.TagFile(file_obj):
+                    yield (IndexType.PACKAGES, section)
+        for filename in self._get_files("Sources"):
+            _progress(
+                "processing sources file %s ...",
+                filename,
+            )
+            with open(
+                filename, encoding="UTF-8", errors="replace"
+            ) as file_obj:
+                for section in apt_pkg.TagFile(file_obj):
+                    yield (IndexType.SOURCES, section)
diff --git a/germinate/scripts/germinate_main.py b/germinate/scripts/germinate_main.py
index e6b9fa2..c6be0af 100644
--- a/germinate/scripts/germinate_main.py
+++ b/germinate/scripts/germinate_main.py
@@ -125,6 +125,7 @@ def parse_options(argv):
         metavar="COMPS",
         help="operate on components COMPS (default: %default)",
     )
+    parser.add_option("-C", "--apt-config", dest="apt_config")
     parser.add_option(
         "--vcs",
         dest="vcs",
@@ -211,15 +212,18 @@ def main(argv):
     g = Germinator(options.arch)
     g._always_follow_build_depends = options.always_follow_build_depends
 
-    archive = germinate.archive.TagFile(
-        options.dist,
-        options.components,
-        options.arch,
-        options.mirrors,
-        source_mirrors=options.source_mirrors,
-        installer_packages=options.installer,
-        cleanup=options.cleanup,
-    )
+    if options.apt_config:
+        archive = germinate.archive.AptArchive(options.apt_config)
+    else:
+        archive = germinate.archive.TagFile(
+            options.dist,
+            options.components,
+            options.arch,
+            options.mirrors,
+            source_mirrors=options.source_mirrors,
+            installer_packages=options.installer,
+            cleanup=options.cleanup,
+        )
     g.parse_archive(archive)
 
     if os.path.isfile("hints"):
diff --git a/man/germinate.1 b/man/germinate.1
index f1adc23..25e44d9 100644
--- a/man/germinate.1
+++ b/man/germinate.1
@@ -303,6 +303,16 @@ The default is
 Operate on the specified components.
 The default is
 .Li main .
+.It Xo Fl C ,
+.Fl Fl apt-config Ar conf
+.Xc
+Use apt with APT_CONFIG set to the supplied value to find the
+package and source indices to operate on. Overrides any values passed
+for
+.Fl Fl mirror ,
+.Fl Fl source-mirror ,
+.Fl Fl dist , or
+.Fl Fl components .
 .It Fl Fl vcs Ns = Ns Brq Li auto Ns | Ns Li bzr Ns | Ns Li git
 Check out seeds from a version control system rather than fetching them
 directly from a URL.
diff --git a/setup.cfg b/setup.cfg
index 1a20a71..ce0411d 100644
--- a/setup.cfg
+++ b/setup.cfg
@@ -1,6 +1,6 @@
 [metadata]
 name = germinate
-version = 2.41
+version = 2.43
 description = Expand dependencies in a list of seed packages
 author = Scott James Remnant
 author_email = [e-mail]