Merge ~ogayot/curtin:apt-pinning into curtin:master

[Olivier Gayot]

Implement support for APT preferences in apt-config

Implement support for APT preferences in apt-config

apt-config now supports a set of APT preferences (i.e. pinning rules) as in the following example:

  apt:
    preferences:
      - {package: "python3-*", pin: "origin *ubuntu.com*", pin-priority: 200}
      - {package: "python-*", pin: "origin *ubuntu.com*", pin-priority: -1}

These preferences are deployed under <target>/etc/apt/preferences.d/90-curtin.pref using the format specified in apt_preferences(5).

  Package: python3-*
  Pin: origin *ubuntu.com*
  Pin-Priority: 200

  Package: python-*
  Pin: origin *ubuntu.com*
  Pin-Priority: -1

If no preferences are configured, we drop the file 90-curtin.pref if it exists.

[Server Team CI bot (server-team-bot)]

FAILED: Continuous integration, rev:fd826f643aea700409e517a18577131d4800bc72

No commit message was specified in the merge proposal. Click on the following link and set the commit message (if you want jenkins to rebuild you need to trigger it yourself):
https://code.launchpad.net/~ogayot/curtin/+git/curtin/+merge/413788/+edit-commit-message

https://jenkins.ubuntu.com/server/job/curtin-ci/220/
Executed test runs:
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-amd64/220/
    FAILURE: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-arm64/220/
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-ppc64el/220/
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-s390x/220/

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/curtin-ci/220//rebuild

review: Needs Fixing (continuous-integration)

[Dan Bungert (dbungert)]

review: Approve

[Server Team CI bot (server-team-bot)]

FAILED: Continuous integration, rev:4a5bede1b0d8b09817c42c52b0027a2c4dbd4fa2

No commit message was specified in the merge proposal. Click on the following link and set the commit message (if you want jenkins to rebuild you need to trigger it yourself):
https://code.launchpad.net/~ogayot/curtin/+git/curtin/+merge/413788/+edit-commit-message

https://jenkins.ubuntu.com/server/job/curtin-ci/223/
Executed test runs:
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-amd64/223/
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-arm64/223/
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-ppc64el/223/
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-s390x/223/

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/curtin-ci/223//rebuild

review: Needs Fixing (continuous-integration)

[Server Team CI bot (server-team-bot)]

review: Approve (continuous-integration)

diff --git a/curtin/commands/apt_config.py b/curtin/commands/apt_config.py
index 9ea2d30..be4a039 100644
--- a/curtin/commands/apt_config.py
+++ b/curtin/commands/apt_config.py
@@ -28,6 +28,9 @@ APT_LISTS = "/var/lib/apt/lists"
 APT_CONFIG_FN = "/etc/apt/apt.conf.d/94curtin-config"
 APT_PROXY_FN = "/etc/apt/apt.conf.d/90curtin-aptproxy"
 
+# Files to store pinning information
+APT_PREFERENCES_FN = "/etc/apt/preferences.d/90curtin.pref"
+
 # Default keyserver to use
 DEFAULT_KEYSERVER = "keyserver.ubuntu.com"
 
@@ -81,6 +84,11 @@ def handle_apt(cfg, target=None):
     except (IOError, OSError):
         LOG.exception("Failed to apply proxy or apt config info:")
 
+    try:
+        apply_apt_preferences(cfg, target + APT_PREFERENCES_FN)
+    except (IOError, OSError):
+        LOG.exception("Failed to apply apt preferences.")
+
     # Process 'apt_source -> sources {dict}'
     if 'sources' in cfg:
         params = mirrors
@@ -595,6 +603,36 @@ def apply_apt_proxy_config(cfg, proxy_fname, config_fname):
         LOG.debug("no apt config configured, removed %s", config_fname)
 
 
+def preference_to_str(preference):
+    """ Return a textual representation of a given preference as specified in
+    apt_preferences(5).
+    """
+
+    return """\
+Package: {package}
+Pin: {pin}
+Pin-Priority: {pin_priority}
+""".format(package=preference["package"],
+           pin=preference["pin"],
+           pin_priority=preference["pin-priority"])
+
+
+def apply_apt_preferences(cfg, pref_fname):
+    """ Apply apt preferences if any is provided.
+    """
+
+    prefs = cfg.get("preferences")
+    if not prefs:
+        if os.path.isfile(pref_fname):
+            util.del_file(pref_fname)
+            LOG.debug("no apt preferences configured, removed %s", pref_fname)
+        return
+    prefs_as_strings = [preference_to_str(pref) for pref in prefs]
+    print(prefs_as_strings)
+    LOG.debug("write apt preferences info to %s.", pref_fname)
+    util.write_file(pref_fname, "\n".join(prefs_as_strings))
+
+
 def apt_command(args):
     """ Main entry point for curtin apt-config standalone command
         This does not read the global config as handled by curthooks, but
diff --git a/doc/topics/apt_source.rst b/doc/topics/apt_source.rst
index cf0f8bd..924ee80 100644
--- a/doc/topics/apt_source.rst
+++ b/doc/topics/apt_source.rst
@@ -31,6 +31,8 @@ Features
 
   - add arbitrary apt.conf settings
 
+  - add arbitrary apt preferences
+
   - provide debconf configurations
 
   - disabling suites (=pockets)
diff --git a/examples/apt-source.yaml b/examples/apt-source.yaml
index f0f7108..e9543ae 100644
--- a/examples/apt-source.yaml
+++ b/examples/apt-source.yaml
@@ -152,6 +152,18 @@ apt:
   # The following example is also the builtin default if nothing is specified
   add_apt_repo_match: '^[\w-]+:\w'
 
+  # 1.9 preferences
+  #
+  # Any apt preferences that will be made available to apt
+  # see the APT_PREFERENCES(5) man page for details about what can be specified
+  preferences:
+    - package: python3-*
+      pin: origin *ubuntu.com*
+      pin-priority: 200
+    - package: python-*
+      pin: origin *ubuntu.com*
+      pin-priority: -1
+
 
   ##############################################################################
   # Section 2: source list entries
diff --git a/tests/unittests/test_apt_source.py b/tests/unittests/test_apt_source.py
index 48fb820..267711f 100644
--- a/tests/unittests/test_apt_source.py
+++ b/tests/unittests/test_apt_source.py
@@ -572,6 +572,55 @@ class TestAptSourceConfig(CiTestCase):
                                     'Acquire::ftp::Proxy "foobar3";\n'
                                     'Acquire::https::Proxy "foobar4";\n'))
 
+    def test_preference_to_str(self):
+        """ test_preference_to_str - Test converting a preference dict to
+        textual representation.
+        """
+        preference = {
+            "package": "*",
+            "pin": "release a=unstable",
+            "pin-priority": 50,
+        }
+
+        expected = """\
+Package: *
+Pin: release a=unstable
+Pin-Priority: 50
+"""
+        self.assertEqual(expected, apt_config.preference_to_str(preference))
+
+    @staticmethod
+    def test_apply_apt_preferences():
+        """ test_apply_apt_preferences - Test apt preferences configuration
+        """
+        cfg = {
+            "preferences": [
+                {
+                    "package": "*",
+                    "pin": "release a=unstable",
+                    "pin-priority": 50,
+                }, {
+                    "package": "dummy-unwanted-package",
+                    "pin": "origin *ubuntu.com*",
+                    "pin-priority": -1,
+                }
+            ]
+        }
+
+        expected_content = """\
+Package: *
+Pin: release a=unstable
+Pin-Priority: 50
+
+Package: dummy-unwanted-package
+Pin: origin *ubuntu.com*
+Pin-Priority: -1
+"""
+        with mock.patch.object(util, "write_file") as mockobj:
+            apt_config.apply_apt_preferences(cfg, "preferencesfn")
+
+        mockobj.assert_called_with("preferencesfn", expected_content)
+
     def test_mirror(self):
         """test_mirror - Test defining a mirror"""
         pmir = "http://us.archive.ubuntu.com/ubuntu/"