Merge ~chad.smith/cloud-init:bug/1840080-ubuntu-drivers-emit-latelink-v2 into cloud-init:master

[Chad Smith]

ubuntu-drivers: call db_x_loadtemplatefile to accept NVIDIA EULA

Emit a script allowing cloud-init to set linux/nvidia/latelink
debconf selection to true. This avoids having to call
debconf-set-selections and allows cloud-init to pre-confgure
linux-restricted-modules to link NVIDIA drivers to the running kernel.

Cloud-init loads this debconf template and sets the value to true in the
debconf database by sourcing debconf's /usr/share/debconf/confmodule and
uses db_x_loadtemplatefile to register cloud-init's setting for
linux/nvidia/latelink.

LP: #1840080

[Server Team CI bot (server-team-bot)]

FAILED: Continuous integration, rev:044f9d692ebca82186b9159a69e75816ad99f998
https://jenkins.ubuntu.com/server/job/cloud-init-ci/1060/
Executed test runs:
    SUCCESS: Checkout
    FAILED: Unit & Style Tests

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/1060//rebuild

review: Needs Fixing (continuous-integration)

[Ryan Harper (raharper)]

some inline comments/suggestions.

[Server Team CI bot (server-team-bot)]

FAILED: Continuous integration, rev:71fcd42be6330033bcb731080053a885786f2130
https://jenkins.ubuntu.com/server/job/cloud-init-ci/1061/
Executed test runs:
    SUCCESS: Checkout
    FAILED: Unit & Style Tests

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/1061//rebuild

review: Needs Fixing (continuous-integration)

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:48733aa42ac30013db4ce3daa7643ee941b51b63
https://jenkins.ubuntu.com/server/job/cloud-init-ci/1062/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Unit & Style Tests
    SUCCESS: Ubuntu LTS: Build
    SUCCESS: Ubuntu LTS: Integration
    IN_PROGRESS: Declarative: Post Actions

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/1062//rebuild

review: Approve (continuous-integration)

[Server Team CI bot (server-team-bot)]

FAILED: Continuous integration, rev:cdd846983dc8d3dca53d3c37d69c0c967787a7ae
https://jenkins.ubuntu.com/server/job/cloud-init-ci/1063/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Unit & Style Tests
    SUCCESS: Ubuntu LTS: Build
    FAILED: Ubuntu LTS: Integration

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/1063//rebuild

review: Needs Fixing (continuous-integration)

[Chad Smith (chad.smith)]

Just validated on Eoan that this package properly sets up linux/nvidia/latelink true via debconf-get-selections.

The ubuntu-drivers-common package fixes has not yet published to Eoan. So, end-to-end validation will not be possible until an updated ubuntu-drivers-common and linux-modules-nvidia* is published.

As it is, this behavior paves the way for the ubuntu-drivers-common/linux-modules-nvidia package features.

# cloud-init properly registers and sets linux/nvidia/latelink : true
ubuntu@ip-172-31-30-68:~$ sudo debconf-get-selections | egrep 'cloud-init|late'
cloud-init linux/nvidia/latelink boolean true
cloud-init cloud-init/datasources multiselect NoCloud, ConfigDrive, OpenNebula, DigitalOcean, Azure, AltCloud, OVF, MAAS, GCE, OpenStack, CloudSigma, SmartOS, Bigstep, Scaleway, AliYun, Ec2, CloudStack, Hetzner, IBMCloud, None

# ubuntu-drivers-common was installed, but no linux-modules-nvidia-* yet
dpkg -l | egrep 'cloud-init|nvidia|ubuntu-drivers'
ii cloud-init 19.2-21-gcdd84698-1~bddeb all Init scripts for cloud instances
ii cloud-initramfs-copymods 0.44ubuntu1 all copy initramfs modules into root filesystem for later use
ii cloud-initramfs-dyn-netconf 0.44ubuntu1 all write a network interface file in /run for BOOTIF
ii libnvidia-cfg1-430:amd64 430.40-0ubuntu2 amd64 NVIDIA binary OpenGL/GLX configuration library
ii libnvidia-compute-430:amd64 430.40-0ubuntu2 amd64 NVIDIA libcompute package
ii nvidia-compute-utils-430 430.40-0ubuntu2 amd64 NVIDIA compute utilities
ii nvidia-headless-no-dkms-430 430.40-0ubuntu2 amd64 NVIDIA headless metapackage - no DKMS
ii nvidia-kernel-source-430 430.40-0ubuntu2 amd64 NVIDIA kernel source package
ii ubuntu-drivers-common 1:0.7.5 amd64 Detect and install additional Ubuntu driver packages

### If we manually install linux-modules-nvidia-430-generic, latelink appears set true
ubuntu@ip-172-31-16-66:~$ sudo debconf-get-selections | grep late
cloud-init linux/nvidia/latelink boolean true
linux-modules-nvidia-430-5.2.0-13-generic linux/nvidia/latelink booleantrue

# from dpkg -l on eoan-proposed
ii linux-modules-nvidia-430-5.2.0-13-generic 5.2.0-13.14 amd64 Linux kernel nvidia modules for version 5.2.0-13

[Ryan Harper (raharper)]

Implementation looks great. One suggestion on the template description.

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:cdd846983dc8d3dca53d3c37d69c0c967787a7ae
https://jenkins.ubuntu.com/server/job/cloud-init-ci/1064/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Unit & Style Tests
    SUCCESS: Ubuntu LTS: Build
    SUCCESS: Ubuntu LTS: Integration
    IN_PROGRESS: Declarative: Post Actions

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/1064//rebuild

review: Approve (continuous-integration)

[Server Team CI bot (server-team-bot)]

FAILED: Continuous integration, rev:8da37261c56c583b1631cbae9910a100c7894c6e
https://jenkins.ubuntu.com/server/job/cloud-init-ci/1065/
Executed test runs:
    SUCCESS: Checkout
    FAILED: Unit & Style Tests

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/1065//rebuild

review: Needs Fixing (continuous-integration)

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:19dbf5e1449cfa59b9f8fbfb056d182516543eb1
https://jenkins.ubuntu.com/server/job/cloud-init-ci/1066/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Unit & Style Tests
    SUCCESS: Ubuntu LTS: Build
    SUCCESS: Ubuntu LTS: Integration
    IN_PROGRESS: Declarative: Post Actions

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/1066//rebuild

review: Approve (continuous-integration)

[Server Team CI bot (server-team-bot)]

Autolanding: FAILED
More details in the following jenkins job:
https://jenkins.ubuntu.com/server/job/cloud-init-autoland-test/294/
Executed test runs:
    FAILED: Checkout

review: Needs Fixing (continuous-integration)

[Server Team CI bot (server-team-bot)]

Autolanding: FAILED
More details in the following jenkins job:
https://jenkins.ubuntu.com/server/job/cloud-init-autoland-test/295/
Executed test runs:
    FAILED: Checkout

review: Needs Fixing (continuous-integration)

[Server Team CI bot (server-team-bot)]

Autolanding: FAILED
More details in the following jenkins job:
https://jenkins.ubuntu.com/server/job/cloud-init-autoland-test/296/
Executed test runs:
    FAILED: Checkout

review: Needs Fixing (continuous-integration)

[Server Team CI bot (server-team-bot)]

review: Approve (continuous-integration)

[Server Team CI bot (server-team-bot)]

Autolanding: FAILED
Unapproved changes made after approval.
https://jenkins.ubuntu.com/server/job/cloud-init-autoland-test/297/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Unit & Style Tests
    SUCCESS: Ubuntu LTS: Build
    SUCCESS: Ubuntu LTS: Integration
    IN_PROGRESS: Declarative: Post Actions

review: Needs Fixing (continuous-integration)

[Server Team CI bot (server-team-bot)]

review: Approve (continuous-integration)

diff --git a/cloudinit/config/cc_ubuntu_drivers.py b/cloudinit/config/cc_ubuntu_drivers.py
index 4da34ee..297451d 100644
--- a/cloudinit/config/cc_ubuntu_drivers.py
+++ b/cloudinit/config/cc_ubuntu_drivers.py
@@ -2,13 +2,14 @@
 
 """Ubuntu Drivers: Interact with third party drivers in Ubuntu."""
 
+import os
 from textwrap import dedent
 
-from cloudinit.config import cc_apt_configure
 from cloudinit.config.schema import (
     get_schema_doc, validate_cloudconfig_schema)
 from cloudinit import log as logging
 from cloudinit.settings import PER_INSTANCE
+from cloudinit import temp_utils
 from cloudinit import type_utils
 from cloudinit import util
 
@@ -65,6 +66,33 @@ OLD_UBUNTU_DRIVERS_STDERR_NEEDLE = (
 __doc__ = get_schema_doc(schema)  # Supplement python help()
 
 
+# Use a debconf template to configure a global debconf variable
+# (linux/nvidia/latelink) setting this to "true" allows the
+# 'linux-restricted-modules' deb to accept the NVIDIA EULA and the package
+# will automatically link the drivers to the running kernel.
+
+# EOL_XENIAL: can then drop this script and use python3-debconf which is only
+# available in Bionic and later. Can't use python3-debconf currently as it
+# isn't in Xenial and doesn't yet support X_LOADTEMPLATEFILE debconf command.
+
+NVIDIA_DEBCONF_CONTENT = """\
+Template: linux/nvidia/latelink
+Type: boolean
+Default: true
+Description: Late-link NVIDIA kernel modules?
+ Enable this to link the NVIDIA kernel modules in cloud-init and
+ make them available for use.
+"""
+
+NVIDIA_DRIVER_LATELINK_DEBCONF_SCRIPT = """\
+#!/bin/sh
+# Allow cloud-init to trigger EULA acceptance via registering a debconf
+# template to set linux/nvidia/latelink true
+. /usr/share/debconf/confmodule
+db_x_loadtemplatefile "$1" cloud-init
+"""
+
+
 def install_drivers(cfg, pkg_install_func):
     if not isinstance(cfg, dict):
         raise TypeError(
@@ -90,17 +118,27 @@ def install_drivers(cfg, pkg_install_func):
     if version_cfg:
         driver_arg += ':{}'.format(version_cfg)
 
-    LOG.debug("Installing NVIDIA drivers (%s=%s, version=%s)",
+    LOG.debug("Installing and activating NVIDIA drivers (%s=%s, version=%s)",
               cfgpath, nv_acc, version_cfg if version_cfg else 'latest')
 
-    # Setting NVIDIA latelink confirms acceptance of EULA for the package
-    # linux-restricted-modules
-    # Reference code defining debconf variable is here
-    # https://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/
-    # linux-restricted-modules/+git/eoan/tree/debian/templates/
-    # nvidia.templates.in
-    selections = b'linux-restricted-modules linux/nvidia/latelink boolean true'
-    cc_apt_configure.debconf_set_selections(selections)
+    # Register and set debconf selection linux/nvidia/latelink = true
+    tdir = temp_utils.mkdtemp(needs_exe=True)
+    debconf_file = os.path.join(tdir, 'nvidia.template')
+    debconf_script = os.path.join(tdir, 'nvidia-debconf.sh')
+    try:
+        util.write_file(debconf_file, NVIDIA_DEBCONF_CONTENT)
+        util.write_file(
+            debconf_script,
+            util.encode_text(NVIDIA_DRIVER_LATELINK_DEBCONF_SCRIPT),
+            mode=0o755)
+        util.subp([debconf_script, debconf_file])
+    except Exception as e:
+        util.logexc(
+            LOG, "Failed to register NVIDIA debconf template: %s", str(e))
+        raise
+    finally:
+        if os.path.isdir(tdir):
+            util.del_dir(tdir)
 
     try:
         util.subp(['ubuntu-drivers', 'install', '--gpgpu', driver_arg])
diff --git a/cloudinit/config/tests/test_ubuntu_drivers.py b/cloudinit/config/tests/test_ubuntu_drivers.py
index 6a763bd..4695269 100644
--- a/cloudinit/config/tests/test_ubuntu_drivers.py
+++ b/cloudinit/config/tests/test_ubuntu_drivers.py
@@ -1,6 +1,7 @@
 # This file is part of cloud-init. See LICENSE file for license information.
 
 import copy
+import os
 
 from cloudinit.tests.helpers import CiTestCase, skipUnlessJsonSchema, mock
 from cloudinit.config.schema import (
@@ -9,11 +10,27 @@ from cloudinit.config import cc_ubuntu_drivers as drivers
 from cloudinit.util import ProcessExecutionError
 
 MPATH = "cloudinit.config.cc_ubuntu_drivers."
+M_TMP_PATH = MPATH + "temp_utils.mkdtemp"
 OLD_UBUNTU_DRIVERS_ERROR_STDERR = (
     "ubuntu-drivers: error: argument <command>: invalid choice: 'install' "
     "(choose from 'list', 'autoinstall', 'devices', 'debug')\n")
 
 
+class AnyTempScriptAndDebconfFile(object):
+
+    def __init__(self, tmp_dir, debconf_file):
+        self.tmp_dir = tmp_dir
+        self.debconf_file = debconf_file
+
+    def __eq__(self, cmd):
+        if not len(cmd) == 2:
+            return False
+        script, debconf_file = cmd
+        if bool(script.startswith(self.tmp_dir) and script.endswith('.sh')):
+            return debconf_file == self.debconf_file
+        return False
+
+
 class TestUbuntuDrivers(CiTestCase):
     cfg_accepted = {'drivers': {'nvidia': {'license-accepted': True}}}
     install_gpgpu = ['ubuntu-drivers', 'install', '--gpgpu', 'nvidia']
@@ -28,20 +45,23 @@ class TestUbuntuDrivers(CiTestCase):
                 {'drivers': {'nvidia': {'license-accepted': "TRUE"}}},
                 schema=drivers.schema, strict=True)
 
-    @mock.patch(MPATH + "cc_apt_configure.debconf_set_selections")
+    @mock.patch(M_TMP_PATH)
     @mock.patch(MPATH + "util.subp", return_value=('', ''))
     @mock.patch(MPATH + "util.which", return_value=False)
     def _assert_happy_path_taken(
-            self, config, m_which, m_subp, m_debconf_set_selections):
+            self, config, m_which, m_subp, m_tmp):
         """Positive path test through handle. Package should be installed."""
+        tdir = self.tmp_dir()
+        debconf_file = os.path.join(tdir, 'nvidia.template')
+        m_tmp.return_value = tdir
         myCloud = mock.MagicMock()
         drivers.handle('ubuntu_drivers', config, myCloud, None, None)
         self.assertEqual([mock.call(['ubuntu-drivers-common'])],
                          myCloud.distro.install_packages.call_args_list)
-        self.assertEqual([mock.call(self.install_gpgpu)],
-                         m_subp.call_args_list)
-        m_debconf_set_selections.assert_called_with(
-            b'linux-restricted-modules linux/nvidia/latelink boolean true')
+        self.assertEqual(
+            [mock.call(AnyTempScriptAndDebconfFile(tdir, debconf_file)),
+             mock.call(self.install_gpgpu)],
+            m_subp.call_args_list)
 
     def test_handle_does_package_install(self):
         self._assert_happy_path_taken(self.cfg_accepted)
@@ -52,20 +72,33 @@ class TestUbuntuDrivers(CiTestCase):
             new_config['drivers']['nvidia']['license-accepted'] = true_value
             self._assert_happy_path_taken(new_config)
 
-    @mock.patch(MPATH + "cc_apt_configure.debconf_set_selections")
-    @mock.patch(MPATH + "util.subp", side_effect=ProcessExecutionError(
-        stdout='No drivers found for installation.\n', exit_code=1))
+    @mock.patch(M_TMP_PATH)
+    @mock.patch(MPATH + "util.subp")
     @mock.patch(MPATH + "util.which", return_value=False)
-    def test_handle_raises_error_if_no_drivers_found(self, m_which, m_subp, _):
+    def test_handle_raises_error_if_no_drivers_found(
+            self, m_which, m_subp, m_tmp):
         """If ubuntu-drivers doesn't install any drivers, raise an error."""
+        tdir = self.tmp_dir()
+        debconf_file = os.path.join(tdir, 'nvidia.template')
+        m_tmp.return_value = tdir
         myCloud = mock.MagicMock()
+
+        def fake_subp(cmd):
+            if cmd[0].startswith(tdir):
+                return
+            raise ProcessExecutionError(
+                stdout='No drivers found for installation.\n', exit_code=1)
+        m_subp.side_effect = fake_subp
+
         with self.assertRaises(Exception):
             drivers.handle(
                 'ubuntu_drivers', self.cfg_accepted, myCloud, None, None)
         self.assertEqual([mock.call(['ubuntu-drivers-common'])],
                          myCloud.distro.install_packages.call_args_list)
-        self.assertEqual([mock.call(self.install_gpgpu)],
-                         m_subp.call_args_list)
+        self.assertEqual(
+            [mock.call(AnyTempScriptAndDebconfFile(tdir, debconf_file)),
+             mock.call(self.install_gpgpu)],
+            m_subp.call_args_list)
         self.assertIn('ubuntu-drivers found no drivers for installation',
                       self.logs.getvalue())
 
@@ -113,19 +146,25 @@ class TestUbuntuDrivers(CiTestCase):
                       myLog.debug.call_args_list[0][0][0])
         self.assertEqual(0, m_install_drivers.call_count)
 
-    @mock.patch(MPATH + "cc_apt_configure.debconf_set_selections")
+    @mock.patch(M_TMP_PATH)
     @mock.patch(MPATH + "util.subp", return_value=('', ''))
     @mock.patch(MPATH + "util.which", return_value=True)
-    def test_install_drivers_no_install_if_present(self, m_which, m_subp, _):
+    def test_install_drivers_no_install_if_present(
+            self, m_which, m_subp, m_tmp):
         """If 'ubuntu-drivers' is present, no package install should occur."""
+        tdir = self.tmp_dir()
+        debconf_file = os.path.join(tdir, 'nvidia.template')
+        m_tmp.return_value = tdir
         pkg_install = mock.MagicMock()
         drivers.install_drivers(self.cfg_accepted['drivers'],
                                 pkg_install_func=pkg_install)
         self.assertEqual(0, pkg_install.call_count)
         self.assertEqual([mock.call('ubuntu-drivers')],
                          m_which.call_args_list)
-        self.assertEqual([mock.call(self.install_gpgpu)],
-                         m_subp.call_args_list)
+        self.assertEqual(
+            [mock.call(AnyTempScriptAndDebconfFile(tdir, debconf_file)),
+             mock.call(self.install_gpgpu)],
+            m_subp.call_args_list)
 
     def test_install_drivers_rejects_invalid_config(self):
         """install_drivers should raise TypeError if not given a config dict"""
@@ -134,21 +173,33 @@ class TestUbuntuDrivers(CiTestCase):
             drivers.install_drivers("mystring", pkg_install_func=pkg_install)
         self.assertEqual(0, pkg_install.call_count)
 
-    @mock.patch(MPATH + "cc_apt_configure.debconf_set_selections")
-    @mock.patch(MPATH + "util.subp", side_effect=ProcessExecutionError(
-        stderr=OLD_UBUNTU_DRIVERS_ERROR_STDERR, exit_code=2))
+    @mock.patch(M_TMP_PATH)
+    @mock.patch(MPATH + "util.subp")
     @mock.patch(MPATH + "util.which", return_value=False)
     def test_install_drivers_handles_old_ubuntu_drivers_gracefully(
-            self, m_which, m_subp, _):
+            self, m_which, m_subp, m_tmp):
         """Older ubuntu-drivers versions should emit message and raise error"""
+        tdir = self.tmp_dir()
+        debconf_file = os.path.join(tdir, 'nvidia.template')
+        m_tmp.return_value = tdir
         myCloud = mock.MagicMock()
+
+        def fake_subp(cmd):
+            if cmd[0].startswith(tdir):
+                return
+            raise ProcessExecutionError(
+                stderr=OLD_UBUNTU_DRIVERS_ERROR_STDERR, exit_code=2)
+        m_subp.side_effect = fake_subp
+
         with self.assertRaises(Exception):
             drivers.handle(
                 'ubuntu_drivers', self.cfg_accepted, myCloud, None, None)
         self.assertEqual([mock.call(['ubuntu-drivers-common'])],
                          myCloud.distro.install_packages.call_args_list)
-        self.assertEqual([mock.call(self.install_gpgpu)],
-                         m_subp.call_args_list)
+        self.assertEqual(
+            [mock.call(AnyTempScriptAndDebconfFile(tdir, debconf_file)),
+             mock.call(self.install_gpgpu)],
+            m_subp.call_args_list)
         self.assertIn('WARNING: the available version of ubuntu-drivers is'
                       ' too old to perform requested driver installation',
                       self.logs.getvalue())
@@ -160,17 +211,21 @@ class TestUbuntuDriversWithVersion(TestUbuntuDrivers):
         'drivers': {'nvidia': {'license-accepted': True, 'version': '123'}}}
     install_gpgpu = ['ubuntu-drivers', 'install', '--gpgpu', 'nvidia:123']
 
-    @mock.patch(MPATH + "cc_apt_configure.debconf_set_selections")
+    @mock.patch(M_TMP_PATH)
     @mock.patch(MPATH + "util.subp", return_value=('', ''))
     @mock.patch(MPATH + "util.which", return_value=False)
-    def test_version_none_uses_latest(self, m_which, m_subp, _):
+    def test_version_none_uses_latest(self, m_which, m_subp, m_tmp):
+        tdir = self.tmp_dir()
+        debconf_file = os.path.join(tdir, 'nvidia.template')
+        m_tmp.return_value = tdir
         myCloud = mock.MagicMock()
         version_none_cfg = {
             'drivers': {'nvidia': {'license-accepted': True, 'version': None}}}
         drivers.handle(
             'ubuntu_drivers', version_none_cfg, myCloud, None, None)
         self.assertEqual(
-            [mock.call(['ubuntu-drivers', 'install', '--gpgpu', 'nvidia'])],
+            [mock.call(AnyTempScriptAndDebconfFile(tdir, debconf_file)),
+             mock.call(['ubuntu-drivers', 'install', '--gpgpu', 'nvidia'])],
             m_subp.call_args_list)
 
     def test_specifying_a_version_doesnt_override_license_acceptance(self):
diff --git a/cloudinit/tests/helpers.py b/cloudinit/tests/helpers.py
index f41180f..23fddd0 100644
--- a/cloudinit/tests/helpers.py
+++ b/cloudinit/tests/helpers.py
@@ -198,7 +198,8 @@ class CiTestCase(TestCase):
                 prefix="ci-%s." % self.__class__.__name__)
         else:
             tmpd = tempfile.mkdtemp(dir=dir)
-        self.addCleanup(functools.partial(shutil.rmtree, tmpd))
+        self.addCleanup(
+            functools.partial(shutil.rmtree, tmpd, ignore_errors=True))
         return tmpd
 
     def tmp_path(self, path, dir=None):