unattended-upgrades + nvidia stack upgrade == dkms fail (package shim-signed 1.32~16.04.1+0.9+1474479173.6c180c6-1ubuntu1 failed to install/upgrade: subprocess installed post-installation script returned error exit status 1)

[Impact]

 * When Secure Boot is enabled and MOK is not set and a new dkms module is installed sim-signed asks for a Secure Boot MOK, or aborts package installation in non-interactive mode. When unattended-upgrades performed the upgrade the aborted installation leaves an unconfigured system behind that may even fail to boot. In nvdidia's special case the new module is actually just a new version of the nvidia module which should be fine to install.

 * The fix in shim-signed now handles nvidia dkms module directory renames as simple upgrades and also does not handle module removals as a reason to abort installation.

[Test Case (shim-signed)]

1. Set up Bionic system with Secure Boot enabled.

2. Install packagages to trigger MOK enrollment and enroll the key:

 apt install dkms shim-signed r8168-dkms

3. Create a new key to be enrolled again:

 rm /var/lib/shim-signed/mok/MOK.der
 update-secureboot-policy --new-key

4. Simulate module removal and test that the command returns 0:

# echo /var/lib/dkms/zzz >> /var/lib/shim-signed/dkms-list
# env DEBIAN_FRONTEND=noninteractive update-secureboot-policy --enroll-key; echo $?
Running in non-interactive mode, doing nothing.
0

5. Simulate nvidia module rname and test that the command returns 0:

# cat > /var/lib/shim-signed/dkms-list <<EOF
/var/lib/dkms
/var/lib/dkms/nvidia-1
/var/lib/dkms/r8168
EOF
# mkdir /var/lib/dkms/nvidia
# env DEBIAN_FRONTEND=noninteractive update-secureboot-policy --enroll-key; echo $?
Running in non-interactive mode, doing nothing.
0

6. Simulate addition of nvidia driver, it should still return 1:
# cat > /var/lib/shim-signed/dkms-list <<EOF
/var/lib/dkms
/var/lib/dkms/r8168
EOF

# env DEBIAN_FRONTEND=noninteractive update-secureboot-policy --enroll-key; echo $?
Running in non-interactive mode, doing nothing.
--- /var/lib/shim-signed/dkms-list 2018-10-26 19:19:19.526697542 +0200
+++ /var/lib/shim-signed/dkms-list.new 2018-10-26 19:19:19.530697604 +0200
@@ -1,2 +1,3 @@
 /var/lib/dkms
+/var/lib/dkms/nvidia
 /var/lib/dkms/r8168
1

[Test Case (unattended-upgrades)]

1. Set up a fully - or almost fully updated Bionic system.

2. Install packagages to trigger the block:
apt install dkms shim-signed r8168-dkms

3. Fake enabled secure boot:
echo "shim-signed shim/enable_secureboot boolean true" | debconf-set-selections

4. Add and enable PPA hosting updated dkms package pulling in a new dkms-like dependency:
add-apt-repository ppa:rbalint/scratch
echo 'Unattended-Upgrade::Allowed-Origins {"LP-PPA-rbalint-scratch:${distro_codename}";}' > /etc/apt/apt.conf.d/51unattended-upgrades-all

5. Observe u-u keeping back the new package:
unattended-upgrade --verbose --dry-run --debug
...
Checking: r8168-dkms ([<Origin component:'main' archive:'bionic' origin:'LP-PPA-rbalint-scratch' label:"Scratch space, don't use" site:'ppa.launchpad.net' isTrusted:True>])
pkg new-dkms-dep may trigger secure boot key prompt
sanity check failed
...

[Regression Potential (shim-signed)]

* The fix lets installation of nvidia dkms module upgrades continue and also lets dkms module removals continue when MOK is not set and those should not cause regressions themselves. In case of an implementation mistake a new module installation could go undetected and could cause the system not load a dkms module on next boot.
In practice not loading new modules rarely cause regressions, but if a module is converted from being in the kernel to a dkms module upon an upgrade this is possible.
 * I tested the module addition, removal, nvidia module upgrade and not module change cases with stubs pretending that the system is secure-boot capable an found the changed script working properly.

[Regression Potential (unattended-upgrades)]

* Since the fix is holding back packages from installation it is expected that systems that would have otherwise broke during the installation would not receive all updates. Since exact detection of the installation failure reported here does not seem possible u-u holds back more packages than it would be absolutely necessary.

* Administrators are expected to set up email notifications about the updates performed by u-u and act on held back packages.

* Since updates pulling in new packages are fairly rare especially in the -security pocket which u-u installs from by default unwanted regressions are unlikely to show up.

[Original Bug Text]

Occurred a minute after logging in

ProblemType: Package
DistroRelease: Ubuntu 16.04
Package: shim-signed 1.32~16.04.1+0.9+1474479173.6c180c6-1ubuntu1
ProcVersionSignature: Ubuntu 4.10.0-37.41~16.04.1-generic 4.10.17
Uname: Linux 4.10.0-37-generic x86_64
.proc.sys.kernel.moksbstate_disabled: 0
ApportVersion: 2.20.1-0ubuntu2.10
Architecture: amd64
Date: Tue Oct 24 11:35:53 2017
EFITables:
 Oct 24 11:33:04 paddy-laptop kernel: efi: EFI v2.40 by American Megatrends
 Oct 24 11:33:04 paddy-laptop kernel: efi: ACPI=0x78660000 ACPI 2.0=0x78660000 SMBIOS=0xf0000 SMBIOS 3.0=0xf0020 ESRT=0x79360598
 Oct 24 11:33:04 paddy-laptop kernel: esrt: Reserving ESRT space from 0x0000000079360598 to 0x00000000793605d0.
 Oct 24 11:33:04 paddy-laptop kernel: Secure boot enabled
ErrorMessage: subprocess installed post-installation script returned error exit status 1
InstallationDate: Installed on 2017-09-11 (42 days ago)
InstallationMedia: Ubuntu 16.04.3 LTS "Xenial Xerus" - Release amd64 (20170801)
RelatedPackageVersions:
 dpkg 1.18.4ubuntu1.2
 apt 1.2.24
SecureBoot: 6 0 0 0 1
SourcePackage: shim-signed
Title: package shim-signed 1.32~16.04.1+0.9+1474479173.6c180c6-1ubuntu1 failed to install/upgrade: subprocess installed post-installation script returned error exit status 1
UpgradeStatus: No upgrade log present (probably fresh install)