Upgrade failed - unauthenticated package (module-init-tools)

This bug has been reported on the Ubuntu ISO testing tracker.

A list of all reports related to this bug can be found here:
http://iso.qa.ubuntu.com/qatracker/reports/bugs/1550741

Status changed to 'Confirmed' because the bug affects multiple users.

Circumvention, copied from Bug #1550922:

Set up a config file in /etc/update-manager/release-upgrades.d/*.cfg.

According to http://askubuntu.com/questions/425355/error-authenticating-some-packages-while-upgrade one needs to create a file /etc/update-manager/release-upgrades.d/unauth.cfg with the contents:

[Distro]
AllowUnauthenticated=yes

Tested and works.

In order to check for the above condition, one need a precursor script (as root) before the update proper, which goes along these lines:

#! /bin/sh
if [[ -n $(dpkg --get-selections | grep module-init-tools | cut -f1) ]]
then
    cat <<- EOF >/etc/update-manager/release-upgrades.d/unauth.cfg
[Distro]
AllowUnauthenticated=yes
EOF
fi

Workaround of post #4 works.
Upgrading going on now... Hope for the best! :-)

Adding some extra debugging (logging.warning() calls) produces the following in /var/log/dist-upgrade/main.log:

2016-03-01 11:59:20,561 WARNING origin is: <Origin component:'main' archive:'xenial' origin:'Ubuntu' label:'Ubuntu' site:'192.168.10.7' isTrusted:True>
2016-03-01 11:59:20,561 WARNING origin is: <Origin component:'main' archive:'xenial' origin:'Ubuntu' label:'Ubuntu' site:'192.168.10.7' isTrusted:True>
2016-03-01 11:59:20,562 WARNING origin is: <Origin component:'' archive:'now' origin:'' label:'' site:'' isTrusted:False>
2016-03-01 11:59:20,562 WARNING not trusted: module-init-tools
2016-03-01 11:59:20,562 WARNING origin is: <Origin component:'main' archive:'xenial' origin:'Ubuntu' label:'Ubuntu' site:'192.168.10.7' isTrusted:True>
2016-03-01 11:59:20,562 WARNING origin is: <Origin component:'main' archive:'xenial' origin:'Ubuntu' label:'Ubuntu' site:'192.168.10.7' isTrusted:True>

I don't know what archive:'now' is though.

Status changed to 'Confirmed' because the bug affects multiple users.

Setting to critical as this is blocking Upgrades which is one of the key install methods for 16.04

I looked at this a bit more, here are some findings:
- the "module-init-tools" package is returned from cache.get_changes() *but* it not marked for install/remove/upgrade/reinstall/downgrade - funny enough its also not marked "keep" so its in some heisenstate
- module-init-tools is not installable because it has a hard depenency on libkmod2=21-1ubuntu1 but the libmod2 version in xenial is 22-1ubuntu1.
- when apts depcache.cc sees a package like module-init-tools that has a hard dependency that can't be satisfied it will reset the candidate version. I (strongly) suspect it also need to set it back into "keep" state.

So to fix this we need:
- make module-init-tools installable again
- fix apt to not go into heisenstate when it encounters a situation like this

Status changed to 'Confirmed' because the bug affects multiple users.

(as I was asked to have a look – only reviewing based on comments and code in this bug through)

I guess setting the state explicit here is okay, I wonder why the package hasn't any state through – isn't that kinda normal for a package not touched at all? I also think it is wrong that .get_changes() returns packages in "non-interesting" states as those are clearly not changes.

Basic sanity checking might be in order here to catch such issues in the future as e.g. a package from archive "now" (which is the designation for a package in the dpkg/status file) can realistically only be removed.

APT has the test-bug-735967-lib32-to-i386-unavailable testcase which produces such a situation, the heisenstate doesn't seem to interest apt through and can't be easily observed. Maybe the tests should be extended to be able to call python – the ability could be handy as a CI test in general.

I believe the fundamental issue here is that as of kmod 22-1 we are expecting to be able to drop the module-init-tools transitional, but because that has a versioned dependancy on kmod we end up in an unresolvable situation. As this package is transitional it seems most appropriate to just force it off on upgrades. We should be able to add Provides/Conflicts/Replaces on kmod to achieve this.

The module-init-tools binary package was sitting NBS in xenial; this has been resolved now by the removal of the package from the archive. So it's possible that no further changes are required to kmod, Andy.

I've just tested this now in a lxc container and we're at least not hitting the module-init-tools complain.
The system is currently running apt-get upgrade (it would previously fail prior to that)

I've also tested an upgrade of a Trusty desktop to Xenial and while the upgrade is in progress, it has gotten past the failure to calculate the upgrade. Subsequently, I'm marking the kmod task as Invalid. It'd still be good to get mvo's apt fix added in some form and make ubuntu-release-upgrader more informative as to the failure.

This bug was fixed in the package kmod - 22-1ubuntu3

---------------
kmod (22-1ubuntu3) xenial; urgency=low

  * reinstate module-init-tools transitional package. (LP: #1550741)
   - as we have versioned dependancies from the kernel to this in 14.04
     removing this package throws the apt in trusty for a loop preventing
     upgrades.
   - note that this reverts the P/C/R combo from the previous upload.

kmod (22-1ubuntu2) xenial; urgency=low

  * Provides/Conflicts/Replaces: module-init-tools to fix upgrades from
    15.10 which has a strict versioned Depends: on kmod. (LP: #1550741)

 -- Andy Whitcroft <email address hidden> Wed, 09 Mar 2016 10:31:51 +0000

This bug was fixed in the package apt - 1.2.7

---------------
apt (1.2.7) unstable; urgency=medium

  "Caesar is dead"

  [ Frans Spiesschaert ]
  * Dutch program translation update (Closes: 817060)
  * Dutch manpages translation update (Closes: 817062)

  [ Julian Andres Klode ]
  * Use native architecture instead of amd64 for build-dep-purge test
  * Do not consider SHA1 usable
  * Test that SHA1-only .diff/Index files are not used
  * test: Use SHA512 digests for GPG, reject SHA1-based signatures
  * methods/gpgv: Reject weak digest algorithms
  * apt-pkg/acquire-worker.cc: Introduce 104 Warning message
  * methods/gpgv: Warn about SHA1 (and RIPEMD-160)

  [ David Kalnischkies ]
  * require $(HASH)-Download field in .diff/Index files
  * flush line-clearing on progress stop before post-invoke (Closes: 793672)
  * enforce verify of filesize in 'apt-get source'

  [ Manuel "Venturi" Porras Peralta ]
  * Spanish apt-mark translation fix (Closes: 817999)

  [ Zhou Mo ]
  * zh_CN.po: fix translation bug. (Closes: #818177)

  [ Michael Vogt ]
  * Fix bug where the problemresolve can put a pkg into a heisenstate
    (LP: #1550741)

 -- Julian Andres Klode <email address hidden> Tue, 15 Mar 2016 19:20:18 +0100