unattended-upgrade-shutdown hangs when /var is a separate filesystem

I have also tried to let the systemd unit file for this service WantedBy the umount.target.
According to the systemd documentation that should make it run before the filesystems
are unmounted, but apparently that doesn't work. systemctl list-dependencies still shows
that the unattended-upgrade.service reverse depends on the shutdown.target.

Status changed to 'Confirmed' because the bug affects multiple users.

I have same problem. I need to kill unattended-upgrades service manually to shutdown machine. This problem comes in 2017 year.

System is Ubuntu 16.04 x86_64.

I have narrowed down. This bug was introduced by the fix in #1618900.
That tries to fix the problem of the the unattended-upgrades.service
by letting it depend on network.target and local-fs.target.
However, that doesn't change any of the dependencies, it only makes
the race window larger.

After some experiments, the correct dependencies in the unit file appear to be:

Before=remote-fs.target local-fs.target

Which make it behave just like the SysV init script.

This change together with the patch I attached to this bug report
together should fix the issue and also fix #1618900 .

Regards,
Erik

FWIW, the init script /etc/init.d/unattended-upgrades should be
removed from systemd systems (Xenial, Yaketty).

I get this sometimes on shutdown but not always.

Erik, until the fix reaches users, should I:
systemctl disable unattended-upgrades.service
?

Hello,

This logic is correct. apt_pkg.get_lock(options.lock_file) is trying to acquire a lockfile (/var/run/unattended-upgrades.lock) which is only possible if no unattended-upgrade is already running. If it can acquire the lock, it means that NO unattended-upgrade is running, hence shutdown may proceed.

 while True:
        res = apt_pkg.get_lock(options.lock_file)
        logging.debug("get_lock returned %i" % res)
        # exit here if there is no lock
        if res > 0:
            logging.debug("lock not taken")
            break
        lock_was_taken = True

The problem is that apt_pkg.get_lock(options.lock_file) will also return -1 if the path to the lock file doesn't exist, which is the case when /var is unmounted, hence /var/run is no longer present (/var/run is a symlink to /run).

I'm working on identifying the proper systemd order to make sure that /var/run is still accessible.

You're right, I thought it was the lockfile for the shutdown helper only.

Somewhere else in this thread I already mentioned that a dependency on the local_fs and remote_fs targets fixes the systemd order:

Before=remote-fs.target local-fs.target

That makes sure the service is run before all file systems are unmounted. If you add the shutdown.target to that rule, systemd somehow thinks it should be run just before shutdown which is obviously too late. I'm not sure that is a systemd bug or a systemd feature, but either way it doesn't do what we want.

Regards,

Erik

--
Erik Mouw

> Op 26 jan. 2017 om 18:00 heeft Louis Bouchard <email address hidden> het volgende geschreven:
>
> Hello,
>
> This logic is correct. apt_pkg.get_lock(options.lock_file) is trying to
> acquire a lockfile (/var/run/unattended-upgrades.lock) which is only
> possible if no unattended-upgrade is already running. If it can acquire
> the lock, it means that NO unattended-upgrade is running, hence shutdown
> may proceed.
>
> while True:
> res = apt_pkg.get_lock(options.lock_file)
> logging.debug("get_lock returned %i" % res)
> # exit here if there is no lock
> if res > 0:
> logging.debug("lock not taken")
> break
> lock_was_taken = True
>
> The problem is that apt_pkg.get_lock(options.lock_file) will also return
> -1 if the path to the lock file doesn't exist, which is the case when
> /var is unmounted, hence /var/run is no longer present (/var/run is a
> symlink to /run).
>
> I'm working on identifying the proper systemd order to make sure that
> /var/run is still accessible.
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1654600
>
> Title:
> unattended-upgrade-shutdown hangs when /var is a separate filesystem
>
> Status in unattended-upgrades package in Ubuntu:
> Confirmed
> Status in unattended-upgrades source package in Xenial:
> Confirmed
> Status in unattended-upgrades source package in Yakkety:
> New
>
> Bug description:
> The systemd unit file unattended-upgrades.service is used to stop a running unattended-upgrade
> process during shutdown. This unit file is running together with all filesystem
> unmount services.
>
> The unattended-upgrades service checks if the lockfile for unattended-upgrade
> (in /var/run) exists, and if it does, there is an unattended-upgrade in progress
> and the service will wait until it finishes (and therefore automatically wait at
> shutdown).
>
> However, if /var is a separate filesystem, it will get unmoun...

Ok, here is how I understand the situation.

According to the systemd documentation about [Unit] Before= :

"Given two units with any ordering dependency between them, if one unit is shut down and the other is started up, the shutdown is ordered before the start-up. It doesn't matter if the ordering dependency is After= or Before=."

To me this means that network.service & local-fs.service will be shutdown _BEFORE_ unattended-upgrades-shutdown runs, hence /var var will be unmounted when it runs.

My current solution is to turn the unattended-upgrades.service ExecStart= into an ExecStop= so the unit will run as a shutdown instead of a start when the system shuts down :

[Unit]
Description=Unattended Upgrades Shutdown
DefaultDependencies=no
Before=shutdown.target reboot.target halt.target network.target local-fs.target
Documentation=man:unattended-upgrade(8)

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStop=/usr/share/unattended-upgrades/unattended-upgrade-shutdown --debug
TimeoutStopSec=900

[Install]
WantedBy=shutdown.target

Preliminary tests seem to run fine but I want to get confirmation on such a change by someone more expert with systemd that I am.

Status changed to 'Confirmed' because the bug affects multiple users.

Is there a good reason not to simply change the path to the lock file from /var/run to /run?

As a workaround I created a symlink to /run inside the mounted /var :
mount -o bind / /mnt
ln -s /run /mnt/var/
umount /mnt

Bug #1661611 seems to be identical

This bug was fixed in the package unattended-upgrades - 0.93.1ubuntu2

---------------
unattended-upgrades (0.93.1ubuntu2) zesty; urgency=medium

  * The systemd unit needs to be an ExecStop since it is is activated on
    shutdown. Otherwise, it will get scheduled after completion of
    the local-fs.target. In the case where /var is a separate
    filesystem, unattended-upgrade-shutdown will hang until timeout
    since /var/run is expected but no longer there (LP: #1654600)

 -- Louis Bouchard <email address hidden> Thu, 02 Mar 2017 16:55:26 +0100

Tested the fix. It works. Thanks!

quick fix:

sed -i "s#ExecStart=#RemainAfterExit=yes\nExecStop=#;" /lib/systemd/system/unattended-upgrades.service

But now it seems that it doesn't run it on shutdown. I used the following debug service:

ExecStop=/bin/bash -c 'echo -e "\nhello######################\n" > /dev/ttyS0'

And it doesn't print message in serial console. But when you start and stop this unit manually - it prints.

[ OK ] Stopped target Graphical Interface.
         Stopping Accounts Service...
         Stopping User Manager for UID 1000...
         Stopping Session 1 of user ubuntu.
         Stopping ACPI event daemon...
[ OK ] Stopped target Cloud-init target.
[ OK ] Stopped Execute cloud user/final scripts.
[ OK ] Stopped target Multi-User System.
         Stopping Deferred execution scheduler...
         Stopping LXD - container startup/shutdown...
         Stopping D-Bus System Message Bus...
         Stopping LSB: daemon to balance interrupts for SMP systems...
         Stopping LSB: Set the CPU Frequency Scaling governor to "ondemand"...
         Stopping LSB: MD monitoring daemon...
         Stopping Regular background program processing daemon...
         Stopping FUSE filesystem for LXC...
         Stopping OpenBSD Secure Shell server...
         Stopping LSB: Record successful boot for GRUB...
[ OK ] Stopped target Timers.
[ OK ] Stopped Timer to automatically refresh installed snaps.
[ OK ] Stopped Daily apt activities.
[ OK ] Stopped Daily Cleanup of Temporary Directories.
[ OK ] Stopped target Login Prompts.
         Stopping Getty on tty1...
         Stopping Serial Getty on ttyS0...
[ OK ] Stopped Apply the settings specified in cloud-config.
[ OK ] Stopped target Cloud-config availability.
         Stopping Snappy daemon...
         Stopping System Logging Service...
[ OK ] Stopped target System Time Synchronized.
         Stopping LSB: automatic crash report generation...
[ OK ] Closed Load/Save RF Kill Switch Status /dev/rfkill Watch.
         Stopping Authenticate and Authorize Users to Run Privileged Tasks...
[ OK ] Unmounted /var/lib/lxcfs.
[ OK ] Stopped System Logging Service.
[ OK ] Stopped Deferred execution scheduler.
[ OK ] Stopped OpenBSD Secure Shell server.
[ OK ] Stopped Accounts Service.
[ OK ] Stopped Snappy daemon.
[ OK ] Stopped ACPI event daemon.
[ OK ] Stopped Authenticate and Authorize Users to Run Privileged Tasks.
[ OK ] Stopped Serial Getty on ttyS0.
[ OK ] Stopped Regular background program processing daemon.
[ OK ] Stopped Getty on tty1.
[ OK ] Stopped User Manager for UID 1000.
[ OK ] Stopped Session 1 of user ubuntu.
[ OK ] Stopped D-Bus System Message Bus.
[ OK ] Stopped FUSE filesystem for LXC.
[ OK ] Stopped LXD - container startup/shutdown.
[ OK ] Stopped LSB: MD monitoring daemon.
[ OK ] Stopped LSB: Record successful boot for GRUB.
[ OK ] Stopped LSB: daemon to balance interrupts for SMP systems.
[ OK ] Stopped LSB: automatic crash report generation.
[ OK ] Stopped LSB: Set the CPU Frequency Scaling governor to "ondemand".
[ OK ] Stopped User Manager for UID 1000.
[ OK ] Removed slice User Slice of ubuntu.
   ...

I have tested this and it does NOT work.

I fixed the issue in couple of different ways:

One way is completely remove the unit (who in right mind does install updates during reboot on production servers?). I will be actively doing that on all machines I encounter.

Another way is this:
sed -i 's#/var/run/#/run/#g' /usr/share/unattended-upgrades/unattended-upgrade-shutdown
So the problem with that sed is that the logging part will fail. So not really a solution.

Systemd has proven to be inconsistent pile of crap so far...

I could not make systemd to run this unit before local-fs.target. It appears that the documentation is misleading...

Before=shutdown.target reboot.target halt.target network.target local-fs.target

should be

Before=shutdown.target reboot.target halt.target network.target
After=local-fs.target

because, AIUI, at shutdown, "Before" means "After" and "After" means "Before".

So unattended-upgrades.service's ExecStop'll start before local-fs.target, so before "/var" is unmounted and "/var/log" and "/var/run" are gone.

[Why is "/var/run" used? This is a native Ubuntu package; are there any versions of Ubuntu where "/var/run" isn't a symlink to "/run"?]

Hello,

I am curious to hear about the context of your tests : my Zesty system with a separated /var does show the problem prior to upgrade to unattended-upgrades-0.93.1ubuntu2 and no longer hangs when the new package is applied. There might be situations that I have not tested where the issue could still exists.

Your suggestion to replace /var/run by /var is incorrect : /var/run/unattended-upgrades.lock is placed by the unattended-upgrades script; Looking in /run for a lock file that has been put in /var/run is simply inadequate.

While replacing /var/run by /run across the board is a valid proposition, it requires more inspection than simply replacing its occurrences in one script.

Hello Erik, or anyone else affected,

Accepted unattended-upgrades into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/unattended-upgrades/0.90ubuntu0.4 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Hello Erik, or anyone else affected,

Accepted unattended-upgrades into yakkety-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/unattended-upgrades/0.92ubuntu1.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

@arges,

I already noticed that upgrade procedure doesn't start at all with the new change. But actual shutdown/reboot hang has disappeared.

@kay-diam

I don't understand your statement. The unit clearly get started. This is from your log :

[ OK ] Stopped target Local File Systems.
[ OK ] Started Unattended Upgrades Shutdown. <<<<<<<
         Unmounting /run/user/1000...
         Unmounting /tmp...
[ OK ] Stopped Apply Kernel Variables.

@louis-bouchard

"Started Unattended Upgrades Shutdown." means that the unit was started. Since the patch assumes that we don't use "ExecStart" this means nothing.

Upgrade procedure should be executed on "ExecStop". But it is not happening. I tried to set debug info inside the "ExecStop" instead of executing "unattended-upgrade-shutdown", but it doesn't work. I can provide you with the detailed step-by-step instructions on how to 100% reproduce the issue, but it will require KVM host, and running my scripts which will deploy VM with LVM.

Basic steps:

* follow https://github.com/kayrus/deploy-vm docs
* git clone https://github.com/kayrus/deploy-vm
* cd deploy-vm
* ./deploy_vms_cluster.sh -o ubuntu -c xenial
* follow https://github.com/kayrus/rescue-initramfs docs
* and reinstall VM from initramfs using "deploy_os_lvm.sh" script

Then your Xenial VM will stuck at reboot

BTW, KVM allows you to use serial console you can use it for debugging. But you have explicitly specify the order of defined "console" kernel parameters, i.e.:

this will specify VGA console as default output:
console=ttyS0 console=tty1

this will specify serial console as default output:
console=tty1 console=ttyS0

@kay-diam:

I get what you mean. I also use KVM to debug the issue.

I'm going to switch to verification-failed to this one does not get released and push the investigation further as there is another issue that side-tracked me :

If Unattended-Upgrade::InstallOnShutdown is set to "true", it will still fail as the network will no longer be available when it gets to access the archive so this must also be fixed.

...Louis

@louis-bouchard

I'd love to help you, but I'm in rush on my current job.

Hello,

Yes, I am fully aware that my /var/run->/run replacement in that script is a hack.

To be clear I am not testing in Zesty but Ubuntu 16.04.2 LTS Xenial.

The setup is following:
/dev/sda1 on / type xfs (rw,relatime,attr2,inode64,noquota)
/dev/sdc on /var type xfs (rw,relatime,attr2,inode64,noquota)

Every single machine we (my and my colleague) tested with the updated unit still hangs.

As var as I can tell they never reboot.

I have posted a question regarding this issue to systemd mailing list, and one of the responses was the following:

"""
> [Unit]
> Description=Unattended Upgrades Shutdown
> DefaultDependencies=no
> Before=shutdown.target reboot.target halt.target network.target
> local-fs.target

Well, you order if before local-fs.target, which means it is stopped
after local-fs.target. You get exactly what you ask for. If this is
wrong, fix unit definition, but we do not really know what is
appropriate for this service.

> Documentation=man:unattended-upgrade(8)
>
> [Service]
> Type=oneshot
> RemainAfterExit=yes
> ExecStop=/usr/share/unattended-upgrades/unattended-upgrade-shutdown --debug
> TimeoutStopSec=900
>
> [Install]
> WantedBy=shutdown.target
>
> It appears no one who is involved in this fix understands systemd (which is
> very worrying!).
>
> How does one would fix this unit so it is ran before the file systems get
> unmounted?
>

Order it

After=local-fs.target
"""
"Obviously" according to systemd mailing list if I want to things to run Before I need to use After!

So I believe the confusion is with what exactly Before and After mean in systemd units.

Today I will be testing various combinations with After/Before to see if I can make sense of it.

Hello!

I have changed Before to After and it is working!!!!
Here is what the unit looks like now:

[Unit]
Description=Unattended Upgrades Shutdown
DefaultDependencies=no
After=shutdown.target reboot.target halt.target network.target local-fs.target
Documentation=man:unattended-upgrade(8)

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStop=/usr/share/unattended-upgrades/unattended-upgrade-shutdown
TimeoutStopSec=900

[Install]
WantedBy=shutdown.target

So the proper fix would be changing the Service to ExecStop= and Unit to After=

@sergei-franco:

From what I can tell, it works for you because changing the unit the way you did simply disable execution of the unit. When running with such a configuration either on Xenial or Zesty, I do not see any mention of "Unattended Upgrades Shutdown" in the console output.

When limiting to "After=network.target local-fs.target", I see the unit being started (which is basically a NOOP) but not stopping which is what we want.

I'm continuing my tests

Here is a recap of my day of work :

The unattended-upgrades.service unit never runs the ExecStop because it is not enabled as we see here :

# systemctl status unattended-upgrades.service
● unattended-upgrades.service - Unattended Upgrades Shutdown
   Loaded: loaded (/lib/systemd/system/unattended-upgrades.service; enabled; vendor preset: enabled)
   Active: inactive (dead) <<<<<<<<<<<<<<<
     Docs: man:unattended-upgrade(8)

This is caused by a missing "Default-Start" header in /etc/init.d/unattended-upgrades. Trying to enable the unit leads to :

# systemctl enable unattended-upgrades.service
Synchronizing state of unattended-upgrades.service with SysV init with /lib/systemd/systemd-sysv-install...
Executing /lib/systemd/systemd-sysv-install enable unattended-upgrades
update-rc.d: error: unattended-upgrades Default-Start contains no runlevels, aborting.

I'm currently working on fixing all this in the packaging so we get a correctly working unit after the upgrade of the package.

I have also reworked the unit :

[Unit]
Description=Unattended Upgrades Shutdown
DefaultDependencies=no
After=network.target local-fs.target
RequiresMountsFor=/var/log /var/run
Documentation=man:unattended-upgrade(8)

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStop=/usr/share/unattended-upgrades/unattended-upgrade-shutdown
TimeoutStopSec=900

[Install]
WantedBy=multi-user.target

With this configuration, the unit runs the ExecStop as expected. I also tested using Unattended-Upgrade::InstallOnShutdown "true" and it works as expected, which is to block the shutdown for upgrade with the network being available.

I should have that available for testing tomorrow.

Still doesn't work for me.

Here is a recap of my work and current status :

#1) when a system has a separated /var, the unit will hang since it is looking
for /var/run to be present and it has been unmounted.

#2) when using Unattended-Upgrade::InstallOnShutdown "true"; the upgrade never
completes as the query to the online archive fails since the network is no
longer available.

#3) it is impossible to enable the unattended-upgrades.service unit. Here
is an example :

> $ systemctl status unattended-upgrades.service
> ● unattended-upgrades.service - Unattended Upgrades Shutdown
> Loaded: loaded (/lib/systemd/system/unattended-upgrades.service; enabled; vendor preset: enabled)
> Active: inactive (dead)
> Docs: man:unattended-upgrade(8)

According to the doc[2], during shutdown, even if Before= or After= is used, the
unit being started will only start after the Shutdown of its dependencies. Prior
to the recent addition of local-fs.target network.target, the unit ran quickly
enough for the /var to be still mounted. But even without these dependencies,
InstallOnShutdown would fail with the following :

> 2017-03-10 13:40:42,803 INFO Starting unattended upgrades script
> 2017-03-10 13:40:42,803 INFO Allowed origins are: ['o=Ubuntu,a=zesty', 'o=Ubuntu,a=zesty-security']
> 2017-03-10 13:41:40,554 ERROR An error occurred: 'Cannot initiate the connection to 192.168.200.3:8000 (192.168.200.3). - connect (101: Network is unreachable)'
> 2017-03-10 13:41:40,555 ERROR The URI 'http://fr.archive.ubuntu.com/ubuntu/pool/main/i/init-system-helpers/init-system-helpers_1.47_all.deb' failed to download, aborting

When trying to switch the unit to an ExecStop=, we find that the Stop never
runs. This is caused by the fact that the unit is disabled (#3). Trying to
enable the unit leads to :

> # systemctl enable unattended-upgrades
> Synchronizing state of unattended-upgrades.service with SysV service script with /lib/systemd/systemd-sysv-install.
> Executing: /lib/systemd/systemd-sysv-install enable unattended-upgrades
> update-rc.d: error: unattended-upgrades Default-Start contains no runlevels, aborting.

Adding runlevels 2 3 4 5 fixes this then the unit can be enabled. So we get
to a unit that looks like this :

> [Unit]
> Description=Unattended Upgrades Shutdown
> DefaultDependencies=no
> After=network.target local-fs.target
> RequiresMountsFor=/var/run /var/log
> Documentation=man:unattended-upgrade(8)
>
> [Service]
> Type=oneshot
> RemainAfterExit=yes
> ExecStop=/usr/share/unattended-upgrades/unattended-upgrade-shutdown
> TimeoutStopSec=900
>
> [Install]
> WantedBy=multi-user.target

Before= is replaced by After= as, during shutdown otherwise the unit does not
run. RequiresMountsFor= is added as both /var/log and /var/run are needed in
order to run correctly.

RemainAfterExit=yes is added so the unit appears as started. There is no longer
a requirement to have an ExecStart present.

WantedBy is switched to multi-user.target as on the way up, we do nothing and we
are no longer depending on anything related to shutdown.

Now this only works IF /var is a separate FS. The reason for that is the
presence of DefaultDependencies=no. I don't think that it is required but was
the...

The Unit file in #33 works for me, ie stops the 10 minute hang, when unattended-upgrades is not running.

- Xenial with /var as a separate FS, InstallOnShutdown="false"

I will try to create a test case where unattended-upgrades is running when shutdown is called.

Follow-up to #34, on a system where unattended-upgrades happens to be running when shutdown is called, the shutdown does not wait, therefore any running dpkg is killed and the package left in an inconsistent state.

 - Xenial with single partition, InstallOnShutdown not defined.

This may be a separate issue, but I'll investigate further before raising it as such.

A quick update on this issue. I have a solution ready to fix this. Unfortunately it requires changing the WantedBy= statement, which triggers a deb-systemd-helper bug : https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=797108

This has the effect of leaving the service disabled after upgrade. I'm working on fixing that.

Targetting Zesty as the fix will come after release so as a SRU.

Brief update to tell everyone that I have a fix ready for all stable releases. I will update it to Artful soon and the SRU will follow soon.

This bug was fixed in the package unattended-upgrades - 0.93.1ubuntu3

---------------
unattended-upgrades (0.93.1ubuntu3) artful; urgency=medium

  * Complete the solution for the unattended-upgrades.service unit not
    correctly working (LP: #1654600):
    - d/rules : Remove the override_dh_installinit. The stop option is no longer
      available so the command falls back to default. This is the normal
      behavior so the override is not required
    - d/unattended-upgrades.init : Add Default-Start runlevels, otherwise the
      unattended-upgrades.service unit cannot be enabled on boot
    - d/postinst : Cleanup the stop symlinks created by the wrong
      override_dh_installinit. Without that, the systemd unit cannot be
      enabled correctly.
      Force disable the service before deb-systemd-helper runs so the old
      symlink is not left dangling (workaround for Debian Bug #797108).
      Force enable and start of the systemd unit to work around Debian Bug #797108
      which fails to enable systemd units correctly when WantedBy= statement
      is changed which is the case here.
    - d/unattended-upgrades.service : Fix the service so it runs correctly on
      shutdown :
        Remove DefaultDependencies=no : Breaks normal shutdown dependencies
        Set After= to network.target and local-fs.target. Since our service is
        now ExecStop, it will run before network and local-fs become unavailable.
        Add RequiresMountsFor=/var/log /var/run /var/lib /boot : Necessary if
        /var is a separate file system. Set WantedBy= to multi-user.target
    - Add DEP8 tests to verify the following :
      Verify that the unattended-upgrades.service unit is enabled and started.
      Verify that InstallOnShutdown works when configured.

 -- Louis Bouchard <email address hidden> Mon, 24 Apr 2017 14:07:19 +0200

Hello Erik, or anyone else affected,

Accepted unattended-upgrades into zesty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/unattended-upgrades/0.93.1ubuntu2.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Hello Erik, or anyone else affected,

Accepted unattended-upgrades into yakkety-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/unattended-upgrades/0.92ubuntu1.4 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Hello Erik, or anyone else affected,

Accepted unattended-upgrades into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/unattended-upgrades/0.90ubuntu0.5 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Test performed for verification-done for :
 - Xenial
 - Yakkety
 - Zesty
1) Installed unattended-upgrades from -proposed
2) Verified the status of the unattended-upgrades service. All displayed :

# systemctl status unattended-upgrades
● unattended-upgrades.service - Unattended Upgrades Shutdown
   Loaded: loaded (/lib/systemd/system/unattended-upgrades.service; enabled; vendor preset: enabled)
   Active: active (exited) since Fri 2017-04-28 14:02:35 CEST; 47s ago
     Docs: man:unattended-upgrade(8)

3) Enabled Unattended-Upgrade::InstallOnShutdown "true"; in /etc/apt/apt.conf.d/50unattended-upgrades and rebooted. All releases applied the upgradeable packages upon reboot (some had too many and the timeout ran out before end of update)

4) Ran apt -y dist-upgrade on all releases. Rebooted. systems rebooted normally.

5) Enabled trusty-proposed on a Trusty system and ran do-release-upgrade to go to Xenial. After completion of the full upgrade, unattended-upgrades service was active (as seen in #2)

6) Enabled zesty-proposed on a system with /var as a separate file system. Upgraded unattended-upgrades. After upgrade, unattended-upgrades service was active (as seen in #2)

7) Verified that the workaround for Debian Bug #809669 left the upgraded system in an adequate configuration :

cat /var/lib/systemd/deb-systemd-helper-enabled/unattended-upgrades.service.dsh-also
/etc/systemd/system/multi-user.target.wants/unattended-upgrades.service

# find /etc/systemd | grep unatt
/etc/systemd/system/multi-user.target.wants/unattended-upgrades.service

This resolved the issue.
Version of the package testes: unattended-upgrades 0.90ubuntu0.5
Ubuntu version - 16.04 xenial
verification-done

The verification of the Stable Release Update for unattended-upgrades has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

This bug was fixed in the package unattended-upgrades - 0.92ubuntu1.4

---------------
unattended-upgrades (0.92ubuntu1.4) yakkety; urgency=medium

  * Complete the solution for the unattended-upgrades.service unit not
    correctly working (LP: #1654600):
    - d/rules : Remove the override_dh_installinit. The stop option is no longer
      available so the command falls back to default. This is the normal
      behavior so the override is not required
    - d/unattended-upgrades.init : Add Default-Start runlevels, otherwise the
      unattended-upgrades.service unit cannot be enabled on boot
    - d/postinst : Cleanup the stop symlinks created by the wrong
      override_dh_installinit. Without that, the systemd unit cannot be
      enabled correctly.
      Force disable the service before deb-systemd-helper runs so the old
      symlink is not left dangling (workaround for Debian Bug #797108).
      Force enable and start of the systemd unit to work around Debian Bug #797108
      which fails to enable systemd units correctly when WantedBy= statement
      is changed which is the case here.
    - d/unattended-upgrades.service : Fix the service so it runs correctly on
      shutdown :
        Remove DefaultDependencies=no : Breaks normal shutdown dependencies
        Set After= to network.target and local-fs.target. Since our service is
        now ExecStop, it will run before network and local-fs become unavailable.
        Add RequiresMountsFor=/var/log /var/run /var/lib /boot : Necessary if
        /var is a separate file system. Set WantedBy= to multi-user.target
    - Add DEP8 tests to verify the following :
      Verify that the unattended-upgrades.service unit is enabled and started.
      Verify that InstallOnShutdown works when configured.

 -- Louis Bouchard <email address hidden> Tue, 04 Apr 2017 18:31:42 +0200

This bug was fixed in the package unattended-upgrades - 0.90ubuntu0.5

---------------
unattended-upgrades (0.90ubuntu0.5) xenial; urgency=medium

  * Complete the solution for the unattended-upgrades.service unit not
    correctly working (LP: #1654600):
    - d/rules : Remove the override_dh_installinit. The stop option is no longer
      available so the command falls back to default. This is the normal
      behavior so the override is not required
    - d/unattended-upgrades.init : Add Default-Start runlevels, otherwise the
      unattended-upgrades.service unit cannot be enabled on boot
    - d/postinst : Cleanup the stop symlinks created by the wrong
      override_dh_installinit. Without that, the systemd unit cannot be
      enabled correctly.
      Force disable the service before deb-systemd-helper runs so the old
      symlink is not left dangling (workaround for Debian Bug #797108).
      Force enable and start of the systemd unit to work around Debian Bug #797108
      which fails to enable systemd units correctly when WantedBy= statement
      is changed which is the case here.
      Both systemctl enable AND start are needed as enable --now fails on
      Xenial.
    - d/unattended-upgrades.service : Fix the service so it runs correctly on
      shutdown :
        Remove DefaultDependencies=no : Breaks normal shutdown dependencies
        Set After= to network.target and local-fs.target. Since our service is
        now ExecStop, it will run before network and local-fs become unavailable.
        Add RequiresMountsFor=/var/log /var/run /var/lib /boot : Necessary if
        /var is a separate file system. Set WantedBy= to multi-user.target
    - Add DEP8 tests to verify the following :
      Verify that the unattended-upgrades.service unit is enabled and started.
      Verify that InstallOnShutdown works when configured.

 -- Louis Bouchard <email address hidden> Fri, 21 Apr 2017 16:47:52 +0200

This bug was fixed in the package unattended-upgrades - 0.93.1ubuntu2.1

---------------
unattended-upgrades (0.93.1ubuntu2.1) zesty; urgency=medium

  * Complete the solution for the unattended-upgrades.service unit not
    correctly working (LP: #1654600):
    - d/rules : Remove the override_dh_installinit. The stop option is no longer
      available so the command falls back to default. This is the normal
      behavior so the override is not required
    - d/unattended-upgrades.init : Add Default-Start runlevels, otherwise the
      unattended-upgrades.service unit cannot be enabled on boot
    - d/postinst : Cleanup the stop symlinks created by the wrong
      override_dh_installinit. Without that, the systemd unit cannot be
      enabled correctly.
      Force disable the service before deb-systemd-helper runs so the old
      symlink is not left dangling (workaround for Debian Bug #797108).
      Force enable and start of the systemd unit to work around Debian Bug #797108
      which fails to enable systemd units correctly when WantedBy= statement
      is changed which is the case here.
    - d/unattended-upgrades.service : Fix the service so it runs correctly on
      shutdown :
        Remove DefaultDependencies=no : Breaks normal shutdown dependencies
        Set After= to network.target and local-fs.target. Since our service is
        now ExecStop, it will run before network and local-fs become unavailable.
        Add RequiresMountsFor=/var/log /var/run /var/lib /boot : Necessary if
        /var is a separate file system. Set WantedBy= to multi-user.target
    - Add DEP8 tests to verify the following :
      Verify that the unattended-upgrades.service unit is enabled and started.
      Verify that InstallOnShutdown works when configured.

 -- Louis Bouchard <email address hidden> Fri, 21 Apr 2017 12:40:29 +0200

Hello Erik, or anyone else affected,

Accepted unattended-upgrades into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/unattended-upgrades/1.1ubuntu1.18.04.7~16.04.0 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Hello Erik, or anyone else affected,

Accepted unattended-upgrades into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/unattended-upgrades/1.1ubuntu1.18.04.7~16.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

This bug was fixed in the package unattended-upgrades - 0.90ubuntu0.10

---------------
unattended-upgrades (0.90ubuntu0.10) xenial-security; urgency=medium

  * No change rebuild in the -security pocket (See LP #1686470)

 -- Marc Deslauriers <email address hidden> Fri, 18 Jan 2019 13:34:27 -0500

Hello Erik, or anyone else affected,

Accepted unattended-upgrades into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/unattended-upgrades/1.1ubuntu1.18.04.7~16.04.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Already verified in 0.90ubuntu0.5 for Xenial, not additional test is needed.

This bug was fixed in the package unattended-upgrades - 1.1ubuntu1.18.04.7~16.04.2

---------------
unattended-upgrades (1.1ubuntu1.18.04.7~16.04.2) xenial; urgency=medium

  * Don't check blacklist too early and report updates from not allowed origins
    as kept back. (LP: #1781176)
  * test/test_blacklisted_wrong_origin.py: Fix and enable test
  * Filter out progress indicator from dpkg log (LP: #1599646)
  * Clear cache when autoremoval fails (LP: #1779157)
  * Find autoremovable kernel packages using the patterns in APT's way
    (LP: #1815494)

unattended-upgrades (1.1ubuntu1.18.04.7~16.04.1) xenial; urgency=medium

  * Start service after systemd-logind.service to be able to take inhibition
    lock (LP: #1806487)
  * Handle gracefully when logind is down (LP: #1806487)

unattended-upgrades (1.1ubuntu1.18.04.7~16.04.0) xenial; urgency=medium

  * Backport to Xenial (LP: #1702793)
  * Revert to build-depending on debhelper (>= 9~) and dh-systemd
  * Revert configuration example changes to avoid triggering a debconf question
  * debian/postinst: Update recovery to be triggered on Xenial's package versions

unattended-upgrades (1.1ubuntu1.18.04.7) bionic; urgency=medium

  * Trigger unattended-upgrade-shutdown actions with PrepareForShutdown()
    Performing upgrades in service's ExecStop did not work when the upgrades
    involved restarting services because systemd blocked other stop/start
    actions making maintainer scripts time out and be killed leaving a broken
    system behind.
    Running unattended-upgrades.service before shutdown.target as a oneshot
    service made it run after unmounting filesystems and scheduling services
    properly on shutdown is a complex problem and adding more services to the
    mix make it even more fragile.
    The solution of monitoring PrepareForShutdown() signal from DBus
    allows Unattended Upgrade to run _before_ the jobs related to shutdown are
    queued thus package upgrades can safely restart services without
    risking causing deadlocks or breaking part of the shutdown actions.
    Also ask running unattended-upgrades to stop when shutdown starts even in
    InstallOnShutdown mode and refactor most of unattended-upgrade-shutdown to
    UnattendedUpgradesShutdown class. (LP: #1778219)
  * Increase logind's InhibitDelayMaxSec to 30s. (LP: #1778219)
    This allows more time for unattended-upgrades to shut down gracefully
    or even install a few packages in InstallOnShutdown mode, but is still a
    big step back from the 30 minutes allowed for InstallOnShutdown previously.
    Users enabling InstallOnShutdown node are advised to increase
    InhibitDelayMaxSec even further possibly to 30 minutes.
    - Add NEWS entry about increasing InhibitDelayMaxSec and InstallOnShutdown
      changes
  * Ignore "W503 line break before binary operator"
    because it will become the best practice and breaks the build
  * Stop using ActionGroups, they interfere with apt.Cache.clear()
    causing all autoremovable packages to be handled as newly autoremovable
    ones and be removed by default. Dropping ActionGroup usage does not slow
    down the most frequent case of not having anything to upgrade a...