double virtlogd sockets with services running can trigger issues on upgrade

Pre Upgrade

root@c2:~# systemctl status virtlogd.service virtlogd.socket virtlogd-admin.socket virtlockd.service virtlockd.socket virtlockd-admin.socket --no-pager --lines 1
● virtlogd.service - Virtual machine log manager
   Loaded: loaded (/lib/systemd/system/virtlogd.service; indirect; vendor preset: enabled)
   Active: active (running) since Thu 2018-08-09 05:26:07 UTC; 3h 43min ago
     Docs: man:virtlogd(8)
           https://libvirt.org
 Main PID: 4059 (virtlogd)
    Tasks: 2 (limit: 4915)
   CGroup: /system.slice/virtlogd.service
           └─4059 /usr/sbin/virtlogd

Aug 09 09:07:01 c2 systemd[1]: virtlogd.service: Failed to reset devices.list: Operation not permitted

● virtlogd.socket - Virtual machine log manager socket
   Loaded: loaded (/lib/systemd/system/virtlogd.socket; enabled; vendor preset: enabled)
   Active: active (running) since Thu 2018-08-09 05:26:07 UTC; 3h 43min ago
   Listen: /var/run/libvirt/virtlogd-sock (Stream)
   CGroup: /system.slice/virtlogd.socket
Failed to dump process list, ignoring: No such file or directory

Aug 09 05:26:07 c2 systemd[1]: Listening on Virtual machine log manager socket.
Unit virtlogd-admin.socket could not be found.

● virtlockd.service - Virtual machine lock manager
   Loaded: loaded (/lib/systemd/system/virtlockd.service; indirect; vendor preset: enabled)
   Active: inactive (dead)
     Docs: man:virtlockd(8)
           https://libvirt.org

● virtlockd.socket - Virtual machine lock manager socket
   Loaded: loaded (/lib/systemd/system/virtlockd.socket; enabled; vendor preset: enabled)
   Active: active (listening) since Thu 2018-08-09 05:26:06 UTC; 3h 43min ago
   Listen: /var/run/libvirt/virtlockd-sock (Stream)
   CGroup: /system.slice/virtlockd.socket
Failed to dump process list, ignoring: No such file or directory

Aug 09 05:26:06 c2 systemd[1]: Listening on Virtual machine lock manager socket.
Unit virtlockd-admin.socket could not be found.

The Upgrade
# apt upgrade
Reading package lists... Done
Building dependency tree
Reading state information... Done
Calculating upgrade... Done
The following NEW packages will be installed:
  libjansson4
The following packages will be upgraded:
  libvirt-clients libvirt-daemon libvirt-daemon-driver-storage-rbd libvirt-daemon-system libvirt0
5 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 3790 kB of archives.
After this operation, 1225 kB disk space will be freed.
Do you want to continue? [Y/n] Y
Get:1 http://archive.ubuntu.com/ubuntu cosmic/main amd64 libjansson4 amd64 2.11-1 [29.3 kB]
Get:2 http://ppa.launchpad.net/ci-train-ppa-service/3349/ubuntu cosmic/main amd64 libvirt-daemon-driver-storage-rbd amd64 4.6.0-1ubuntu1~ppa3 [34.3 kB]
Get:3 http://ppa.launchpad.net/ci-train-ppa-service/3349/ubuntu cosmic/main amd64 libvirt-daemon-system amd64 4.6.0-1ubuntu1~ppa3 [77.0 kB]
Get:4 http://ppa.launchpad.net/ci-train-ppa-service/3349/ubuntu cosmic/main amd64 libvirt-daemon amd64 4.6.0-1ubuntu1~ppa3 [1712 kB]
Get:5 http://ppa.launchpad.net/ci-train-ppa-service/3349/ubuntu cosmic/main amd64 libvirt-clients amd64 4.6.0-1ubuntu1~ppa3 [615 kB]
Get:6 http://ppa.launchpad.net/ci-train-ppa-service/3349/ubuntu cosmic/main amd64 libvirt0 amd64 4.6.0-1ubuntu1~ppa3 [1322 kB]
Fetched 3790 kB in 1s (3161 kB/s)
Preconfiguring packages ...
Selecting previously unselected package libjansson4:amd64.
(Reading database ... 44954 files and directories currently installed.)
Preparing to unpack .../0-libjansson4_2.11-1_amd64.deb ...
Unpacking libjansson4:amd64 (2.11-1) ...
Preparing to unpack .../1-libvirt-daemon-driver-storage-rbd_4.6.0-1ubuntu1~ppa3_amd64.deb ...
Unpacking libvirt-daemon-driver-storage-rbd (4.6.0-1ubuntu1~ppa3) over (4.0.0-1ubuntu13) ...
Preparing to unpack .../2-libvirt-daemon-system_4.6.0-1ubuntu1~ppa3_amd64.deb ...
Unpacking libvirt-daemon-system (4.6.0-1ubuntu1~ppa3) over (4.0.0-1ubuntu13) ...
Preparing to unpack .../3-libvirt-daemon_4.6.0-1ubuntu1~ppa3_amd64.deb ...
Unpacking libvirt-daemon (4.6.0-1ubuntu1~ppa3) over (4.0.0-1ubuntu13) ...
Preparing to unpack .../4-libvirt-clients_4.6.0-1ubuntu1~ppa3_amd64.deb ...
Unpacking libvirt-clients (4.6.0-1ubuntu1~ppa3) over (4.0.0-1ubuntu13) ...
Preparing to unpack .../5-libvirt0_4.6.0-1ubuntu1~ppa3_amd64.deb ...
Unpacking libvirt0:amd64 (4.6.0-1ubuntu1~ppa3) over (4.0.0-1ubuntu13) ...
Processing triggers for ureadahead (0.100.0-20) ...
Setting up libjansson4:amd64 (2.11-1) ...
Processing triggers for libc-bin (2.27-3ubuntu1) ...
Processing triggers for systemd (237-3ubuntu10) ...
Processing triggers for man-db (2.8.4-2) ...
Setting up libvirt0:amd64 (4.6.0-1ubuntu1~ppa3) ...
Setting up libvirt-daemon (4.6.0-1ubuntu1~ppa3) ...
Setting up libvirt-clients (4.6.0-1ubuntu1~ppa3) ...
Setting up libvirt-daemon-system (4.6.0-1ubuntu1~ppa3) ...
Installing new version of config file /etc/apparmor.d/abstractions/libvirt-qemu ...
Installing new version of config file /etc/apparmor.d/usr.sbin.libvirtd ...
Installing new version of config file /etc/default/libvirt-guests ...
Installing new version of config file /etc/libvirt/libvirtd.conf ...
Installing new version of config file /etc/...

Post Upgrade:
root@c2:~# systemctl status virtlogd.service virtlogd.socket virtlogd-admin.socket virtlockd.service virtlockd.socket virtlockd-admin.socket --no-pager --lines 2
● virtlogd.service - Virtual machine log manager
   Loaded: loaded (/lib/systemd/system/virtlogd.service; indirect; vendor preset: enabled)
   Active: active (running) since Thu 2018-08-09 05:26:07 UTC; 3h 45min ago
     Docs: man:virtlogd(8)
           https://libvirt.org
 Main PID: 4059 (virtlogd)
    Tasks: 2 (limit: 4915)
   CGroup: /system.slice/virtlogd.service
           └─4059 /usr/sbin/virtlogd

Aug 09 09:10:37 c2 systemd[1]: Dependency failed for Virtual machine log manager.
Aug 09 09:10:37 c2 systemd[1]: virtlogd.service: Job virtlogd.service/start failed with result 'dependency'.

● virtlogd.socket - Virtual machine log manager socket
   Loaded: loaded (/lib/systemd/system/virtlogd.socket; enabled; vendor preset: enabled)
   Active: active (running) since Thu 2018-08-09 05:26:07 UTC; 3h 45min ago
   Listen: /var/run/libvirt/virtlogd-sock (Stream)
   CGroup: /system.slice/virtlogd.socket
Failed to dump process list, ignoring: No such file or directory

Aug 09 05:26:07 c2 systemd[1]: Listening on Virtual machine log manager socket.

● virtlogd-admin.socket - Virtual machine log manager socket
   Loaded: loaded (/lib/systemd/system/virtlogd-admin.socket; enabled; vendor preset: enabled)
   Active: inactive (dead)
   Listen: /var/run/libvirt/virtlogd-admin-sock (Stream)

Aug 09 09:10:37 c2 systemd[1]: virtlogd-admin.socket: Socket service virtlogd.service already active, refusing.
Aug 09 09:10:37 c2 systemd[1]: Failed to listen on Virtual machine log manager socket.

● virtlockd.service - Virtual machine lock manager
   Loaded: loaded (/lib/systemd/system/virtlockd.service; indirect; vendor preset: enabled)
   Active: inactive (dead)
     Docs: man:virtlockd(8)
           https://libvirt.org

● virtlockd.socket - Virtual machine lock manager socket
   Loaded: loaded (/lib/systemd/system/virtlockd.socket; enabled; vendor preset: enabled)
   Active: active (listening) since Thu 2018-08-09 09:10:36 UTC; 31s ago
   Listen: /var/run/libvirt/virtlockd-sock (Stream)
   CGroup: /system.slice/virtlockd.socket
Failed to dump process list, ignoring: No such file or directory

Aug 09 09:10:36 c2 systemd[1]: Listening on Virtual machine lock manager socket.

● virtlockd-admin.socket - Virtual machine lock manager admin socket
   Loaded: loaded (/lib/systemd/system/virtlockd-admin.socket; enabled; vendor preset: enabled)
   Active: active (listening) since Thu 2018-08-09 09:10:37 UTC; 31s ago
   Listen: /var/run/libvirt/virtlockd-admin-sock (Stream)
   CGroup: /system.slice/virtlockd-admin.socket
Failed to dump process list, ignoring: No such file or directory

Aug 09 09:10:37 c2 systemd[1]: Listening on Virtual machine lock manager admin socket.

It seems absolutely reproducible - so I added a bit more verbose output in cae one searches for this.

This is in Debian since April, that would have shown up there.
Why would it only hit us thou?

I was upgrading Debian from 3.0 (stable) to 4.6 (unstable)

PRE
root@debian-stretch:~# systemctl status virtlogd.service virtlogd.socket virtlogd-admin.socket virtlockd.service virtlockd.socket virtlockd-admin.socket --no-pager --lines 2
● virtlogd.service - Virtual machine log manager
   Loaded: loaded (/lib/systemd/system/virtlogd.service; indirect; vendor preset: enabled)
   Active: active (running) since Thu 2018-08-09 09:28:33 UTC; 7s ago
     Docs: man:virtlogd(8)
           http://libvirt.org
 Main PID: 7080 (virtlogd)
    Tasks: 2 (limit: 4915)
   CGroup: /system.slice/virtlogd.service
           └─7080 /usr/sbin/virtlogd

Aug 09 09:28:35 debian-stretch systemd[1]: virtlogd.service: Failed to reset devices.list: Operation not permitted
Aug 09 09:28:37 debian-stretch systemd[1]: virtlogd.service: Failed to reset devices.list: Operation not permitted

● virtlogd.socket - Virtual machine log manager socket
   Loaded: loaded (/lib/systemd/system/virtlogd.socket; enabled; vendor preset: enabled)
   Active: active (running) since Thu 2018-08-09 09:28:33 UTC; 7s ago
   Listen: /var/run/libvirt/virtlogd-sock (Stream)

Aug 09 09:28:33 debian-stretch systemd[1]: Listening on Virtual machine log manager socket.
Unit virtlogd-admin.socket could not be found.

● virtlockd.service - Virtual machine lock manager
   Loaded: loaded (/lib/systemd/system/virtlockd.service; indirect; vendor preset: enabled)
   Active: inactive (dead)
     Docs: man:virtlockd(8)
           http://libvirt.org

● virtlockd.socket - Virtual machine lock manager socket
   Loaded: loaded (/lib/systemd/system/virtlockd.socket; enabled; vendor preset: enabled)
   Active: active (listening) since Thu 2018-08-09 09:28:34 UTC; 7s ago
   Listen: /var/run/libvirt/virtlockd-sock (Stream)

Aug 09 09:28:34 debian-stretch systemd[1]: Listening on Virtual machine lock manager socket.
Unit virtlockd-admin.socket could not be found.

Upgrade just libvirt and dependencies:
This looked similar but not exactly the same, it turned out to be affected by the jannson issue that I proposed a fix to Debian already.
Doing it all over again stretch to 4.5 (testing) which should expose the same issue but not the swarm of red libjansson4 herrings.

[...]

Upgrade just libvirt and dependencies - stretch to testing.
# apt install libvirt-daemon-system
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages were automatically installed and are no longer required:
  libboost-iostreams1.62.0 libboost-random1.62.0 libboost-system1.62.0 libboost-thread1.62.0 librados2 librbd1
Use 'apt autoremove' to remove them.
The following additional packages will be installed:
  ca-certificates libcap-ng0 libcom-err2 libcomerr2 libcurl3-gnutls libidn2-0 libnghttp2-14 libpsl5 librtmp1 libunistring2 libvirt-clients
  libvirt-daemon libvirt0 openssl publicsuffix
Suggested packages:
  libvirt-daemon-driver-storage-gluster libvirt-daemon-driver-storage-rbd libvirt-daemon-driver-storage-sheepdog
  libvirt-daemon-driver-storage-zfs numad apparmor auditd nfs-common open-...

Reported to Debian

And debugging showed that the ordering issue is due to sysV scripts triggering invoke.rc breaking the systemd ordering by making virtlogd.service realize the new dependency (virtlogd-admin.socket) is missing.

Test build in PPA now...

The POC removes the sysV scripts which is probably a good idea anyway.
Backports only go back as far as Bionic for us which is fully systemd.

If this is what we go for we'd also want rm_conffile statements to clean up.

Test with the new (~ppa4) version LGTM:

...
Unpacking libvirt0:amd64 (4.6.0-1ubuntu1~ppa4) over (4.0.0-1ubuntu13) ...
Processing triggers for ureadahead (0.100.0-20) ...
Setting up libjansson4:amd64 (2.11-1) ...
Processing triggers for libc-bin (2.27-3ubuntu1) ...
Processing triggers for systemd (237-3ubuntu10) ...
Processing triggers for man-db (2.8.4-2) ...
Setting up libvirt0:amd64 (4.6.0-1ubuntu1~ppa4) ...
Setting up libvirt-daemon (4.6.0-1ubuntu1~ppa4) ...
Setting up libvirt-clients (4.6.0-1ubuntu1~ppa4) ...
Setting up libvirt-daemon-system (4.6.0-1ubuntu1~ppa4) ...
Installing new version of config file /etc/apparmor.d/abstractions/libvirt-qemu ...
Installing new version of config file /etc/apparmor.d/usr.sbin.libvirtd ...
Installing new version of config file /etc/default/libvirt-guests ...
Installing new version of config file /etc/libvirt/libvirtd.conf ...
Installing new version of config file /etc/libvirt/libxl.conf ...
Installing new version of config file /etc/libvirt/qemu.conf ...
Installing new version of config file /etc/libvirt/virtlockd.conf ...
Installing new version of config file /etc/libvirt/virtlogd.conf ...
Created symlink /etc/systemd/system/sockets.target.wants/virtlockd-admin.socket → /lib/systemd/system/virtlockd-admin.socket.
Created symlink /etc/systemd/system/sockets.target.wants/virtlogd-admin.socket → /lib/systemd/system/virtlogd-admin.socket.
virtlockd.service is a disabled or a static unit, not starting it.
virtlogd.service is a disabled or a static unit, not starting it.
Setting up libvirt-daemon dnsmasq configuration.
Setting up libvirt-daemon-driver-storage-rbd (4.6.0-1ubuntu1~ppa4) ...
Processing triggers for libc-bin (2.27-3ubuntu1) ...

After this even the admin socket started just fine.

Status after upgrade:
root@c2:~# systemctl status virtlogd.service virtlogd.socket virtlogd-admin.socket virtlockd.service virtlockd.socket virtlockd-admin.socket --no-pager --lines 2
● virtlogd.service - Virtual machine log manager
   Loaded: loaded (/lib/systemd/system/virtlogd.service; indirect; vendor preset: enabled)
   Active: active (running) since Thu 2018-08-09 11:57:04 UTC; 43s ago
     Docs: man:virtlogd(8)
           https://libvirt.org
 Main PID: 5889 (virtlogd)
    Tasks: 1 (limit: 4915)
   CGroup: /system.slice/virtlogd.service
           └─5889 /usr/sbin/virtlogd

Aug 09 11:57:05 c2 systemd[1]: virtlogd.service: Failed to reset devices.list: Operation not permitted
Aug 09 11:57:05 c2 systemd[1]: virtlogd.service: Failed to reset devices.list: Operation not permitted

● virtlogd.socket - Virtual machine log manager socket
   Loaded: loaded (/lib/systemd/system/virtlogd.socket; enabled; vendor preset: enabled)
   Active: active (running) since Thu 2018-08-09 11:57:04 UTC; 43s ago
   Listen: /var/run/libvirt/virtlogd-sock (Stream)
   CGroup: /system.slice/virtlogd.socket
Failed to dump process list, ignoring: No such file or directory

Aug 09 11:57:04 c2 systemd[1]: Stopping Virtual machine log manager socket.
Aug 09 11:57:04 c2 systemd[1]: Listening on Virtual machine log manager socket.

● virtlogd-admin.socket - Virtual machine log manager socket
   Loaded: loaded (/lib/systemd/system/virtlogd-admin.so...

Note to myself - command to trigger on fresh cosmic container:

apt update && apt upgrade -y && apt install libvirt-daemon-system -y && add-apt-repository ppa:ci-train-ppa-service/3349 && systemctl status virtlogd.service virtlogd.socket virtlogd-admin.socket virtlockd.service virtlockd.socket virtlockd-admin.socket --no-pager --lines 2; apt upgrade -y; systemctl status virtlogd.service virtlogd.socket virtlogd-admin.socket virtlockd.service virtlockd.socket virtlockd-admin.socket --no-pager --lines 2

It turns out it will do this in postinst now:
 if [ -n $2 ]
  deb-systemd-invoke restart virtlogd.socket

Ignoring all sorts of --no-restart-on-upgrade as the above very literally sounds like "restart on upgrade"

In d/rules we have 2x dh_systemd_start:
1. dh_systemd_start -p libvirt-daemon-system --restart-after-upgrade libvirtd.service
2. dh_systemd_start -p libvirt-daemon-system --no-restart-on-upgrade $(LIBVIRT_SYSTEM_SERVICES)

#2 follows the --no-restart-on-upgrade policy for all services:
  deb-systemd-invoke start 'virtlockd-admin.socket' 'virtlockd.service' 'virtlockd.socket' 'virtlogd-admin.socket' 'virtlogd.service' 'virtlogd.socket'

But #1 seems to pull in all that are required:
From the top of libvirtd.service:
  Requires=virtlogd.socket
  Requires=virtlockd.socket

And the call it generates is:
  deb-systemd-invoke restart 'libvirtd.service' 'virtlockd.socket' 'virtlogd.socket'

This restarts virtlogd.service:
$systemctl status virtlogd.service | grep Active; deb-systemd-invoke restart 'virtlogd.socket'; systemctl status virtlogd.service | grep Active
   Active: active (running) since Thu 2018-08-09 12:40:40 UTC; 5min ago
   Active: active (running) since Thu 2018-08-09 12:46:36 UTC; 20ms ago

I'm not entirely sure what the bug here is, either
- why is virtlogd.socket listed on this line - probably the requires
or
- why is restarting the socket restarting the service - probably because it knows sockets can only be restarted when services are down

As long as it had a sysV init script that is prefered from starting the systemd service.
This is why it is active now.

Arr, such a simple issue and such a messy way to a fix :-/

In general Debian wants to support non-systemd systems more than we do - so dropping sysV might not only be an issue due to the systemd based start pulling off a restart of virtlogd - but also that this might be a derive that is more than we want.

So I was testing "with" sysV scripts again, which step exactly will trigger the initial issue which was:
- something starts virtlogd
- virtlogd realizes it has a new dependency (the admin socket)
- the socket can't start as the service is up
- virtlogd considers itself UP-But-Returns RC=1
- install breaks

I want to make sure the steps there are really understood, and maybe "just" dropping the virtlogd sysV will be the middle path that makes it work.
Will be back after a debug session of the exact step that triggers the initial issue ...

Trying an interim TL;DR:
- Service A - --no-stop-on-upgrade
  A Requires A1.socket
  A Requires A2.socket
- Service B - --restart-after-upgrade
  B Requires=A1.socket
- A and B have also sysV scripts.
- d/rules calls dh_installinit and dh_systemd_start
- maintscripts get

This is confusing enough - re-summarize the approaches I had so far.

Remember:
- when there is a sysV script it will call invoke.rc
- without sysV script it will call deb-systemd-invoke

Original Issue:
- invoke.rc on virtlogd
  - this will realize the new dependency to virtlogd-admin.socket
  - virtlogd-admin.socket can't be started because
    virtlogd-admin.socket: Socket service virtlogd.service already active, refusing.
  - virtlogd.service is running fine, but the start returns RC!=0
  - that makes the upgrade fail

Fix I:
- drop both sysV scripts (virtlogd/libvirtd)
  - virtlogd will be started via deb-systemd-invoke
    This ignores errors in the postinst and is fine on upgrade
  - but libvirtd being taken over by deb-systemd-invoke is bad
    - the dh_systemd_start checks the service file and adds dependencies to the line
      deb-systemd-invoke restart 'libvirtd.service' 'virtlockd.socket' 'virtlogd.socket'
    - it knows that to restart a socket the service has to be stopped and started
    - so virtlogd is restarted ignoring the --no-restart-on-upgrade of the actual virtlogd
      service

Fix II:
- drop only virtlogd sysV script
  - virtlogd will be started via deb-systemd-invoke
    This ignores errors in the postinst and is fine on upgrade
  - libvirtd will continue to be started by invoke.rc which will restart "just" libvirtd
    (from dh_installinit)
  - but dh_systemd_start will have added a restart section
    deb-systemd-invoke $_dh_action 'virtlockd.socket' 'virtlogd.socket' >/dev/null || true
  - this will still restart the virtlogd service which has originally
    dh_installinit -p libvirt-daemon-system --name=virtlogd --no-restart-on-upgrade
  - We have two dh-systemd_start calls in d/rules, assuming retain order
    dh_systemd_start -p libvirt-daemon-system --restart-after-upgrade libvirtd.service
    is the one which makes it generate the restart on the sockets.
    It seems to realize that libvirtd will be started by invoke.rc, so it leaves that out, but
    will trigger the two dependencies.

Fix III:
- drop virtlogd sysV script and change the dh_systemds_start order
  - hope is that the dh_systemd_start of libvirtd considers the .socket
    files already handled and would no more add them with restart due to dependencies.
  - It changed the order in the generated maintainer script
  - But the section triggered by libvirtd.service still adds the sockets to the restart action
    deb-systemd-invoke $_dh_action 'virtlockd.socket' 'virtlogd.socket' >/dev/null || true
  - Please do mind, as in other cases here libvirtd itself is not here
    as it is taken over by the sysV start via invoke.rc

Fix IV:
- drop both sysV script, set compat 11, convert to dh_installsystemd
  - Intention: Use the new code only and check if it does any better?
  - This has so many other unrelated things due to the compat change that I won't go on this path
    unless ...

Was incomplete - once more

Trying an interim TL;DR:
- Service A - --no-stop-on-upgrade
  A Requires A1.socket
  A Requires A2.socket
- Service B - --restart-after-upgrade
  B Requires=A1.socket
- A and B have also sysV scripts.
- d/rules calls dh_installinit and dh_systemd_start
- maintscripts get:
  - invoke.rc start A - ok
  - invoke.rc restart B - ok
  - deb-systemd-invoke A.socket - this will restart A.service and breaks --no-stop-on-upgrade

Fix V:
 - drop virtlogd sysV script (to fix the original issue) and drop the dh_systemd__start call to
   libvirtd (to avoid the secondary issue)
 - Intention: libvirtd (re)start is taken care of by dh_installinit anyway, avoid the bad restarts
   on virtlogd with this tweak
 - with that it seems to work, but it might have other implications that I missed
   - the new virtlogd-admin.socket is down (as it would need to restart the service)
   - service itself is up and still has the old PID so all is good
   - installation works, no more breaking the upgrade.

I'll add this finding to the Debian report to get their opinion on all of this

Upgrade output with that:
Setting up libvirt-daemon-system (4.6.0-1ubuntu1~ppa8) ...
Installing new version of config file /etc/apparmor.d/abstractions/libvirt-qemu ...
Installing new version of config file /etc/apparmor.d/usr.sbin.libvirtd ...
Installing new version of config file /etc/default/libvirt-guests ...
Installing new version of config file /etc/libvirt/libvirtd.conf ...
Installing new version of config file /etc/libvirt/libxl.conf ...
Installing new version of config file /etc/libvirt/qemu.conf ...
Installing new version of config file /etc/libvirt/virtlockd.conf ...
Installing new version of config file /etc/libvirt/virtlogd.conf ...
Created symlink /etc/systemd/system/sockets.target.wants/virtlockd-admin.socket → /lib/systemd/system/virtlockd-admin.socket.
Created symlink /etc/systemd/system/sockets.target.wants/virtlogd-admin.socket → /lib/systemd/system/virtlogd-admin.socket.
virtlockd.service is a disabled or a static unit, not starting it.
virtlogd.service is a disabled or a static unit, not starting it.
Job for virtlogd-admin.socket failed.
See "systemctl status virtlogd-admin.socket" and "journalctl -xe" for details.
Removing obsolete conffile /etc/init.d/virtlogd ...
Setting up libvirt-daemon dnsmasq configuration.
Setting up libvirt-daemon-driver-storage-rbd (4.6.0-1ubuntu1~ppa8) ...
Processing triggers for libc-bin (2.27-3ubuntu1) ...

RC is 0

Status check after upgrade:

● virtlogd.service - Virtual machine log manager
   Loaded: loaded (/lib/systemd/system/virtlogd.service; indirect; vendor preset: enabled)
   Active: active (running) since Fri 2018-08-10 11:21:55 UTC; 56s ago
     Docs: man:virtlogd(8)
           https://libvirt.org
 Main PID: 3191 (virtlogd)
    Tasks: 2 (limit: 4915)
   CGroup: /system.slice/virtlogd.service
           └─3191 /usr/sbin/virtlogd

Aug 10 11:22:50 c2 systemd[1]: virtlogd.service: Failed to reset devices.list: Operation not permitted
Aug 10 11:22:50 c2 systemd[1]: virtlogd.service: Failed to reset devices.list: Operation not permitted

● virtlogd.socket - Virtual machine log manager socket
   Loaded: loaded (/lib/systemd/system/virtlogd.socket; enabled; vendor preset: enabled)
   Active: active (running) since Fri 2018-08-10 11:21:55 UTC; 56s ago
   Listen: /var/run/libvirt/virtlogd-sock (Stream)
   CGroup: /system.slice/virtlogd.socket
Failed to dump process list, ignoring: No such file or directory

Aug 10 11:21:55 c2 systemd[1]: Listening on Virtual machine log manager socket.

● virtlogd-admin.socket - Virtual machine log manager socket
   Loade...

Submitted my updated findings to Debian

Discussion with Debian systemd folks went great.
It seems mid/long term we will get dh_installsystemd to do the right thing.

For now (Freeze ahead) we want something for Ubuntu to make it actually work now as it is super-unlikely to get the dh_ changes and pick them up in time for this Compar 8 package as it is now.

The solution for us might be to drop both sysV scripts (as tried before) but also drop the "Also=" lines from the install section of libvirtd.service.
That should stop the tools from doing the wrong thing and was also found to potentially actively harm e.g. enable/disable will do so for all sockets as well.
The service already has a requires dependency to the sockets, so we should loose nothing.

I have built and tested this from a PPA.
It works as it should now.

- the new virtlogd-admin.socket is installed and enabled, but off (can't start due to service running already)
- after a reboot or full service restart that is all fine
- the virtlogd service is staying up through the upgrades
- libvirtd itself is restarted as it should (after upgrade)
- the upgrade works with info messages, but not failing the upgrade

I'll suggest that to Debian, but they might wait for the fix in the dh_*systemd* tools.
We will probably have to take the solution as outlined to be in the Feature Freeze.

If pre cosmic release there is a nice other way we can still pick it up as a fix, but that would make it releasable for now.

Also made available to Debian via: https://salsa.debian.org/libvirt-team/libvirt/merge_requests/5

This bug was fixed in the package libvirt - 4.6.0-2ubuntu1

---------------
libvirt (4.6.0-2ubuntu1) cosmic; urgency=medium

  * Merged with Debian unstable (LP: #1786957).
    Among many other new features and fixes this includes fixes
    for (LP: #1754871), Remaining changes:
    - Disable libssh2 support (universe dependency)
    - Disable firewalld support (universe dependency)
    - Set qemu-group to kvm (for compat with older ubuntu)
    - Additional apport package-hook
    - Autostart default bridged network (As upstream does, but not Debian).
      In addition to just enabling it our solution provides:
      + do not autostart if subnet is already taken (e.g. in guests).
      + iterate some alternative subnets before giving up
    - d/p/ubuntu/Allow-libvirt-group-to-access-the-socket.patch: This is
      the group based access to libvirt functions as it was used in Ubuntu
      for quite long.
      + d/p/ubuntu/daemon-augeas-fix-expected.patch fix some related tests
        due to the group access change.
      + d/libvirt-daemon-system.postinst: add users in sudo to the libvirt
        group.
    - ubuntu/parallel-shutdown.patch: set parallel shutdown by default.
    - d/p/ubuntu/enable-kvm-spice.patch: compat with older Ubuntu qemu/kvm
      which provided a separate kvm-spice.
    - Xen related
      - d/p/ubuntu/ubuntu-libxl-qemu-path.patch: this change was split. The
        section that adapts the path of the emulator to the Debian/Ubuntu
        packaging is kept.
      - d/p/ubuntu/ubuntu-libxl-Fix-up-VRAM-to-minimum-requirements.patch: auto
        set VRAM to minimum requirements
      - d/p/ubuntu/xen-default-uri.patch: set default URI on xen hosts
      - Add libxl log directory
      - libvirt-uri.sh: Automatically switch default libvirt URI for users on
        Xen dom0 via user profile (was missing on changelogs before)
    - d/p/ubuntu/apibuild-skip-libvirt-common.h: drop libvirt-common.h from
      included_files to avoid build failures due to duplicate definitions.
    - Update README.Debian with Ubuntu changes
    - Convert libvirt0, libnss_libvirt and libvirt-dev to multi-arch.
    - Enable some additional features on ppc64el and s390x (for arch parity)
      + systemtap, zfs, numa and numad on s390x.
      + systemtap on ppc64el.
    - d/t/control, d/t/smoke-qemu-session: fixup smoke-qemu-session by making
      vmlinuz available and accessible (Debian bug 848314)
    - d/t/control, d/t/smoke-lxc: fix up lxc smoke test isolation
    - Add dnsmasq configuration to work with system wide dnsmasq (drop >18.04,
      no more UCA onto Xenial then which has global dnsmasq by default).
    - d/p/ubuntu/ubuntu_machine_type.patch: accept ubuntu types as pci440fx
    - Further upstreamed apparmor Delta, especially any new one
      Our former delta is split into logical pieces and is either Ubuntu only
      or is part of a continuous upstreaming effort.
      Listing related remaining changes in debian/patches/ubuntu-aa/:
      + 0001-apparmor-Allow-pygrub-to-run-on-Debian-Ubuntu.patch: apparmor:
        Allow pygrub to run on Debian/Ubuntu
      + 0003-apparmor-libvirt-qemu-Allow-read-access-to-overcommi.patch:
        ...