uninstalling chrony leaves no time service running

Lets break this into use cases in Bionic:

I was not sure who should win in each case.
We might either want the clear "order" chrony > ntp > openntp > systemd-timesyncd
Or we might want a "last installed" approach, but that is hard as upgrades to not count here only real "install". What would "--reinstall" be in these cases?
Maybe we should stick with the clear order, that at least seems more deterministic.
Cases 4-6 try to cover testing that order invariancy.

This is an "ideal world" approach, not sure if we can achieve that in the short term.
After the "=>" assignment is the service that should run (and only this one).

0. default install - systemd-timesyncd

1. default install - install chrony => Chrony
1b. - remove chrony => systemd-timesyncd

2. default install - install ntp => NTP
2b. - remove ntp => systemd-timesyncd

3. default install - install openntp => openntp
3b. - remove openntp => systemd-timesyncd

4. default install - install ntp, install chrony => Chrony
4b. remove chrony => NTP
4c. remove NTP => systemd-timesyncd

5. default install - install chrony, install NTP => Chrony
5b. remove Chrony => NTP
5c. remove NTP => systemd-timesyncd

6. default install - install openntp => openntp
6b. install NTP => NTP
6c. install chrony => chrony
6d. remote NTP & Chrony => openntp
6e. remove openntp => systemd-timesyncd

7. xenial with ntp - upgrade to B => NTP

8. xenial with ntp - upgrade to B, install chrony => Chrony

9. xenial with ntp - upgrade to B, remove NTP => systemd-timesyncd

10. xenial without ntp - upgrade to B => systemd-timesyncd

Status right now:
ok - 0. systemd-timesyncd

ok - 1. default install - install chrony => Chrony
fail - 1b. - remove chrony => none (ok after reboot)

ok - 2. default install - install ntp => NTP
fail - 2b. - remove ntp => none (ok after reboot)

ok - 3. default install - install openntp => openntp
fail - 3b. - remove openntp => none (ok after reboot)

ok - 4. default install - install ntp, install chrony => Chrony
fail - 4b. remove chrony => none (NTP masked by chrony install)
fail - 4c. remove NTP => systemd-timesyncd

fail - 5. default install - install chrony, install NTP => NTP (masked chrony on install)
ok - 5b. remove Chrony => NTP
fail - 5c. remove NTP => systemd-timesyncd

ok - 6. default install - install openntp => openntp
ok - 6b. install NTP => NTP
ok - 6c. install chrony => chrony
fail - 6d. remote NTP & Chrony => openntp (due to conflicts now nothing is her anymore)
fail - 6e. remove openntp => systemd-timesyncd

ok - 7. xenial with ntp - upgrade to B => NTP

ok - 8. xenial with ntp - upgrade to B, install chrony => Chrony

fail - 9. xenial with ntp - upgrade to B, remove NTP => systemd-timesyncd (ok after reboot)

ok - 10. xenial without ntp - upgrade to B => systemd-timesyncd

Several insights on all of this:

"who wins":
install ntp, chrony or openntp (can only be one so take chrony) - reboot - who wins?
-> Seems to work always to be chrony, but why.
-> also matches some tests by waveform
I started a discussion with systemd upstream if there are guarantees.
On IRC it was suggested that it might just be alphanumeric.
I made some tests to check the influence of other things like enabling in different orders.
But it stuck to alphanumeric wins.
In our case that is great as every NTP service name is >> systemd-timesyncd.
While opening a systemd github issue to be sure I found that it is actually random.
And it seems there is no way ton control it.
Until that exists we can re-arrange our POV:
- this is only a "problem between systemd-timesyncd and the servers"
- the old condition to not start when installed would avoid that completely
- a perfect solution (that detects not "installed" but running) could wait until upstream solves it

"masking":
I found that the "Masking" of chrony/ntp services is actually a conflicts in packaging- that is good (because it is the right thing to do).
That means that one who installs one of the NTP servers is removing the others (that is just right so). The packages solve that via a virtual package to not have to list all of each other.
openntp and chrony and ntpsec provide "time-daemon"
They all conflict with time-daemon, so only one of them can be installed even not listing all of them.
The only outlier is ntp (did I mentione we went away from it), but for that all of them explicitly conflict with ntp as well.

"Severity" -> low/med:
Of the above of case 1-3 were worse on Xenial
Xenial => systemd-timesyncd
Xenial - install ntpd => ntpd AND systemd-timesyncd running
Xenial - then after reboot, only ntpd
This is due to ntp being a sysV script in Xenial and has no way to express the conflicts.
Due to that both run, only after a reboot the condition in the old systemd-timesyncd would prevent it from starting.
But it would even have done so if ntp was only installed, but not running.
And while broken we never had bugs about the above in the last two years.
And the conflicts behavior is better than that at least.

Summarizing what we really need now:
1. better completeness of those services conflicting with each other (simple add to their .service files) - one "direction" of conflicts is enough so fix those in main (chrony in that case)
2. determinism (which starts if multiple are installed)
   For now re-vive what https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=873185 dropped
3. after removal of the NTP daemons we'd want systemd-timesyncd to pick up again immediately (without a reboot)

Linked the upstream discussion on the determinism of conflicting services.

#1 is solved by the package conflicts
#2 might be solved by very suble wording in systemd
#3 might be low prio as it was never different

Going on to check #2 atm

#2 here some theory by smoser (thanks)
http://paste.ubuntu.com/p/6JDQ9ycJhw/

I tested on that and it works reliable.

TL;DR as long as we have ONE base service "=conflictee" and ONE extra service "=conflicter".
then systemd will always run the second.

Fortunately in Debian/Ubuntu the apt level conflicts will make sure the above constraint is true.
I'm now on more than 50 tests with my former case modified and we are good.

So #2 is ok as well.

This leaves only #3 to be considered to think about a potential fix.

Added the related Debian bug I opened.
Also renamed as (for now) here we only address #3 of the issues described.

If conclusion with Debian and/or Upstream is reached we can use that as well.
But with the remaining uncertainty (and all real tests being good) we won't change anything for #2 yet.

For the remaining issue #3 - on Chrony remove we know that:
1. no other time server is there anymore (due to conflicts)
2. we are going away

So we could on postrm do something like this:

if systemctl is-enabled systemd-timesyncd
    systemctl start systemd-timesyncd
fi

Test PPA (for #3) building in https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/3242

Fix uploaded to bionic-unapproved in 3.2-4ubuntu4

This bug was fixed in the package chrony - 3.2-4ubuntu4

---------------
chrony (3.2-4ubuntu4) bionic; urgency=medium

  * d/postrm: re-establish systemd-timesyncd on removal (LP: #1764357)
  * Notify chrony to update sources in response to systemd-networkd
    events (LP: #1718227)
    - d/links: link dispatcher script to networkd-dispatcher events routable
      and off
    - d/control: set Recommends to networkd-dispatcher
    - d/p/lp-1718227-ignore-non-up-down-events-in-nm-dispatcher.patch
    - d/p/lp-1718227-nm-dispatcher-for-networkd.patch

 -- Christian Ehrhardt <email address hidden> Mon, 16 Apr 2018 17:04:06 +0200