Pacemaker package upgrades stop but fail to start pacemaker resulting in HA outage

From the charm perspective we need to determine if the charm does anything beyond the packaging that could lead to this.

The charm runs:
update-rc.d -f pacemaker defaults

Testing.

Status changed to 'Confirmed' because the bug affects multiple users.

Hmm

https://github.com/ClusterLabs/pacemaker/commit/fc11a4651b971bd3b9dfc15b8b5c538c7ee9ab75#diff-06baf804b1f5edee72fb63df4b884b7c

And

https://github.com/ClusterLabs/pacemaker/pull/656

Other related bugs - bug 1322899 and bug 1052449

I think this is a packaging/upstream bug which I appear to have had a go at before (see #4); however I've not seen this specific symptom before.

Marking charm bug task as invalid - its doing the right things, however the package does not declare default start/stop levels so its all foobar.

Hmmm - however pacemaker@xenial ships with a native systemd unit; so I'm baffled as to the behaviour of the package upgrade.

This issue can be fixed by re-adding the required Start/Stop bits to the LSB header; however the behaviour of update-rc.d when native systemd unit files are in use looks odd to me.

Raising an init-system-helpers bug task for foundations team input on this.

Actually this only impacts Xenial; Debian carried a patch from:

pacemaker (1.1.15~rc3-1) unstable; urgency=medium

  [ Christoph Berg ]
  * [23ee108] libcrmservice3.symbols: Exclude systemd symbol on non-linux
  * [41dea05] Fix time formatting on x32
  * [533c5cc] Fix FTBFS on GNU Hurd

  [ Arturo Borrero Gonzalez ]
  * [698053d] d/tests/control: add isolation-container restriction

  [ Ferenc Wágner ]
  * [065159d] New patch Enable-the-init-scripts-on-multi-user-runlevels.patch
  * [7a5008b] New patch Make-the-asciidoc-documentation-reproducible.patch
  * [08a4162] New patch Add-remote_fs-dependencies-to-the-init-scripts.patch
  * [7a65d2c] New upstream release (1.1.15~rc1)
  * [dd9f5f4] Remove upstreamed patches, refresh the rest
  * [0cdd116] Update symbol files
  * [3de7a21] New patch Add-documentation-URIs-to-the-service-files.patch
  * [9faaa02] Move documentation generators into Build-Depends-Indep
  * [9362a46] Move documentation into /usr/share/doc/pacemaker
  * [64c0e84] Move misc documentation files into the pacemaker-doc package
  * [837e74d] New patch Read-default-files-in-pacemaker.service.patch
  * [72fef80] New upstream release (1.1.15~rc2)
  * [69ee575] Remove patch included in 1.1.15~rc2
  * [46dce98] Also add documentation URI to our version of crm_mon.service
  * [8c25aff] New patch to avoid using WIFCONTINUED entirely
  * [a28e0ae] New upstream release (1.1.15~rc3)
  * [a8976fe] Remove freshly upstreamed patches, refresh the rest

 -- Ferenc Wágner <email address hidden> Sat, 28 May 2016 22:28:49 +0200

Which re-enables the default runlevels.

Proposed update for xenial

Doing some testing via:

  https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/3010

before upload to xenial-proposed; not that the act of fixing this problem will in itself cause the pacemaker daemons to be started on upgrade. However it won't be auto-applied on a default install (unlike the security update that caused the original issue).

Tested OK (pacemaker was restarted after upgrade)

Uploaded to unapproved queue for SRU Team review - I'd suggest we fasttrack this ASAP into -updates and -security to catch auto updates for those systems not already updated.

Updates building in:

  https://launchpad.net/~ubuntu-security-proposed/+archive/ubuntu/ppa

Testing from security-proposed PPA:

Broken install (dpkg-reconfigure pacemaker already run, daemons stopped): Packages updated and pacemaker daemon restarted by postinst: OK

Running install (pacemaker daemons running prior to pkg upgrade): Packages update, pacemaker daemon stop prior to unpack and restarted by postinst: OK

From my perspective the update looks good - but would appreciate a second pair of eyes on this one.

Also tested proposed fix on a three unit gnocchi HA deployment; all pacemaker updates applied and restarted pacemaker postinst as desired.

Tested proposed package upgrade and works for me - http://pastebin.ubuntu.com/25815820/

Also validated that the update fixes an existing machine with the 1.2 update on it (with pacemaker services not running).

Pacemaker started after the update.

Did some touch testing in a VM and confirmed that the 1.2 package loses its daemons on any kind of reconfigured. This behaviour is rectified with the 1.3 package.

This bug was fixed in the package pacemaker - 1.1.14-2ubuntu1.3

---------------
pacemaker (1.1.14-2ubuntu1.3) xenial; urgency=medium

  * Fix default start/stop levels for init scripts, ensuring that
    running pacemaker daemons are restarted after package upgrades
    (LP: #1727063):
    - d/p/Enable-the-init-scripts-on-multi-user-runlevels.patch: Cherry
      picked fix from later package versions.

 -- James Page <email address hidden> Wed, 25 Oct 2017 10:04:30 +0100

The verification of the Stable Release Update for pacemaker has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

re: init-system-helpers, I noticed the oddity of the init script on a systemd system as well and found that there's a systemd hack in /lib/lsb/init-functions.d/40-systemd that allows for multi-startup compatibility. I believe invoke-rc.d should check systemd "enabled/disabled" state instead of just the S/K links in /etc/rcX.d.

Status changed to 'Confirmed' because the bug affects multiple users.

Is systemd's sysv-generator prioritized over regular systemd unit files ? My question raises from this fix. It looks like unit files were automatically created because of the existence of the wrong LSB parameters in init files - by the generator - but at the same time Xenial should be using systemd unit files instead (as they are present in this package).

From a high level, it appears that invoke-rc.d script used for compatibility falls back to checking for /etc/rcX.d symlinks for a "policy" check if there is no $POLICYHELPER installed. Perhaps the actual shortcoming is not having the policy-rc.d installed to prefer systemd over init.d on Xenial.

Confirmed with Drew that this bug has not been hit since the pacemaker fix released.

Can remove field-high team, but keep for the discussion on init-system-helpers.