[MIR] make pcs the default management tool for Corosync/Pacemaker clusters

Bug #1953341 reported by Lucas Kanashiro
10
This bug affects 1 person
Affects Status Importance Assigned to Milestone
pcs (Ubuntu)
Fix Released
Undecided
Unassigned

Bug Description

[Availability]

pcs is already in universe since Ubuntu Xenial. It is an arch:all package and it builds fine in amd64. Package LP page:

https://launchpad.net/ubuntu/+source/pcs

[Rationale]

Nowadays, pcs is the main tool used to manage Corosync/Pacemaker clusters out there. Redhat is the upstream for most of this stack and they do maintain and use it. Right now, we do recommend the other option which is crmsh (this is still used by Suse), but the Server team has been working to move to pcs which is a more modern and easier tool to use when the subject is managing clusters.

It has a nice CLI to set up a cluster and manage resources and stonith agents, also a nice web UI to manage all of that. Recently, the Server team migrated all the HA testing infrastructure to use pcs instead of crmsh, to validate that the tool is working well in some expected scenarios:

https://github.com/canonical/server-test-scripts/pull/164

Those tests are running as Jenkins jobs here:

https://jenkins.canonical.com/server-team/view/ha/

Ideally, we expect that pcs will be promoted in the "L" development cycle. The idea is to promote only the pcs binary, and not pcs-snmp binary for now.

[Security]

Required links:

https://cve.mitre.org/cgi-bin/cvekey.cgi?keyword=pcsd

I looked for pcsd (which is the daemon name) since pcs brings a bunch of unrelated CVEs. As you can see in the CVE database, most of the vulnerabilities were discovered prior 2019. Since then, the project seems more stable.

https://www.openwall.com/lists/oss-security/2018/04/09/2

In the oss-security mailing list I was able to find only the email above.

https://ubuntu.com/security/cves?package=pcs

And in the Ubuntu security tracker only 6 results, most of them seems to be considered not too important. Moreover, this is in universe.

pcs does install two executable files in /usr/sbin:

- /usr/sbin/pcs
- /usr/sbin/pcsd

It also installs two systemd service files:

- /lib/systemd/system/pcsd-ruby.service
- /lib/systemd/system/pcsd.service

It also provides a PAM config file (/etc/pam.d/pcsd) with the following content:

#%PAM-1.0
@include common-auth
@include common-account
@include common-password
@include common-session

And the pcsd daemon runs a service binded to port 2224 by default.

[Quality assurance - function/usage]

Once the package is installed in a system it is ready to use. It configures a single node Pacemaker cluster and one can play with it out-of-the-box.

[Quality assurance - maintenance]

The package is well maintained in Debian and Ubuntu. The Server team has been in touch with the Debian maintainer and working closely in some bug fixes and improvements.

https://bugs.launchpad.net/ubuntu/+source/pcs/+bugs

Only 2 bugs reported in Ubuntu (one there is this MIR bug), both of them against Xenial.

And no bugs reported against Debian:

https://bugs.debian.org/cgi-bin/pkgreport.cgi?src=pcs

Upstream is very responsive. They are moving away from the legacy ruby code, the code base will be python only in the future, which will also ease the maintenance of the package.

[Quality assurance - testing]

Upstream has a good test suite which is executed during package build time:

https://launchpadlibrarian.net/618937705/buildlog_ubuntu-kinetic-amd64.pcs_0.11.3-1ubuntu1_BUILDING.txt.gz

The package contains some DEP-8 tests which are not superficial, they test the cluster setup and checking its status, and also the pcs-snmp binary feature:

https://autopkgtest.ubuntu.com/packages/pcs

The tests is being skipped in armhf because it seems to run in unprivileged namespaces. And failing in i386 because some dependencies are not installable. The tests are passing in all other supported architectures.

As explained in the first section, the Server team also does some testing on our own testing infrastructure.

[Quality assurance - packaging]

debian/watch is present and works

debian/control defines a correct Maintainer field

Lintian overrides are not present. Here is the output of `lintian --pedantic` against the latest version in kinetic:

W: pcs source: newer-standards-version 4.6.1 (current is 4.6.0.1)
P: pcs source: very-long-line-length-in-source-file pcs/pcs.8.in line 99 is 911 characters long (>512)
P: pcs source: very-long-line-length-in-source-file pcs_test/tier1/legacy/test_resource.py line 179 is 613 characters long (>512)
P: pcs source: very-long-line-length-in-source-file pcs_test/tier1/legacy/test_stonith.py line 683 is 528 characters long (>512)
P: pcs source: very-long-line-length-in-source-file ... use --no-tag-display-limit to see all (or pipe to a file/program)

This package does not rely on obsolete or about to be demoted packages.

The package will not be installed by default

Packaging and build is easy: https://git.launchpad.net/ubuntu/+source/pcs/tree/debian/rules

[UI standards]

N/A

[Dependencies]

There are further dependencies that are not yet in main, the MIR process for them is handled as part of this bug here (TODO).

The dependencies of the pcs binary only (not considering build and pcs-snmp dependencies) needing promotion (according to check-mir script and apt-cache depends) are:

## Ready to be promoted:

- python3-dacite
  + MIR bug: https://bugs.launchpad.net/ubuntu/+source/dacite/+bug/1989628
- python3-tornado
  + MIR bug: https://bugs.launchpad.net/ubuntu/+source/python-tornado/+bug/1990191
- ruby-ethon
  + MIR bug: https://bugs.launchpad.net/ubuntu/+source/ruby-ethon/+bug/1990571
- ruby-mime-types
  + MIR bug: https://bugs.launchpad.net/ubuntu/+source/ruby-mime-types/+bug/1990569
- ruby-mime-types-data
  + MIR bug: https://bugs.launchpad.net/ubuntu/+source/ruby-mime-types-data/+bug/1990568
- ruby-sinatra
  + MIR bug: https://bugs.launchpad.net/ubuntu/+source/ruby-sinatra/+bug/1990579
- ruby-rack
   + MIR bug: https://bugs.launchpad.net/ubuntu/+source/ruby-rack/+bug/1990575
 - ruby-mustermann
   + MIR bug: https://bugs.launchpad.net/ubuntu/+source/ruby-mustermann/+bug/1990574
 - ruby-ruby2-keywords
   + MIR bug: https://bugs.launchpad.net/ubuntu/+source/ruby-ruby2-keywords/+bug/1990573
 - ruby-tilt
   + MIR bug: https://bugs.launchpad.net/ubuntu/+source/ruby-tilt/+bug/1990576
- ruby-ffi (ruby-ethon dep)
  + MIR bug: https://bugs.launchpad.net/ubuntu/+source/ruby-ffi/+bug/1990570
- ruby-json
  + MIR bug: https://bugs.launchpad.net/ubuntu/+source/ruby-json/+bug/1990572
- ruby-childprocess
  + MIR bug: https://bugs.launchpad.net/ubuntu/+source/ruby-childprocess/+bug/1991839
- ruby-nio4r (puma dep)
  + MIR bug: https://bugs.launchpad.net/ubuntu/+source/ruby-nio4r/+bug/2006464
- puma
  + MIR bug: https://bugs.launchpad.net/ubuntu/+source/puma/+bug/2006461

[Standards compliance]

This package correctly follows FHS and Debian Policy

[Maintenance/Owner]

Owning Team will be the Server team
Team is not yet, but will subscribe to the package before promotion

This does not use static builds

This does not use vendored code

This package is not rust based

The package has been built in the archive more recently than the last test rebuild

[Background information]

The Package description explains the package well.

Upstream Name is: pcs

Link to upstream project: https://github.com/ClusterLabs/pcs

[Original description]

Nowadays, crmsh is the default and supported tool to manage Corosync/Pacemaker clusters, but in the future we plan to move to pcs. We need to put some tests in place and make sure the entire stack is sill working fine with it, moreover, we need to update all the docs to tell users about the changes.

Tags: sec-1414

CVE References

description: updated
description: updated
description: updated
description: updated
description: updated
description: updated
description: updated
Changed in pcs (Ubuntu):
assignee: nobody → Ioanna Alifieraki (joalif)
Revision history for this message
Seth Arnold (seth-arnold) wrote :

One concern I have is that upstream is moving away from the Ruby versions of packages towards Python versions of packages: will this transition be complete before 24.04 LTS release? Will it be done earlier enough than 24.04 LTS for us to have confidence in it?

I'd like to avoid us shipping software that upstream plans to abandon either shortly after we ship it, or shortly before we ship it.

Thanks

Revision history for this message
Lucas Kanashiro (lucaskanashiro) wrote :

This is a valid concern Seth, thanks for the comment. I was thinking about the same and I asked upstream about it here [1], and their answer is basically that there is no timeline [2]. They are moving some pieces to python in their own pace, I do not think we should block this MIR because of this code rewrite.

[1] https://github.com/ClusterLabs/pcs/issues/264#issuecomment-1226266729
[2] https://github.com/ClusterLabs/pcs/issues/264#issuecomment-1235654386

Revision history for this message
Mark Esler (eslerm) wrote :

My take away from the MIR Team is that it doesn't matter if upstream abandons Ruby packages.

Revision history for this message
Ioanna Alifieraki (joalif) wrote (last edit ):
Download full text (5.5 KiB)

Review for Package: pcs

[Summary]
The package does not have any issues apart from having many dependencies
that need to be pulled into main and needing a sec review.
I noticed that there are 2 depencies (ruby-open4 and python3-pyagentx) reported
by check-mir that are not in the long list of depencencies and there's no
MIR bug filed for those.

MIR team ACK under the constraint to resolve the below listed
required TODOs and as much as possible having a look at the
recommended TODOs.
This does need a security review, so I'll assign ubuntu-security.
List of specific binary packages to be promoted to main: pcs
Specific binary packages built, but NOT to be promoted to main: pcs-snmp

Notes:
Required TODOs:
1. There are 2 packages that need MIR (not yet in the known long list of dependencies):
   - ruby-open4 https://bugs.launchpad.net/ubuntu/+source/ruby-open4
   - python3-pyagentx https://launchpad.net/ubuntu/+source/pyagentx

2. The long list of ruby* packages that MIR review is in process :
- python3-dacite
  + MIR bug: https://bugs.launchpad.net/ubuntu/+source/dacite/+bug/1989628
- python3-tornado
  + MIR bug: https://bugs.launchpad.net/ubuntu/+source/python-tornado/+bug/1990191
- ruby-backports
  + MIR bug: https://bugs.launchpad.net/ubuntu/+source/ruby-backports/+bug/1990565
- ruby-ethon
  + MIR bug: https://bugs.launchpad.net/ubuntu/+source/ruby-ethon/+bug/1990571
  + ruby-ffi
    * MIR bug: https://bugs.launchpad.net/ubuntu/+source/ruby-ffi/+bug/1990570
  + ruby-mime-types
    * MIR bug: https://bugs.launchpad.net/ubuntu/+source/ruby-mime-types/+bug/1990569
    * ruby-mime-types-data
      - MIR bug: https://bugs.launchpad.net/ubuntu/+source/ruby-mime-types-data/+bug/1990568
- ruby-json
  + MIR bug: https://bugs.launchpad.net/ubuntu/+source/ruby-json/+bug/1990572
  + ruby-childprocess
    * MIR bug: https://bugs.launchpad.net/ubuntu/+source/ruby-childprocess/+bug/1991839
- ruby-sinatra
  + MIR bug: https://bugs.launchpad.net/ubuntu/+source/ruby-sinatra/+bug/1990579
  + ruby-rack
    * MIR bug: https://bugs.launchpad.net/ubuntu/+source/ruby-rack/+bug/1990575
  + ruby-rack-protection (provided by src:ruby-sinatra)
    * MIR bug: https://bugs.launchpad.net/ubuntu/+source/ruby-sinatra/+bug/1990579
  + ruby-mustermann
    * MIR bug: https://bugs.launchpad.net/ubuntu/+source/ruby-mustermann/+bug/1990574
    * ruby-ruby2-keywords
      - MIR bug: https://bugs.launchpad.net/ubuntu/+source/ruby-ruby2-keywords/+bug/1990573
  + ruby-tilt
    * MIR bug: https://bugs.launchpad.net/ubuntu/+source/ruby-tilt/+bug/1990576
- thin
  + MIR bug: https://bugs.launchpad.net/ubuntu/+source/thin/+bug/1990582
  + ruby-daemons
    * MIR bug: https://bugs.launchpad.net/ubuntu/+source/ruby-daemons/+bug/1990581
  + ruby-eventmachine
    * MIR bug: https://bugs.launchpad.net/ubuntu/+source/ruby-eventmachine/+bug/1990580

3. The package should get a team bug subscriber before being promoted

[Duplication]
There is the crmsh package already in universe providing the same functionality.
However, the [Rational] section justifies pcs to be promoted to main.

[Dependencies]
OK:
- no -dev/-debug/-doc packages that need exclusion
- No dependencies in main that are only supe...

Read more...

Changed in pcs (Ubuntu):
status: New → Incomplete
Revision history for this message
Lucas Kanashiro (lucaskanashiro) wrote :

Thanks for the review Ioanna!

Comments about the TODO list:

1. The 2 packages you mentioned will not require promotion. I still need to prepare an upload to Ubuntu with some changes, but let me explain the rationale:

- ruby-open4: This ruby gem is not well maintained upstream (the last commit is from many years ago), so we decided to work with upstream to move to a more well maintained gem. We decided then to use childprocess, I submitted a patch upstream with the replace and it was already merged:

https://github.com/ClusterLabs/pcs/pull/568

I need to apply this patch to our pcs package and upload to Lunar, this is in my TODO list and will be done soon. FWIW the ruby-childprocess MIR is also ongoing, waiting for the Security team review.

- python3-pyagentx: This is a dependency only for pcs-snmp, and as I explicitly mentioned in the last paragraph of the "Rationale" section in the bug description, we do not want to promote pcs-snmp, only pcs.

2. As you may know, all the mentioned MIRs are ongoing and hopefully everything will be ready to be promoted soon.

3. The Server team is already subscribed to the package.

With all the explaination above, could we assign the bug to the Security team so they can perform the needed review?

Revision history for this message
Ioanna Alifieraki (joalif) wrote :

@Lucas,

Thank you very much for the clarifications!
I missed the bit that pcs-snmp will not be promoted (I also fixed that in my review).
I'll assign to security now.

Changed in pcs (Ubuntu):
assignee: Ioanna Alifieraki (joalif) → nobody
status: Incomplete → Confirmed
assignee: nobody → Ubuntu Security Team (ubuntu-security)
Revision history for this message
Ioanna Alifieraki (joalif) wrote :

This is ACKed from MIR team and requires a security review.

Steve Beattie (sbeattie)
tags: added: sec-1414
description: updated
Revision history for this message
Mark Esler (eslerm) wrote :

pcs is replacing thin with puma https://github.com/ClusterLabs/pcs/pull/632

I believe we need to drop MIRs for thin and ruby-eventmachine and add MIRs for puma and ruby-nio4r.

Revision history for this message
Lucas Kanashiro (lucaskanashiro) wrote :

I removed thin, ruby-eventmachine and ruby-daemons from the description of the bug (they are not needed anymore due to the comment above), and added puma and ruby-nio4r MIR bugs link:

- https://bugs.launchpad.net/ubuntu/+source/puma/+bug/2006461
- https://bugs.launchpad.net/ubuntu/+source/ruby-nio4r/+bug/2006464

Waiting for the MIR and Security teams review.

description: updated
description: updated
Revision history for this message
Lucas Kanashiro (lucaskanashiro) wrote :

I uploaded pcs/0.11.4-1ubuntu3 replacing thin with puma.

description: updated
description: updated
Revision history for this message
Mark Esler (eslerm) wrote :
Download full text (5.6 KiB)

I reviewed pcs 0.11.4-1ubuntu3 as checked into lunar. This shouldn't be considered a full audit but rather a quick gauge of maintainability.

- CVE History:
  - eleven past CVEs
  - lunar package currenty affected by CVE-2022-2735
  - has a Security Policy \o/
    - https://github.com/ClusterLabs/pcs/blob/main/SECURITY.md
  - upstream communicates vulnerabilities well
  - upsteam responds to reports quickly
- Build-Depends?
  - lunar main
    - debhelper-compat (debhelper)
    - libpam0g-dev (pam)
    - pkg-config
    - psmisc
    - puma (mir ack'd)
    - python3
    - python3-cryptography (python-cryptography)
    - python3-dacite (dacite, ack'd to main)
    - python3-dateutil (python-dateutil)
    - python3-lxml (lxml)
    - python3-pycurl (pycurl)
    - python3-pyparsing (pyparsing)
    - python3-tornado (python-tornado, ack'd to main)
    - ruby
    - ruby-childprocess (ack'd to main)
    - ruby-dev
    - ruby-ethon (ack'd to main)
    - ruby-sinatra (ack'd to main)
    - ruby-rubygems (rubygems)
    - ruby-webrick
    - systemd
    - wget
  - lunar universe
    - dh-python
    - python3-pip (python-pip)
    - python3-pyagentx (pyagentx)
    - python3-setuptools (python-setuptools)
    - python3-setuptools-scm (setuptools-scm)
    - ruby-json
    - ruby-rack-test
- pre/post inst/rm scripts?
  - sets permissions and links services
- init scripts?
  - from debian/pcs.*.init
    - /etc/init.d/pcsd
    - /etc/init.d/pcsd-ruby
    - /etc/init.d/pcs_snmp_agent
  - like systemd services, but more rudimentary
    - no env loading
- systemd units?
  - minimal systemd services to load env and manage processes
    - /lib/systemd/system/pcsd-ruby.service
    - /lib/systemd/system/pcsd.service
    - /lib/systemd/system/pcs_snmp_agent.service
- dbus services?
  - none
- setuid binaries?
  - none
- binaries in PATH?
  - /usr/sbin/pcs
  - /usr/sbin/pcsd
- etc files
  - /etc/default/pcsd
  - /etc/logrotate.d/pcsd
  - /etc/pam.d/pcsd
  - /etc/default/pcs_snmp_agent
- sudo fragments?
  - none
- polkit files?
  - none
- udev rules?
  - none
- unit tests / autopkgtests?
  - contains (many) build tests
    - 6230 total
    - they are even distinguished by tier
  - has autopkgtests
    - lunar failing amd64!
- cron jobs?
  - none
- Build logs:
  - E: pcs: depends-on-obsolete-package Depends: lsb-base (>= 3.0-6)

- Processes spawned?
  - heavy use
  - kill_services effectively runs killall -9
  - most handled by lib.external.CommandRunner which is used by lib.utils.cmd_runner()
    - lib.utils.run() is present and marked as deprecated in favor of CommandRunner
    - api inherits this power
    - extremely powerful if misused
- Memory management?
  - standard python and ruby
- File IO?
  - heavy use
  - mostly standard python and some ruby
- Logging?
  - lots of logging, reporting, and debug support
  - looks well thought out and safe
- Environment variable usage?
  - heavy use
    - especially in the context of executing processes
  - low sanitization for env or other inputs generally
  - systemd services load specific env file
- Use of privileged functions?
  - ability to change file permissions
  - seven if branches depend on uid/euid 0
    - this encourages r...

Read more...

Changed in pcs (Ubuntu):
assignee: Ubuntu Security Team (ubuntu-security) → nobody
status: Confirmed → In Progress
Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

Showing up in component mismatch now as planned.

Changed in pcs (Ubuntu):
status: In Progress → Fix Committed
Revision history for this message
Christian Ehrhardt  (paelzer) wrote :
Download full text (10.0 KiB)

I'm collecting and ensuring the correct status in all related cases right now.
Most needed just a status update to match.

A few are special
- Dacite was already promoted in https://bugs.launchpad.net/ubuntu/+source/dacite/+bug/1989628 2h ago.
- ruby-json was fully processed but is (Seems due to changes in dependencies by changes needed to be done as part of the other review feedback) no more showing up in component mismatches. I checked and it isn't needed to install and run PCS in Lunar. Hence that will not be promoted. I'll update bug 1990572
- python-tornado was in main, demoted later and now comes back, bug updated accordingly
- Some needed special care as e.g. the component mismatch checker only finds the oldest related MIR bug - On tornado I updated and reverse linked them from the old case, just in case one sees this and wants to track down later changes.

All package subscriptions are ok (on the server team), all cases MIR and Security processing is complete.

All 14 cases are "only" in lunar, there are no builds of them in proposed we'd also need to consider right now (could overwrite once they migrate).

None of the cases lists related binaries that are supposed to be withheld.
But actually the binary ruby-sinatra-contrib would not have a dependency path, AFAICS it would be auto-demoted just after - so not promoting it in the first place.

python-tornado-doc would be held via auto-include and has no dependencies in universe - so it is ok to move as well.

The only related thing in proposed is pacemaker changing the dependency, that LGTM (tests ok as far as they ran) and could migrate once these component mismatches are resolved.

Promoting them to main:
Override component to main
pcs 0.11.4-1ubuntu3 in lunar: universe/misc -> main
pcs 0.11.4-1ubuntu3 in lunar amd64: universe/admin/extra/100% -> main
pcs 0.11.4-1ubuntu3 in lunar arm64: universe/admin/extra/100% -> main
pcs 0.11.4-1ubuntu3 in lunar armhf: universe/admin/extra/100% -> main
pcs 0.11.4-1ubuntu3 in lunar i386: universe/admin/extra/100% -> main
pcs 0.11.4-1ubuntu3 in lunar ppc64el: universe/admin/extra/100% -> main
pcs 0.11.4-1ubuntu3 in lunar riscv64: universe/admin/extra/100% -> main
pcs 0.11.4-1ubuntu3 in lunar s390x: universe/admin/extra/100% -> main
pcs-snmp 0.11.4-1ubuntu3 in lunar amd64: universe/admin/optional/100% -> main
pcs-snmp 0.11.4-1ubuntu3 in lunar arm64: universe/admin/optional/100% -> main
pcs-snmp 0.11.4-1ubuntu3 in lunar armhf: universe/admin/optional/100% -> main
pcs-snmp 0.11.4-1ubuntu3 in lunar i386: universe/admin/optional/100% -> main
pcs-snmp 0.11.4-1ubuntu3 in lunar ppc64el: universe/admin/optional/100% -> main
pcs-snmp 0.11.4-1ubuntu3 in lunar riscv64: universe/admin/optional/100% -> main
pcs-snmp 0.11.4-1ubuntu3 in lunar s390x: universe/admin/optional/100% -> main
ruby-ethon 0.16.0-1 in lunar: universe/misc -> main
ruby-ethon 0.16.0-1 in lunar amd64: universe/ruby/optional/100% -> main
ruby-ethon 0.16.0-1 in lunar arm64: universe/ruby/optional/100% -> main
ruby-ethon 0.16.0-1 in lunar armhf: universe/ruby/optional/100% -> main
ruby-ethon 0.16.0-1 in lunar i386: universe/ruby/optional/100% -> main
ruby-ethon 0.16.0-1 in lunar ppc64el: universe/...

Changed in pcs (Ubuntu):
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.