[SRU] Add LPAR fence agent to Pacemaker (fence-agents)

------- Comment From <email address hidden> 2020-07-29 10:27 EDT-------
There's a bug fix pull request "fence_ibmz: fix handling of API response when LPAR is not found": https://github.com/ClusterLabs/fence-agents/pull/352

Please keep in mind to include it in your builds.

Related commits here are:

commit 06c6cea9
Author: Oyvind Albrigtsen <email address hidden>
Date: Tue Aug 25 11:25:25 2020

    spec: fix ibmz subpackage referring to fence_z instead of fence_ibmz

commit 218d826c
Author: Paulo de Rezende Pinatti <email address hidden>
Date: Tue Jul 28 10:06:42 2020

    fence_ibmz: fix handling of API response when LPAR is not found

Both commits are released in version v4.6.0.

v4.5.2-81-g06c6cea9 - v4.6.0
v4.5.2-74-g218d826c - v4.6.0

So this, together with LP: #1894323 should justify a FFe for v4.6.0.

The bug taking care of the FFe for fence-agents is:

https://bugs.launchpad.net/ubuntu/+source/fence-agents/+bug/1894325

and the merge of this new version will occur there. I'll mark this bug as Fix Released as well, once it is fixed there. Later I'll work in backporting the agent to Focal.

A PPA is now available (thx to Rafael) with groovy's full HA stack (incl. some extras) backported to Focal and Bionic for further testing:
https://launchpad.net/~ubuntu-ha/+archive/ubuntu/groovy-ha-stack
See the packages here for the fence agents:
https://launchpad.net/~ubuntu-ha/+archive/ubuntu/groovy-ha-stack/+packages

------- Comment From <email address hidden> 2020-10-20 09:05 EDT-------
Sorry, this fell a bit through the cracks.

I have verified that all basic operations are working with the backported version on Ubuntu 20.04.1. Do you require explicit testing on Ubuntu 20.10?

Thx '<email address hidden>' for testing on 20.04.1!

Testing on 20.10 is not needed, since the packages are already in groovy/20.10 (groovy was actually the base for the PPA with the backports to focal and bionic).

I backported the needed code to support the fence agent to Bionic and Focal here in this PPA:

https://launchpad.net/~lucaskanashiro/+archive/ubuntu/ha-stack-ibm

The idea would be to SRU those patches and release them to Bionic and Focal, instead of relying only on the PPA with Groovy's full HA stack backported. However, to be able to do that we would need some detailed steps on how to test it. If someone from IBM could confirm the packages from the PPA above are working as expected and provide the steps of what was done to validate it would be awesome, and we could move forward with the SRU process.

------- Comment From <email address hidden> 2021-02-24 05:32 EDT-------
I have done some rudimentary tests with the package on Ubuntu 20.04.1.

# cat /etc/os-release
NAME="Ubuntu"
VERSION="20.04.1 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.1 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal

# apt info fence-agents
Package: fence-agents
Version: 4.5.2-1ubuntu1~ppa2
Priority: optional
Section: admin
Maintainer: Ubuntu Developers <email address hidden>
Original-Maintainer: Debian HA Maintainers <email address hidden>
Installed-Size: 839 kB
Depends: python3:any, libc6 (>= 2.15), python3-boto3, python3-googleapi, python3-pycurl, python3-pexpect, python3-requests, python3-suds
Recommends: libnet-telnet-perl, openssh-client, sg3-utils, snmp
Suggests: python3-adal, python3-azure, python3-keystoneauth1, python3-keystoneclient, python3-novaclient
Breaks: cman (<= 3.0.12-2ubuntu4)
Replaces: cman (<= 3.0.12-2ubuntu4)
Download-Size: 236 kB
APT-Manual-Installed: yes
APT-Sources: http://ppa.launchpad.net/lucaskanashiro/ha-stack-ibm/ubuntu focal/main s390x Packages
Description: Fence Agents for Red Hat Cluster
Red Hat Fence Agents is a collection of scripts to handle remote
power management for several devices. They allow failed or
unreachable nodes to be forcibly restarted and removed from the
cluster.

# fence_ibmz --ip=<hmc> --username=<user> --password=<password> --plug=M46/m46lp64 -o status
Status: ON
# fence_ibmz --ip=<hmc> --username=<user> --password=<password> --plug=M46/m46lp64 -o off
Success: Powered OFF
# fence_ibmz --ip=<hmc> --username=<user> --password=<password> --plug=M46/m46lp64 -o status
Status: OFF
# fence_ibmz --ip=<hmc> --username=<user> --password=<password> --plug=M46/m46lp64 -o on
Success: Powered ON
# fence_ibmz --ip=<hmc> --username=<user> --password=<password> --plug=M46/m46lp64 -o status
Status: ON
# fence_ibmz --ip=localhost --username=sysprog --password=o8cpav4u --plug=M46/m46lp64 -o list-status
[long list of all CECs and LPARs on the HMC]

Life-cycle changes/reporting seems to be working fine. I am not sure how the other actions (monitor/validate-all) are supposed to work, let me know if you need me to verify this as well.

Thanks "<email address hidden>" for the quick testing on focal / 20.04.

Would you mind quickly trying out the 18.04 / bionic version as well?

------- Comment From <email address hidden> 2021-03-01 10:45 EDT-------
# cat /etc/os-release
NAME="Ubuntu"
VERSION="18.04.5 LTS (Bionic Beaver)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 18.04.5 LTS"
VERSION_ID="18.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=bionic
UBUNTU_CODENAME=bionic

# apt show fence-agents
Package: fence-agents
Version: 4.0.25-2ubuntu1.3~ppa2
Priority: optional
Section: admin
Maintainer: Ubuntu Developers <email address hidden>
Original-Maintainer: Debian HA Maintainers <email address hidden>
Installed-Size: 686 kB
Depends: python:any (>= 2.7.5-5~), libc6 (>= 2.15), python-pycurl, python-pexpect
Recommends: libnet-telnet-perl, openssh-client, sg3-utils, snmp
Suggests: python-requests, python-suds, python-boto3
Conflicts: cman (<= 3.0.12-2ubuntu4)
Replaces: cman (<= 3.0.12-2ubuntu4)
Download-Size: 198 kB
APT-Manual-Installed: yes
APT-Sources: http://ppa.launchpad.net/lucaskanashiro/ha-stack-ibm/ubuntu bionic/main s390x Packages
Description: Fence Agents for Red Hat Cluster
Red Hat Fence Agents is a collection of scripts to handle remote
power management for several devices. They allow failed or
unreachable nodes to be forcibly restarted and removed from the
cluster.

I've ran the same commands on Ubuntu 18.04.5 without issues. The only thing that I had to do was manually install python-requests. There's probably a package dependency missing.

Thanks for testing the packages "<email address hidden>", appreciated! We will need to double-check them when the SRU team accepts them into the proposed pocket. For now, we have enough testing :)

Regarding the python-requests package: it is already a suggested dependency in Bionic. In Focal you did not have this issue because it was promoted to a hard runtime dependency. That was done by the Debian maintainer due to this bug:

https://bugs.debian.org/899381

A way to avoid this issue is to pass --install-suggests to your apt install command. For me it makes sense to be a suggested dependency since just 2 fence agents (IBMz LPAR and VMWare soup agents) out of around 50 (Bionic package) depend on python-requests, which makes me believe that most users installing the fence-agents binary package do not need this python module installed.

If you think we should promote python-requests in Bionic to a runtime dependency let's open a new bug and start a discussion there. For now, I'll leave it as is, a suggested runtime dependency.

Hello bugproxy, or anyone else affected,

Accepted fence-agents into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/fence-agents/4.5.2-1ubuntu0.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Hello bugproxy, or anyone else affected,

Accepted fence-agents into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/fence-agents/4.0.25-2ubuntu1.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Hi "<email address hidden>", could you perform the same tests you did previously but now using the fence-agents package from focal-proposed instead of my PPA? You can check the "Test plan" section in the bug description but it is basically what you already did. This is part of the SRU process and we need that to move forward. TIA!

------- Comment From <email address hidden> 2021-04-12 03:34 EDT-------
Both life-cycle tests from above complete successfully on both bionic and focal.

# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 18.04.5 LTS
Release: 18.04
Codename: bionic

# apt show fence-agents
Package: fence-agents
Version: 4.0.25-2ubuntu1.3
Priority: optional
Section: universe/admin
Origin: Ubuntu
Maintainer: Ubuntu Developers <email address hidden>
Original-Maintainer: Debian HA Maintainers
<email address hidden>
Bugs: https://bugs.launchpad.net/ubuntu/+filebug
Installed-Size: 686 kB
Depends: python:any (>= 2.7.5-5~), libc6 (>= 2.15), python-pycurl,
python-pexpect
Recommends: libnet-telnet-perl, openssh-client, sg3-utils, snmp
Suggests: python-requests, python-suds, python-boto3
Conflicts: cman (<= 3.0.12-2ubuntu4)
Replaces: cman (<= 3.0.12-2ubuntu4)
Homepage: https://github.com/ClusterLabs/fence-agents
Download-Size: 197 kB
APT-Manual-Installed: yes
APT-Sources: http://ports.ubuntu.com/ubuntu-ports bionic-proposed/universe
s390x Packages

Dependency on python-requests missing.

# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 20.04.2 LTS
Release: 20.04
Codename: focal

# apt show fence-agents
Package: fence-agents
Version: 4.5.2-1ubuntu0.1
Priority: optional
Section: universe/admin
Origin: Ubuntu
Maintainer: Ubuntu Developers <email address hidden>
Original-Maintainer: Debian HA Maintainers <email address hidden>
Bugs: https://bugs.launchpad.net/ubuntu/+filebug
Installed-Size: 839 kB
Depends: python3:any, libc6 (>= 2.15), python3-boto3, python3-googleapi, python3-pycurl, python3-pexpect, python3-requests, python3-suds
Recommends: libnet-telnet-perl, openssh-client, sg3-utils, snmp
Suggests: python3-adal, python3-azure, python3-keystoneauth1, python3-keystoneclient, python3-novaclient
Breaks: cman (<= 3.0.12-2ubuntu4)
Replaces: cman (<= 3.0.12-2ubuntu4)
Homepage: https://github.com/ClusterLabs/fence-agents
Download-Size: 234 kB
APT-Manual-Installed: yes
APT-Sources: http://ports.ubuntu.com/ubuntu-ports focal-proposed/universe s390x Packages

Thx 'BWALK' for the SRU verification on B and F! I'm adjusting the tags accordingly ...

This bug was fixed in the package fence-agents - 4.5.2-1ubuntu0.1

---------------
fence-agents (4.5.2-1ubuntu0.1) focal; urgency=medium

  * Backport upstream patch to add agent for IBM z LPARs (LP: #1889070).
    - d/p/0003-Add-fence-agent-for-IBM-z-LPARs.patch
    - d/copyright: add the IBM z LPARs agent.
    - d/rules: skip fence_ibmz in dh_auto_test. The test metadata file was
      removed from the upstream patch and it makes "make check" fail for
      fence_ibmz.

 -- Lucas Kanashiro <email address hidden> Wed, 10 Feb 2021 16:15:15 -0300

The verification of the Stable Release Update for fence-agents has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

This bug was fixed in the package fence-agents - 4.0.25-2ubuntu1.3

---------------
fence-agents (4.0.25-2ubuntu1.3) bionic; urgency=medium

  * Backport upstream patch to add agent for IBM z LPARs (LP: #1889070).
    - d/p/0013-Add-fence-agent-for-IBM-z-LPARs.patch
    - d/copyright: add the IBM z LPARs agent.
    - d/rules: skip fence_ibmz in dh_auto_test. The test metadata file was
      removed from the upstream patch and it makes "make check" fail for
      fence_ibmz.

 -- Lucas Kanashiro <email address hidden> Wed, 10 Feb 2021 16:39:54 -0300

------- Comment From <email address hidden> 2021-04-15 07:11 EDT-------
IBM bugzilla status->closed, Fix Released with all requested distros