Comment 15 for bug 1972730

Hello all,

I went through the verification of the SRU successfully, I found evidence of regressions with my cluster, here it the log of my terminal going through it.

$ juju status
Model Controller Cloud/Region Version SLA Timestamp
hacluster serverstack serverstack/serverstack 2.9.29 unsupported 09:36:25-04:00

App Version Status Scale Charm Channel Rev Exposed Message
hacluster active 3 hacluster edge 104 no Unit is ready and clustered
keystone 21.0.0 active 3 keystone edge 575 no Application Ready
keystone-mysql-router 8.0.29 active 3 mysql-router 8.0/stable 30 no Unit is ready
mysql-innodb-cluster 8.0.29 active 3 mysql-innodb-cluster 8.0/stable 26 no Unit is ready: Mode: R/W, Cluster is ONLINE and can tolerate up to ONE failure.

Unit Workload Agent Machine Public address Ports Message
keystone/0 active idle 3 10.5.0.147 5000/tcp Unit is ready
  hacluster/1 active idle 10.5.0.147 Unit is ready and clustered
  keystone-mysql-router/1 active idle 10.5.0.147 Unit is ready
keystone/1 active idle 4 10.5.2.108 5000/tcp Unit is ready
  hacluster/2 active idle 10.5.2.108 Unit is ready and clustered
  keystone-mysql-router/2 active idle 10.5.2.108 Unit is ready
keystone/2* active idle 5 10.5.0.30 5000/tcp Unit is ready
  hacluster/0* active idle 10.5.0.30 Unit is ready and clustered
  keystone-mysql-router/0* active idle 10.5.0.30 Unit is ready
mysql-innodb-cluster/0* active idle 0 10.5.1.86 Unit is ready: Mode: R/W, Cluster is ONLINE and can tolerate up to ONE failure.
mysql-innodb-cluster/1 active idle 1 10.5.2.126 Unit is ready: Mode: R/O, Cluster is ONLINE and can tolerate up to ONE failure.
mysql-innodb-cluster/2 active idle 2 10.5.2.100 Unit is ready: Mode: R/O, Cluster is ONLINE and can tolerate up to ONE failure.

Machine State Address Inst id Series AZ Message
0 started 10.5.1.86 12b64376-138c-44ca-8873-bcaa7e6e28de jammy nova ACTIVE
1 started 10.5.2.126 88874312-5829-40ec-b9c3-9d7c15a6d2bf jammy nova ACTIVE
2 started 10.5.2.100 e619a2d4-e7cc-4970-8dfd-10a67a84ee39 jammy nova ACTIVE
3 started 10.5.0.147 17c62da7-b9e5-4a1f-aab5-988a8dd55d28 jammy nova ACTIVE
4 started 10.5.2.108 9c17bd02-6bef-42e6-a5ae-e97bfac50014 jammy nova ACTIVE
5 started 10.5.0.30 b9f599a5-3687-45e5-a0fc-ca68f7e16daf jammy nova ACTIVE

$ juju ssh keystone/leader
...
ubuntu@juju-d7d86e-hacluster-5:~$ sudo crm status
Cluster Summary:
  * Stack: corosync
  * Current DC: juju-d7d86e-hacluster-3 (version 2.1.2-ada5c3b36e2) - partition with quorum
  * Last updated: Tue Jul 19 13:25:05 2022
  * Last change: Tue Jul 19 13:17:45 2022 by hacluster via crmd on juju-d7d86e-hacluster-5
  * 3 nodes configured
  * 4 resource instances configured

Node List:
  * Online: [ juju-d7d86e-hacluster-3 juju-d7d86e-hacluster-4 juju-d7d86e-hacluster-5 ]

Full List of Resources:
  * Resource Group: grp_ks_vips:
    * res_ks_553c26b_vip (ocf:heartbeat:IPaddr2): Started juju-d7d86e-hacluster-3
  * Clone Set: cl_ks_haproxy [res_ks_haproxy]:
    * Started: [ juju-d7d86e-hacluster-3 juju-d7d86e-hacluster-4 juju-d7d86e-hacluster-5 ]

ubuntu@juju-d7d86e-hacluster-5:~$ sudo crm -w -F node maintenance $(hostname)
WARNING: crmadmin -S juju-d7d86e-hacluster-3 unexpected output: Controller on juju-d7d86e-hacluster-3 in state S_IDLE: ok (exit code: 0)
ubuntu@juju-d7d86e-hacluster-5:~$ sudo apt policy crmsh
crmsh:
  Installed: 4.3.1-1ubuntu2
  Candidate: 4.3.1-1ubuntu2
  Version table:
 *** 4.3.1-1ubuntu2 500
        500 http://nova.clouds.archive.ubuntu.com/ubuntu jammy/main amd64 Packages
        100 /var/lib/dpkg/status
ubuntu@juju-d7d86e-hacluster-5:~$ sudo vim /etc/apt/sources.list # enabling proposed
ubuntu@juju-d7d86e-hacluster-5:~$ sudo apt update -qqq
ubuntu@juju-d7d86e-hacluster-5:~$ sudo apt policy crmsh
crmsh:
  Installed: 4.3.1-1ubuntu2
  Candidate: 4.3.1-1ubuntu3
  Version table:
     4.3.1-1ubuntu3 500
        500 http://nova.clouds.archive.ubuntu.com/ubuntu jammy-proposed/main amd64 Packages
 *** 4.3.1-1ubuntu2 500
        500 http://nova.clouds.archive.ubuntu.com/ubuntu jammy/main amd64 Packages
        100 /var/lib/dpkg/status
ubuntu@juju-d7d86e-hacluster-5:~$ sudo apt install --only-upgrade crmsh
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Suggested packages:
  csync2 ocfs2-tools sbd vim-addon-manager
The following packages will be upgraded:
  crmsh
1 upgraded, 0 newly installed, 0 to remove and 22 not upgraded.
Need to get 559 kB of archives.
After this operation, 0 B of additional disk space will be used.
Get:1 http://nova.clouds.archive.ubuntu.com/ubuntu jammy-proposed/main amd64 crmsh all 4.3.1-1ubuntu3 [559 kB]
Fetched 559 kB in 1s (522 kB/s)
(Reading database ... 82022 files and directories currently installed.)
Preparing to unpack .../crmsh_4.3.1-1ubuntu3_all.deb ...
Unpacking crmsh (4.3.1-1ubuntu3) over (4.3.1-1ubuntu2) ...
Setting up crmsh (4.3.1-1ubuntu3) ...
Processing triggers for man-db (2.10.2-1) ...
Scanning processes...
Scanning candidates...
Scanning linux images...

Running kernel seems to be up-to-date.

Restarting services...
Service restarts being deferred:
 systemctl restart networkd-dispatcher.service
 systemctl restart unattended-upgrades.service

No containers need to be restarted.

No user sessions are running outdated binaries.

No VM guests are running outdated hypervisor (qemu) binaries on this host.
ubuntu@juju-d7d86e-hacluster-5:~$ sudo apt policy crmsh
crmsh:
  Installed: 4.3.1-1ubuntu3
  Candidate: 4.3.1-1ubuntu3
  Version table:
 *** 4.3.1-1ubuntu3 500
        500 http://nova.clouds.archive.ubuntu.com/ubuntu jammy-proposed/main amd64 Packages
        100 /var/lib/dpkg/status
     4.3.1-1ubuntu2 500
        500 http://nova.clouds.archive.ubuntu.com/ubuntu jammy/main amd64 Packages
ubuntu@juju-d7d86e-hacluster-5:~$ sudo crm status
Cluster Summary:
  * Stack: corosync
  * Current DC: juju-d7d86e-hacluster-3 (version 2.1.2-ada5c3b36e2) - partition with quorum
  * Last updated: Tue Jul 19 13:29:30 2022
  * Last change: Tue Jul 19 13:25:17 2022 by root via cibadmin on juju-d7d86e-hacluster-5
  * 3 nodes configured
  * 4 resource instances configured

Node List:
  * Node juju-d7d86e-hacluster-5: maintenance
  * Online: [ juju-d7d86e-hacluster-3 juju-d7d86e-hacluster-4 ]

Full List of Resources:
  * Resource Group: grp_ks_vips:
    * res_ks_553c26b_vip (ocf:heartbeat:IPaddr2): Started juju-d7d86e-hacluster-3
  * Clone Set: cl_ks_haproxy [res_ks_haproxy]:
    * res_ks_haproxy (lsb:haproxy): Started juju-d7d86e-hacluster-5 (unmanaged)
    * Started: [ juju-d7d86e-hacluster-3 juju-d7d86e-hacluster-4 ]
ubuntu@juju-d7d86e-hacluster-5:~$ sudo crm -w -F node ready $(hostname) # to make the resource managed again.
ubuntu@juju-d7d86e-hacluster-5:~$ sudo crm status
Cluster Summary:
  * Stack: corosync
  * Current DC: juju-d7d86e-hacluster-3 (version 2.1.2-ada5c3b36e2) - partition with quorum
  * Last updated: Tue Jul 19 13:31:55 2022
  * Last change: Tue Jul 19 13:31:50 2022 by root via crm_attribute on juju-d7d86e-hacluster-5
  * 3 nodes configured
  * 4 resource instances configured

Node List:
  * Online: [ juju-d7d86e-hacluster-3 juju-d7d86e-hacluster-4 juju-d7d86e-hacluster-5 ]

Full List of Resources:
  * Resource Group: grp_ks_vips:
    * res_ks_553c26b_vip (ocf:heartbeat:IPaddr2): Started juju-d7d86e-hacluster-3
  * Clone Set: cl_ks_haproxy [res_ks_haproxy]:
    * Started: [ juju-d7d86e-hacluster-3 juju-d7d86e-hacluster-4 juju-d7d86e-hacluster-5 ]

ubuntu@juju-d7d86e-hacluster-5:~$ sudo crm -w -F node maintenance $(hostname)
ubuntu@juju-d7d86e-hacluster-5:~$ echo $?
0
ubuntu@juju-d7d86e-hacluster-5:~$ sudo crm status
Cluster Summary:
  * Stack: corosync
  * Current DC: juju-d7d86e-hacluster-3 (version 2.1.2-ada5c3b36e2) - partition with quorum
  * Last updated: Tue Jul 19 13:32:33 2022
  * Last change: Tue Jul 19 13:32:24 2022 by root via cibadmin on juju-d7d86e-hacluster-5
  * 3 nodes configured
  * 4 resource instances configured

Node List:
  * Node juju-d7d86e-hacluster-5: maintenance
  * Online: [ juju-d7d86e-hacluster-3 juju-d7d86e-hacluster-4 ]

Full List of Resources:
  * Resource Group: grp_ks_vips:
    * res_ks_553c26b_vip (ocf:heartbeat:IPaddr2): Started juju-d7d86e-hacluster-3
  * Clone Set: cl_ks_haproxy [res_ks_haproxy]:
    * res_ks_haproxy (lsb:haproxy): Started juju-d7d86e-hacluster-5 (unmanaged)
    * Started: [ juju-d7d86e-hacluster-3 juju-d7d86e-hacluster-4 ]