Host swact failed. Active controller host did not change

Bug #1812108 reported by Peng Peng
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
StarlingX
Fix Released
Medium
Bin Qian

Bug Description

Brief Description
-----------------
Connect system with floating IP, Run host-swact on active controller. After active controller disconnected and reconnected, the active controller is not changed.

Severity
--------
Major

Steps to Reproduce
------------------
host-swact

Expected Behaviour
------------------
active controller is changed

Actual Behaviour
----------------
no change

Reproducibility
---------------
Reproducible
4/10

System Configuration
--------------------
Dedicated storage

Branch/Pull Time/Commit
-----------------------
Master of 2019-01-15_20-18-00

Timestamp/Logs
--------------
[2019-01-16 09:09:22,480] 263 DEBUG MainThread ssh.send :: Send 'system --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://192.168.204.2:5000/v3 --os-user-domain-name Default --os-project-domain-name Default --os-endpoint-type internalURL --os-region-name RegionOne host-swact controller-1'
[2019-01-16 09:09:25,818] 388 DEBUG MainThread ssh.expect :: Output:
+---------------------+-----------------------------------------------------------------+
| Property | Value |
+---------------------+-----------------------------------------------------------------+
| action | none |
| administrative | unlocked |
| availability | available |
| bm_ip | None |
| bm_type | None |
| bm_username | |
| boot_device | /dev/disk/by-path/pci-0000:03:00.0-sas-0x5001e67cd7837001-lun-0 |
| capabilities | {u'stor_function': u'monitor'} |
| config_applied | 145bdb65-b815-4568-9126-5d7ead542cae |
| config_status | None |
| config_target | 145bdb65-b815-4568-9126-5d7ead542cae |
| console | ttyS0,115200 |
| created_at | 2019-01-16T06:44:30.414178+00:00 |
| hostname | controller-1 |
| id | 2 |
| install_output | text |
| install_state | completed |
| install_state_info | None |
| invprovision | provisioned |
| location | {} |
| mgmt_ip | 192.168.204.4 |
| mgmt_mac | 90:e2:ba:a3:9c:38 |
| operational | enabled |
| personality | controller |
| reserved | False |
| rootfs_device | /dev/disk/by-path/pci-0000:03:00.0-sas-0x5001e67cd7837001-lun-0 |
| serialid | None |
| software_load | 19.01 |
| task | Swacting |
| tboot | false |
| ttys_dcd | None |
| updated_at | 2019-01-16T09:09:05.990110+00:00 |
| uptime | 1688 |
| uuid | 0344ba4a-5af3-4f6d-904f-b428d3590c38 |
| vim_progress_status | services-enabled |
+---------------------+-----------------------------------------------------------------+
controller-1:~$
[2019-01-16 09:09:25,819] 263 DEBUG MainThread ssh.send :: Send 'echo $?'
[2019-01-16 09:09:25,922] 388 DEBUG MainThread ssh.expect :: Output:
0
controller-1:~$
[2019-01-16 09:09:25,922] 1337 DEBUG MainThread ssh.get_active_controller:: Getting active controller client for pv0
[2019-01-16 09:09:25,922] 263 DEBUG MainThread ssh.send :: Send ''
[2019-01-16 09:09:26,025] 388 DEBUG MainThread ssh.expect :: Output:
controller-1:~$
[2019-01-16 09:09:31,030] 263 DEBUG MainThread ssh.send :: Send ''
[2019-01-16 09:09:31,133] 388 DEBUG MainThread ssh.expect :: Output:
controller-1:~$
[2019-01-16 09:09:36,138] 263 DEBUG MainThread ssh.send :: Send ''
[2019-01-16 09:09:39,243] 1331 INFO MainThread host_helper.wait_for_swact_complete:: ssh to 128.224.150.73 disconnected, indicating swacting initiated.
[2019-01-16 09:10:09,265] 134 INFO MainThread ssh.connect :: Attempt to connect to host - 128.224.150.73
[2019-01-16 09:10:25,794] 263 DEBUG MainThread ssh.send :: Send ''
[2019-01-16 09:10:25,896] 388 DEBUG MainThread ssh.expect :: Output:
controller-1:~$
[2019-01-16 09:10:25,897] 162 INFO MainThread ssh.connect :: Login successful!
[2019-01-16 09:10:25,897] 263 DEBUG MainThread ssh.send :: Send ''
[2019-01-16 09:10:25,998] 388 DEBUG MainThread ssh.expect :: Output:
controller-1:~$
[2019-01-16 09:10:25,999] 388 DEBUG MainThread ssh.expect :: Output:

[2019-01-16 09:10:45,610] 263 DEBUG MainThread ssh.send :: Send 'system --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://192.168.204.2:5000/v3 --os-user-domain-name Default --os-project-domain-name Default --os-endpoint-type internalURL --os-region-name RegionOne servicegroup-list'
[2019-01-16 09:10:47,937] 388 DEBUG MainThread ssh.expect :: Output:
+--------------------------------------+-----------------------------+--------------+-----------------+
| uuid | service_group_name | hostname | state |
+--------------------------------------+-----------------------------+--------------+-----------------+
| b3f31ab2-e34f-449a-804c-a60f5cb64c55 | cloud-services | controller-1 | active |
| 8422fcb0-cd46-4fe5-b847-9542addfa5df | cloud-services | controller-0 | standby |
| a3fd2f7a-e4fc-4a16-8325-7fa7547f7a5b | controller-services | controller-1 | active-degraded |
| b1b225eb-3751-4274-94e9-ec28cc4b8945 | controller-services | controller-0 | standby |

Tags: stx.2.0 stx.ha
Ghada Khalil (gkhalil)
Changed in starlingx:
assignee: nobody → Bin Qian (bqian20)
Revision history for this message
Ghada Khalil (gkhalil) wrote :

Marking as release gating until further investigation.

Changed in starlingx:
importance: Undecided → Medium
status: New → Triaged
tags: added: stx.2019.05 stx.ha
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to stx-ha (master)

Fix proposed to branch: master
Review: https://review.openstack.org/634416

Changed in starlingx:
status: Triaged → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to stx-ha (master)

Reviewed: https://review.openstack.org/634416
Committed: https://git.openstack.org/cgit/openstack/stx-ha/commit/?id=1066d26e9e150d4f93f404740b526d00d7255062
Submitter: Zuul
Branch: master

commit 1066d26e9e150d4f93f404740b526d00d7255062
Author: Bin Qian <email address hidden>
Date: Thu Jan 31 08:23:05 2019 -0500

    Fixed host-swact failed

    Adding new domain event SM_SERVICE_DOMAIN_EVENT_CHANGING_LEADER
    to handle an on demand switching of service scheduler leader.

    Closes-Bug: 1812108

    Change-Id: I6796d8efcb1ef0c7fa835ed34028c8e6a2b5dcae
    Signed-off-by: Bin Qian <email address hidden>

Changed in starlingx:
status: In Progress → Fix Released
Ken Young (kenyis)
tags: added: stx.2.0
removed: stx.2019.05
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.