tripleo

Barbican tempest test failing to ssh to cirros image

Bug #1755485 reported by John Trowbridge
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
Triaged
Critical
Unassigned
tripleo rocky-1

Bug Description

Seeing this on Queens featureset017 in the promote job. The test is failing to SSH to the cirros instance:

2018-03-13 13:00:37 | 2018-03-13 12:43:44,292 1358 INFO [tempest.lib.common.ssh] Creating ssh connection to '192.168.24.104:22' as 'cirros' with public key authentication
2018-03-13 13:00:37 | 2018-03-13 12:43:47,299 1358 WARNING [tempest.lib.common.ssh] Failed to establish authenticated ssh connection to cirros@192.168.24.104 ([Errno None] Unable to connect to port 22 on 192.168.24.104). Number attempts: 1. Retry after 2 seconds.
2018-03-13 13:00:37 | 2018-03-13 12:43:50,305 1358 WARNING [tempest.lib.common.ssh] Failed to establish authenticated ssh connection to cirros@192.168.24.104 ([Errno None] Unable to connect to port 22 on 192.168.24.104). Number attempts: 2. Retry after 3 seconds.
2018-03-13 13:00:37 | 2018-03-13 12:43:56,815 1358 WARNING [tempest.lib.common.ssh] Failed to establish authenticated ssh connection to cirros@192.168.24.104 ([Errno None] Unable to connect to port 22 on 192.168.24.104). Number attempts: 3. Retry after 4 seconds.
2018-03-13 13:00:37 | 2018-03-13 12:44:04,327 1358 WARNING [tempest.lib.common.ssh] Failed to establish authenticated ssh connection to cirros@192.168.24.104 ([Errno None] Unable to connect to port 22 on 192.168.24.104). Number attempts: 4. Retry after 5 seconds.
2018-03-13 13:00:37 | 2018-03-13 12:44:12,839 1358 WARNING [tempest.lib.common.ssh] Failed to establish authenticated ssh connection to cirros@192.168.24.104 ([Errno None] Unable to connect to port 22 on 192.168.24.104). Number attempts: 5. Retry after 6 seconds.
2018-03-13 13:00:37 | 2018-03-13 12:44:22,347 1358 WARNING [tempest.lib.common.ssh] Failed to establish authenticated ssh connection to cirros@192.168.24.104 ([Errno None] Unable to connect to port 22 on 192.168.24.104). Number attempts: 6. Retry after 7 seconds.
2018-03-13 13:00:37 | 2018-03-13 12:44:32,859 1358 WARNING [tempest.lib.common.ssh] Failed to establish authenticated ssh connection to cirros@192.168.24.104 ([Errno None] Unable to connect to port 22 on 192.168.24.104). Number attempts: 7. Retry after 8 seconds.
2018-03-13 13:00:37 | 2018-03-13 12:44:44,371 1358 WARNING [tempest.lib.common.ssh] Failed to establish authenticated ssh connection to cirros@192.168.24.104 ([Errno None] Unable to connect to port 22 on 192.168.24.104). Number attempts: 8. Retry after 9 seconds.
2018-03-13 13:00:37 | 2018-03-13 12:44:56,885 1358 WARNING [tempest.lib.common.ssh] Failed to establish authenticated ssh connection to cirros@192.168.24.104 ([Errno None] Unable to connect to port 22 on 192.168.24.104). Number attempts: 9. Retry after 10 seconds.
2018-03-13 13:00:37 | 2018-03-13 12:45:10,397 1358 WARNING [tempest.lib.common.ssh] Failed to establish authenticated ssh connection to cirros@192.168.24.104 ([Errno None] Unable to connect to port 22 on 192.168.24.104). Number attempts: 10. Retry after 11 seconds.
2018-03-13 13:00:37 | 2018-03-13 12:45:24,911 1358 WARNING [tempest.lib.common.ssh] Failed to establish authenticated ssh connection to cirros@192.168.24.104 ([Errno None] Unable to connect to port 22 on 192.168.24.104). Number attempts: 11. Retry after 12 seconds.
2018-03-13 13:00:37 | 2018-03-13 12:45:37,651 1358 WARNING [tempest.lib.common.ssh] Failed to establish authenticated ssh connection to cirros@192.168.24.104 (Authentication failed.). Number attempts: 12. Retry after 13 seconds.
2018-03-13 13:00:37 | 2018-03-13 12:45:51,382 1358 WARNING [tempest.lib.common.ssh] Failed to establish authenticated ssh connection to cirros@192.168.24.104 (Authentication failed.). Number attempts: 13. Retry after 14 seconds.
2018-03-13 13:00:37 | 2018-03-13 12:46:06,147 1358 WARNING [tempest.lib.common.ssh] Failed to establish authenticated ssh connection to cirros@192.168.24.104 (Authentication failed.). Number attempts: 14. Retry after 15 seconds.
2018-03-13 13:00:37 | 2018-03-13 12:46:21,879 1358 WARNING [tempest.lib.common.ssh] Failed to establish authenticated ssh connection to cirros@192.168.24.104 (Authentication failed.). Number attempts: 15. Retry after 16 seconds.
2018-03-13 13:00:37 | 2018-03-13 12:46:38,640 1358 WARNING [tempest.lib.common.ssh] Failed to establish authenticated ssh connection to cirros@192.168.24.104 (Authentication failed.). Number attempts: 16. Retry after 17 seconds.
2018-03-13 13:00:37 | 2018-03-13 12:46:56,408 1358 WARNING [tempest.lib.common.ssh] Failed to establish authenticated ssh connection to cirros@192.168.24.104 (Authentication failed.). Number attempts: 17. Retry after 18 seconds.
2018-03-13 13:00:37 | 2018-03-13 12:47:15,173 1358 WARNING [tempest.lib.common.ssh] Failed to establish authenticated ssh connection to cirros@192.168.24.104 (Authentication failed.). Number attempts: 18. Retry after 19 seconds.
2018-03-13 13:00:37 | 2018-03-13 12:47:34,939 1358 WARNING [tempest.lib.common.ssh] Failed to establish authenticated ssh connection to cirros@192.168.24.104 (Authentication failed.). Number attempts: 19. Retry after 20 seconds.
2018-03-13 13:00:37 | 2018-03-13 12:47:55,707 1358 WARNING [tempest.lib.common.ssh] Failed to establish authenticated ssh connection to cirros@192.168.24.104 (Authentication failed.). Number attempts: 20. Retry after 21 seconds.
2018-03-13 13:00:37 | 2018-03-13 12:48:17,479 1358 WARNING [tempest.lib.common.ssh] Failed to establish authenticated ssh connection to cirros@192.168.24.104 (Authentication failed.). Number attempts: 21. Retry after 22 seconds.
2018-03-13 13:00:37 | 2018-03-13 12:48:40,234 1358 WARNING [tempest.lib.common.ssh] Failed to establish authenticated ssh connection to cirros@192.168.24.104 (Authentication failed.). Number attempts: 22. Retry after 23 seconds.
2018-03-13 13:00:37 | 2018-03-13 12:49:04,007 1358 ERROR [tempest.lib.common.ssh] Failed to establish authenticated ssh connection to cirros@192.168.24.104 after 22 attempts
2018-03-13 13:00:37 | 2018-03-13 12:49:04.007 1358 ERROR tempest.lib.common.ssh Traceback (most recent call last):
2018-03-13 13:00:37 | 2018-03-13 12:49:04.007 1358 ERROR tempest.lib.common.ssh File "/usr/lib/python2.7/site-packages/tempest/lib/common/ssh.py", line 107, in _get_ssh_connection
2018-03-13 13:00:37 | 2018-03-13 12:49:04.007 1358 ERROR tempest.lib.common.ssh sock=proxy_chan)
2018-03-13 13:00:37 | 2018-03-13 12:49:04.007 1358 ERROR tempest.lib.common.ssh File "/usr/lib/python2.7/site-packages/paramiko/client.py", line 380, in connect
2018-03-13 13:00:37 | 2018-03-13 12:49:04.007 1358 ERROR tempest.lib.common.ssh look_for_keys, gss_auth, gss_kex, gss_deleg_creds, gss_host)
2018-03-13 13:00:37 | 2018-03-13 12:49:04.007 1358 ERROR tempest.lib.common.ssh File "/usr/lib/python2.7/site-packages/paramiko/client.py", line 621, in _auth
2018-03-13 13:00:37 | 2018-03-13 12:49:04.007 1358 ERROR tempest.lib.common.ssh raise saved_exception
2018-03-13 13:00:37 | 2018-03-13 12:49:04.007 1358 ERROR tempest.lib.common.ssh AuthenticationException: Authentication failed.
2018-03-13 13:00:37 | 2018-03-13 12:49:04.007 1358 ERROR tempest.lib.common.ssh
2018-03-13 13:00:37 | 2018-03-13 12:49:04,011 1358 ERROR [barbican_tempest_plugin.tests.scenario.manager] (VolumeEncryptionTest:test_encrypted_cinder_volumes_cryptsetup) Initializing SSH connection to 192.168.24.104 failed. Error: Connection to the 192.168.24.104 via SSH timed out.
2018-03-13 13:00:37 | User: cirros, Password: None
2018-03-13 13:00:37 | 2018-03-13 12:49:04.011 1358 ERROR barbican_tempest_plugin.tests.scenario.manager Traceback (most recent call last):
2018-03-13 13:00:37 | 2018-03-13 12:49:04.011 1358 ERROR barbican_tempest_plugin.tests.scenario.manager File "/usr/lib/python2.7/site-packages/barbican_tempest_plugin/tests/scenario/manager.py", line 431, in get_remote_client
2018-03-13 13:00:37 | 2018-03-13 12:49:04.011 1358 ERROR barbican_tempest_plugin.tests.scenario.manager linux_client.validate_authentication()
2018-03-13 13:00:37 | 2018-03-13 12:49:04.011 1358 ERROR barbican_tempest_plugin.tests.scenario.manager File "/usr/lib/python2.7/site-packages/tempest/lib/common/utils/linux/remote_client.py", line 60, in wrapper
2018-03-13 13:00:37 | 2018-03-13 12:49:04.011 1358 ERROR barbican_tempest_plugin.tests.scenario.manager six.reraise(*original_exception)
2018-03-13 13:00:37 | 2018-03-13 12:49:04.011 1358 ERROR barbican_tempest_plugin.tests.scenario.manager File "/usr/lib/python2.7/site-packages/tempest/lib/common/utils/linux/remote_client.py", line 33, in wrapper
2018-03-13 13:00:37 | 2018-03-13 12:49:04.011 1358 ERROR barbican_tempest_plugin.tests.scenario.manager return function(self, *args, **kwargs)
2018-03-13 13:00:37 | 2018-03-13 12:49:04.011 1358 ERROR barbican_tempest_plugin.tests.scenario.manager File "/usr/lib/python2.7/site-packages/tempest/lib/common/utils/linux/remote_client.py", line 116, in validate_authentication
2018-03-13 13:00:37 | 2018-03-13 12:49:04.011 1358 ERROR barbican_tempest_plugin.tests.scenario.manager self.ssh_client.test_connection_auth()
2018-03-13 13:00:37 | 2018-03-13 12:49:04.011 1358 ERROR barbican_tempest_plugin.tests.scenario.manager File "/usr/lib/python2.7/site-packages/tempest/lib/common/ssh.py", line 207, in test_connection_auth
2018-03-13 13:00:37 | 2018-03-13 12:49:04.011 1358 ERROR barbican_tempest_plugin.tests.scenario.manager connection = self._get_ssh_connection()
2018-03-13 13:00:37 | 2018-03-13 12:49:04.011 1358 ERROR barbican_tempest_plugin.tests.scenario.manager File "/usr/lib/python2.7/site-packages/tempest/lib/common/ssh.py", line 121, in _get_ssh_connection
2018-03-13 13:00:37 | 2018-03-13 12:49:04.011 1358 ERROR barbican_tempest_plugin.tests.scenario.manager password=self.password)
2018-03-13 13:00:37 | 2018-03-13 12:49:04.011 1358 ERROR barbican_tempest_plugin.tests.scenario.manager SSHTimeout: Connection to the 192.168.24.104 via SSH timed out.
2018-03-13 13:00:37 | 2018-03-13 12:49:04.011 1358 ERROR barbican_tempest_plugin.tests.scenario.manager User: cirros, Password: None
2018-03-13 13:00:37 | 2018-03-13 12:49:04.011 1358 ERROR barbican_tempest_plugin.tests.scenario.manager

Logs: https://logs.rdoproject.org/openstack-periodic/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset017-queens/da920d8

Tempest logs: https://logs.rdoproject.org/openstack-periodic/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset017-queens/da920d8/undercloud/home/jenkins/tempest_output.log.txt.gz

Reproducer script: https://logs.rdoproject.org/openstack-periodic/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset017-queens/da920d8/reproducer-quickstart.sh

Revision history for this message
Matt Young (halcyondude) wrote :

Adding barbican_tempest_plugin.tests.scenario.test_volume_encryption.VolumeEncryptionTest to queens skiplist

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to tripleo-quickstart-extras (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/552529

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to tripleo-quickstart-extras (master)

Reviewed: https://review.openstack.org/552529
Committed: https://git.openstack.org/cgit/openstack/tripleo-quickstart-extras/commit/?id=ec8bbe2b64e6e172fbe8529220c31326ab910a6f
Submitter: Zuul
Branch: master

commit ec8bbe2b64e6e172fbe8529220c31326ab910a6f
Author: Matt Young <email address hidden>
Date: Tue Mar 13 10:27:10 2018 -0400

    queens: Add failing barbican tempest test to skip list

    Add failing barbican tempest test to queens skip list.

    Change-Id: Ifc1f843b3380075b1cecf05feeb5df3c9420f90d
    Related-Bug: #1755485

Emilien Macchi (emilienm)
tags: added: tech-debt
Revision history for this message
yatin (yatinkarel) wrote :

The reverted patch merged: https://review.openstack.org/#/c/553820/.
Tempest test failing after revert as well:- https://logs.rdoproject.org/openstack-periodic/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset017-queens/6e4ffba/undercloud/home/jenkins/tempest/tempest.html.gz

Revision history for this message
Damien Ciabrini (dciabrin) wrote :
Download full text (4.0 KiB)

For the record, it seems I'm hitting the same problem in test "tripleo-ci-centos-7-3nodes-multinode" when gating tripleo-heat-templates.

This is apparently not limited to my review, e.g.:
  . https://review.openstack.org/#/c/539503/
  . https://review.openstack.org/#/c/547476/

I don't have other t-h-t example right now but both reviews above fail like:
2018-03-20 18:23:27,063 23530 ERROR [tempest.lib.common.ssh] Failed to establish authenticated ssh connection to cirros@192.168.24.107 after 23 attempts
2018-03-20 18:23:27.063 23530 ERROR tempest.lib.common.ssh Traceback (most recent call last):
2018-03-20 18:23:27.063 23530 ERROR tempest.lib.common.ssh File "/usr/lib/python2.7/site-packages/tempest/lib/common/ssh.py", line 107, in _get_ssh_connection
2018-03-20 18:23:27.063 23530 ERROR tempest.lib.common.ssh sock=proxy_chan)
2018-03-20 18:23:27.063 23530 ERROR tempest.lib.common.ssh File "/usr/lib/python2.7/site-packages/paramiko/client.py", line 380, in connect
2018-03-20 18:23:27.063 23530 ERROR tempest.lib.common.ssh look_for_keys, gss_auth, gss_kex, gss_deleg_creds, gss_host)
2018-03-20 18:23:27.063 23530 ERROR tempest.lib.common.ssh File "/usr/lib/python2.7/site-packages/paramiko/client.py", line 621, in _auth
2018-03-20 18:23:27.063 23530 ERROR tempest.lib.common.ssh raise saved_exception
2018-03-20 18:23:27.063 23530 ERROR tempest.lib.common.ssh AuthenticationException: Authentication failed.
2018-03-20 18:23:27.063 23530 ERROR tempest.lib.common.ssh
2018-03-20 18:23:27,066 23530 ERROR [tempest.scenario.manager] Public network connectivity check failed
2018-03-20 18:23:27.066 23530 ERROR tempest.scenario.manager Traceback (most recent call last):
2018-03-20 18:23:27.066 23530 ERROR tempest.scenario.manager File "/usr/lib/python2.7/site-packages/tempest/scenario/manager.py", line 585, in check_public_network_connectivity
2018-03-20 18:23:27.066 23530 ERROR tempest.scenario.manager mtu=mtu)
2018-03-20 18:23:27.066 23530 ERROR tempest.scenario.manager File "/usr/lib/python2.7/site-packages/tempest/scenario/manager.py", line 571, in check_vm_connectivity
2018-03-20 18:23:27.066 23530 ERROR tempest.scenario.manager self.get_remote_client(ip_address, username, private_key)
2018-03-20 18:23:27.066 23530 ERROR tempest.scenario.manager File "/usr/lib/python2.7/site-packages/tempest/scenario/manager.py", line 358, in get_remote_client
2018-03-20 18:23:27.066 23530 ERROR tempest.scenario.manager linux_client.validate_authentication()
2018-03-20 18:23:27.066 23530 ERROR tempest.scenario.manager File "/usr/lib/python2.7/site-packages/tempest/lib/common/utils/linux/remote_client.py", line 60, in wrapper
2018-03-20 18:23:27.066 23530 ERROR tempest.scenario.manager six.reraise(*original_exception)
2018-03-20 18:23:27.066 23530 ERROR tempest.scenario.manager File "/usr/lib/python2.7/site-packages/tempest/lib/common/utils/linux/remote_client.py", line 33, in wrapper
2018-03-20 18:23:27.066 23530 ERROR tempest.scenario.manager return function(self, *args, **kwargs)
2018-03-20 18:23:27.066 23530 ERROR tempest.scenario.manager File "/usr/lib/python2.7/site-packages/tempest/lib/common/utils/linux/re...

Read more...

Revision history for this message
Damien Ciabrini (dciabrin) wrote :

Could it be a dup of https://bugs.launchpad.net/tripleo/+bug/1751180 ?

Revision history for this message
Matt Young (halcyondude) wrote :

Per bug triage with tripleo CI squad, marking as dup

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.