tripleo nova containers restarting in a loop causing - 503 errors - starting container trunk.registry.rdoproject.org/tripleomaster/centos-binary-nova-compute-ironic

Last known working
===================================

https://logs.rdoproject.org/openstack-periodic/periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master/0830d9b/undercloud/home/jenkins/undercloud_install.log.txt.gz#_2018-06-05_05_55_03

openstack-nova-18.0.0-0.20180601221704.f902e0d.el7.noarch.rpm
===================================

Latest Failed job installs is at head of nova as of 6/07/2018
https://github.com/openstack/nova/commit/54c9a944c618dc173bd1214be4de9a44479c8959
https://trunk.rdoproject.org/centos7-master/d5/96/d596a4ea9f0b0d9255f21bbd4c3757eec6017669_ceffe88f/
openstack-nova-18.0.0-0.20180607163313.54c9a94.el7.noarch.rpm

Nova is clearly down or unreachable

https://logs.rdoproject.org/openstack-periodic/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset016-master/cc496dd/undercloud/var/log/extra/nova_list.txt.gz

ERROR (ConnectFailure): Unable to establish connection to http://192.168.24.1:8774/v2.1: HTTPConnectionPool(host='192.168.24.1', port=8774): Max retries exceeded with url: /v2.1 (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x7fabdd786190>: Failed to establish a new connection: [Errno 111] Connection refused',))

The issue looks not related to ironic. Nova-api container instead, and a few more, failing by a healthcheck a several seconds after it starts. The strange thing is that there is no logs stored in the nova api log file. And the container stdout looks normal.

That I can see with recreated nova api container via paunch, for debugging disabled health checks:

* nova api works normally then I start it after kolla init steps in the container.
* healthcheck script connects to undercloud.internalapi.localdomain:8774 taken from /etc/httpd/conf.d/10-nova_api_wsgi.conf, it is resolved to 192.168.24.1:8774
* nothing listens there (haproxy issue)? there is only 192.168.24.3:8774 available.

haproxy config stanza:
 listen nova_osapi
   bind 192.168.24.2:13774 transparent ssl crt /etc/pki/tls/private/overcloud_endpoint.pem
   bind 192.168.24.3:8774 transparent
   mode http
   http-request set-header X-Forwarded-Proto https if { ssl_fc }
   http-request set-header X-Forwarded-Proto http if !{ ssl_fc }
   option httpchk
   option httplog
   redirect scheme https code 301 if { hdr(host) -i 192.168.24.2 } !{ ssl_fc }
   rsprep ^Location:\ http:(.*) Location:\ https:\1
   server undercloud.internalapi.localdomain 192.168.24.1:8774 check fall 5 inter 2000 rise 2

The commands I used to start a debug container w/o healthchecks (edited /var/lib/tripleo-config/docker-container-startup-config-step_4.json manually for nova_api)

$ sudo paunch --debug debug --file /var/lib/tripleo-config/docker-container-startup-config-step_4.json --interactive --shell --user root --container nova_api --action run #--config-id tripleo_step4 --managed-by tripleo-Controller

$ (in the container) sudo -E kolla_set_configs; nova-api

This is happening after switched to containerized undercloud:- https://review.openstack.org/#/c/571529/
Happening only in promotion jobs(new container image for nova-api), caused by:- https://review.openstack.org/#/c/558765. The patch is wrong, proposing fix for it, let's see how it goes.

Proposed:- https://review.openstack.org/#/c/574145/

Fix released, master promoted.

This issue was fixed in the openstack/kolla 7.0.0.0b3 development milestone.