Code review comment for lp:~yamahata/glance/lp802883

On Wed, Jul 06, 2011 at 03:12:38PM -0000, OpenStack Hudson wrote:

> ======================================================================
> FAIL: Ensure marker and limit query params work
> ----------------------------------------------------------------------
> Traceback (most recent call last):
> File "/tmp/tmpJjk6OQ/tests/functional/test_curl_api.py", line 1017, in test_limited_images
> self.assertEqual('{"images": []}', out.strip())
> AssertionError: '{"images": []}' != 'Traceback (most recent call last):\n File "/usr/lib/python2.6/dist-packages/eventlet/wsgi.py", line 336, in handle_one_response\n result = self.application(self.environ, start_response)\n File "/usr/lib/pymodules/python2.6/webob/dec.py", line 147, in __call__\n resp = self.call_func(req, *args, **self.kwargs)\n File "/usr/lib/pymodules/python2.6/webob/dec.py", line 208, in call_func\n return self.func(req, *args, **kwargs)\n File "/tmp/tmpJjk6OQ/glance/common/wsgi.py", line 113, in __call__\n response = req.get_response(self.application)\n File "/usr/lib/pymodules/python2.6/webob/request.py", line 1053, in get_response\n application, catch_exc_info=False)\n File "/usr/lib/pymodules/python2.6/webob/request.py", line 1022, in call_application\n app_iter = application(self.environ, start_response)\n File "/usr/lib/pymodules/python2.6/webob/dec.py", line 159, in __call__\n return resp(environ, start_response)\n File "/usr/local/lib/python2.6/dist-packages/Routes-1.12.3-py2.6.egg/routes/middleware.py", line 131, in __call__\n response = self.app(environ, start_response)\n File "/usr/lib/pymodules/python2.6/webob/dec.py", line 159, in __call__\n return resp(environ, start_response)\n File "/usr/lib/pymodules/python2.6/webob/dec.py", line 147, in __call__\n resp = self.call_func(req, *args, **self.kwargs)\n File "/usr/lib/pymodules/python2.6/webob/dec.py", line 208, in call_func\n return self.func(req, *args, **kwargs)\n File "/tmp/tmpJjk6OQ/glance/common/wsgi.py", line 311, in __call__\n request, **action_args)\n File "/tmp/tmpJjk6OQ/glance/common/wsgi.py", line 328, in dispatch\n return method(*args, **kwargs)\n File "/tmp/tmpJjk6OQ/glance/api/v1/images.py", line 99, in index\n images = registry.get_images_list(self.options, **params)\n File "/tmp/tmpJjk6OQ/glance/registry/__init__.py", line 37, in get_images_list\n return c.get_images(**kwargs)\n File "/tmp/tmpJjk6OQ/glance/registry/client.py", line 58, in get_images\n res = self.do_request("GET", "/images", params=params)\n File "/tmp/tmpJjk6OQ/glance/common/client.py", line 148, in do_request\n "server. Got error: %s" % e)\nClientConnectionError: Unable to connect to server. Got error: [Errno 111] ECONNREFUSED'

All test passes for me after merging glance trunk(revno 151).

The above shows curl received the following trace back.
ECONNREFUSED means that no one was listening the port of glance registry,
so glance image server couldn't connect to glance registry server.

My guess is, tests.utils.get_unused_port() and
tests.functional.FunctionlTet.ping_server() are very prone to race
condition. So the testing machine is somewhat overloaded unluckily
when the test was run.

The race scenario

- get_unsed_port()
  allocate the port, and then free the socket freeing theport.

- some other process in the test system used the port.

- wait_for_servers()
  checks the port is used. Yes, it's surely used by the third program.
  Then go ahead.
  But this isn't what the test expected.

- the third program freed the socket.

- Here glance api server started, but glance register server is under
  booting up. (Or aborted abnormally due to unsed port)

- curl is executed and access api service

- the api service tries to connect the register service.
  => ECONNREFUSED

Comments?

thanks,

> Traceback (most recent call last):
> File "/usr/lib/python2.6/dist-packages/eventlet/wsgi.py", line 336, in handle_one_response
> result = self.application(self.environ, start_response)
> File "/usr/lib/pymodules/python2.6/webob/dec.py", line 147, in __call__
> resp = self.call_func(req, *args, **self.kwargs)
> File "/usr/lib/pymodules/python2.6/webob/dec.py", line 208, in call_func
> return self.func(req, *args, **kwargs)
> File "/tmp/tmpJjk6OQ/glance/common/wsgi.py", line 113, in __call__
> response = req.get_response(self.application)
> File "/usr/lib/pymodules/python2.6/webob/request.py", line 1053, in get_response
> application, catch_exc_info=False)
> File "/usr/lib/pymodules/python2.6/webob/request.py", line 1022, in call_application
> app_iter = application(self.environ, start_response)
> File "/usr/lib/pymodules/python2.6/webob/dec.py", line 159, in __call__
> return resp(environ, start_response)
> File "/usr/local/lib/python2.6/dist-packages/Routes-1.12.3-py2.6.egg/routes/middleware.py", line 131, in __call__
> response = self.app(environ, start_response)
> File "/usr/lib/pymodules/python2.6/webob/dec.py", line 159, in __call__
> return resp(environ, start_response)
> File "/usr/lib/pymodules/python2.6/webob/dec.py", line 147, in __call__
> resp = self.call_func(req, *args, **self.kwargs)
> File "/usr/lib/pymodules/python2.6/webob/dec.py", line 208, in call_func
> return self.func(req, *args, **kwargs)
> File "/tmp/tmpJjk6OQ/glance/common/wsgi.py", line 311, in __call__
> request, **action_args)
> File "/tmp/tmpJjk6OQ/glance/common/wsgi.py", line 328, in dispatch
> return method(*args, **kwargs)
> File "/tmp/tmpJjk6OQ/glance/api/v1/images.py", line 99, in index
> images = registry.get_images_list(self.options, **params)
> File "/tmp/tmpJjk6OQ/glance/registry/__init__.py", line 37, in get_images_list
> return c.get_images(**kwargs)
> File "/tmp/tmpJjk6OQ/glance/registry/client.py", line 58, in get_images
> res = self.do_request("GET", "/images", params=params)
> File "/tmp/tmpJjk6OQ/glance/common/client.py", line 148, in do_request
> "server. Got error: %s" % e)
> ClientConnectionError: Unable to connect to server. Got error: [Errno 111] ECONNREFUSED

--
yamahata