Glance

multiprocess glance-api failed to exit when stopped by ctrl+c

Bug #978130 reported by Huang Zhiteng on 2012-04-10

This bug affects 1 person

	Status	Importance	Assigned to	Milestone
Glance	Fix Released	High	Eoghan Glynn	Glance 2012.2 "folsom"
Essex	Fix Released	Low	Eoghan Glynn	Glance 2012.1.1
glance (Ubuntu)	Fix Released	Undecided	Unassigned
Precise	Fix Released	Undecided	Unassigned

Bug Description

I was running glance-api with DevStack on Fedora 16. I found multiprocess glance-api failed to exit when stopped by ctrl+c.

The version I was using is: commit 4255bbfb2ee80abdca9c9858a7076bf4b52180be

Here's how to reproduce this issue:

1. Config # of workers > 1 (which means glance-api will run in eventlet pool of one standalone process);
2. Start glance-api by '/PATH_TO_GLANCE_BIN/glance-api --config-file=/PATH_TO_GLANCE_CONFIG';
3. Stop glance-api by pressing Ctrl+C

Then the stdout or log (?) will end-up error messages like:

File "/opt/stack/glance/glance/common/wsgi.py", line 235, in run_server
    log=WritableLogger(self.logger), custom_pool=self.pool)
  File "/usr/lib/python2.7/site-packages/eventlet/wsgi.py", line 562, in server
    serv = Server(sock, sock.getsockname(),
  File "/usr/lib64/python2.7/socket.py", line 224, in meth
    return getattr(self._sock,name)(*args)
  File "/usr/lib64/python2.7/socket.py", line 170, in _dummy
    raise error(EBADF, 'Bad file descriptor')
socket.error: [Errno 9] Bad file descriptor
2012-04-10 23:09:12 374 INFO [eventlet.wsgi.server] Started child 1240
Traceback (most recent call last):
  File "bin/glance-api", line 52, in <module>
    server.wait()
  File "/opt/stack/glance/glance/common/wsgi.py", line 209, in wait
    self.wait_on_children()
  File "/opt/stack/glance/glance/common/wsgi.py", line 193, in wait_on_children
    self.run_child()
  File "/opt/stack/glance/glance/common/wsgi.py", line 220, in run_child
    self.run_server()
  File "/opt/stack/glance/glance/common/wsgi.py", line 235, in run_server
    log=WritableLogger(self.logger), custom_pool=self.pool)
  File "/usr/lib/python2.7/site-packages/eventlet/wsgi.py", line 562, in server
    serv = Server(sock, sock.getsockname(),
  File "/usr/lib64/python2.7/socket.py", line 224, in meth
    return getattr(self._sock,name)(*args)
  File "/usr/lib64/python2.7/socket.py", line 170, in _dummy
    raise error(EBADF, 'Bad file descriptor')
socket.error: [Errno 9] Bad file descriptor
2012-04-10 23:09:12 373 ERROR [eventlet.wsgi.server] Removing dead child 1239
2012-04-10 23:09:12 373 INFO [eventlet.wsgi.server] Started child 1241
Traceback (most recent call last):
  File "bin/glance-api", line 52, in <module>
    server.wait()
  File "/opt/stack/glance/glance/common/wsgi.py", line 209, in wait
    self.wait_on_children()
  File "/opt/stack/glance/glance/common/wsgi.py", line 193, in wait_on_children
    self.run_child()
  File "/opt/stack/glance/glance/common/wsgi.py", line 220, in run_child
    self.run_server()
  File "/opt/stack/glance/glance/common/wsgi.py", line 235, in run_server
    log=WritableLogger(self.logger), custom_pool=self.pool)
  File "/usr/lib/python2.7/site-packages/eventlet/wsgi.py", line 562, in server
    serv = Server(sock, sock.getsockname(),
  File "/usr/lib64/python2.7/socket.py", line 224, in meth
    return getattr(self._sock,name)(*args)
  File "/usr/lib64/python2.7/socket.py", line 170, in _dummy
    raise error(EBADF, 'Bad file descriptor')
socket.error: [Errno 9] Bad file descriptor
2012-04-10 23:09:12 373 ERROR [eventlet.wsgi.server] Removing dead child 1238
2012-04-10 23:09:12 374 ERROR [eventlet.wsgi.server] Removing dead child 1240
2012-04-10 23:09:12 373 INFO [eventlet.wsgi.server] Started child 1242

Tags:

Related branches

lp:~openstack-ubuntu-testing/glance/precise-essex-proposed

Merged into lp:~ubuntu-server-dev/glance/essex at revision 123

Ubuntu Server Developers: Pending requested 2012-03-16

lp:ubuntu/precise-proposed/glance

Jay Pipes (jaypipes) on 2012-04-10

Changed in glance:
status:	New → Confirmed
importance:	Undecided → High

Eoghan Glynn (eglynn) on 2012-04-11

Changed in glance:
assignee:	nobody → Eoghan Glynn (eglynn)
status:	Confirmed → In Progress

Eoghan Glynn (eglynn) on 2012-04-11

Changed in glance:
milestone:	none → folsom-1

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2012-04-11: Fix proposed to glance (master)

Fix proposed to branch: master
Review: https://review.openstack.org/6468

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2012-04-12: Fix merged to glance (master)

Reviewed: https://review.openstack.org/6468
Committed: http://github.com/openstack/glance/commit/93013cc5d5d3ab1a574e3110f80ff63f2ff76674
Submitter: Jenkins
Branch: master

commit 93013cc5d5d3ab1a574e3110f80ff63f2ff76674
Author: Eoghan Glynn <email address hidden>
Date: Wed Apr 11 17:44:07 2012 +0100

Kill glance-api child workers on SIGINT

Fixes bug 978130

    Previously glance-api configured to run multiple child
    workers went in a respawn tail-spin when the parent is
    interrupted with crtl+c.

This is now avoided by terminating the process group when
SIGINT is received, in the same way as SIGTERM is handled.

Change-Id: Ieca90f45e07cc3645a997bba493e0b9a9f08a470

Changed in glance:
status:	In Progress → Fix Committed

Thierry Carrez (ttx) on 2012-05-23

Changed in glance:
status:	Fix Committed → Fix Released

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2012-06-15: Fix proposed to glance (stable/essex)

Fix proposed to branch: stable/essex
Review: https://review.openstack.org/8595

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2012-06-15: Fix merged to glance (stable/essex)

Reviewed: https://review.openstack.org/8595
Committed: http://github.com/openstack/glance/commit/57db6c58ba2a207507e54b2a92ce8d407659f363
Submitter: Jenkins
Branch: stable/essex

commit 57db6c58ba2a207507e54b2a92ce8d407659f363
Author: Eoghan Glynn <email address hidden>
Date: Wed Apr 11 17:44:07 2012 +0100

Kill glance-api child workers on SIGINT

Fixes bug 978130

    Previously glance-api configured to run multiple child
    workers went in a respawn tail-spin when the parent is
    interrupted with crtl+c.

This is now avoided by terminating the process group when
SIGINT is received, in the same way as SIGTERM is handled.

Change-Id: Ieca90f45e07cc3645a997bba493e0b9a9f08a470

Dave Walker (davewalker) on 2012-08-24

Changed in glance (Ubuntu):
status:	New → Fix Released
Changed in glance (Ubuntu Precise):
status:	New → Confirmed

Revision history for this message

Adam Gandelman (gandelman-a) wrote on 2012-08-30: Verification report.

Please find the attached test log from the Ubuntu Server Team's CI infrastructure. As part of the verification process for this bug, Glance has been deployed and configured across multiple nodes using precise-proposed as an installation source. After successful bring-up and configuration of the cluster, a number of exercises and smoke tests have be invoked to ensure the updated package did not introduce any regressions. A number of test iterations were carried out to catch any possible transient errors.

Please Note the list of installed packages at the top and bottom of the report.

For records of upstream test coverage of this update, please see the Jenkins links in the comments of the relevant upstream code-review(s):

Trunk review: https://review.openstack.org/6468
Stable review: https://review.openstack.org/8595

As per the provisional Micro Release Exception granted to this package by the Technical Board, we hope this contributes toward verification of this update.

Revision history for this message

Adam Gandelman (gandelman-a) wrote on 2012-08-30:

2012.1.3+stable~20120821-120fcf-0ubuntu1.log Edit (82.0 KiB, text/plain)

Test coverage log.

tags:

added: verification-done

Revision history for this message

Clint Byrum (clint-fewbar) wrote on 2012-09-03: Update Released

The verification of this Stable Release Update has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regresssions.

Revision history for this message

Launchpad Janitor (janitor) wrote on 2012-09-03:

This bug was fixed in the package glance - 2012.1.3+stable~20120821-120fcf-0ubuntu1

---------------
glance (2012.1.3+stable~20120821-120fcf-0ubuntu1) precise-proposed; urgency=low

  * New upstream snapshot. (LP: #1041120)
  * Resynchronzie with stable/essex:
    - Glance add uploads a double image if using ssl and images is smaller
      than 4k. (LP: #1007093)
    - If response.environ is None, instance fails to spawn.
      (LP: #1010560)
    - exception.BadStoreURL exposes sensitive information to end users.
      (LP: #1012268)
    - glance-cache.conf needs metadata encryption key (LP: #1012752)
    - image.upload notification doesn't report size (LP: #1018246)
    - Admins should be able to share image regardless of ownership.
      (LP: #1021054)
    - Glance scrubber date formatting fails with postgres (LP: #1022369)
    - Support zero-size image creation. (LP: #1025353)
    - Image id not contained in swift chunk debug message. (LP: #1028433)
    - qpid_heartbeat setting is ineffective. (LP: #1032314)
    - Image properties that reference image ids are not updated to UUIDs.
      (LP: #975651)
    - Migration 012_id_to_uuid attempts to convert IDs twice for non-sqlite
      databases. (LP: #975655)
    - multiprocess glance-api failed to exit when stopped by ctrl+c.
      (LP: #978130)
    - /usr/bin/glance's built-in pager breaks redirection.
      (LP: #978610)
    - Content-Length and Transfer-Encoding are mutually exclusive HTTP headers
      (LP: #981332)
    - glance add command - incorrect help text (LP: #997565)
  * debian/patches/convert_properties_to_uuid.patch: Dropped no longer
    needed.
  * debian/patches/fix-pep8-ubuntu.patch: Dropped no longer needed.
-- Adam Gandelman <email address hidden> Fri, 24 Jun 2012 03:14:33 -0400