Glance image-delete HTTPInternalServerError HTTP 500

Bug #1075580 reported by bullardza@gmail.com
20
This bug affects 3 people
Affects Status Importance Assigned to Milestone
Glance
Fix Released
Undecided
Josh Durgin
Folsom
Fix Released
Medium
Josh Durgin
Grizzly
Fix Released
Undecided
Josh Durgin
glance (Ubuntu)
Fix Released
Undecided
Unassigned
Quantal
Fix Released
Undecided
Unassigned

Bug Description

When trying to glance image-delete I get the --debug output below, along with an HTTP 500. There doesn't seem to be a rhyme or reason for getting this.

Any help would be appreciated.

'X-Auth-Token: 6c3a2309546340bc92ef28e6e3573e09' -H 'Content-Type: application/octet-stream' -H 'User-Agent: python-glanceclient' http://10.101.54.11:9292/v1/images/102b2ad1-724d-4a71-a14a-7c46ec779800

HTTP/1.1 500 Internal Server Error
date: Tue, 06 Nov 2012 14:52:38 GMT
content-length: 3450
content-type: text/plain
connection: close

Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/eventlet/wsgi.py", line 382, in handle_one_response
    result = self.application(self.environ, start_response)
  File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 147, in __call__
    resp = self.call_func(req, *args, **self.kwargs)
  File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 210, in call_func
    return self.func(req, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/glance/common/wsgi.py", line 326, in __call__
    response = req.get_response(self.application)
  File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1086, in get_response
    application, catch_exc_info=False)
  File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1055, in call_application
    app_iter = application(self.environ, start_response)
  File "/usr/lib/python2.7/dist-packages/keystone/middleware/auth_token.py", line 278, in __call__
    return self.app(env, start_response)
  File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 147, in __call__
    resp = self.call_func(req, *args, **self.kwargs)
  File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 210, in call_func
    return self.func(req, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/glance/common/wsgi.py", line 326, in __call__
    response = req.get_response(self.application)
  File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1086, in get_response
    application, catch_exc_info=False)
  File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1055, in call_application
    app_iter = application(self.environ, start_response)
  File "/usr/lib/python2.7/dist-packages/paste/urlmap.py", line 203, in __call__
    return app(environ, start_response)
  File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 159, in __call__
    return resp(environ, start_response)
  File "/usr/lib/python2.7/dist-packages/routes/middleware.py", line 131, in __call__
    response = self.app(environ, start_response)
  File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 159, in __call__
    return resp(environ, start_response)
  File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 147, in __call__
    resp = self.call_func(req, *args, **self.kwargs)
  File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 210, in call_func
    return self.func(req, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/glance/common/wsgi.py", line 532, in __call__
    request, **action_args)
  File "/usr/lib/python2.7/dist-packages/glance/common/wsgi.py", line 549, in dispatch
    return method(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/glance/common/utils.py", line 408, in wrapped
    return func(self, req, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/glance/api/v1/images.py", line 836, in delete
    req.context, id)
  File "/usr/lib/python2.7/dist-packages/glance/store/__init__.py", line 259, in safe_delete_from_backend
    return delete_from_backend(context, uri, **kwargs)
  File "/usr/lib/python2.7/dist-packages/glance/store/__init__.py", line 239, in delete_from_backend
    return store.delete(loc)
  File "/usr/lib/python2.7/dist-packages/glance/store/rbd.py", line 267, in delete
    with rbd.Image(ioctx, loc.image) as image:
  File "/usr/lib/python2.7/dist-packages/rbd.py", line 299, in __init__
    raise TypeError('name must be a string')
TypeError: name must be a string

Request returned failure status.
None
HTTPInternalServerError (HTTP 500)

Revision history for this message
bullardza@gmail.com (bullardza) wrote :

I should add that I can delete some images, but not all, and there doesn't seem to be a rhyme or reason for how they were created as it relates to being able to delete them.

Revision history for this message
Iccha Sethi (iccha-sethi) wrote :

Can you paste the entire glance command?

Also are you the owner of the images you are trying to delete which dont get deleted?

Revision history for this message
Jay Pipes (jaypipes) wrote :

My guess is that image identifier UUIDs that start with integers vs. letters are being misinterpreted by the RBD library as integers...

Revision history for this message
bullardza@gmail.com (bullardza) wrote :
Download full text (7.3 KiB)

Sure.

Here is my glance image-list

root@cloud1:~# glance image-list
+--------------------------------------+-----------------------------------+-------------+------------------+-------------+--------+
| ID | Name | Disk Format | Container Format | Size | Status |
+--------------------------------------+-----------------------------------+-------------+------------------+-------------+--------+
| 102b2ad1-724d-4a71-a14a-7c46ec779800 | another-cirros-raw | raw | bare | 41126400 | active |
| 143c8e08-5e63-4c12-98f8-5281b4dccd74 | w2k8x64virtio | raw | ovf | 34359738368 | active |
| 26913719-a3dd-47d0-9dd3-ab3c43e1a832 | rhel-6.3-x86_64-minimal-net-net | raw | bare | 5368709120 | active |
| 2793ec6d-c705-4201-b7e0-bc9e3e437e99 | precise-server-cloudimg-amd64-raw | raw | bare | 2147483648 | active |
| 29cadc20-9b7b-479d-9cf3-781e2e730323 | smoke-test-17199 | raw | bare | 41126400 | active |
| 33413113-bdbd-4c83-9395-edccdcb47b3b | cirros-0.3.0-x86_64 | qcow2 | bare | 9761280 | active |
| 47348b19-8d1f-46d7-a9cf-b4b5592f6276 | smoke-test-13678 | raw | bare | 41126400 | active |
| 508116f8-aaa9-4dea-80c3-e06f55c20255 | w2k8x64virtioproper | raw | ovf | 34359738368 | active |
| 621e0155-5e72-4380-bd78-031be17a8dae | precise-server-cloudimg-amd64 | qcow2 | bare | 232914944 | active |
| 742b10c2-98ba-4077-834f-c5a9031aafbf | rhel-6.3-x86_64-minimal | raw | bare | 5368709120 | active |
| 7665d5cb-d9eb-4107-9c86-34425e3d6a53 | w2k8-iaas | raw | ovf | 53687091200 | active |
| 78924e97-b5ad-47f8-8aec-a0b6951292f9 | windows2008r2-raw | raw | bare | 34359738368 | active |
| 83fb108a-03ce-493b-af22-ed5557eefafc | cirros-0.3.0.raw | raw | bare | 41126400 | active |
| 842fa5b1-7458-4db2-9ae4-391d2f943437 | windows2008r2-virtio.raw | raw | bare | 34359738368 | active |
| 8ed374fa-e950-4d0b-8644-c23f93ce6062 | rhel-6.3-x86_64-minimal-net | raw | bare | 5368709120 | active |
| a3f1473d-ce67-4a14-978c-36fd015625b8 | windows2008r2-raw | raw | bare | 34359738368 | active |
| a6ec9ddb-eb7c-4cd8-93ce-63f5261f30aa | cirros_cmd_vol_snap | raw | bare | | active |
| b2210588-7e80-4d4c-9c72-8f68d2bb49d0 | w2k8x64 | raw | bare | 53687091200 | active |
| c7ec2f2b-5ea3-46ea-939b-90bf358a81e3 | yet-another-cirros-raw | raw | bare | 41126400 | active |
| cbeefdf3-6e02-44be-8582-baf5976d8d47 | smoke-test-13512 | raw | bare | 41126400 | active |
| d43bf4ae-e728-40d1-8855-395371033c87 | smoke-test-8360 ...

Read more...

Revision history for this message
Josh Durgin (jdurgin) wrote :

librbd doesn't handle unicode strings, and that's the exception it throws when given one. I'll probably change it to only throw the exception when given a unicode string that contains non-ascii in the future, but for current versions of librbd, this patch should work.

I'll submit this through gerrit with unit tests shortly.

Changed in glance:
assignee: nobody → Josh Durgin (jdurgin)
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to glance (master)

Fix proposed to branch: master
Review: https://review.openstack.org/15804

Josh Durgin (jdurgin)
tags: added: folsom-backport-potential
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to glance (master)

Reviewed: https://review.openstack.org/15804
Committed: http://github.com/openstack/glance/commit/3aea9320d20d0291c5b4b027a585370d8e658a95
Submitter: Jenkins
Branch: master

commit 3aea9320d20d0291c5b4b027a585370d8e658a95
Author: Josh Durgin <email address hidden>
Date: Fri Nov 9 09:28:19 2012 -0800

    Ensure strings passed to librbd are not unicode

    librbd rejects unicode by throwing an exception. If the location uri
    was unicode, this would result in deletes failing. Convert all rbd
    StoreLocation properties to ascii strings to avoid this problem.

    Fixes bug #1075580
    Signed-off-by: Josh Durgin <email address hidden>

    Change-Id: I445902bb1b72b8c814585f1481e6bf9e758b4e07

Changed in glance:
status: In Progress → Fix Committed
Thierry Carrez (ttx)
Changed in glance:
milestone: none → grizzly-1
status: Fix Committed → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to glance (stable/folsom)

Fix proposed to branch: stable/folsom
Review: https://review.openstack.org/16996

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to glance (stable/folsom)

Reviewed: https://review.openstack.org/16996
Committed: http://github.com/openstack/glance/commit/49408e9f846ab08ed4f1ac32c3e4e934cd448990
Submitter: Jenkins
Branch: stable/folsom

commit 49408e9f846ab08ed4f1ac32c3e4e934cd448990
Author: Josh Durgin <email address hidden>
Date: Fri Nov 9 09:28:19 2012 -0800

    Ensure strings passed to librbd are not unicode

    librbd rejects unicode by throwing an exception. If the location uri
    was unicode, this would result in deletes failing. Convert all rbd
    StoreLocation properties to ascii strings to avoid this problem.

    Fixes bug #1075580
    Signed-off-by: Josh Durgin <email address hidden>

    Change-Id: I445902bb1b72b8c814585f1481e6bf9e758b4e07
    (cherry picked from commit 3aea9320d20d0291c5b4b027a585370d8e658a95)

Changed in glance (Ubuntu):
status: New → Fix Released
Changed in glance (Ubuntu Quantal):
status: New → Confirmed
Revision history for this message
Clint Byrum (clint-fewbar) wrote : Please test proposed package

Hello <email address hidden>, or anyone else affected,

Accepted glance into quantal-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/glance/2012.2.1-0ubuntu1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in glance (Ubuntu Quantal):
status: Confirmed → Fix Committed
tags: added: verification-needed
Mark McLoughlin (markmc)
tags: removed: folsom-backport-potential
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package glance - 2012.2.1-0ubuntu1

---------------
glance (2012.2.1-0ubuntu1) quantal-proposed; urgency=low

  * Dropped patches, applied upstream:
    - debian/patches/CVE-2012-4573.patch
    - debian/patches/CVE-2012-4573b.patch
  * Resynchronize with stable/folsom (199783ce) (LP: #1085255):
    - [49408e9] Glance image-delete HTTPInternalServerError HTTP 500
      (LP: #1075580)
    - [91aaa48] Image fails to upload to swift: TypeError: object of type
      'CooperativeReader' has no len( (LP: #1057322)
    - [a296a5b] Return 403 when admin deletes a deleted image (LP: #1060944)
    - [3e58a6a] Disallow updating deleted images. (LP: #1060930)
    - [26c8085] admins can see deleted images in v2 api (LP: #1071446)
    - [8321ca6] No exclude option to skip tests in run_tests.sh (LP: #1065758)
    - [c3bea11] Badly named stable/folsom Glance tarballs (LP: #1059634)
    - [fc0ee76] Non-admin users can cause public glance images to be deleted
      from the backend storage repository in the v2 api (LP: #1076506)
    - [90bcdc5] Non-admin users can cause public glance images to be deleted
      from the backend storage repository (LP: #1065187)
    - [7841cc9] FakeAuth not always admin
    - [ddad275] Jenkins jobs fail because of incompatibility between sqlalchemy-
      migrate and the newest sqlalchemy-0.8.0b1 (LP: #1073569)
    - [1d5c651] nosetest options cause no such option errors (LP: #1056420)
    - [ac223e2] Set defaultbranch in .gitreview to stable/folsom
 -- Adam Gandelman <email address hidden> Tue, 04 Dec 2012 09:19:35 -0800

Changed in glance (Ubuntu Quantal):
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.