Uploading a new object fails with Ceph as object storage backend using RadosGW

Bug #1352256 reported by Ashish Chandra
78
This bug affects 16 people
Affects Status Importance Assigned to Milestone
OpenStack Dashboard (Horizon)
Fix Released
Low
Paul Karikh
Kilo
Fix Released
Undecided
Unassigned
Ubuntu Cloud Archive
Invalid
Medium
Corey Bryant
Kilo
Fix Released
Medium
Corey Bryant

Bug Description

While uploading a new Object using Horizon, with Ceph as object storage backend, it fails with error mesage "Error: Unable to upload object"

Ceph Release : Firefly

Error in horizon_error.log:

[Wed Jul 23 09:04:46.840751 2014] [:error] [pid 30045:tid 140685813683968] INFO:urllib3.connectionpool:Starting new HTTP connection (1): firefly-master.ashish.com
[Wed Jul 23 09:04:46.842984 2014] [:error] [pid 30045:tid 140685813683968] WARNING:urllib3.connectionpool:HttpConnectionPool is full, discarding connection: firefly-master.ashish.com
[Wed Jul 23 09:04:46.843118 2014] [:error] [pid 30045:tid 140685813683968] REQ: curl -i http://firefly-master.ashish.com/swift/v1/new-cont-dash/test -X PUT -H "X-Auth-Token: 91fc8466ce17e0d22af86de9b3343b2d"
[Wed Jul 23 09:04:46.843227 2014] [:error] [pid 30045:tid 140685813683968] RESP STATUS: 411 Length Required
[Wed Jul 23 09:04:46.843584 2014] [:error] [pid 30045:tid 140685813683968] RESP HEADERS: [('date', 'Wed, 23 Jul 2014 09:04:46 GMT'), ('content-length', '238'), ('content-type', 'text/html; charset=iso-8859-1'), ('connection', 'close'), ('server', 'Apache/2.4.7 (Ubuntu)')]
[Wed Jul 23 09:04:46.843783 2014] [:error] [pid 30045:tid 140685813683968] RESP BODY: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
[Wed Jul 23 09:04:46.843907 2014] [:error] [pid 30045:tid 140685813683968] <html><head>
[Wed Jul 23 09:04:46.843930 2014] [:error] [pid 30045:tid 140685813683968] <title>411 Length Required</title>
[Wed Jul 23 09:04:46.843937 2014] [:error] [pid 30045:tid 140685813683968] </head><body>
[Wed Jul 23 09:04:46.843944 2014] [:error] [pid 30045:tid 140685813683968] <h1>Length Required</h1>
[Wed Jul 23 09:04:46.843951 2014] [:error] [pid 30045:tid 140685813683968] <p>A request of the requested method PUT requires a valid Content-length.<br />
[Wed Jul 23 09:04:46.843957 2014] [:error] [pid 30045:tid 140685813683968] </p>
[Wed Jul 23 09:04:46.843963 2014] [:error] [pid 30045:tid 140685813683968] </body></html>
[Wed Jul 23 09:04:46.843969 2014] [:error] [pid 30045:tid 140685813683968]
[Wed Jul 23 09:04:46.844530 2014] [:error] [pid 30045:tid 140685813683968] Object PUT failed: http://firefly-master.ashish.com/swift/v1/new-cont-dash/test 411 Length Required [first 60 chars of response] <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
[Wed Jul 23 09:04:46.844555 2014] [:error] [pid 30045:tid 140685813683968] <html><he
[Wed Jul 23 09:04:46.844567 2014] [:error] [pid 30045:tid 140685813683968] Traceback (most recent call last):
[Wed Jul 23 09:04:46.844573 2014] [:error] [pid 30045:tid 140685813683968] File "/opt/stack/python-swiftclient/swiftclient/client.py", line 1208, in _retry
[Wed Jul 23 09:04:46.844582 2014] [:error] [pid 30045:tid 140685813683968] rv = func(self.url, self.token, *args, **kwargs)
[Wed Jul 23 09:04:46.844588 2014] [:error] [pid 30045:tid 140685813683968] File "/opt/stack/python-swiftclient/swiftclient/client.py", line 981, in put_object
[Wed Jul 23 09:04:46.844594 2014] [:error] [pid 30045:tid 140685813683968] http_response_content=body)
[Wed Jul 23 09:04:46.844601 2014] [:error] [pid 30045:tid 140685813683968] ClientException: Object PUT failed: http://firefly-master.ashish.com/swift/v1/new-cont-dash/test 411 Length Required [first 60 chars of response] <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
[Wed Jul 23 09:04:46.844607 2014] [:error] [pid 30045:tid 140685813683968] <html><he
[Wed Jul 23 09:04:46.844879 2014] [:error] [pid 30045:tid 140685813683968] Recoverable error: Object PUT failed: http://firefly-master.ashish.com/swift/v1/new-cont-dash/test 411 Length Required [first 60 chars of response] <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
[Wed Jul 23 09:04:46.844900 2014] [:error] [pid 30045:tid 140685813683968] <html><he
[Wed Jul 23 09:04:46.854643 2014] [:error] [pid 30045:tid 140685813683968] INFO:urllib3.connectionpool:Starting new HTTP connection (1): 10.0.1.60
[Wed Jul 23 09:04:46.855247 2014] [:error] [pid 30045:tid 140685813683968] DEBUG:urllib3.connectionpool:Setting read timeout to None
[Wed Jul 23 09:04:46.888503 2014] [:error] [pid 30045:tid 140685813683968] DEBUG:urllib3.connectionpool:"POST /v2.0/tokens HTTP/1.1" 200 1358
[Wed Jul 23 09:04:46.892722 2014] [:error] [pid 30045:tid 140685813683968] INFO:urllib3.connectionpool:Starting new HTTP connection (1): 10.0.1.60
[Wed Jul 23 09:04:46.894144 2014] [:error] [pid 30045:tid 140685813683968] DEBUG:urllib3.connectionpool:Setting read timeout to None
[Wed Jul 23 09:04:46.910724 2014] [:error] [pid 30045:tid 140685813683968] DEBUG:urllib3.connectionpool:"GET /v2.0/tenants HTTP/1.1" 200 231

Revision history for this message
Ashish Chandra (ashish-chandra) wrote :

The object can be added using CLI:

ashish@firefly-master:~/devstack$ swift upload new-cont openrc
openrc
ashish@firefly-master:~/devstack$ swift stat new-cont
       Account: v1
     Container: new-cont
       Objects: 1
         Bytes: 3463
      Read ACL:
     Write ACL:
       Sync To:
      Sync Key:
        Server: Apache/2.4.7 (Ubuntu)
X-Container-Bytes-Used-Actual: 4096
  Content-Type: text/plain; charset=utf-8
ashish@firefly-master:~/devstack$ swift stat new-cont openrc
       Account: v1
     Container: new-cont
        Object: openrc
Content Length: 3463
 Last Modified: Wed, 23 Jul 2014 09:44:31 GMT
          ETag: 9204776814ca62c92c7996de725ecc6b
    Meta Mtime: 1406103867.326814
 Accept-Ranges: bytes
        Server: Apache/2.4.7 (Ubuntu)

The stats for the object misses "ContentType" and "X-Timestamp" header, which is not returned by RadosGW.

Revision history for this message
Jiri Suchomel (jsuchome) wrote :
Revision history for this message
Gary W. Smith (gary-w-smith) wrote :

This appears to be closely related to https://bugs.launchpad.net/horizon/+bug/1297173

Revision history for this message
Akihiro Motoki (amotoki) wrote :

It is a bug of Ceph side. bug 1297173 is marked as Invalid and the bug itself is handled in Ceph.

Revision history for this message
Fabricio Costi (fabricio-9) wrote :

I believe this bug is not related to 1297173. In Juno, Horizon fails to upload an object with error message "Unable to upload object", but it properly shows the detail pane of objects uploaded from cli. Looks like a Horizon issue.

From /var/log/apache2/error.log

[Wed Jan 28 00:41:27.879758 2015] [:error] [pid 1372:tid 140018748028672] Container HEAD failed: http://10.114.18.64:80/swift/v1/nebula 404 Not Found
[Wed Jan 28 00:41:27.879795 2015] [:error] [pid 1372:tid 140018748028672] Traceback (most recent call last):
[Wed Jan 28 00:41:27.879803 2015] [:error] [pid 1372:tid 140018748028672] File "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 1236, in _retry
[Wed Jan 28 00:41:27.879809 2015] [:error] [pid 1372:tid 140018748028672] rv = func(self.url, self.token, *args, **kwargs)
[Wed Jan 28 00:41:27.879816 2015] [:error] [pid 1372:tid 140018748028672] File "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 678, in head_container
[Wed Jan 28 00:41:27.879822 2015] [:error] [pid 1372:tid 140018748028672] http_response_content=body)
[Wed Jan 28 00:41:27.879828 2015] [:error] [pid 1372:tid 140018748028672] ClientException: Container HEAD failed: http://10.114.18.64:80/swift/v1/nebula 404 Not Found
[Wed Jan 28 00:41:34.065103 2015] [:error] [pid 1374:tid 140018815170304] Object PUT failed: http://10.114.18.64:80/swift/v1/nebula/synergy.conf 411 Length Required [$
[Wed Jan 28 00:41:34.065135 2015] [:error] [pid 1374:tid 140018815170304] <html><he
[Wed Jan 28 00:41:34.065142 2015] [:error] [pid 1374:tid 140018815170304] Traceback (most recent call last):
[Wed Jan 28 00:41:34.065148 2015] [:error] [pid 1374:tid 140018815170304] File "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 1236, in _retry
[Wed Jan 28 00:41:34.065155 2015] [:error] [pid 1374:tid 140018815170304] rv = func(self.url, self.token, *args, **kwargs)
[Wed Jan 28 00:41:34.065178 2015] [:error] [pid 1374:tid 140018815170304] File "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 1009, in put_object
[Wed Jan 28 00:41:34.065185 2015] [:error] [pid 1374:tid 140018815170304] http_response_content=body)
[Wed Jan 28 00:41:34.065191 2015] [:error] [pid 1374:tid 140018815170304] ClientException: Object PUT failed: http://10.114.18.64:80/swift/v1/nebula/synergy.conf 411 L$
[Wed Jan 28 00:41:34.065197 2015] [:error] [pid 1374:tid 140018815170304] <html><he
[Wed Jan 28 00:41:34.065464 2015] [:error] [pid 1374:tid 140018815170304] Recoverable error: Object PUT failed: http://10.114.18.64:80/swift/v1/nebula/synergy.conf 411$
[Wed Jan 28 00:41:34.065478 2015] [:error] [pid 1374:tid 140018815170304] <html><he

Yet as I said I can view the details of any object.

Paul Karikh (pkarikh)
Changed in horizon:
assignee: nobody → Paul Karikh (pkarikh)
Revision history for this message
Paul Karikh (pkarikh) wrote :

Looks like this bug is still valid.

I've tracked handling of upload request and found out that there is no place where Horizon passes length header into the swift client. `put_object` method from swift API supports `content_length ` param and Horizon calculates size of the file in the `put_object` method. So we can use this 'size' and pass it as a param into 'put_object'.

Revision history for this message
Rohit Karajgi (rohitk) wrote :

I was just able to reproduce it on Kilo, with radosgw object storage.

Revision history for this message
Rohit Karajgi (rohitk) wrote :

pkarikh: Can you please upload a patch so that I can test it in my environment?

Revision history for this message
Paul Karikh (pkarikh) wrote :

@rohitk: yep, I'm going to upload it soon. Right now I'm trying to fix failing unit-tests.

Revision history for this message
Rohit Karajgi (rohitk) wrote :

pkarikh: You're right, passing content_length=size in the put_object fixed the issue
for me. Waiting for your patch!
I suggest you also backport it to icehouse/juno. Because I confirm this issue exists there too.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to horizon (master)

Fix proposed to branch: master
Review: https://review.openstack.org/185403

Changed in horizon:
status: New → In Progress
Revision history for this message
Akihiro Motoki (amotoki) wrote :

Is it specific to Ceph implementation?
If so, I think it is a bug of Ceph and it should be fixed in Ceph side too.

On the other hand, it does not necessarily mean I oppose to Horizon change.

Revision history for this message
Jiri Suchomel (jsuchome) wrote :

I've been trying to reproduce this (and test the patch), but uploading objects through Horizon UI seems to work for me.

Do you need to do anything special to reproduce it?

Changed in horizon:
importance: Undecided → Low
milestone: none → liberty-2
Changed in horizon:
status: In Progress → Fix Committed
Changed in horizon:
status: Fix Committed → Fix Released
Thierry Carrez (ttx)
Changed in horizon:
milestone: liberty-2 → 8.0.0
Revision history for this message
Valery Tschopp (valery-tschopp) wrote :

Could it be possible to have this fix applied in kilo too?

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to horizon (stable/kilo)

Fix proposed to branch: stable/kilo
Review: https://review.openstack.org/287928

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to horizon (stable/kilo)

Reviewed: https://review.openstack.org/287928
Committed: https://git.openstack.org/cgit/openstack/horizon/commit/?id=181e43fe008a1dc6da4ba94de2b0e72912f955ba
Submitter: Jenkins
Branch: stable/kilo

commit 181e43fe008a1dc6da4ba94de2b0e72912f955ba
Author: Paul Karikh <email address hidden>
Date: Mon May 25 17:48:45 2015 +0300

    Add missing content-length header

    This patch adds missing content-length header
    param into swift_upload_object() method.
    Without it object upload handling could be failed
    with "411 Length Required" error if Ceph is used
    as an object storage.

    Change-Id: Id0813806abb36a6f015efd13ec987492a3701476
    Closes-Bug: #1352256
    (cherry picked from commit 8201d65cf983f9c995dd653953ad1719f6df8459)

tags: added: in-stable-kilo
Revision history for this message
James Page (james-page) wrote :

Adding task for Ubuntu Cloud Archive - we'll pick this up in the next set of kilo updates.

Revision history for this message
James Page (james-page) wrote : Update Released

The verification of the Stable Release Update for horizon has completed successfully and the package has now been released to -updates. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Changed in cloud-archive:
status: Triaged → Invalid
Revision history for this message
James Page (james-page) wrote :

This bug was fixed in the package horizon - 1:2015.1.3-0ubuntu1
---------------

 horizon (1:2015.1.3-0ubuntu1) trusty-kilo; urgency=medium
 .
   * New upstream stable release (LP: #1559215, #1352256).

Revision history for this message
Doug Hellmann (doug-hellmann) wrote : Fix included in openstack/horizon 2015.1.4

This issue was fixed in the openstack/horizon 2015.1.4 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

This issue was fixed in the openstack/horizon 2015.1.4 release.

Revision history for this message
Ryan Beisner (1chb1n) wrote : Please test proposed package

Hello Ashish, or anyone else affected,

Accepted horizon into kilo-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:kilo-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-kilo-needed to verification-kilo-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-kilo-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

tags: added: verification-kilo-needed
Revision history for this message
Corey Bryant (corey.bryant) wrote : Update Released

The verification of the Stable Release Update for horizon has completed successfully and the package has now been released to -updates. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Revision history for this message
Corey Bryant (corey.bryant) wrote :
Download full text (18.2 KiB)

This bug was fixed in the package horizon - 1:2015.1.4-0ubuntu3
---------------

 horizon (1:2015.1.4-0ubuntu3) trusty-kilo; urgency=medium
 .
   * debian/theme/ubuntu/_styles.scss: Ensure btn-primary text color takes
     precedence (LP: #1481216).
 .
 horizon (1:2015.1.4-0ubuntu2) trusty-kilo; urgency=medium
 .
   * d/p/remove-can-access-caching.patch (LP: #1382079): Remove session
     caching of can_access call results which was disabling the project
     selector.
 .
 horizon (1:2015.1.4-0ubuntu1) trusty-kilo; urgency=medium
 .
   * New upstream stable release (LP: #1580334).
 .
 horizon (1:2015.1.3-0ubuntu1) trusty-kilo; urgency=medium
 .
   * New upstream stable release (LP: #1559215, #1352256).
 .
 horizon (1:2015.1.2-0ubuntu1) vivid; urgency=medium
 .
   [ Chuck Short ]
   * Resynchronize with stable/kilo (5b9bdca) (LP: #1481008):
     - [5b9bdca] Use security group id not name in the create instance dialog
     - [74d885f] Fix not opening region selector on first click
     - [b8e1ce8] Neutron Quota Settings Flag Disables Neutron GUI
     - [853f0ec] Fix inability to switch region via Switch Region dropdown
     - [0b3c92d] Updated from global requirements
     - [d6be306] Fix a unit test for create instance
     - [8ed7624] Compatibility fix for pyscss 1.3.4
     - [7c8fc8f] Skip test_routerrule_detail blocking the gate
     - [fea681e] Remove 'shared' from Neutron Firewall table
     - [1d20ea0] Fix static file paths in heat topology view
     - [8da65b6] Don't get descript if attribute doesn't exist.
     - [905e92b] Fix attribute error with old glance backend
     - [c079705] Added correct URL to edit QoS spec
     - [900ca63] Updated from global requirements
     - [3becb61] Port cleanup on failed VM instance launch
     - [b84218d] N1Kv: Fix failing network and port creates
     - [0f9d94e] Add OPENSTACK_NOVA_EXTENSIONS_BLACK
 .
   [ Corey Byrant ]
   * d/rules: Prevent dh_python2 from guessing dependencies.
   * d/openstack-dashboard-ubuntu-theme.postrm: Ensure that purge and
     reload of static assets is only done on purge/remove, resolving
     problems when upgrading to new versions (LP: #1506826).
 .
 horizon (1:2015.1.1-0ubuntu1) vivid; urgency=medium
 .
   * Resynchronize with stable/kilo (bb05237) (LP: #1481008):
     - [d29a68c] Removing Network Profile support in stable/kilo
     - [cfb16ec] Add a conditional judgement to avoid invalid dict index
     - [824df76] Fix mock issues found due to new mock release
     - [82f63a5] Fix Create User form's fields sorting for Django 1.7
     - [f0691cf] changing email from user list deletes user passwd
     - [25c961f] fix delete action always cause error
     - [ec27165] Fix date pickers in metering modal
     - [8ed6a85] Wait until the image is registered in Sahara image test
     - [fcff73c] Skip test_{floatingip,image_register_unregister,create_delete_user} for now
     - [1e84c93] Handle errors during glance image updates correctly
     - [2fc3fcd] corrected the create volume from snapshot
     - [84da479] Escape the description param from heat template
     - [bafdabb] Prevent multiple form submission on table row actions
     - [0642fdd] Fixing data processing operat...

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.