Swift uploads through Glance using ridiculously small chunks

"By the looks of it, the code before this would have attempted 200 MB reads, which would have been almost as bad -- you'd get 200 MB allocated for buffering per incoming stream."

Firstly, it would be great to leave the snark at the door, Ewan...

Secondly, the original code only used 200MB reads when the image size passed to the storage backend was >5GB. So saying that every incoming stream would allocate 200MB is not true...

It seems that, as you say, the 57c4e9b6c686f1303c815698cc68b95bf9b033e7 patch removed the multiplication of the config variable by (1024 * 1024), resulting in, as you say, "ridiculously small chunks". The solution is to return the multiplication of the megabyte to the config value.

FYI, 200MB is the default segment size for large object uploads in Swift, which is why 200MB was used originally.

Fix proposed to branch: master
Review: https://review.openstack.org/2728

I got halfway through implementing the simple fix, and then realised that the entire disk buffering process was unnecessary. The branch that I've proposed removes that as well as fixing the config variable.

Reviewed: https://review.openstack.org/2728
Committed: http://github.com/openstack/glance/commit/48449171a73e9b470b3caa136f20d56cd1757bdc
Submitter: Jenkins
Branch: master

commit 48449171a73e9b470b3caa136f20d56cd1757bdc
Author: Ewan Mellor <email address hidden>
Date: Wed Dec 28 21:13:55 2011 -0800

    Bug #909533: Swift uploads through Glance using ridiculously small chunks

    Remove the use of a temporary disk buffer when streaming to Swift. This
    was added on the assumption that it was not possible to stream chunks to
    Swift direct from webob.Request.body_file. That's not true -- a simple
    file-like object wrapping body_file does the job perfectly.

    This removes the need for swift_store_object_buffer_dir in the config file.
    It is also significantly cheaper, since there is one copy fewer.

    Fix the parsing of the swift_store_large_object_size and
    swift_store_large_object_chunk_size options. These are
    specified in MB in the config file, but needs to be in bytes internally,
    because they are compared against the image_size parameter given to add().

    Update the unit tests to match. This includes an additional check that
    put_object is called the correct number of times. The unit tests missed the
    latter problem problem because there was no check that a small object is only
    uploaded in one chunk, and despite the comment to the contrary there was no
    check that the right number of chunks were written in the large-object case
    either. Added these in all places where store.add is called.

    Change-Id: Ieb4cf68516b53bd16d2671e49e805b26118b3671