Merge lp:~gaurav-gangalwar/swift/copy_manifest into lp:~hudson-openstack/swift/trunk

[Gaurav Gangalwar]

We are consolidating the segments of manifest object in COPY operation.
This will not work if the object is large object (>5GB).
It will also affect manifest objects on POST operations in "post_as_copy".

I have made the changes to honour segmentation in COPY operation also.

[gholt (gholt)]

Actually, that was sort of the point of the way the code currently works.

A COPY of a segmented object will make a new consolidated version of the object, if the total length is less than the single object cluster max (5G).

If you just want another segmented object copy, it's pretty quick and easy to PUT a new manifest file with X-Object-Manifest pointing to the same segments list.

review: Disapprove

[Gaurav Gangalwar (gaurav-gangalwar)]

Ok..but is there any specific reason for this kind of behaviour?
and what will be the behaviour in case of large objects?

On Thu, Aug 4, 2011 at 7:04 PM, gholt <email address hidden> wrote:

> Review: Disapprove
> Actually, that was sort of the point of the way the code currently works.
>
> A COPY of a segmented object will make a new consolidated version of the
> object, if the total length is less than the single object cluster max (5G).
>
> If you just want another segmented object copy, it's pretty quick and easy
> to PUT a new manifest file with X-Object-Manifest pointing to the same
> segments list.
> --
>
> https://code.launchpad.net/~gaurav-gangalwar/swift/copy_manifest/+merge/70436<https://code.launchpad.net/%7Egaurav-gangalwar/swift/copy_manifest/+merge/70436>
> You are the owner of lp:~gaurav-gangalwar/swift/copy_manifest.
>

[gholt (gholt)]

Yes, the specific reason is that there are a few things you may want to do with an existing segmented object:

1) Make a "shallow" copy where there's just a new manifest file pointing to the same segments.

2) Make a "deep" copy where the segments are all copied to new locations as well as a new manifest.

3) Make a "consolidated" copy where a bunch of segments are concatenated into one new object. The original object may have been segmented to make uploading easier (smaller chunks) or faster (parallel segment uploading) but the final object might be better to work with consolidated.

For 1) it is as simple as a 0-byte PUT with X-Object-Manifest set to the same segments path.

For 2) you have to copy each segment independently and then do the new 0-byte PUT with the X-Object-Manifest set to the new segments path.

For 3) you can do a COPY from the segmented object and the Destination will end up with a single segment object.

If you make COPY do essentially what 1) is, there is no way to do 3) server-side.

[Gaurav Gangalwar (gaurav-gangalwar)]

Ok, but in case(3) if object is copied to itself, we are keeping segments as
it is, even after consolidating the object data, this means user has to
explicitly delete those segments, right?
So user has to delete segments after POST and COPY operations.
But after POST and COPY manifest info is lost, so how user will be able to
delete those segments?

Thanks,
Gaurav

On Thu, Aug 4, 2011 at 7:41 PM, gholt <email address hidden> wrote:

> Yes, the specific reason is that there are a few things you may want to do
> with an existing segmented object:
>
> 1) Make a "shallow" copy where there's just a new manifest file pointing to
> the same segments.
>
> 2) Make a "deep" copy where the segments are all copied to new locations as
> well as a new manifest.
>
> 3) Make a "consolidated" copy where a bunch of segments are concatenated
> into one new object. The original object may have been segmented to make
> uploading easier (smaller chunks) or faster (parallel segment uploading) but
> the final object might be better to work with consolidated.
>
> For 1) it is as simple as a 0-byte PUT with X-Object-Manifest set to the
> same segments path.
>
> For 2) you have to copy each segment independently and then do the new
> 0-byte PUT with the X-Object-Manifest set to the new segments path.
>
> For 3) you can do a COPY from the segmented object and the Destination will
> end up with a single segment object.
>
> If you make COPY do essentially what 1) is, there is no way to do 3)
> server-side.
>
> --
>
> https://code.launchpad.net/~gaurav-gangalwar/swift/copy_manifest/+merge/70436<https://code.launchpad.net/%7Egaurav-gangalwar/swift/copy_manifest/+merge/70436>
> You are the owner of lp:~gaurav-gangalwar/swift/copy_manifest.
>

[gholt (gholt)]

Segments are really just regular objects, so they can be deleted however, they will show up in listings where they were uploaded whether or not a manifest points to them.

[Gaurav Gangalwar (gaurav-gangalwar)]

Ok, just thought of deleting these segments after consolidation, it will
make users life easy and also save that extra disk space.

Thanks,
Gaurav

On Thu, Aug 4, 2011 at 9:46 PM, gholt <email address hidden> wrote:

> Segments are really just regular objects, so they can be deleted however,
> they will show up in listings where they were uploaded whether or not a
> manifest points to them.
> --
>
> https://code.launchpad.net/~gaurav-gangalwar/swift/copy_manifest/+merge/70436<https://code.launchpad.net/%7Egaurav-gangalwar/swift/copy_manifest/+merge/70436>
> You are the owner of lp:~gaurav-gangalwar/swift/copy_manifest.
>

Unmerged revisions

338. By <email address hidden> on 2011-08-04

Don't consolidate segments for manifest object copy.

=== modified file 'swift/common/utils.py'
--- swift/common/utils.py	2011-07-22 19:58:22 +0000
+++ swift/common/utils.py	2011-08-04 13:24:39 +0000
@@ -1011,3 +1011,10 @@
     if index == -1:
         return '%d' % value
     return '%d%si' % (round(value), suffixes[index])
+
+
+
+def dummy_app_iter(data):
+    for index in range(len(data)):
+        yield data[index]
+

=== modified file 'swift/proxy/server.py'
--- swift/proxy/server.py	2011-07-14 18:43:08 +0000
+++ swift/proxy/server.py	2011-08-04 13:24:39 +0000
@@ -42,7 +42,8 @@
 
 from swift.common.ring import Ring
 from swift.common.utils import cache_from_env, ContextPool, get_logger, \
-    get_remote_client, normalize_timestamp, split_path, TRUE_VALUES
+    get_remote_client, normalize_timestamp, split_path, TRUE_VALUES, \
+    dummy_app_iter
 from swift.common.bufferedhttp import http_connect
 from swift.common.constraints import check_metadata, check_object_creation, \
     check_utf8, CONTAINER_LISTING_LIMIT, MAX_ACCOUNT_NAME_LENGTH, \
@@ -874,7 +875,6 @@
                 resp.content_length = content_length
                 resp.last_modified = last_modified
             resp.headers['accept-ranges'] = 'bytes'
-
         return resp
 
     @public
@@ -1044,14 +1044,22 @@
             self.container_name = orig_container_name
             new_req = Request.blank(req.path_info,
                         environ=req.environ, headers=req.headers)
-            data_source = source_resp.app_iter
-            new_req.content_length = source_resp.content_length
-            if new_req.content_length is None:
+            
+            if source_resp.content_length is None:
                 # This indicates a transfer-encoding: chunked source object,
                 # which currently only happens because there are more than
                 # CONTAINER_LISTING_LIMIT segments in a segmented object. In
                 # this case, we're going to refuse to do the server-side copy.
                 return HTTPRequestEntityTooLarge(request=req)
+            
+            if 'x-object-manifest' not in source_resp.headers:
+                data_source = source_resp.app_iter
+                new_req.content_length = source_resp.content_length
+            else:
+                new_req.headers['X-Object-Manifest'] = source_resp.headers['x-object-manifest']
+                data_source = source_resp.app_iter = dummy_app_iter('')
+                new_req.content_length = 0
+                
             new_req.etag = source_resp.etag
             # we no longer need the X-Copy-From header
             del new_req.headers['X-Copy-From']

=== modified file 'test/unit/proxy/test_server.py'
--- test/unit/proxy/test_server.py	2011-07-22 17:54:54 +0000
+++ test/unit/proxy/test_server.py	2011-08-04 13:24:39 +0000
@@ -2512,7 +2512,7 @@
         headers = readuntil2crlfs(fd)
         exp = 'HTTP/1.1 200'
         self.assertEquals(headers[:len(exp)], exp)
-        self.assert_('x-object-manifest:' not in headers.lower())
+        self.assert_('x-object-manifest:' in headers.lower())
         self.assert_('Content-Length: 25\r' in headers)
         body = fd.read()
         self.assertEquals(body, '1234 1234 1234 1234 1234 ')