Merge lp:~gholt/swift/lobjects5 into lp:~hudson-openstack/swift/trunk

[gholt]

Change copies of x-object-manifest objects to copy the actual contents of the object, not just the manifest marker itself.

Change copies of x-object-manifest objects to copy the actual contents of the object, not just the manifest marker itself. In this way, you can consolidate the segments of a manifest object into a single whole object, assuming it meets the max-object-size limitations of the cluster.

This seems to be controversial. I'll let others chime in on their views. Here's my explanation:

A copy of a manifest could really do any of the following:
    A) just make a new manifest pointing to the same segments
    B) make a new object with the same contents as the source object (ignoring that it's a manifest)
    C) make a new manifest pointing to newly made copies of the segments

My logic for picking B is:
    A) is really easy/cheap to do client-side
    B) is expensive to do client-side and would actually be useful if done server-side
    C) is crazy

[lp:~gholt/swift/lobjects5]

150. By gholt on 2010-12-28

Made copies of ridiculously segmented objects error.

151. By gholt on 2010-12-30

Merge from trunk

152. By gholt on 2011-01-05

Merge from trunk

[John Dickinson (notmyname)]

seems good to me

review: Approve

[Chuck Thier (cthier)]

Looks good

review: Approve

=== modified file 'swift/proxy/server.py'
--- swift/proxy/server.py	2011-01-04 23:34:43 +0000
+++ swift/proxy/server.py	2011-01-05 15:19:45 +0000
@@ -950,15 +950,15 @@
             self.container_name = orig_container_name
             new_req = Request.blank(req.path_info,
                         environ=req.environ, headers=req.headers)
-            if 'x-object-manifest' in source_resp.headers:
-                data_source = iter([''])
-                new_req.content_length = 0
-                new_req.headers['X-Object-Manifest'] = \
-                    source_resp.headers['x-object-manifest']
-            else:
-                data_source = source_resp.app_iter
-                new_req.content_length = source_resp.content_length
-                new_req.etag = source_resp.etag
+            data_source = source_resp.app_iter
+            new_req.content_length = source_resp.content_length
+            if new_req.content_length is None:
+                # This indicates a transfer-encoding: chunked source object,
+                # which currently only happens because there are more than
+                # CONTAINER_LISTING_LIMIT segments in a segmented object. In
+                # this case, we're going to refuse to do the server-side copy.
+                return HTTPRequestEntityTooLarge(request=req)
+            new_req.etag = source_resp.etag
             # we no longer need the X-Copy-From header
             del new_req.headers['X-Copy-From']
             for k, v in source_resp.headers.items():

=== modified file 'test/unit/proxy/test_server.py'
--- test/unit/proxy/test_server.py	2011-01-04 23:34:43 +0000
+++ test/unit/proxy/test_server.py	2011-01-05 15:19:45 +0000
@@ -1995,6 +1995,48 @@
                 # will be sent in a single chunk.
                 self.assertEquals(body,
                     '19\r\n1234 1234 1234 1234 1234 \r\n0\r\n\r\n')
+                # Make a copy of the manifested object, which should
+                # error since the number of segments exceeds
+                # CONTAINER_LISTING_LIMIT.
+                sock = connect_tcp(('localhost', prolis.getsockname()[1]))
+                fd = sock.makefile()
+                fd.write('PUT /v1/a/segmented/copy HTTP/1.1\r\nHost: '
+                    'localhost\r\nConnection: close\r\nX-Auth-Token: '
+                    't\r\nX-Copy-From: segmented/name\r\nContent-Length: '
+                    '0\r\n\r\n')
+                fd.flush()
+                headers = readuntil2crlfs(fd)
+                exp = 'HTTP/1.1 413'
+                self.assertEquals(headers[:len(exp)], exp)
+                body = fd.read()
+                # After adjusting the CONTAINER_LISTING_LIMIT, make a copy of
+                # the manifested object which should consolidate the segments.
+                proxy_server.CONTAINER_LISTING_LIMIT = 10000
+                sock = connect_tcp(('localhost', prolis.getsockname()[1]))
+                fd = sock.makefile()
+                fd.write('PUT /v1/a/segmented/copy HTTP/1.1\r\nHost: '
+                    'localhost\r\nConnection: close\r\nX-Auth-Token: '
+                    't\r\nX-Copy-From: segmented/name\r\nContent-Length: '
+                    '0\r\n\r\n')
+                fd.flush()
+                headers = readuntil2crlfs(fd)
+                exp = 'HTTP/1.1 201'
+                self.assertEquals(headers[:len(exp)], exp)
+                body = fd.read()
+                # Retrieve and validate the copy.
+                sock = connect_tcp(('localhost', prolis.getsockname()[1]))
+                fd = sock.makefile()
+                fd.write('GET /v1/a/segmented/copy HTTP/1.1\r\nHost: '
+                    'localhost\r\nConnection: close\r\nX-Auth-Token: '
+                    't\r\n\r\n')
+                fd.flush()
+                headers = readuntil2crlfs(fd)
+                exp = 'HTTP/1.1 200'
+                self.assertEquals(headers[:len(exp)], exp)
+                self.assert_('x-object-manifest:' not in headers.lower())
+                self.assert_('Content-Length: 25\r' in headers)
+                body = fd.read()
+                self.assertEquals(body, '1234 1234 1234 1234 1234 ')
             finally:
                 prospa.kill()
                 acc1spa.kill()