Merge lp:~gholt/swift/lobjects4 into lp:~hudson-openstack/swift/trunk

Proposed by gholt on 2010-11-16
Status: Superseded
Proposed branch: lp:~gholt/swift/lobjects4
Merge into: lp:~hudson-openstack/swift/trunk
Diff against target: 2282 lines (+1404/-78)
11 files modified
bin/st (+180/-39)
doc/source/index.rst (+1/-0)
doc/source/overview_large_objects.rst (+177/-0)
swift/common/client.py (+17/-10)
swift/common/constraints.py (+11/-0)
swift/obj/server.py (+7/-2)
swift/proxy/server.py (+265/-3)
test/functionalnosetests/test_object.py (+279/-0)
test/unit/common/test_constraints.py (+27/-0)
test/unit/obj/test_server.py (+82/-22)
test/unit/proxy/test_server.py (+358/-2)
To merge this branch: bzr merge lp:~gholt/swift/lobjects4
Reviewer Review Type Date Requested Status
John Dickinson Approve on 2010-12-13
Greg Lange Approve on 2010-12-13
gholt (community) Approve on 2010-12-08
clayg Approve on 2010-12-06
Review via email: mp+41020@code.launchpad.net

This proposal has been superseded by a proposal from 2010-12-14.

Commit Message

Large object support by allowing the client to upload the object in segments and download them all as a single object.

Description of the Change

Added large object support by allowing the client to upload the object in segments and download them all as a single object. Also, made updates client.py and st to support and provide an example of how to use the feature. Finally, there is an overview document that needs reviewing.

To post a comment you must log in.
lp:~gholt/swift/lobjects4 updated on 2010-11-29
134. By gholt on 2010-11-24

Merged from trunk

135. By gholt on 2010-11-29

Added history section for large object support docs

136. By gholt on 2010-11-29

Fixed bug dfg found in st

clayg (clay-gerrard) wrote :

great work gholt.

In SegmentedIterable, _load_next_segment, __iter__ and app_iter_range all log the same message ("ERROR: While processing manifest") - if the exception happens in _load_next_segment it gets logged twice.

Eitherway, the posthooklogger will log the status of the response as successful, even though the client likely blew up cause it didn't get all the expected data.

Just for the purpose of logging, since SegmentedIterable has a backref to the Response - I would suggest we update the status_int to a 503. Even though we can't do anything about the 200 we already sent to the client, we know that whatever bytes were transferred so far won't be billable.

Just a thought - personal preference in control structure - I don't care for the except Exception: if not isisntance(StopIteration): log, raise - I would prefer except StopIteration: pass except Exception: log, raise.

Also in SegmentedIterable, instead of initializing self.response to None, how about an empty Response(). It *should* be overridden (assuming the caller read the docstring) - but initializing to "something" removes the need for an extra test in all the "if self.response: update attribute on response" code. Alternatively you could make response a required named parameter to __init__ and then just require the calling method update it's response's app_iter instead of its new SegmentedIterable's backref.

resp = Response()
resp.app_iter = SegIter(resp)

^ does that work?

The history section mentions the "implied user manifest" that we have now with the consistency issue where the proxy can't ever know if it has all of the users intended objects, but not the rejected "explicit user manifest" when the body of the manifest file includes all of the objects in the order the should be glued together - like amazon uses. Is it worth mentioning why our solution is better?

Thanks for working on this essential feature, you deserve much karama!

gholt (gholt) wrote :

Cool, will apply your stuff tomorrow. The except Exception: if not isinstance was pure lame on my part. The other stuff you already know I like. :) I dunno on the history doc thing; wanna type up what you'd like to see? Amazon's thing is really quite different since with them you still can't exceed the maximum 5G limit they have, glued or not.

gholt (gholt) wrote :

Eh, I was bored so I just pushed the changes described.

lp:~gholt/swift/lobjects4 updated on 2010-12-03
137. By gholt on 2010-11-30

Merged from trunk

138. By gholt on 2010-12-03

SegmentIterable: logs exceptions just once; 503s on exception; fix except syntax; make sure self.response is always *something*

clayg (clay-gerrard) wrote :

I think this is ready!

I put up an ether on the large objects history section if anyone wants to suggest any changes: http://etherpad.openstack.org/SwiftLargeObjectSupport

review: Approve
lp:~gholt/swift/lobjects4 updated on 2010-12-06
139. By gholt on 2010-12-06

Merge from trunk

140. By gholt on 2010-12-06

Merge from lp:~clay-gerrard/swift/lobjects_history

gholt (gholt) wrote :

I'm not sure how I ended up a reviewer on my own branch; but I approve!

review: Approve
lp:~gholt/swift/lobjects4 updated on 2010-12-10
141. By gholt on 2010-12-10

Merged from trunk

Greg Lange (greglange) wrote :

I like the change. My only gripe would be that an explicit handling of the 10,000 segment limit would be better, either allow unlimited segments or error when more than 10,000 segments are uploaded as appropriate. This is being addressed in a bug report.

review: Approve
John Dickinson (notmyname) wrote :

looks great

review: Approve
lp:~gholt/swift/lobjects4 updated on 2010-12-16
142. By gholt on 2010-12-13

Now supports infinite objects!

143. By gholt on 2010-12-14

Fixed a bug where a HEAD on a really, really large object would give a content-length of 0 instead of transfer-encoding: chunked

144. By gholt on 2010-12-14

x-copy-from now understands manifest sources and copies details rather than contents

145. By gholt on 2010-12-14

Limit manifest gets to one segment per second; prevents amplification attacks of tons of tiny segments

146. By gholt on 2010-12-14

Changed to only limit manifest gets after first 10 segments. Makes tests run faster but does allow amplification 1:10. At least it's not 1:infinity like before.

147. By gholt on 2010-12-14

Even though isn't 100% related, made st emit a warning if there's a / in a container name

148. By gholt on 2010-12-16

st: Works with chunked transfer encoded downloads now

149. By gholt on 2010-12-16

Made stat display of objects suppress content-length, last-modified, and etag if they aren't in the headers

150. By gholt on 2010-12-16

lobjects: The Last-Modified header is now determined for reasonably segmented objects.

Unmerged revisions

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1=== modified file 'bin/st'
2--- bin/st 2010-12-09 17:10:37 +0000
3+++ bin/st 2010-12-13 22:17:44 +0000
4@@ -581,7 +581,8 @@
5 :param container: container name that the object is in
6 :param name: object name to put
7 :param contents: a string or a file like object to read object data from
8- :param content_length: value to send as content-length header
9+ :param content_length: value to send as content-length header; also limits
10+ the amount read from contents
11 :param etag: etag of contents
12 :param chunk_size: chunk size of data to write
13 :param content_type: value to send as content-type header
14@@ -611,18 +612,24 @@
15 conn.putrequest('PUT', path)
16 for header, value in headers.iteritems():
17 conn.putheader(header, value)
18- if not content_length:
19+ if content_length is None:
20 conn.putheader('Transfer-Encoding', 'chunked')
21- conn.endheaders()
22- chunk = contents.read(chunk_size)
23- while chunk:
24- if not content_length:
25+ conn.endheaders()
26+ chunk = contents.read(chunk_size)
27+ while chunk:
28 conn.send('%x\r\n%s\r\n' % (len(chunk), chunk))
29- else:
30+ chunk = contents.read(chunk_size)
31+ conn.send('0\r\n\r\n')
32+ else:
33+ conn.endheaders()
34+ left = content_length
35+ while left > 0:
36+ size = chunk_size
37+ if size > left:
38+ size = left
39+ chunk = contents.read(size)
40 conn.send(chunk)
41- chunk = contents.read(chunk_size)
42- if not content_length:
43- conn.send('0\r\n\r\n')
44+ left -= len(chunk)
45 else:
46 conn.request('PUT', path, contents, headers)
47 resp = conn.getresponse()
48@@ -860,15 +867,20 @@
49
50
51 st_delete_help = '''
52-delete --all OR delete container [object] [object] ...
53+delete --all OR delete container [--leave-segments] [object] [object] ...
54 Deletes everything in the account (with --all), or everything in a
55- container, or a list of objects depending on the args given.'''.strip('\n')
56+ container, or a list of objects depending on the args given. Segments of
57+ manifest objects will be deleted as well, unless you specify the
58+ --leave-segments option.'''.strip('\n')
59
60
61 def st_delete(parser, args, print_queue, error_queue):
62 parser.add_option('-a', '--all', action='store_true', dest='yes_all',
63 default=False, help='Indicates that you really want to delete '
64 'everything in the account')
65+ parser.add_option('', '--leave-segments', action='store_true',
66+ dest='leave_segments', default=False, help='Indicates that you want '
67+ 'the segments of manifest objects left alone')
68 (options, args) = parse_args(parser, args)
69 args = args[1:]
70 if (not args and not options.yes_all) or (args and options.yes_all):
71@@ -876,11 +888,42 @@
72 (basename(argv[0]), st_delete_help))
73 return
74
75+ def _delete_segment((container, obj), conn):
76+ conn.delete_object(container, obj)
77+ if options.verbose:
78+ print_queue.put('%s/%s' % (container, obj))
79+
80 object_queue = Queue(10000)
81
82 def _delete_object((container, obj), conn):
83 try:
84+ old_manifest = None
85+ if not options.leave_segments:
86+ try:
87+ old_manifest = conn.head_object(container, obj).get(
88+ 'x-object-manifest')
89+ except ClientException, err:
90+ if err.http_status != 404:
91+ raise
92 conn.delete_object(container, obj)
93+ if old_manifest:
94+ segment_queue = Queue(10000)
95+ scontainer, sprefix = old_manifest.split('/', 1)
96+ for delobj in conn.get_container(scontainer,
97+ prefix=sprefix)[1]:
98+ segment_queue.put((scontainer, delobj['name']))
99+ if not segment_queue.empty():
100+ segment_threads = [QueueFunctionThread(segment_queue,
101+ _delete_segment, create_connection()) for _ in
102+ xrange(10)]
103+ for thread in segment_threads:
104+ thread.start()
105+ while not segment_queue.empty():
106+ sleep(0.01)
107+ for thread in segment_threads:
108+ thread.abort = True
109+ while thread.isAlive():
110+ thread.join(0.01)
111 if options.verbose:
112 path = options.yes_all and join(container, obj) or obj
113 if path[:1] in ('/', '\\'):
114@@ -891,6 +934,7 @@
115 raise
116 error_queue.put('Object %s not found' %
117 repr('%s/%s' % (container, obj)))
118+
119 container_queue = Queue(10000)
120
121 def _delete_container(container, conn):
122@@ -976,7 +1020,7 @@
123
124
125 st_download_help = '''
126-download --all OR download container [object] [object] ...
127+download --all OR download container [options] [object] [object] ...
128 Downloads everything in the account (with --all), or everything in a
129 container, or a list of objects depending on the args given. For a single
130 object download, you may use the -o [--output] <filename> option to
131@@ -1020,15 +1064,18 @@
132 path = options.yes_all and join(container, obj) or obj
133 if path[:1] in ('/', '\\'):
134 path = path[1:]
135+ md5sum = None
136 make_dir = out_file != "-"
137 if content_type.split(';', 1)[0] == 'text/directory':
138 if make_dir and not isdir(path):
139 mkdirs(path)
140 read_length = 0
141- md5sum = md5()
142+ if 'x-object-manifest' not in headers:
143+ md5sum = md5()
144 for chunk in body:
145 read_length += len(chunk)
146- md5sum.update(chunk)
147+ if md5sum:
148+ md5sum.update(chunk)
149 else:
150 dirpath = dirname(path)
151 if make_dir and dirpath and not isdir(dirpath):
152@@ -1040,13 +1087,15 @@
153 else:
154 fp = open(path, 'wb')
155 read_length = 0
156- md5sum = md5()
157+ if 'x-object-manifest' not in headers:
158+ md5sum = md5()
159 for chunk in body:
160 fp.write(chunk)
161 read_length += len(chunk)
162- md5sum.update(chunk)
163+ if md5sum:
164+ md5sum.update(chunk)
165 fp.close()
166- if md5sum.hexdigest() != etag:
167+ if md5sum and md5sum.hexdigest() != etag:
168 error_queue.put('%s: md5sum != etag, %s != %s' %
169 (path, md5sum.hexdigest(), etag))
170 if read_length != content_length:
171@@ -1267,6 +1316,9 @@
172 headers.get('content-length'),
173 headers.get('last-modified'),
174 headers.get('etag')))
175+ if 'x-object-manifest' in headers:
176+ print_queue.put(' Manifest: %s' %
177+ headers['x-object-manifest'])
178 for key, value in headers.items():
179 if key.startswith('x-object-meta-'):
180 print_queue.put('%14s: %s' % ('Meta %s' %
181@@ -1274,7 +1326,7 @@
182 for key, value in headers.items():
183 if not key.startswith('x-object-meta-') and key not in (
184 'content-type', 'content-length', 'last-modified',
185- 'etag', 'date'):
186+ 'etag', 'date', 'x-object-manifest'):
187 print_queue.put(
188 '%14s: %s' % (key.title(), value))
189 except ClientException, err:
190@@ -1363,23 +1415,48 @@
191 upload [options] container file_or_directory [file_or_directory] [...]
192 Uploads to the given container the files and directories specified by the
193 remaining args. -c or --changed is an option that will only upload files
194- that have changed since the last upload.'''.strip('\n')
195+ that have changed since the last upload. -S <size> or --segment-size <size>
196+ and --leave-segments are options as well (see --help for more).
197+'''.strip('\n')
198
199
200 def st_upload(options, args, print_queue, error_queue):
201 parser.add_option('-c', '--changed', action='store_true', dest='changed',
202 default=False, help='Will only upload files that have changed since '
203 'the last upload')
204+ parser.add_option('-S', '--segment-size', dest='segment_size', help='Will '
205+ 'upload files in segments no larger than <size> and then create a '
206+ '"manifest" file that will download all the segments as if it were '
207+ 'the original file. The segments will be uploaded to a '
208+ '<container>_segments container so as to not pollute the main '
209+ '<container> listings.')
210+ parser.add_option('', '--leave-segments', action='store_true',
211+ dest='leave_segments', default=False, help='Indicates that you want '
212+ 'the older segments of manifest objects left alone (in the case of '
213+ 'overwrites)')
214 (options, args) = parse_args(parser, args)
215 args = args[1:]
216 if len(args) < 2:
217 error_queue.put('Usage: %s [options] %s' %
218 (basename(argv[0]), st_upload_help))
219 return
220-
221- file_queue = Queue(10000)
222-
223- def _upload_file((path, dir_marker), conn):
224+ object_queue = Queue(10000)
225+
226+ def _segment_job(job, conn):
227+ if job.get('delete', False):
228+ conn.delete_object(job['container'], job['obj'])
229+ else:
230+ fp = open(job['path'], 'rb')
231+ fp.seek(job['segment_start'])
232+ conn.put_object(job.get('container', args[0] + '_segments'),
233+ job['obj'], fp, content_length=job['segment_size'])
234+ if options.verbose and 'log_line' in job:
235+ print_queue.put(job['log_line'])
236+
237+ def _object_job(job, conn):
238+ path = job['path']
239+ container = job.get('container', args[0])
240+ dir_marker = job.get('dir_marker', False)
241 try:
242 obj = path
243 if obj.startswith('./') or obj.startswith('.\\'):
244@@ -1388,7 +1465,7 @@
245 if dir_marker:
246 if options.changed:
247 try:
248- headers = conn.head_object(args[0], obj)
249+ headers = conn.head_object(container, obj)
250 ct = headers.get('content-type')
251 cl = int(headers.get('content-length'))
252 et = headers.get('etag')
253@@ -1401,24 +1478,86 @@
254 except ClientException, err:
255 if err.http_status != 404:
256 raise
257- conn.put_object(args[0], obj, '', content_length=0,
258+ conn.put_object(container, obj, '', content_length=0,
259 content_type='text/directory',
260 headers=put_headers)
261 else:
262- if options.changed:
263+ # We need to HEAD all objects now in case we're overwriting a
264+ # manifest object and need to delete the old segments
265+ # ourselves.
266+ old_manifest = None
267+ if options.changed or not options.leave_segments:
268 try:
269- headers = conn.head_object(args[0], obj)
270+ headers = conn.head_object(container, obj)
271 cl = int(headers.get('content-length'))
272 mt = headers.get('x-object-meta-mtime')
273- if cl == getsize(path) and \
274+ if options.changed and cl == getsize(path) and \
275 mt == put_headers['x-object-meta-mtime']:
276 return
277+ if not options.leave_segments:
278+ old_manifest = headers.get('x-object-manifest')
279 except ClientException, err:
280 if err.http_status != 404:
281 raise
282- conn.put_object(args[0], obj, open(path, 'rb'),
283- content_length=getsize(path),
284- headers=put_headers)
285+ if options.segment_size and \
286+ getsize(path) < options.segment_size:
287+ full_size = getsize(path)
288+ segment_queue = Queue(10000)
289+ segment_threads = [QueueFunctionThread(segment_queue,
290+ _segment_job, create_connection()) for _ in xrange(10)]
291+ for thread in segment_threads:
292+ thread.start()
293+ segment = 0
294+ segment_start = 0
295+ while segment_start < full_size:
296+ segment_size = int(options.segment_size)
297+ if segment_start + segment_size > full_size:
298+ segment_size = full_size - segment_start
299+ segment_queue.put({'path': path,
300+ 'obj': '%s/%s/%s/%08d' % (obj,
301+ put_headers['x-object-meta-mtime'], full_size,
302+ segment),
303+ 'segment_start': segment_start,
304+ 'segment_size': segment_size,
305+ 'log_line': '%s segment %s' % (obj, segment)})
306+ segment += 1
307+ segment_start += segment_size
308+ while not segment_queue.empty():
309+ sleep(0.01)
310+ for thread in segment_threads:
311+ thread.abort = True
312+ while thread.isAlive():
313+ thread.join(0.01)
314+ new_object_manifest = '%s_segments/%s/%s/%s/' % (
315+ container, obj, put_headers['x-object-meta-mtime'],
316+ full_size)
317+ if old_manifest == new_object_manifest:
318+ old_manifest = None
319+ put_headers['x-object-manifest'] = new_object_manifest
320+ conn.put_object(container, obj, '', content_length=0,
321+ headers=put_headers)
322+ else:
323+ conn.put_object(container, obj, open(path, 'rb'),
324+ content_length=getsize(path), headers=put_headers)
325+ if old_manifest:
326+ segment_queue = Queue(10000)
327+ scontainer, sprefix = old_manifest.split('/', 1)
328+ for delobj in conn.get_container(scontainer,
329+ prefix=sprefix)[1]:
330+ segment_queue.put({'delete': True,
331+ 'container': scontainer, 'obj': delobj['name']})
332+ if not segment_queue.empty():
333+ segment_threads = [QueueFunctionThread(segment_queue,
334+ _segment_job, create_connection()) for _ in
335+ xrange(10)]
336+ for thread in segment_threads:
337+ thread.start()
338+ while not segment_queue.empty():
339+ sleep(0.01)
340+ for thread in segment_threads:
341+ thread.abort = True
342+ while thread.isAlive():
343+ thread.join(0.01)
344 if options.verbose:
345 print_queue.put(obj)
346 except OSError, err:
347@@ -1429,22 +1568,22 @@
348 def _upload_dir(path):
349 names = listdir(path)
350 if not names:
351- file_queue.put((path, True)) # dir_marker = True
352+ object_queue.put({'path': path, 'dir_marker': True})
353 else:
354 for name in listdir(path):
355 subpath = join(path, name)
356 if isdir(subpath):
357 _upload_dir(subpath)
358 else:
359- file_queue.put((subpath, False)) # dir_marker = False
360+ object_queue.put({'path': subpath})
361
362 url, token = get_auth(options.auth, options.user, options.key,
363 snet=options.snet)
364 create_connection = lambda: Connection(options.auth, options.user,
365 options.key, preauthurl=url, preauthtoken=token, snet=options.snet)
366- file_threads = [QueueFunctionThread(file_queue, _upload_file,
367+ object_threads = [QueueFunctionThread(object_queue, _object_job,
368 create_connection()) for _ in xrange(10)]
369- for thread in file_threads:
370+ for thread in object_threads:
371 thread.start()
372 conn = create_connection()
373 # Try to create the container, just in case it doesn't exist. If this
374@@ -1453,6 +1592,8 @@
375 # it'll surface on the first object PUT.
376 try:
377 conn.put_container(args[0])
378+ if options.segment_size is not None:
379+ conn.put_container(args[0] + '_segments')
380 except:
381 pass
382 try:
383@@ -1460,10 +1601,10 @@
384 if isdir(arg):
385 _upload_dir(arg)
386 else:
387- file_queue.put((arg, False)) # dir_marker = False
388- while not file_queue.empty():
389+ object_queue.put({'path': arg})
390+ while not object_queue.empty():
391 sleep(0.01)
392- for thread in file_threads:
393+ for thread in object_threads:
394 thread.abort = True
395 while thread.isAlive():
396 thread.join(0.01)
397
398=== modified file 'doc/source/index.rst'
399--- doc/source/index.rst 2010-11-12 18:54:07 +0000
400+++ doc/source/index.rst 2010-12-13 22:17:44 +0000
401@@ -44,6 +44,7 @@
402 overview_replication
403 overview_stats
404 ratelimit
405+ overview_large_objects
406
407 Developer Documentation
408 =======================
409
410=== added file 'doc/source/overview_large_objects.rst'
411--- doc/source/overview_large_objects.rst 1970-01-01 00:00:00 +0000
412+++ doc/source/overview_large_objects.rst 2010-12-13 22:17:44 +0000
413@@ -0,0 +1,177 @@
414+====================
415+Large Object Support
416+====================
417+
418+--------
419+Overview
420+--------
421+
422+Swift has a limit on the size of a single uploaded object; by default this is
423+5GB. However, the download size of a single object is virtually unlimited with
424+the concept of segmentation. Segments of the larger object are uploaded and a
425+special manifest file is created that, when downloaded, sends all the segments
426+concatenated as a single object. This also offers much greater upload speed
427+with the possibility of parallel uploads of the segments.
428+
429+----------------------------------
430+Using ``st`` for Segmented Objects
431+----------------------------------
432+
433+The quickest way to try out this feature is use the included ``st`` Swift Tool.
434+You can use the ``-S`` option to specify the segment size to use when splitting
435+a large file. For example::
436+
437+ st upload test_container -S 1073741824 large_file
438+
439+This would split the large_file into 1G segments and begin uploading those
440+segments in parallel. Once all the segments have been uploaded, ``st`` will
441+then create the manifest file so the segments can be downloaded as one.
442+
443+So now, the following ``st`` command would download the entire large object::
444+
445+ st download test_container large_file
446+
447+``st`` uses a strict convention for its segmented object support. In the above
448+example it will upload all the segments into a second container named
449+test_container_segments. These segments will have names like
450+large_file/1290206778.25/21474836480/00000000,
451+large_file/1290206778.25/21474836480/00000001, etc.
452+
453+The main benefit for using a separate container is that the main container
454+listings will not be polluted with all the segment names. The reason for using
455+the segment name format of <name>/<timestamp>/<size>/<segment> is so that an
456+upload of a new file with the same name won't overwrite the contents of the
457+first until the last moment when the manifest file is updated.
458+
459+``st`` will manage these segment files for you, deleting old segments on
460+deletes and overwrites, etc. You can override this behavior with the
461+``--leave-segments`` option if desired; this is useful if you want to have
462+multiple versions of the same large object available.
463+
464+----------
465+Direct API
466+----------
467+
468+You can also work with the segments and manifests directly with HTTP requests
469+instead of having ``st`` do that for you. You can just upload the segments like
470+you would any other object and the manifest is just a zero-byte file with an
471+extra ``X-Object-Manifest`` header.
472+
473+All the object segments need to be in the same container, have a common object
474+name prefix, and their names sort in the order they should be concatenated.
475+They don't have to be in the same container as the manifest file will be, which
476+is useful to keep container listings clean as explained above with ``st``.
477+
478+The manifest file is simply a zero-byte file with the extra
479+``X-Object-Manifest: <container>/<prefix>`` header, where ``<container>`` is
480+the container the object segments are in and ``<prefix>`` is the common prefix
481+for all the segments.
482+
483+It is best to upload all the segments first and then create or update the
484+manifest. In this way, the full object won't be available for downloading until
485+the upload is complete. Also, you can upload a new set of segments to a second
486+location and then update the manifest to point to this new location. During the
487+upload of the new segments, the original manifest will still be available to
488+download the first set of segments.
489+
490+Here's an example using ``curl`` with tiny 1-byte segments::
491+
492+ # First, upload the segments
493+ curl -X PUT -H 'X-Auth-Token: <token>' \
494+ http://<storage_url>/container/myobject/1 --data-binary '1'
495+ curl -X PUT -H 'X-Auth-Token: <token>' \
496+ http://<storage_url>/container/myobject/2 --data-binary '2'
497+ curl -X PUT -H 'X-Auth-Token: <token>' \
498+ http://<storage_url>/container/myobject/3 --data-binary '3'
499+
500+ # Next, create the manifest file
501+ curl -X PUT -H 'X-Auth-Token: <token>' \
502+ -H 'X-Object-Manifest: container/myobject/' \
503+ http://<storage_url>/container/myobject --data-binary ''
504+
505+ # And now we can download the segments as a single object
506+ curl -H 'X-Auth-Token: <token>' \
507+ http://<storage_url>/container/myobject
508+
509+----------------
510+Additional Notes
511+----------------
512+
513+* With a ``GET`` or ``HEAD`` of a manifest file, the ``X-Object-Manifest:
514+ <container>/<prefix>`` header will be returned with the concatenated object
515+ so you can tell where it's getting its segments from.
516+
517+* The response's ``Content-Length`` for a ``GET`` or ``HEAD`` on the manifest
518+ file will be the sum of all the segments in the ``<container>/<prefix>``
519+ listing, dynamically. So, uploading additional segments after the manifest is
520+ created will cause the concatenated object to be that much larger; there's no
521+ need to recreate the manifest file.
522+
523+* The response's ``Content-Type`` for a ``GET`` or ``HEAD`` on the manifest
524+ will be the same as the ``Content-Type`` set during the ``PUT`` request that
525+ created the manifest. You can easily change the ``Content-Type`` by reissuing
526+ the ``PUT``.
527+
528+* The response's ``ETag`` for a ``GET`` or ``HEAD`` on the manifest file will
529+ be the MD5 sum of the concatenated string of ETags for each of the segments
530+ in the ``<container>/<prefix>`` listing, dynamically. Usually in Swift the
531+ ETag is the MD5 sum of the contents of the object, and that holds true for
532+ each segment independently. But, it's not feasible to generate such an ETag
533+ for the manifest itself, so this method was chosen to at least offer change
534+ detection.
535+
536+-------
537+History
538+-------
539+
540+Large object support has gone through various iterations before settling on
541+this implementation.
542+
543+The primary factor driving the limitation of object size in swift is
544+maintaining balance among the partitions of the ring. To maintain an even
545+dispersion of disk usage throughout the cluster the obvious storage pattern
546+was to simply split larger objects into smaller segments, which could then be
547+glued together during a read.
548+
549+Before the introduction of large object support some applications were already
550+splitting their uploads into segments and re-assembling them on the client
551+side after retrieving the individual pieces. This design allowed the client
552+to support backup and archiving of large data sets, but was also frequently
553+employed to improve performance or reduce errors due to network interruption.
554+The major disadvantage of this method is that knowledge of the original
555+partitioning scheme is required to properly reassemble the object, which is
556+not practical for some use cases, such as CDN origination.
557+
558+In order to eliminate any barrier to entry for clients wanting to store
559+objects larger than 5GB, initially we also prototyped fully transparent
560+support for large object uploads. A fully transparent implementation would
561+support a larger max size by automatically splitting objects into segments
562+during upload within the proxy without any changes to the client API. All
563+segments were completely hidden from the client API.
564+
565+This solution introduced a number of challenging failure conditions into the
566+cluster, wouldn't provide the client with any option to do parallel uploads,
567+and had no basis for a resume feature. The transparent implementation was
568+deemed just too complex for the benefit.
569+
570+The current "user manifest" design was chosen in order to provide a
571+transparent download of large objects to the client and still provide the
572+uploading client a clean API to support segmented uploads.
573+
574+Alternative "explicit" user manifest options were discussed which would have
575+required a pre-defined format for listing the segments to "finalize" the
576+segmented upload. While this may offer some potential advantages, it was
577+decided that pushing an added burden onto the client which could potentially
578+limit adoption should be avoided in favor of a simpler "API" (essentially just
579+the format of the 'X-Object-Manifest' header).
580+
581+During development it was noted that this "implicit" user manifest approach
582+which is based on the path prefix can be potentially affected by the eventual
583+consistency window of the container listings, which could theoretically cause
584+a GET on the manifest object to return an invalid whole object for that short
585+term. In reality you're unlikely to encounter this scenario unless you're
586+running very high concurrency uploads against a small testing environment
587+which isn't running the object-updaters or container-replicators.
588+
589+Like all of swift, Large Object Support is living feature which will continue
590+to improve and may change over time.
591
592=== modified file 'swift/common/client.py'
593--- swift/common/client.py 2010-11-18 21:40:40 +0000
594+++ swift/common/client.py 2010-12-13 22:17:44 +0000
595@@ -569,7 +569,8 @@
596 :param container: container name that the object is in
597 :param name: object name to put
598 :param contents: a string or a file like object to read object data from
599- :param content_length: value to send as content-length header
600+ :param content_length: value to send as content-length header; also limits
601+ the amount read from contents
602 :param etag: etag of contents
603 :param chunk_size: chunk size of data to write
604 :param content_type: value to send as content-type header
605@@ -599,18 +600,24 @@
606 conn.putrequest('PUT', path)
607 for header, value in headers.iteritems():
608 conn.putheader(header, value)
609- if not content_length:
610+ if content_length is None:
611 conn.putheader('Transfer-Encoding', 'chunked')
612- conn.endheaders()
613- chunk = contents.read(chunk_size)
614- while chunk:
615- if not content_length:
616+ conn.endheaders()
617+ chunk = contents.read(chunk_size)
618+ while chunk:
619 conn.send('%x\r\n%s\r\n' % (len(chunk), chunk))
620- else:
621+ chunk = contents.read(chunk_size)
622+ conn.send('0\r\n\r\n')
623+ else:
624+ conn.endheaders()
625+ left = content_length
626+ while left > 0:
627+ size = chunk_size
628+ if size > left:
629+ size = left
630+ chunk = contents.read(size)
631 conn.send(chunk)
632- chunk = contents.read(chunk_size)
633- if not content_length:
634- conn.send('0\r\n\r\n')
635+ left -= len(chunk)
636 else:
637 conn.request('PUT', path, contents, headers)
638 resp = conn.getresponse()
639
640=== modified file 'swift/common/constraints.py'
641--- swift/common/constraints.py 2010-10-26 15:13:14 +0000
642+++ swift/common/constraints.py 2010-12-13 22:17:44 +0000
643@@ -113,6 +113,17 @@
644 if not check_utf8(req.headers['Content-Type']):
645 return HTTPBadRequest(request=req, body='Invalid Content-Type',
646 content_type='text/plain')
647+ if 'x-object-manifest' in req.headers:
648+ value = req.headers['x-object-manifest']
649+ container = prefix = None
650+ try:
651+ container, prefix = value.split('/', 1)
652+ except ValueError:
653+ pass
654+ if not container or not prefix or '?' in value or '&' in value or \
655+ prefix[0] == '/':
656+ return HTTPBadRequest(request=req,
657+ body='X-Object-Manifest must in the format container/prefix')
658 return check_metadata(req, 'object')
659
660
661
662=== modified file 'swift/obj/server.py'
663--- swift/obj/server.py 2010-11-01 21:47:48 +0000
664+++ swift/obj/server.py 2010-12-13 22:17:44 +0000
665@@ -391,6 +391,9 @@
666 'ETag': etag,
667 'Content-Length': str(os.fstat(fd).st_size),
668 }
669+ if 'x-object-manifest' in request.headers:
670+ metadata['X-Object-Manifest'] = \
671+ request.headers['x-object-manifest']
672 metadata.update(val for val in request.headers.iteritems()
673 if val[0].lower().startswith('x-object-meta-') and
674 len(val[0]) > 14)
675@@ -460,7 +463,8 @@
676 'application/octet-stream'), app_iter=file,
677 request=request, conditional_response=True)
678 for key, value in file.metadata.iteritems():
679- if key.lower().startswith('x-object-meta-'):
680+ if key == 'X-Object-Manifest' or \
681+ key.lower().startswith('x-object-meta-'):
682 response.headers[key] = value
683 response.etag = file.metadata['ETag']
684 response.last_modified = float(file.metadata['X-Timestamp'])
685@@ -488,7 +492,8 @@
686 response = Response(content_type=file.metadata['Content-Type'],
687 request=request, conditional_response=True)
688 for key, value in file.metadata.iteritems():
689- if key.lower().startswith('x-object-meta-'):
690+ if key == 'X-Object-Manifest' or \
691+ key.lower().startswith('x-object-meta-'):
692 response.headers[key] = value
693 response.etag = file.metadata['ETag']
694 response.last_modified = float(file.metadata['X-Timestamp'])
695
696=== modified file 'swift/proxy/server.py'
697--- swift/proxy/server.py 2010-12-13 18:25:09 +0000
698+++ swift/proxy/server.py 2010-12-13 22:17:44 +0000
699@@ -14,6 +14,10 @@
700 # limitations under the License.
701
702 from __future__ import with_statement
703+try:
704+ import simplejson as json
705+except ImportError:
706+ import json
707 import mimetypes
708 import os
709 import time
710@@ -22,6 +26,7 @@
711 from urllib import unquote, quote
712 import uuid
713 import functools
714+from hashlib import md5
715
716 from eventlet.timeout import Timeout
717 from webob.exc import HTTPBadRequest, HTTPMethodNotAllowed, \
718@@ -36,8 +41,8 @@
719 cache_from_env
720 from swift.common.bufferedhttp import http_connect
721 from swift.common.constraints import check_metadata, check_object_creation, \
722- check_utf8, MAX_ACCOUNT_NAME_LENGTH, MAX_CONTAINER_NAME_LENGTH, \
723- MAX_FILE_SIZE
724+ check_utf8, CONTAINER_LISTING_LIMIT, MAX_ACCOUNT_NAME_LENGTH, \
725+ MAX_CONTAINER_NAME_LENGTH, MAX_FILE_SIZE
726 from swift.common.exceptions import ChunkReadTimeout, \
727 ChunkWriteTimeout, ConnectionTimeout
728
729@@ -95,6 +100,154 @@
730 return 'container/%s/%s' % (account, container)
731
732
733+class SegmentedIterable(object):
734+ """
735+ Iterable that returns the object contents for a segmented object in Swift.
736+
737+ If set, the response's `bytes_transferred` value will be updated (used to
738+ log the size of the request). Also, if there's a failure that cuts the
739+ transfer short, the response's `status_int` will be updated (again, just
740+ for logging since the original status would have already been sent to the
741+ client).
742+
743+ :param controller: The ObjectController instance to work with.
744+ :param container: The container the object segments are within.
745+ :param listing: The listing of object segments to iterate over; this may
746+ be an iterator or list that returns dicts with 'name' and
747+ 'bytes' keys.
748+ :param response: The webob.Response this iterable is associated with, if
749+ any (default: None)
750+ """
751+
752+ def __init__(self, controller, container, listing, response=None):
753+ self.controller = controller
754+ self.container = container
755+ self.listing = iter(listing)
756+ self.segment = -1
757+ self.segment_dict = None
758+ self.segment_peek = None
759+ self.seek = 0
760+ self.segment_iter = None
761+ self.position = 0
762+ self.response = response
763+ if not self.response:
764+ self.response = Response()
765+
766+ def _load_next_segment(self):
767+ """
768+ Loads the self.segment_iter with the next object segment's contents.
769+
770+ :raises: StopIteration when there are no more object segments.
771+ """
772+ try:
773+ self.segment += 1
774+ self.segment_dict = self.segment_peek or self.listing.next()
775+ self.segment_peek = None
776+ partition, nodes = self.controller.app.object_ring.get_nodes(
777+ self.controller.account_name, self.container,
778+ self.segment_dict['name'])
779+ path = '/%s/%s/%s' % (self.controller.account_name, self.container,
780+ self.segment_dict['name'])
781+ req = Request.blank(path)
782+ if self.seek:
783+ req.range = 'bytes=%s-' % self.seek
784+ self.seek = 0
785+ resp = self.controller.GETorHEAD_base(req, 'Object', partition,
786+ self.controller.iter_nodes(partition, nodes,
787+ self.controller.app.object_ring), path,
788+ self.controller.app.object_ring.replica_count)
789+ if resp.status_int // 100 != 2:
790+ raise Exception('Could not load object segment %s: %s' % (path,
791+ resp.status_int))
792+ self.segment_iter = resp.app_iter
793+ except StopIteration:
794+ raise
795+ except Exception, err:
796+ if not getattr(err, 'swift_logged', False):
797+ self.controller.app.logger.exception('ERROR: While processing '
798+ 'manifest /%s/%s/%s %s' % (self.controller.account_name,
799+ self.controller.container_name,
800+ self.controller.object_name, self.controller.trans_id))
801+ err.swift_logged = True
802+ self.response.status_int = 503
803+ raise
804+
805+ def __iter__(self):
806+ """ Standard iterator function that returns the object's contents. """
807+ try:
808+ while True:
809+ if not self.segment_iter:
810+ self._load_next_segment()
811+ while True:
812+ with ChunkReadTimeout(self.controller.app.node_timeout):
813+ try:
814+ chunk = self.segment_iter.next()
815+ break
816+ except StopIteration:
817+ self._load_next_segment()
818+ self.position += len(chunk)
819+ self.response.bytes_transferred = getattr(self.response,
820+ 'bytes_transferred', 0) + len(chunk)
821+ yield chunk
822+ except StopIteration:
823+ raise
824+ except Exception, err:
825+ if not getattr(err, 'swift_logged', False):
826+ self.controller.app.logger.exception('ERROR: While processing '
827+ 'manifest /%s/%s/%s %s' % (self.controller.account_name,
828+ self.controller.container_name,
829+ self.controller.object_name, self.controller.trans_id))
830+ err.swift_logged = True
831+ self.response.status_int = 503
832+ raise
833+
834+ def app_iter_range(self, start, stop):
835+ """
836+ Non-standard iterator function for use with Webob in serving Range
837+ requests more quickly. This will skip over segments and do a range
838+ request on the first segment to return data from, if needed.
839+
840+ :param start: The first byte (zero-based) to return. None for 0.
841+ :param stop: The last byte (zero-based) to return. None for end.
842+ """
843+ try:
844+ if start:
845+ self.segment_peek = self.listing.next()
846+ while start >= self.position + self.segment_peek['bytes']:
847+ self.segment += 1
848+ self.position += self.segment_peek['bytes']
849+ self.segment_peek = self.listing.next()
850+ self.seek = start - self.position
851+ else:
852+ start = 0
853+ if stop is not None:
854+ length = stop - start
855+ else:
856+ length = None
857+ for chunk in self:
858+ if length is not None:
859+ length -= len(chunk)
860+ if length < 0:
861+ # Chop off the extra:
862+ self.response.bytes_transferred = \
863+ getattr(self.response, 'bytes_transferred', 0) \
864+ + length
865+ yield chunk[:length]
866+ break
867+ yield chunk
868+ except StopIteration:
869+ raise
870+ except Exception, err:
871+ if not getattr(err, 'swift_logged', False):
872+ self.controller.app.logger.exception('ERROR: While processing '
873+ 'manifest /%s/%s/%s %s' % (self.controller.account_name,
874+ self.controller.container_name,
875+ self.controller.object_name, self.controller.trans_id))
876+ err.swift_logged = True
877+ self.response.status_int = 503
878+ raise
879+
880+
881 class Controller(object):
882 """Base WSGI controller class for the proxy"""
883
884@@ -538,9 +691,118 @@
885 return aresp
886 partition, nodes = self.app.object_ring.get_nodes(
887 self.account_name, self.container_name, self.object_name)
888- return self.GETorHEAD_base(req, 'Object', partition,
889+ resp = self.GETorHEAD_base(req, 'Object', partition,
890 self.iter_nodes(partition, nodes, self.app.object_ring),
891 req.path_info, self.app.object_ring.replica_count)
892+ # If we get a 416 Requested Range Not Satisfiable we have to check if
893+ # we were actually requesting a manifest object and then redo the range
894+ # request on the whole object.
895+ if resp.status_int == 416:
896+ req_range = req.range
897+ req.range = None
898+ resp2 = self.GETorHEAD_base(req, 'Object', partition,
899+ self.iter_nodes(partition, nodes, self.app.object_ring),
900+ req.path_info, self.app.object_ring.replica_count)
901+ if 'x-object-manifest' not in resp2.headers:
902+ return resp
903+ resp = resp2
904+ req.range = req_range
905+
906+ if 'x-object-manifest' in resp.headers:
907+ lcontainer, lprefix = \
908+ resp.headers['x-object-manifest'].split('/', 1)
909+ lpartition, lnodes = self.app.container_ring.get_nodes(
910+ self.account_name, lcontainer)
911+ marker = ''
912+ listing = []
913+ while True:
914+ lreq = Request.blank('/%s/%s?prefix=%s&format=json&marker=%s' %
915+ (quote(self.account_name), quote(lcontainer),
916+ quote(lprefix), quote(marker)))
917+ lresp = self.GETorHEAD_base(lreq, 'Container', lpartition,
918+ lnodes, lreq.path_info,
919+ self.app.container_ring.replica_count)
920+ if lresp.status_int // 100 != 2:
921+ lresp = HTTPNotFound(request=req)
922+ lresp.headers['X-Object-Manifest'] = \
923+ resp.headers['x-object-manifest']
924+ return lresp
925+ if 'swift.authorize' in req.environ:
926+ req.acl = lresp.headers.get('x-container-read')
927+ aresp = req.environ['swift.authorize'](req)
928+ if aresp:
929+ return aresp
930+ sublisting = json.loads(lresp.body)
931+ if not sublisting:
932+ break
933+ listing.extend(sublisting)
934+ if len(listing) > CONTAINER_LISTING_LIMIT:
935+ break
936+ marker = sublisting[-1]['name']
937+
938+ if len(listing) > CONTAINER_LISTING_LIMIT:
939+ # We will serve large objects with a ton of segments with
940+ # chunked transfer encoding.
941+
942+ def listing_iter():
943+ marker = ''
944+ while True:
945+ lreq = Request.blank(
946+ '/%s/%s?prefix=%s&format=json&marker=%s' %
947+ (quote(self.account_name), quote(lcontainer),
948+ quote(lprefix), quote(marker)))
949+ lresp = self.GETorHEAD_base(lreq, 'Container',
950+ lpartition, lnodes, lreq.path_info,
951+ self.app.container_ring.replica_count)
952+ if lresp.status_int // 100 != 2:
953+ raise Exception('Object manifest GET could not '
954+ 'continue listing: %s %s' %
955+ (req.path, lreq.path))
956+ if 'swift.authorize' in req.environ:
957+ req.acl = lresp.headers.get('x-container-read')
958+ aresp = req.environ['swift.authorize'](req)
959+ if aresp:
960+ raise Exception('Object manifest GET could '
961+ 'not continue listing: %s %s' %
962+ (req.path, aresp))
963+ sublisting = json.loads(lresp.body)
964+ if not sublisting:
965+ break
966+ for obj in sublisting:
967+ yield obj
968+ marker = sublisting[-1]['name']
969+
970+ headers = {
971+ 'X-Object-Manifest': resp.headers['x-object-manifest'],
972+ 'Content-Type': resp.content_type}
973+ for key, value in resp.headers.iteritems():
974+ if key.lower().startswith('x-object-meta-'):
975+ headers[key] = value
976+ resp = Response(headers=headers, request=req,
977+ conditional_response=True)
978+ resp.app_iter = SegmentedIterable(self, lcontainer,
979+ listing_iter(), resp)
980+
981+ else:
982+ # For objects with a reasonable number of segments, we'll serve
983+ # them with a set content-length and computed etag.
984+ content_length = sum(o['bytes'] for o in listing)
985+ etag = md5('"'.join(o['hash'] for o in listing)).hexdigest()
986+ headers = {
987+ 'X-Object-Manifest': resp.headers['x-object-manifest'],
988+ 'Content-Type': resp.content_type,
989+ 'Content-Length': content_length,
990+ 'ETag': etag}
991+ for key, value in resp.headers.iteritems():
992+ if key.lower().startswith('x-object-meta-'):
993+ headers[key] = value
994+ resp = Response(headers=headers, request=req,
995+ conditional_response=True)
996+ resp.app_iter = SegmentedIterable(self, lcontainer, listing,
997+ resp)
998+ resp.content_length = content_length
999+
1000+ return resp
1001
1002 @public
1003 @delay_denial
1004
1005=== modified file 'test/functionalnosetests/test_object.py'
1006--- test/functionalnosetests/test_object.py 2010-11-04 19:39:29 +0000
1007+++ test/functionalnosetests/test_object.py 2010-12-13 22:17:44 +0000
1008@@ -16,6 +16,7 @@
1009 if skip:
1010 raise SkipTest
1011 self.container = uuid4().hex
1012+
1013 def put(url, token, parsed, conn):
1014 conn.request('PUT', parsed.path + '/' + self.container, '',
1015 {'X-Auth-Token': token})
1016@@ -24,6 +25,7 @@
1017 resp.read()
1018 self.assertEquals(resp.status, 201)
1019 self.obj = uuid4().hex
1020+
1021 def put(url, token, parsed, conn):
1022 conn.request('PUT', '%s/%s/%s' % (parsed.path, self.container,
1023 self.obj), 'test', {'X-Auth-Token': token})
1024@@ -35,6 +37,7 @@
1025 def tearDown(self):
1026 if skip:
1027 raise SkipTest
1028+
1029 def delete(url, token, parsed, conn):
1030 conn.request('DELETE', '%s/%s/%s' % (parsed.path, self.container,
1031 self.obj), '', {'X-Auth-Token': token})
1032@@ -42,6 +45,7 @@
1033 resp = retry(delete)
1034 resp.read()
1035 self.assertEquals(resp.status, 204)
1036+
1037 def delete(url, token, parsed, conn):
1038 conn.request('DELETE', parsed.path + '/' + self.container, '',
1039 {'X-Auth-Token': token})
1040@@ -53,6 +57,7 @@
1041 def test_public_object(self):
1042 if skip:
1043 raise SkipTest
1044+
1045 def get(url, token, parsed, conn):
1046 conn.request('GET',
1047 '%s/%s/%s' % (parsed.path, self.container, self.obj))
1048@@ -62,6 +67,7 @@
1049 raise Exception('Should not have been able to GET')
1050 except Exception, err:
1051 self.assert_(str(err).startswith('No result after '))
1052+
1053 def post(url, token, parsed, conn):
1054 conn.request('POST', parsed.path + '/' + self.container, '',
1055 {'X-Auth-Token': token,
1056@@ -73,6 +79,7 @@
1057 resp = retry(get)
1058 resp.read()
1059 self.assertEquals(resp.status, 200)
1060+
1061 def post(url, token, parsed, conn):
1062 conn.request('POST', parsed.path + '/' + self.container, '',
1063 {'X-Auth-Token': token, 'X-Container-Read': ''})
1064@@ -89,6 +96,7 @@
1065 def test_private_object(self):
1066 if skip or skip3:
1067 raise SkipTest
1068+
1069 # Ensure we can't access the object with the third account
1070 def get(url, token, parsed, conn):
1071 conn.request('GET', '%s/%s/%s' % (parsed.path, self.container,
1072@@ -98,8 +106,10 @@
1073 resp = retry(get, use_account=3)
1074 resp.read()
1075 self.assertEquals(resp.status, 403)
1076+
1077 # create a shared container writable by account3
1078 shared_container = uuid4().hex
1079+
1080 def put(url, token, parsed, conn):
1081 conn.request('PUT', '%s/%s' % (parsed.path,
1082 shared_container), '',
1083@@ -110,6 +120,7 @@
1084 resp = retry(put)
1085 resp.read()
1086 self.assertEquals(resp.status, 201)
1087+
1088 # verify third account can not copy from private container
1089 def copy(url, token, parsed, conn):
1090 conn.request('PUT', '%s/%s/%s' % (parsed.path,
1091@@ -123,6 +134,7 @@
1092 resp = retry(copy, use_account=3)
1093 resp.read()
1094 self.assertEquals(resp.status, 403)
1095+
1096 # verify third account can write "obj1" to shared container
1097 def put(url, token, parsed, conn):
1098 conn.request('PUT', '%s/%s/%s' % (parsed.path, shared_container,
1099@@ -131,6 +143,7 @@
1100 resp = retry(put, use_account=3)
1101 resp.read()
1102 self.assertEquals(resp.status, 201)
1103+
1104 # verify third account can copy "obj1" to shared container
1105 def copy2(url, token, parsed, conn):
1106 conn.request('COPY', '%s/%s/%s' % (parsed.path,
1107@@ -143,6 +156,7 @@
1108 resp = retry(copy2, use_account=3)
1109 resp.read()
1110 self.assertEquals(resp.status, 201)
1111+
1112 # verify third account STILL can not copy from private container
1113 def copy3(url, token, parsed, conn):
1114 conn.request('COPY', '%s/%s/%s' % (parsed.path,
1115@@ -155,6 +169,7 @@
1116 resp = retry(copy3, use_account=3)
1117 resp.read()
1118 self.assertEquals(resp.status, 403)
1119+
1120 # clean up "obj1"
1121 def delete(url, token, parsed, conn):
1122 conn.request('DELETE', '%s/%s/%s' % (parsed.path, shared_container,
1123@@ -163,6 +178,7 @@
1124 resp = retry(delete)
1125 resp.read()
1126 self.assertEquals(resp.status, 204)
1127+
1128 # clean up shared_container
1129 def delete(url, token, parsed, conn):
1130 conn.request('DELETE',
1131@@ -173,6 +189,269 @@
1132 resp.read()
1133 self.assertEquals(resp.status, 204)
1134
1135+ def test_manifest(self):
1136+ if skip:
1137+ raise SkipTest
1138+ # Data for the object segments
1139+ segments1 = ['one', 'two', 'three', 'four', 'five']
1140+ segments2 = ['six', 'seven', 'eight']
1141+ segments3 = ['nine', 'ten', 'eleven']
1142+
1143+ # Upload the first set of segments
1144+ def put(url, token, parsed, conn, objnum):
1145+ conn.request('PUT', '%s/%s/segments1/%s' % (parsed.path,
1146+ self.container, str(objnum)), segments1[objnum],
1147+ {'X-Auth-Token': token})
1148+ return check_response(conn)
1149+ for objnum in xrange(len(segments1)):
1150+ resp = retry(put, objnum)
1151+ resp.read()
1152+ self.assertEquals(resp.status, 201)
1153+
1154+ # Upload the manifest
1155+ def put(url, token, parsed, conn):
1156+ conn.request('PUT', '%s/%s/manifest' % (parsed.path,
1157+ self.container), '', {'X-Auth-Token': token,
1158+ 'X-Object-Manifest': '%s/segments1/' % self.container,
1159+ 'Content-Type': 'text/jibberish', 'Content-Length': '0'})
1160+ return check_response(conn)
1161+ resp = retry(put)
1162+ resp.read()
1163+ self.assertEquals(resp.status, 201)
1164+
1165+ # Get the manifest (should get all the segments as the body)
1166+ def get(url, token, parsed, conn):
1167+ conn.request('GET', '%s/%s/manifest' % (parsed.path,
1168+ self.container), '', {'X-Auth-Token': token})
1169+ return check_response(conn)
1170+ resp = retry(get)
1171+ self.assertEquals(resp.read(), ''.join(segments1))
1172+ self.assertEquals(resp.status, 200)
1173+ self.assertEquals(resp.getheader('content-type'), 'text/jibberish')
1174+
1175+ # Get with a range at the start of the second segment
1176+ def get(url, token, parsed, conn):
1177+ conn.request('GET', '%s/%s/manifest' % (parsed.path,
1178+ self.container), '', {'X-Auth-Token': token, 'Range':
1179+ 'bytes=3-'})
1180+ return check_response(conn)
1181+ resp = retry(get)
1182+ self.assertEquals(resp.read(), ''.join(segments1[1:]))
1183+ self.assertEquals(resp.status, 206)
1184+
1185+ # Get with a range in the middle of the second segment
1186+ def get(url, token, parsed, conn):
1187+ conn.request('GET', '%s/%s/manifest' % (parsed.path,
1188+ self.container), '', {'X-Auth-Token': token, 'Range':
1189+ 'bytes=5-'})
1190+ return check_response(conn)
1191+ resp = retry(get)
1192+ self.assertEquals(resp.read(), ''.join(segments1)[5:])
1193+ self.assertEquals(resp.status, 206)
1194+
1195+ # Get with a full start and stop range
1196+ def get(url, token, parsed, conn):
1197+ conn.request('GET', '%s/%s/manifest' % (parsed.path,
1198+ self.container), '', {'X-Auth-Token': token, 'Range':
1199+ 'bytes=5-10'})
1200+ return check_response(conn)
1201+ resp = retry(get)
1202+ self.assertEquals(resp.read(), ''.join(segments1)[5:11])
1203+ self.assertEquals(resp.status, 206)
1204+
1205+ # Upload the second set of segments
1206+ def put(url, token, parsed, conn, objnum):
1207+ conn.request('PUT', '%s/%s/segments2/%s' % (parsed.path,
1208+ self.container, str(objnum)), segments2[objnum],
1209+ {'X-Auth-Token': token})
1210+ return check_response(conn)
1211+ for objnum in xrange(len(segments2)):
1212+ resp = retry(put, objnum)
1213+ resp.read()
1214+ self.assertEquals(resp.status, 201)
1215+
1216+ # Get the manifest (should still be the first segments of course)
1217+ def get(url, token, parsed, conn):
1218+ conn.request('GET', '%s/%s/manifest' % (parsed.path,
1219+ self.container), '', {'X-Auth-Token': token})
1220+ return check_response(conn)
1221+ resp = retry(get)
1222+ self.assertEquals(resp.read(), ''.join(segments1))
1223+ self.assertEquals(resp.status, 200)
1224+
1225+ # Update the manifest
1226+ def put(url, token, parsed, conn):
1227+ conn.request('PUT', '%s/%s/manifest' % (parsed.path,
1228+ self.container), '', {'X-Auth-Token': token,
1229+ 'X-Object-Manifest': '%s/segments2/' % self.container,
1230+ 'Content-Length': '0'})
1231+ return check_response(conn)
1232+ resp = retry(put)
1233+ resp.read()
1234+ self.assertEquals(resp.status, 201)
1235+
1236+ # Get the manifest (should be the second set of segments now)
1237+ def get(url, token, parsed, conn):
1238+ conn.request('GET', '%s/%s/manifest' % (parsed.path,
1239+ self.container), '', {'X-Auth-Token': token})
1240+ return check_response(conn)
1241+ resp = retry(get)
1242+ self.assertEquals(resp.read(), ''.join(segments2))
1243+ self.assertEquals(resp.status, 200)
1244+
1245+ if not skip3:
1246+
1247+ # Ensure we can't access the manifest with the third account
1248+ def get(url, token, parsed, conn):
1249+ conn.request('GET', '%s/%s/manifest' % (parsed.path,
1250+ self.container), '', {'X-Auth-Token': token})
1251+ return check_response(conn)
1252+ resp = retry(get, use_account=3)
1253+ resp.read()
1254+ self.assertEquals(resp.status, 403)
1255+
1256+ # Grant access to the third account
1257+ def post(url, token, parsed, conn):
1258+ conn.request('POST', '%s/%s' % (parsed.path, self.container),
1259+ '', {'X-Auth-Token': token, 'X-Container-Read':
1260+ swift_test_user[2]})
1261+ return check_response(conn)
1262+ resp = retry(post)
1263+ resp.read()
1264+ self.assertEquals(resp.status, 204)
1265+
1266+ # The third account should be able to get the manifest now
1267+ def get(url, token, parsed, conn):
1268+ conn.request('GET', '%s/%s/manifest' % (parsed.path,
1269+ self.container), '', {'X-Auth-Token': token})
1270+ return check_response(conn)
1271+ resp = retry(get, use_account=3)
1272+ self.assertEquals(resp.read(), ''.join(segments2))
1273+ self.assertEquals(resp.status, 200)
1274+
1275+ # Create another container for the third set of segments
1276+ acontainer = uuid4().hex
1277+
1278+ def put(url, token, parsed, conn):
1279+ conn.request('PUT', parsed.path + '/' + acontainer, '',
1280+ {'X-Auth-Token': token})
1281+ return check_response(conn)
1282+ resp = retry(put)
1283+ resp.read()
1284+ self.assertEquals(resp.status, 201)
1285+
1286+ # Upload the third set of segments in the other container
1287+ def put(url, token, parsed, conn, objnum):
1288+ conn.request('PUT', '%s/%s/segments3/%s' % (parsed.path,
1289+ acontainer, str(objnum)), segments3[objnum],
1290+ {'X-Auth-Token': token})
1291+ return check_response(conn)
1292+ for objnum in xrange(len(segments3)):
1293+ resp = retry(put, objnum)
1294+ resp.read()
1295+ self.assertEquals(resp.status, 201)
1296+
1297+ # Update the manifest
1298+ def put(url, token, parsed, conn):
1299+ conn.request('PUT', '%s/%s/manifest' % (parsed.path,
1300+ self.container), '', {'X-Auth-Token': token,
1301+ 'X-Object-Manifest': '%s/segments3/' % acontainer,
1302+ 'Content-Length': '0'})
1303+ return check_response(conn)
1304+ resp = retry(put)
1305+ resp.read()
1306+ self.assertEquals(resp.status, 201)
1307+
1308+ # Get the manifest to ensure it's the third set of segments
1309+ def get(url, token, parsed, conn):
1310+ conn.request('GET', '%s/%s/manifest' % (parsed.path,
1311+ self.container), '', {'X-Auth-Token': token})
1312+ return check_response(conn)
1313+ resp = retry(get)
1314+ self.assertEquals(resp.read(), ''.join(segments3))
1315+ self.assertEquals(resp.status, 200)
1316+
1317+ if not skip3:
1318+
1319+ # Ensure we can't access the manifest with the third account
1320+ # (because the segments are in a protected container even if the
1321+ # manifest itself is not).
1322+
1323+ def get(url, token, parsed, conn):
1324+ conn.request('GET', '%s/%s/manifest' % (parsed.path,
1325+ self.container), '', {'X-Auth-Token': token})
1326+ return check_response(conn)
1327+ resp = retry(get, use_account=3)
1328+ resp.read()
1329+ self.assertEquals(resp.status, 403)
1330+
1331+ # Grant access to the third account
1332+ def post(url, token, parsed, conn):
1333+ conn.request('POST', '%s/%s' % (parsed.path, acontainer),
1334+ '', {'X-Auth-Token': token, 'X-Container-Read':
1335+ swift_test_user[2]})
1336+ return check_response(conn)
1337+ resp = retry(post)
1338+ resp.read()
1339+ self.assertEquals(resp.status, 204)
1340+
1341+ # The third account should be able to get the manifest now
1342+ def get(url, token, parsed, conn):
1343+ conn.request('GET', '%s/%s/manifest' % (parsed.path,
1344+ self.container), '', {'X-Auth-Token': token})
1345+ return check_response(conn)
1346+ resp = retry(get, use_account=3)
1347+ self.assertEquals(resp.read(), ''.join(segments3))
1348+ self.assertEquals(resp.status, 200)
1349+
1350+ # Delete the manifest
1351+ def delete(url, token, parsed, conn, objnum):
1352+ conn.request('DELETE', '%s/%s/manifest' % (parsed.path,
1353+ self.container), '', {'X-Auth-Token': token})
1354+ return check_response(conn)
1355+ resp = retry(delete, objnum)
1356+ resp.read()
1357+ self.assertEquals(resp.status, 204)
1358+
1359+ # Delete the third set of segments
1360+ def delete(url, token, parsed, conn, objnum):
1361+ conn.request('DELETE', '%s/%s/segments3/%s' % (parsed.path,
1362+ acontainer, str(objnum)), '', {'X-Auth-Token': token})
1363+ return check_response(conn)
1364+ for objnum in xrange(len(segments3)):
1365+ resp = retry(delete, objnum)
1366+ resp.read()
1367+ self.assertEquals(resp.status, 204)
1368+
1369+ # Delete the second set of segments
1370+ def delete(url, token, parsed, conn, objnum):
1371+ conn.request('DELETE', '%s/%s/segments2/%s' % (parsed.path,
1372+ self.container, str(objnum)), '', {'X-Auth-Token': token})
1373+ return check_response(conn)
1374+ for objnum in xrange(len(segments2)):
1375+ resp = retry(delete, objnum)
1376+ resp.read()
1377+ self.assertEquals(resp.status, 204)
1378+
1379+ # Delete the first set of segments
1380+ def delete(url, token, parsed, conn, objnum):
1381+ conn.request('DELETE', '%s/%s/segments1/%s' % (parsed.path,
1382+ self.container, str(objnum)), '', {'X-Auth-Token': token})
1383+ return check_response(conn)
1384+ for objnum in xrange(len(segments1)):
1385+ resp = retry(delete, objnum)
1386+ resp.read()
1387+ self.assertEquals(resp.status, 204)
1388+
1389+ # Delete the extra container
1390+ def delete(url, token, parsed, conn):
1391+ conn.request('DELETE', '%s/%s' % (parsed.path, acontainer), '',
1392+ {'X-Auth-Token': token})
1393+ return check_response(conn)
1394+ resp = retry(delete)
1395+ resp.read()
1396+ self.assertEquals(resp.status, 204)
1397+
1398
1399 if __name__ == '__main__':
1400 unittest.main()
1401
1402=== modified file 'test/unit/common/test_constraints.py'
1403--- test/unit/common/test_constraints.py 2010-08-16 22:30:27 +0000
1404+++ test/unit/common/test_constraints.py 2010-12-13 22:17:44 +0000
1405@@ -22,6 +22,7 @@
1406
1407 from swift.common import constraints
1408
1409+
1410 class TestConstraints(unittest.TestCase):
1411
1412 def test_check_metadata_empty(self):
1413@@ -137,6 +138,32 @@
1414 self.assert_(isinstance(resp, HTTPBadRequest))
1415 self.assert_('Content-Type' in resp.body)
1416
1417+ def test_check_object_manifest_header(self):
1418+ resp = constraints.check_object_creation(Request.blank('/',
1419+ headers={'X-Object-Manifest': 'container/prefix', 'Content-Length':
1420+ '0', 'Content-Type': 'text/plain'}), 'manifest')
1421+ self.assert_(not resp)
1422+ resp = constraints.check_object_creation(Request.blank('/',
1423+ headers={'X-Object-Manifest': 'container', 'Content-Length': '0',
1424+ 'Content-Type': 'text/plain'}), 'manifest')
1425+ self.assert_(isinstance(resp, HTTPBadRequest))
1426+ resp = constraints.check_object_creation(Request.blank('/',
1427+ headers={'X-Object-Manifest': '/container/prefix',
1428+ 'Content-Length': '0', 'Content-Type': 'text/plain'}), 'manifest')
1429+ self.assert_(isinstance(resp, HTTPBadRequest))
1430+ resp = constraints.check_object_creation(Request.blank('/',
1431+ headers={'X-Object-Manifest': 'container/prefix?query=param',
1432+ 'Content-Length': '0', 'Content-Type': 'text/plain'}), 'manifest')
1433+ self.assert_(isinstance(resp, HTTPBadRequest))
1434+ resp = constraints.check_object_creation(Request.blank('/',
1435+ headers={'X-Object-Manifest': 'container/prefix&query=param',
1436+ 'Content-Length': '0', 'Content-Type': 'text/plain'}), 'manifest')
1437+ self.assert_(isinstance(resp, HTTPBadRequest))
1438+ resp = constraints.check_object_creation(Request.blank('/',
1439+ headers={'X-Object-Manifest': 'http://host/container/prefix',
1440+ 'Content-Length': '0', 'Content-Type': 'text/plain'}), 'manifest')
1441+ self.assert_(isinstance(resp, HTTPBadRequest))
1442+
1443 def test_check_mount(self):
1444 self.assertFalse(constraints.check_mount('', ''))
1445 constraints.os = MockTrue() # mock os module
1446
1447=== modified file 'test/unit/obj/test_server.py'
1448--- test/unit/obj/test_server.py 2010-10-13 21:26:43 +0000
1449+++ test/unit/obj/test_server.py 2010-12-13 22:17:44 +0000
1450@@ -42,7 +42,7 @@
1451 self.path_to_test_xfs = os.environ.get('PATH_TO_TEST_XFS')
1452 if not self.path_to_test_xfs or \
1453 not os.path.exists(self.path_to_test_xfs):
1454- print >>sys.stderr, 'WARNING: PATH_TO_TEST_XFS not set or not ' \
1455+ print >> sys.stderr, 'WARNING: PATH_TO_TEST_XFS not set or not ' \
1456 'pointing to a valid directory.\n' \
1457 'Please set PATH_TO_TEST_XFS to a directory on an XFS file ' \
1458 'system for testing.'
1459@@ -77,7 +77,8 @@
1460 self.assertEquals(resp.status_int, 201)
1461
1462 timestamp = normalize_timestamp(time())
1463- req = Request.blank('/sda1/p/a/c/o', environ={'REQUEST_METHOD': 'POST'},
1464+ req = Request.blank('/sda1/p/a/c/o',
1465+ environ={'REQUEST_METHOD': 'POST'},
1466 headers={'X-Timestamp': timestamp,
1467 'X-Object-Meta-3': 'Three',
1468 'X-Object-Meta-4': 'Four',
1469@@ -95,7 +96,8 @@
1470 if not self.path_to_test_xfs:
1471 raise SkipTest
1472 timestamp = normalize_timestamp(time())
1473- req = Request.blank('/sda1/p/a/c/fail', environ={'REQUEST_METHOD': 'POST'},
1474+ req = Request.blank('/sda1/p/a/c/fail',
1475+ environ={'REQUEST_METHOD': 'POST'},
1476 headers={'X-Timestamp': timestamp,
1477 'X-Object-Meta-1': 'One',
1478 'X-Object-Meta-2': 'Two',
1479@@ -116,29 +118,37 @@
1480 def test_POST_container_connection(self):
1481 if not self.path_to_test_xfs:
1482 raise SkipTest
1483+
1484 def mock_http_connect(response, with_exc=False):
1485+
1486 class FakeConn(object):
1487+
1488 def __init__(self, status, with_exc):
1489 self.status = status
1490 self.reason = 'Fake'
1491 self.host = '1.2.3.4'
1492 self.port = '1234'
1493 self.with_exc = with_exc
1494+
1495 def getresponse(self):
1496 if self.with_exc:
1497 raise Exception('test')
1498 return self
1499+
1500 def read(self, amt=None):
1501 return ''
1502+
1503 return lambda *args, **kwargs: FakeConn(response, with_exc)
1504+
1505 old_http_connect = object_server.http_connect
1506 try:
1507 timestamp = normalize_timestamp(time())
1508- req = Request.blank('/sda1/p/a/c/o', environ={'REQUEST_METHOD': 'POST'},
1509- headers={'X-Timestamp': timestamp, 'Content-Type': 'text/plain',
1510- 'Content-Length': '0'})
1511+ req = Request.blank('/sda1/p/a/c/o', environ={'REQUEST_METHOD':
1512+ 'POST'}, headers={'X-Timestamp': timestamp, 'Content-Type':
1513+ 'text/plain', 'Content-Length': '0'})
1514 resp = self.object_controller.PUT(req)
1515- req = Request.blank('/sda1/p/a/c/o', environ={'REQUEST_METHOD': 'POST'},
1516+ req = Request.blank('/sda1/p/a/c/o',
1517+ environ={'REQUEST_METHOD': 'POST'},
1518 headers={'X-Timestamp': timestamp,
1519 'X-Container-Host': '1.2.3.4:0',
1520 'X-Container-Partition': '3',
1521@@ -148,7 +158,8 @@
1522 object_server.http_connect = mock_http_connect(202)
1523 resp = self.object_controller.POST(req)
1524 self.assertEquals(resp.status_int, 202)
1525- req = Request.blank('/sda1/p/a/c/o', environ={'REQUEST_METHOD': 'POST'},
1526+ req = Request.blank('/sda1/p/a/c/o',
1527+ environ={'REQUEST_METHOD': 'POST'},
1528 headers={'X-Timestamp': timestamp,
1529 'X-Container-Host': '1.2.3.4:0',
1530 'X-Container-Partition': '3',
1531@@ -158,7 +169,8 @@
1532 object_server.http_connect = mock_http_connect(202, with_exc=True)
1533 resp = self.object_controller.POST(req)
1534 self.assertEquals(resp.status_int, 202)
1535- req = Request.blank('/sda1/p/a/c/o', environ={'REQUEST_METHOD': 'POST'},
1536+ req = Request.blank('/sda1/p/a/c/o',
1537+ environ={'REQUEST_METHOD': 'POST'},
1538 headers={'X-Timestamp': timestamp,
1539 'X-Container-Host': '1.2.3.4:0',
1540 'X-Container-Partition': '3',
1541@@ -226,7 +238,8 @@
1542 timestamp + '.data')
1543 self.assert_(os.path.isfile(objfile))
1544 self.assertEquals(open(objfile).read(), 'VERIFY')
1545- self.assertEquals(pickle.loads(getxattr(objfile, object_server.METADATA_KEY)),
1546+ self.assertEquals(pickle.loads(getxattr(objfile,
1547+ object_server.METADATA_KEY)),
1548 {'X-Timestamp': timestamp,
1549 'Content-Length': '6',
1550 'ETag': '0b4c12d7e0a73840c1c4f148fda3b037',
1551@@ -258,7 +271,8 @@
1552 timestamp + '.data')
1553 self.assert_(os.path.isfile(objfile))
1554 self.assertEquals(open(objfile).read(), 'VERIFY TWO')
1555- self.assertEquals(pickle.loads(getxattr(objfile, object_server.METADATA_KEY)),
1556+ self.assertEquals(pickle.loads(getxattr(objfile,
1557+ object_server.METADATA_KEY)),
1558 {'X-Timestamp': timestamp,
1559 'Content-Length': '10',
1560 'ETag': 'b381a4c5dab1eaa1eb9711fa647cd039',
1561@@ -270,17 +284,17 @@
1562 if not self.path_to_test_xfs:
1563 raise SkipTest
1564 req = Request.blank('/sda1/p/a/c/o', environ={'REQUEST_METHOD': 'PUT'},
1565- headers={'X-Timestamp': normalize_timestamp(time()),
1566- 'Content-Type': 'text/plain'})
1567+ headers={'X-Timestamp': normalize_timestamp(time()),
1568+ 'Content-Type': 'text/plain'})
1569 req.body = 'test'
1570 resp = self.object_controller.PUT(req)
1571 self.assertEquals(resp.status_int, 201)
1572
1573 def test_PUT_invalid_etag(self):
1574 req = Request.blank('/sda1/p/a/c/o', environ={'REQUEST_METHOD': 'PUT'},
1575- headers={'X-Timestamp': normalize_timestamp(time()),
1576- 'Content-Type': 'text/plain',
1577- 'ETag': 'invalid'})
1578+ headers={'X-Timestamp': normalize_timestamp(time()),
1579+ 'Content-Type': 'text/plain',
1580+ 'ETag': 'invalid'})
1581 req.body = 'test'
1582 resp = self.object_controller.PUT(req)
1583 self.assertEquals(resp.status_int, 422)
1584@@ -304,7 +318,8 @@
1585 timestamp + '.data')
1586 self.assert_(os.path.isfile(objfile))
1587 self.assertEquals(open(objfile).read(), 'VERIFY THREE')
1588- self.assertEquals(pickle.loads(getxattr(objfile, object_server.METADATA_KEY)),
1589+ self.assertEquals(pickle.loads(getxattr(objfile,
1590+ object_server.METADATA_KEY)),
1591 {'X-Timestamp': timestamp,
1592 'Content-Length': '12',
1593 'ETag': 'b114ab7b90d9ccac4bd5d99cc7ebb568',
1594@@ -316,25 +331,33 @@
1595 def test_PUT_container_connection(self):
1596 if not self.path_to_test_xfs:
1597 raise SkipTest
1598+
1599 def mock_http_connect(response, with_exc=False):
1600+
1601 class FakeConn(object):
1602+
1603 def __init__(self, status, with_exc):
1604 self.status = status
1605 self.reason = 'Fake'
1606 self.host = '1.2.3.4'
1607 self.port = '1234'
1608 self.with_exc = with_exc
1609+
1610 def getresponse(self):
1611 if self.with_exc:
1612 raise Exception('test')
1613 return self
1614+
1615 def read(self, amt=None):
1616 return ''
1617+
1618 return lambda *args, **kwargs: FakeConn(response, with_exc)
1619+
1620 old_http_connect = object_server.http_connect
1621 try:
1622 timestamp = normalize_timestamp(time())
1623- req = Request.blank('/sda1/p/a/c/o', environ={'REQUEST_METHOD': 'POST'},
1624+ req = Request.blank('/sda1/p/a/c/o',
1625+ environ={'REQUEST_METHOD': 'POST'},
1626 headers={'X-Timestamp': timestamp,
1627 'X-Container-Host': '1.2.3.4:0',
1628 'X-Container-Partition': '3',
1629@@ -555,7 +578,8 @@
1630 self.assertEquals(resp.status_int, 200)
1631 self.assertEquals(resp.etag, etag)
1632
1633- req = Request.blank('/sda1/p/a/c/o2', environ={'REQUEST_METHOD': 'GET'},
1634+ req = Request.blank('/sda1/p/a/c/o2',
1635+ environ={'REQUEST_METHOD': 'GET'},
1636 headers={'If-Match': '*'})
1637 resp = self.object_controller.GET(req)
1638 self.assertEquals(resp.status_int, 412)
1639@@ -715,7 +739,8 @@
1640 """ Test swift.object_server.ObjectController.DELETE """
1641 if not self.path_to_test_xfs:
1642 raise SkipTest
1643- req = Request.blank('/sda1/p/a/c', environ={'REQUEST_METHOD': 'DELETE'})
1644+ req = Request.blank('/sda1/p/a/c',
1645+ environ={'REQUEST_METHOD': 'DELETE'})
1646 resp = self.object_controller.DELETE(req)
1647 self.assertEquals(resp.status_int, 400)
1648
1649@@ -916,21 +941,26 @@
1650 def test_disk_file_mkstemp_creates_dir(self):
1651 tmpdir = os.path.join(self.testdir, 'sda1', 'tmp')
1652 os.rmdir(tmpdir)
1653- with object_server.DiskFile(self.testdir, 'sda1', '0', 'a', 'c', 'o').mkstemp():
1654+ with object_server.DiskFile(self.testdir, 'sda1', '0', 'a', 'c',
1655+ 'o').mkstemp():
1656 self.assert_(os.path.exists(tmpdir))
1657
1658 def test_max_upload_time(self):
1659 if not self.path_to_test_xfs:
1660 raise SkipTest
1661+
1662 class SlowBody():
1663+
1664 def __init__(self):
1665 self.sent = 0
1666+
1667 def read(self, size=-1):
1668 if self.sent < 4:
1669 sleep(0.1)
1670 self.sent += 1
1671 return ' '
1672 return ''
1673+
1674 req = Request.blank('/sda1/p/a/c/o',
1675 environ={'REQUEST_METHOD': 'PUT', 'wsgi.input': SlowBody()},
1676 headers={'X-Timestamp': normalize_timestamp(time()),
1677@@ -946,14 +976,18 @@
1678 self.assertEquals(resp.status_int, 408)
1679
1680 def test_short_body(self):
1681+
1682 class ShortBody():
1683+
1684 def __init__(self):
1685 self.sent = False
1686+
1687 def read(self, size=-1):
1688 if not self.sent:
1689 self.sent = True
1690 return ' '
1691 return ''
1692+
1693 req = Request.blank('/sda1/p/a/c/o',
1694 environ={'REQUEST_METHOD': 'PUT', 'wsgi.input': ShortBody()},
1695 headers={'X-Timestamp': normalize_timestamp(time()),
1696@@ -1001,11 +1035,37 @@
1697 resp = self.object_controller.GET(req)
1698 self.assertEquals(resp.status_int, 200)
1699 self.assertEquals(resp.headers['content-encoding'], 'gzip')
1700- req = Request.blank('/sda1/p/a/c/o', environ={'REQUEST_METHOD': 'HEAD'})
1701+ req = Request.blank('/sda1/p/a/c/o', environ={'REQUEST_METHOD':
1702+ 'HEAD'})
1703 resp = self.object_controller.HEAD(req)
1704 self.assertEquals(resp.status_int, 200)
1705 self.assertEquals(resp.headers['content-encoding'], 'gzip')
1706
1707+ def test_manifest_header(self):
1708+ if not self.path_to_test_xfs:
1709+ raise SkipTest
1710+ timestamp = normalize_timestamp(time())
1711+ req = Request.blank('/sda1/p/a/c/o', environ={'REQUEST_METHOD': 'PUT'},
1712+ headers={'X-Timestamp': timestamp,
1713+ 'Content-Type': 'text/plain',
1714+ 'Content-Length': '0',
1715+ 'X-Object-Manifest': 'c/o/'})
1716+ resp = self.object_controller.PUT(req)
1717+ self.assertEquals(resp.status_int, 201)
1718+ objfile = os.path.join(self.testdir, 'sda1',
1719+ storage_directory(object_server.DATADIR, 'p', hash_path('a', 'c',
1720+ 'o')), timestamp + '.data')
1721+ self.assert_(os.path.isfile(objfile))
1722+ self.assertEquals(pickle.loads(getxattr(objfile,
1723+ object_server.METADATA_KEY)), {'X-Timestamp': timestamp,
1724+ 'Content-Length': '0', 'Content-Type': 'text/plain', 'name':
1725+ '/a/c/o', 'X-Object-Manifest': 'c/o/', 'ETag':
1726+ 'd41d8cd98f00b204e9800998ecf8427e'})
1727+ req = Request.blank('/sda1/p/a/c/o', environ={'REQUEST_METHOD': 'GET'})
1728+ resp = self.object_controller.GET(req)
1729+ self.assertEquals(resp.status_int, 200)
1730+ self.assertEquals(resp.headers.get('x-object-manifest'), 'c/o/')
1731+
1732
1733 if __name__ == '__main__':
1734 unittest.main()
1735
1736=== modified file 'test/unit/proxy/test_server.py'
1737--- test/unit/proxy/test_server.py 2010-12-06 20:02:43 +0000
1738+++ test/unit/proxy/test_server.py 2010-12-13 22:17:44 +0000
1739@@ -35,8 +35,8 @@
1740 from eventlet import sleep, spawn, TimeoutError, util, wsgi, listen
1741 from eventlet.timeout import Timeout
1742 import simplejson
1743-from webob import Request
1744-from webob.exc import HTTPUnauthorized
1745+from webob import Request, Response
1746+from webob.exc import HTTPNotFound, HTTPUnauthorized
1747
1748 from test.unit import connect_tcp, readuntil2crlfs
1749 from swift.proxy import server as proxy_server
1750@@ -53,7 +53,9 @@
1751
1752
1753 def fake_http_connect(*code_iter, **kwargs):
1754+
1755 class FakeConn(object):
1756+
1757 def __init__(self, status, etag=None, body=''):
1758 self.status = status
1759 self.reason = 'Fake'
1760@@ -160,6 +162,7 @@
1761
1762
1763 class FakeMemcache(object):
1764+
1765 def __init__(self):
1766 self.store = {}
1767
1768@@ -372,9 +375,12 @@
1769 class TestProxyServer(unittest.TestCase):
1770
1771 def test_unhandled_exception(self):
1772+
1773 class MyApp(proxy_server.Application):
1774+
1775 def get_controller(self, path):
1776 raise Exception('this shouldnt be caught')
1777+
1778 app = MyApp(None, FakeMemcache(), account_ring=FakeRing(),
1779 container_ring=FakeRing(), object_ring=FakeRing())
1780 req = Request.blank('/account', environ={'REQUEST_METHOD': 'HEAD'})
1781@@ -497,8 +503,11 @@
1782 test_status_map((200, 200, 204, 500, 404), 503)
1783
1784 def test_PUT_connect_exceptions(self):
1785+
1786 def mock_http_connect(*code_iter, **kwargs):
1787+
1788 class FakeConn(object):
1789+
1790 def __init__(self, status):
1791 self.status = status
1792 self.reason = 'Fake'
1793@@ -518,6 +527,7 @@
1794 if self.status == -3:
1795 return FakeConn(507)
1796 return FakeConn(100)
1797+
1798 code_iter = iter(code_iter)
1799
1800 def connect(*args, **ckwargs):
1801@@ -525,7 +535,9 @@
1802 if status == -1:
1803 raise HTTPException()
1804 return FakeConn(status)
1805+
1806 return connect
1807+
1808 with save_globals():
1809 controller = proxy_server.ObjectController(self.app, 'account',
1810 'container', 'object')
1811@@ -546,8 +558,11 @@
1812 test_status_map((200, 200, 503, 503, -1), 503)
1813
1814 def test_PUT_send_exceptions(self):
1815+
1816 def mock_http_connect(*code_iter, **kwargs):
1817+
1818 class FakeConn(object):
1819+
1820 def __init__(self, status):
1821 self.status = status
1822 self.reason = 'Fake'
1823@@ -611,8 +626,11 @@
1824 self.assertEquals(res.status_int, 413)
1825
1826 def test_PUT_getresponse_exceptions(self):
1827+
1828 def mock_http_connect(*code_iter, **kwargs):
1829+
1830 class FakeConn(object):
1831+
1832 def __init__(self, status):
1833 self.status = status
1834 self.reason = 'Fake'
1835@@ -807,6 +825,7 @@
1836 dev['port'] = 1
1837
1838 class SlowBody():
1839+
1840 def __init__(self):
1841 self.sent = 0
1842
1843@@ -816,6 +835,7 @@
1844 self.sent += 1
1845 return ' '
1846 return ''
1847+
1848 req = Request.blank('/a/c/o',
1849 environ={'REQUEST_METHOD': 'PUT', 'wsgi.input': SlowBody()},
1850 headers={'Content-Length': '4', 'Content-Type': 'text/plain'})
1851@@ -854,11 +874,13 @@
1852 dev['port'] = 1
1853
1854 class SlowBody():
1855+
1856 def __init__(self):
1857 self.sent = 0
1858
1859 def read(self, size=-1):
1860 raise Exception('Disconnected')
1861+
1862 req = Request.blank('/a/c/o',
1863 environ={'REQUEST_METHOD': 'PUT', 'wsgi.input': SlowBody()},
1864 headers={'Content-Length': '4', 'Content-Type': 'text/plain'})
1865@@ -1508,7 +1530,9 @@
1866
1867 def test_chunked_put(self):
1868 # quick test of chunked put w/o PATH_TO_TEST_XFS
1869+
1870 class ChunkedFile():
1871+
1872 def __init__(self, bytes):
1873 self.bytes = bytes
1874 self.read_bytes = 0
1875@@ -1576,6 +1600,7 @@
1876 mkdirs(os.path.join(testdir, 'sdb1'))
1877 mkdirs(os.path.join(testdir, 'sdb1', 'tmp'))
1878 try:
1879+ orig_container_listing_limit = proxy_server.CONTAINER_LISTING_LIMIT
1880 conf = {'devices': testdir, 'swift_dir': testdir,
1881 'mount_check': 'false'}
1882 prolis = listen(('localhost', 0))
1883@@ -1669,8 +1694,10 @@
1884 self.assertEquals(headers[:len(exp)], exp)
1885 # Check unhandled exception
1886 orig_update_request = prosrv.update_request
1887+
1888 def broken_update_request(env, req):
1889 raise Exception('fake')
1890+
1891 prosrv.update_request = broken_update_request
1892 sock = connect_tcp(('localhost', prolis.getsockname()[1]))
1893 fd = sock.makefile()
1894@@ -1719,8 +1746,10 @@
1895 # in a test for logging x-forwarded-for (first entry only).
1896
1897 class Logger(object):
1898+
1899 def info(self, msg):
1900 self.msg = msg
1901+
1902 orig_logger = prosrv.logger
1903 prosrv.logger = Logger()
1904 sock = connect_tcp(('localhost', prolis.getsockname()[1]))
1905@@ -1742,8 +1771,10 @@
1906 # Turn on header logging.
1907
1908 class Logger(object):
1909+
1910 def info(self, msg):
1911 self.msg = msg
1912+
1913 orig_logger = prosrv.logger
1914 prosrv.logger = Logger()
1915 prosrv.log_headers = True
1916@@ -1900,6 +1931,70 @@
1917 self.assertEquals(headers[:len(exp)], exp)
1918 body = fd.read()
1919 self.assertEquals(body, 'oh hai123456789abcdef')
1920+ # Create a container for our segmented/manifest object testing
1921+ sock = connect_tcp(('localhost', prolis.getsockname()[1]))
1922+ fd = sock.makefile()
1923+ fd.write('PUT /v1/a/segmented HTTP/1.1\r\nHost: localhost\r\n'
1924+ 'Connection: close\r\nX-Storage-Token: t\r\n'
1925+ 'Content-Length: 0\r\n\r\n')
1926+ fd.flush()
1927+ headers = readuntil2crlfs(fd)
1928+ exp = 'HTTP/1.1 201'
1929+ self.assertEquals(headers[:len(exp)], exp)
1930+ # Create the object segments
1931+ for segment in xrange(5):
1932+ sock = connect_tcp(('localhost', prolis.getsockname()[1]))
1933+ fd = sock.makefile()
1934+ fd.write('PUT /v1/a/segmented/name/%s HTTP/1.1\r\nHost: '
1935+ 'localhost\r\nConnection: close\r\nX-Storage-Token: '
1936+ 't\r\nContent-Length: 5\r\n\r\n1234 ' % str(segment))
1937+ fd.flush()
1938+ headers = readuntil2crlfs(fd)
1939+ exp = 'HTTP/1.1 201'
1940+ self.assertEquals(headers[:len(exp)], exp)
1941+ # Create the object manifest file
1942+ sock = connect_tcp(('localhost', prolis.getsockname()[1]))
1943+ fd = sock.makefile()
1944+ fd.write('PUT /v1/a/segmented/name HTTP/1.1\r\nHost: '
1945+ 'localhost\r\nConnection: close\r\nX-Storage-Token: '
1946+ 't\r\nContent-Length: 0\r\nX-Object-Manifest: '
1947+ 'segmented/name/\r\nContent-Type: text/jibberish\r\n\r\n')
1948+ fd.flush()
1949+ headers = readuntil2crlfs(fd)
1950+ exp = 'HTTP/1.1 201'
1951+ self.assertEquals(headers[:len(exp)], exp)
1952+ # Ensure retrieving the manifest file gets the whole object
1953+ sock = connect_tcp(('localhost', prolis.getsockname()[1]))
1954+ fd = sock.makefile()
1955+ fd.write('GET /v1/a/segmented/name HTTP/1.1\r\nHost: '
1956+ 'localhost\r\nConnection: close\r\nX-Auth-Token: '
1957+ 't\r\n\r\n')
1958+ fd.flush()
1959+ headers = readuntil2crlfs(fd)
1960+ exp = 'HTTP/1.1 200'
1961+ self.assertEquals(headers[:len(exp)], exp)
1962+ self.assert_('X-Object-Manifest: segmented/name/' in headers)
1963+ self.assert_('Content-Type: text/jibberish' in headers)
1964+ body = fd.read()
1965+ self.assertEquals(body, '1234 1234 1234 1234 1234 ')
1966+ # Do it again but exceeding the container listing limit
1967+ proxy_server.CONTAINER_LISTING_LIMIT = 2
1968+ sock = connect_tcp(('localhost', prolis.getsockname()[1]))
1969+ fd = sock.makefile()
1970+ fd.write('GET /v1/a/segmented/name HTTP/1.1\r\nHost: '
1971+ 'localhost\r\nConnection: close\r\nX-Auth-Token: '
1972+ 't\r\n\r\n')
1973+ fd.flush()
1974+ headers = readuntil2crlfs(fd)
1975+ exp = 'HTTP/1.1 200'
1976+ self.assertEquals(headers[:len(exp)], exp)
1977+ self.assert_('X-Object-Manifest: segmented/name/' in headers)
1978+ self.assert_('Content-Type: text/jibberish' in headers)
1979+ body = fd.read()
1980+ # A bit fragile of a test; as it makes the assumption that all
1981+ # will be sent in a single chunk.
1982+ self.assertEquals(body,
1983+ '19\r\n1234 1234 1234 1234 1234 \r\n0\r\n\r\n')
1984 finally:
1985 prospa.kill()
1986 acc1spa.kill()
1987@@ -1909,6 +2004,7 @@
1988 obj1spa.kill()
1989 obj2spa.kill()
1990 finally:
1991+ proxy_server.CONTAINER_LISTING_LIMIT = orig_container_listing_limit
1992 rmtree(testdir)
1993
1994 def test_mismatched_etags(self):
1995@@ -2111,6 +2207,7 @@
1996 res = controller.COPY(req)
1997 self.assert_(called[0])
1998
1999+
2000 class TestContainerController(unittest.TestCase):
2001 "Test swift.proxy_server.ContainerController"
2002
2003@@ -2254,7 +2351,9 @@
2004 self.assertEquals(resp.status_int, 404)
2005
2006 def test_put_locking(self):
2007+
2008 class MockMemcache(FakeMemcache):
2009+
2010 def __init__(self, allow_lock=None):
2011 self.allow_lock = allow_lock
2012 super(MockMemcache, self).__init__()
2013@@ -2265,6 +2364,7 @@
2014 yield True
2015 else:
2016 raise MemcacheLockError()
2017+
2018 with save_globals():
2019 controller = proxy_server.ContainerController(self.app, 'account',
2020 'container')
2021@@ -2870,5 +2970,261 @@
2022 test_status_map((204, 500, 404), 503)
2023
2024
2025+class FakeObjectController(object):
2026+
2027+ def __init__(self):
2028+ self.app = self
2029+ self.logger = self
2030+ self.account_name = 'a'
2031+ self.container_name = 'c'
2032+ self.object_name = 'o'
2033+ self.trans_id = 'tx1'
2034+ self.object_ring = FakeRing()
2035+ self.node_timeout = 1
2036+
2037+ def exception(self, *args):
2038+ self.exception_args = args
2039+ self.exception_info = sys.exc_info()
2040+
2041+ def GETorHEAD_base(self, *args):
2042+ self.GETorHEAD_base_args = args
2043+ req = args[0]
2044+ path = args[4]
2045+ body = data = path[-1] * int(path[-1])
2046+ if req.range and req.range.ranges:
2047+ body = ''
2048+ for start, stop in req.range.ranges:
2049+ body += data[start:stop]
2050+ resp = Response(app_iter=iter(body))
2051+ return resp
2052+
2053+ def iter_nodes(self, partition, nodes, ring):
2054+ for node in nodes:
2055+ yield node
2056+ for node in ring.get_more_nodes(partition):
2057+ yield node
2058+
2059+
2060+class Stub(object):
2061+ pass
2062+
2063+
2064+class TestSegmentedIterable(unittest.TestCase):
2065+
2066+ def setUp(self):
2067+ self.controller = FakeObjectController()
2068+
2069+ def test_load_next_segment_unexpected_error(self):
2070+ # Iterator value isn't a dict
2071+ self.assertRaises(Exception,
2072+ proxy_server.SegmentedIterable(self.controller, None,
2073+ [None])._load_next_segment)
2074+ self.assertEquals(self.controller.exception_args[0],
2075+ 'ERROR: While processing manifest /a/c/o tx1')
2076+
2077+ def test_load_next_segment_with_no_segments(self):
2078+ self.assertRaises(StopIteration,
2079+ proxy_server.SegmentedIterable(self.controller, 'lc',
2080+ [])._load_next_segment)
2081+
2082+ def test_load_next_segment_with_one_segment(self):
2083+ segit = proxy_server.SegmentedIterable(self.controller, 'lc', [{'name':
2084+ 'o1'}])
2085+ segit._load_next_segment()
2086+ self.assertEquals(self.controller.GETorHEAD_base_args[4], '/a/lc/o1')
2087+ data = ''.join(segit.segment_iter)
2088+ self.assertEquals(data, '1')
2089+
2090+ def test_load_next_segment_with_two_segments(self):
2091+ segit = proxy_server.SegmentedIterable(self.controller, 'lc', [{'name':
2092+ 'o1'}, {'name': 'o2'}])
2093+ segit._load_next_segment()
2094+ self.assertEquals(self.controller.GETorHEAD_base_args[4], '/a/lc/o1')
2095+ data = ''.join(segit.segment_iter)
2096+ self.assertEquals(data, '1')
2097+ segit._load_next_segment()
2098+ self.assertEquals(self.controller.GETorHEAD_base_args[4], '/a/lc/o2')
2099+ data = ''.join(segit.segment_iter)
2100+ self.assertEquals(data, '22')
2101+
2102+ def test_load_next_segment_with_two_segments_skip_first(self):
2103+ segit = proxy_server.SegmentedIterable(self.controller, 'lc', [{'name':
2104+ 'o1'}, {'name': 'o2'}])
2105+ segit.segment = 0
2106+ segit.listing.next()
2107+ segit._load_next_segment()
2108+ self.assertEquals(self.controller.GETorHEAD_base_args[4], '/a/lc/o2')
2109+ data = ''.join(segit.segment_iter)
2110+ self.assertEquals(data, '22')
2111+
2112+ def test_load_next_segment_with_seek(self):
2113+ segit = proxy_server.SegmentedIterable(self.controller, 'lc', [{'name':
2114+ 'o1'}, {'name': 'o2'}])
2115+ segit.segment = 0
2116+ segit.listing.next()
2117+ segit.seek = 1
2118+ segit._load_next_segment()
2119+ self.assertEquals(self.controller.GETorHEAD_base_args[4], '/a/lc/o2')
2120+ self.assertEquals(str(self.controller.GETorHEAD_base_args[0].range),
2121+ 'bytes=1-')
2122+ data = ''.join(segit.segment_iter)
2123+ self.assertEquals(data, '2')
2124+
2125+ def test_load_next_segment_with_get_error(self):
2126+
2127+ def local_GETorHEAD_base(*args):
2128+ return HTTPNotFound()
2129+
2130+ self.controller.GETorHEAD_base = local_GETorHEAD_base
2131+ self.assertRaises(Exception,
2132+ proxy_server.SegmentedIterable(self.controller, 'lc', [{'name':
2133+ 'o1'}])._load_next_segment)
2134+ self.assertEquals(self.controller.exception_args[0],
2135+ 'ERROR: While processing manifest /a/c/o tx1')
2136+ self.assertEquals(str(self.controller.exception_info[1]),
2137+ 'Could not load object segment /a/lc/o1: 404')
2138+
2139+ def test_iter_unexpected_error(self):
2140+ # Iterator value isn't a dict
2141+ self.assertRaises(Exception, ''.join,
2142+ proxy_server.SegmentedIterable(self.controller, None, [None]))
2143+ self.assertEquals(self.controller.exception_args[0],
2144+ 'ERROR: While processing manifest /a/c/o tx1')
2145+
2146+ def test_iter_with_no_segments(self):
2147+ segit = proxy_server.SegmentedIterable(self.controller, 'lc', [])
2148+ self.assertEquals(''.join(segit), '')
2149+
2150+ def test_iter_with_one_segment(self):
2151+ segit = proxy_server.SegmentedIterable(self.controller, 'lc', [{'name':
2152+ 'o1'}])
2153+ segit.response = Stub()
2154+ self.assertEquals(''.join(segit), '1')
2155+ self.assertEquals(segit.response.bytes_transferred, 1)
2156+
2157+ def test_iter_with_two_segments(self):
2158+ segit = proxy_server.SegmentedIterable(self.controller, 'lc', [{'name':
2159+ 'o1'}, {'name': 'o2'}])
2160+ segit.response = Stub()
2161+ self.assertEquals(''.join(segit), '122')
2162+ self.assertEquals(segit.response.bytes_transferred, 3)
2163+
2164+ def test_iter_with_get_error(self):
2165+
2166+ def local_GETorHEAD_base(*args):
2167+ return HTTPNotFound()
2168+
2169+ self.controller.GETorHEAD_base = local_GETorHEAD_base
2170+ self.assertRaises(Exception, ''.join,
2171+ proxy_server.SegmentedIterable(self.controller, 'lc', [{'name':
2172+ 'o1'}]))
2173+ self.assertEquals(self.controller.exception_args[0],
2174+ 'ERROR: While processing manifest /a/c/o tx1')
2175+ self.assertEquals(str(self.controller.exception_info[1]),
2176+ 'Could not load object segment /a/lc/o1: 404')
2177+
2178+ def test_app_iter_range_unexpected_error(self):
2179+ # Iterator value isn't a dict
2180+ self.assertRaises(Exception,
2181+ proxy_server.SegmentedIterable(self.controller, None,
2182+ [None]).app_iter_range(None, None).next)
2183+ self.assertEquals(self.controller.exception_args[0],
2184+ 'ERROR: While processing manifest /a/c/o tx1')
2185+
2186+ def test_app_iter_range_with_no_segments(self):
2187+ self.assertEquals(''.join(proxy_server.SegmentedIterable(
2188+ self.controller, 'lc', []).app_iter_range(None, None)), '')
2189+ self.assertEquals(''.join(proxy_server.SegmentedIterable(
2190+ self.controller, 'lc', []).app_iter_range(3, None)), '')
2191+ self.assertEquals(''.join(proxy_server.SegmentedIterable(
2192+ self.controller, 'lc', []).app_iter_range(3, 5)), '')
2193+ self.assertEquals(''.join(proxy_server.SegmentedIterable(
2194+ self.controller, 'lc', []).app_iter_range(None, 5)), '')
2195+
2196+ def test_app_iter_range_with_one_segment(self):
2197+ listing = [{'name': 'o1', 'bytes': 1}]
2198+
2199+ segit = proxy_server.SegmentedIterable(self.controller, 'lc', listing)
2200+ segit.response = Stub()
2201+ self.assertEquals(''.join(segit.app_iter_range(None, None)), '1')
2202+ self.assertEquals(segit.response.bytes_transferred, 1)
2203+
2204+ segit = proxy_server.SegmentedIterable(self.controller, 'lc', listing)
2205+ self.assertEquals(''.join(segit.app_iter_range(3, None)), '')
2206+
2207+ segit = proxy_server.SegmentedIterable(self.controller, 'lc', listing)
2208+ self.assertEquals(''.join(segit.app_iter_range(3, 5)), '')
2209+
2210+ segit = proxy_server.SegmentedIterable(self.controller, 'lc', listing)
2211+ segit.response = Stub()
2212+ self.assertEquals(''.join(segit.app_iter_range(None, 5)), '1')
2213+ self.assertEquals(segit.response.bytes_transferred, 1)
2214+
2215+ def test_app_iter_range_with_two_segments(self):
2216+ listing = [{'name': 'o1', 'bytes': 1}, {'name': 'o2', 'bytes': 2}]
2217+
2218+ segit = proxy_server.SegmentedIterable(self.controller, 'lc', listing)
2219+ segit.response = Stub()
2220+ self.assertEquals(''.join(segit.app_iter_range(None, None)), '122')
2221+ self.assertEquals(segit.response.bytes_transferred, 3)
2222+
2223+ segit = proxy_server.SegmentedIterable(self.controller, 'lc', listing)
2224+ segit.response = Stub()
2225+ self.assertEquals(''.join(segit.app_iter_range(1, None)), '22')
2226+ self.assertEquals(segit.response.bytes_transferred, 2)
2227+
2228+ segit = proxy_server.SegmentedIterable(self.controller, 'lc', listing)
2229+ segit.response = Stub()
2230+ self.assertEquals(''.join(segit.app_iter_range(1, 5)), '22')
2231+ self.assertEquals(segit.response.bytes_transferred, 2)
2232+
2233+ segit = proxy_server.SegmentedIterable(self.controller, 'lc', listing)
2234+ segit.response = Stub()
2235+ self.assertEquals(''.join(segit.app_iter_range(None, 2)), '12')
2236+ self.assertEquals(segit.response.bytes_transferred, 2)
2237+
2238+ def test_app_iter_range_with_many_segments(self):
2239+ listing = [{'name': 'o1', 'bytes': 1}, {'name': 'o2', 'bytes': 2},
2240+ {'name': 'o3', 'bytes': 3}, {'name': 'o4', 'bytes': 4}, {'name':
2241+ 'o5', 'bytes': 5}]
2242+
2243+ segit = proxy_server.SegmentedIterable(self.controller, 'lc', listing)
2244+ segit.response = Stub()
2245+ self.assertEquals(''.join(segit.app_iter_range(None, None)),
2246+ '122333444455555')
2247+ self.assertEquals(segit.response.bytes_transferred, 15)
2248+
2249+ segit = proxy_server.SegmentedIterable(self.controller, 'lc', listing)
2250+ segit.response = Stub()
2251+ self.assertEquals(''.join(segit.app_iter_range(3, None)),
2252+ '333444455555')
2253+ self.assertEquals(segit.response.bytes_transferred, 12)
2254+
2255+ segit = proxy_server.SegmentedIterable(self.controller, 'lc', listing)
2256+ segit.response = Stub()
2257+ self.assertEquals(''.join(segit.app_iter_range(5, None)), '3444455555')
2258+ self.assertEquals(segit.response.bytes_transferred, 10)
2259+
2260+ segit = proxy_server.SegmentedIterable(self.controller, 'lc', listing)
2261+ segit.response = Stub()
2262+ self.assertEquals(''.join(segit.app_iter_range(None, 6)), '122333')
2263+ self.assertEquals(segit.response.bytes_transferred, 6)
2264+
2265+ segit = proxy_server.SegmentedIterable(self.controller, 'lc', listing)
2266+ segit.response = Stub()
2267+ self.assertEquals(''.join(segit.app_iter_range(None, 7)), '1223334')
2268+ self.assertEquals(segit.response.bytes_transferred, 7)
2269+
2270+ segit = proxy_server.SegmentedIterable(self.controller, 'lc', listing)
2271+ segit.response = Stub()
2272+ self.assertEquals(''.join(segit.app_iter_range(3, 7)), '3334')
2273+ self.assertEquals(segit.response.bytes_transferred, 4)
2274+
2275+ segit = proxy_server.SegmentedIterable(self.controller, 'lc', listing)
2276+ segit.response = Stub()
2277+ self.assertEquals(''.join(segit.app_iter_range(5, 7)), '34')
2278+ self.assertEquals(segit.response.bytes_transferred, 2)
2279+
2280+
2281 if __name__ == '__main__':
2282 unittest.main()