Merge lp:~gholt/swift/largefiles into lp:~hudson-openstack/swift/trunk
- largefiles
- Merge into trunk
Status: | Rejected |
---|---|
Rejected by: | gholt |
Proposed branch: | lp:~gholt/swift/largefiles |
Merge into: | lp:~hudson-openstack/swift/trunk |
Diff against target: |
1491 lines (+930/-231) 10 files modified
swift/common/constraints.py (+8/-2) swift/obj/auditor.py (+1/-1) swift/obj/hashes.py (+179/-0) swift/obj/hashes.py.moved (+179/-0) swift/obj/replicator.py (+37/-177) swift/obj/server.py (+81/-30) swift/proxy/server.py (+387/-21) test/unit/__init__.py (+2/-0) test/unit/obj/test_hashes.py (+28/-0) test/unit/obj/test_hashes.py.moved (+28/-0) |
To merge this branch: | bzr merge lp:~gholt/swift/largefiles |
Related bugs: |
Reviewer | Review Type | Date Requested | Status |
---|---|---|---|
gholt (community) | Disapprove | ||
Review via email: mp+35493@code.launchpad.net |
Commit message
Description of the change
Support for very large files by breaking them into segments and distributing the segments across the cluster.
- 72. By gholt
-
Fixed typo
- 73. By gholt
-
Merged changes from trunk
- 74. By gholt
-
very-large-files: now distributes segments across cluster and will not overwrite existing segments during the upload
- 75. By gholt
-
Updating object auditor to clean up old orphaned large file segments
- 76. By gholt
-
Merged changes from trunk
- 77. By gholt
-
Merged changes from trunk
- 78. By gholt
-
Merged from trunk
- 79. By gholt
-
Renovation to store object segments separate from whole objects
- 80. By gholt
-
Just a bit of DRY
- 81. By gholt
-
Support very large chunked transfer encoding uploads
- 82. By gholt
-
Update import and new test stub for moved obj hashes code
- 83. By gholt
-
Merge from trunk
- 84. By gholt
-
Merged from trunk
- 85. By gholt
-
Merged from trunk
- 86. By gholt
-
Merged with trunk
- 87. By gholt
-
Merged from trunk
- 88. By gholt
-
Refactor of container_updater calling; removed outdated imports
- 89. By gholt
-
Merged with refactorhashes
Unmerged revisions
- 91. By gholt
-
Merged from refactorobjasync
- 90. By gholt
-
Merge from trunk
- 89. By gholt
-
Merged with refactorhashes
- 88. By gholt
-
Refactor of container_updater calling; removed outdated imports
- 87. By gholt
-
Merged from trunk
- 86. By gholt
-
Merged with trunk
- 85. By gholt
-
Merged from trunk
- 84. By gholt
-
Merged from trunk
- 83. By gholt
-
Merge from trunk
- 82. By gholt
-
Update import and new test stub for moved obj hashes code
Preview Diff
1 | === modified file 'swift/common/constraints.py' | |||
2 | --- swift/common/constraints.py 2010-09-22 19:53:38 +0000 | |||
3 | +++ swift/common/constraints.py 2010-10-19 18:43:49 +0000 | |||
4 | @@ -19,8 +19,12 @@ | |||
5 | 19 | HTTPRequestEntityTooLarge | 19 | HTTPRequestEntityTooLarge |
6 | 20 | 20 | ||
7 | 21 | 21 | ||
8 | 22 | # FIXME: This will get bumped way up once very large file support is added. | ||
9 | 22 | #: Max file size allowed for objects | 23 | #: Max file size allowed for objects |
10 | 23 | MAX_FILE_SIZE = 5 * 1024 * 1024 * 1024 + 2 | 24 | MAX_FILE_SIZE = 5 * 1024 * 1024 * 1024 + 2 |
11 | 25 | # FIXME: Yeah, I know this is a silly low value; just for testing right now. | ||
12 | 26 | #: File size for segments of large objects | ||
13 | 27 | SEGMENT_SIZE = 1024 | ||
14 | 24 | #: Max length of the name of a key for metadata | 28 | #: Max length of the name of a key for metadata |
15 | 25 | MAX_META_NAME_LENGTH = 128 | 29 | MAX_META_NAME_LENGTH = 128 |
16 | 26 | #: Max length of the value of a key for metadata | 30 | #: Max length of the value of a key for metadata |
17 | @@ -29,14 +33,16 @@ | |||
18 | 29 | MAX_META_COUNT = 90 | 33 | MAX_META_COUNT = 90 |
19 | 30 | #: Max overall size of metadata | 34 | #: Max overall size of metadata |
20 | 31 | MAX_META_OVERALL_SIZE = 4096 | 35 | MAX_META_OVERALL_SIZE = 4096 |
21 | 36 | #: Max account name length | ||
22 | 37 | MAX_ACCOUNT_NAME_LENGTH = 256 | ||
23 | 38 | #: Max container name length | ||
24 | 39 | MAX_CONTAINER_NAME_LENGTH = 256 | ||
25 | 32 | #: Max object name length | 40 | #: Max object name length |
26 | 33 | MAX_OBJECT_NAME_LENGTH = 1024 | 41 | MAX_OBJECT_NAME_LENGTH = 1024 |
27 | 34 | #: Max object list length of a get request for a container | 42 | #: Max object list length of a get request for a container |
28 | 35 | CONTAINER_LISTING_LIMIT = 10000 | 43 | CONTAINER_LISTING_LIMIT = 10000 |
29 | 36 | #: Max container list length of a get request for an account | 44 | #: Max container list length of a get request for an account |
30 | 37 | ACCOUNT_LISTING_LIMIT = 10000 | 45 | ACCOUNT_LISTING_LIMIT = 10000 |
31 | 38 | MAX_ACCOUNT_NAME_LENGTH = 256 | ||
32 | 39 | MAX_CONTAINER_NAME_LENGTH = 256 | ||
33 | 40 | 46 | ||
34 | 41 | 47 | ||
35 | 42 | def check_metadata(req, target_type): | 48 | def check_metadata(req, target_type): |
36 | 43 | 49 | ||
37 | === modified file 'swift/obj/auditor.py' | |||
38 | --- swift/obj/auditor.py 2010-10-18 22:30:26 +0000 | |||
39 | +++ swift/obj/auditor.py 2010-10-19 18:43:49 +0000 | |||
40 | @@ -19,7 +19,7 @@ | |||
41 | 19 | from random import random | 19 | from random import random |
42 | 20 | 20 | ||
43 | 21 | from swift.obj import server as object_server | 21 | from swift.obj import server as object_server |
45 | 22 | from swift.obj.replicator import invalidate_hash | 22 | from swift.obj.hashes import invalidate_hash |
46 | 23 | from swift.common.utils import get_logger, renamer, audit_location_generator | 23 | from swift.common.utils import get_logger, renamer, audit_location_generator |
47 | 24 | from swift.common.exceptions import AuditException | 24 | from swift.common.exceptions import AuditException |
48 | 25 | from swift.common.daemon import Daemon | 25 | from swift.common.daemon import Daemon |
49 | 26 | 26 | ||
50 | === added file 'swift/obj/hashes.py' | |||
51 | --- swift/obj/hashes.py 1970-01-01 00:00:00 +0000 | |||
52 | +++ swift/obj/hashes.py 2010-10-19 18:43:49 +0000 | |||
53 | @@ -0,0 +1,179 @@ | |||
54 | 1 | # Copyright (c) 2010 OpenStack, LLC. | ||
55 | 2 | # | ||
56 | 3 | # Licensed under the Apache License, Version 2.0 (the "License"); | ||
57 | 4 | # you may not use this file except in compliance with the License. | ||
58 | 5 | # You may obtain a copy of the License at | ||
59 | 6 | # | ||
60 | 7 | # http://www.apache.org/licenses/LICENSE-2.0 | ||
61 | 8 | # | ||
62 | 9 | # Unless required by applicable law or agreed to in writing, software | ||
63 | 10 | # distributed under the License is distributed on an "AS IS" BASIS, | ||
64 | 11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or | ||
65 | 12 | # implied. | ||
66 | 13 | # See the License for the specific language governing permissions and | ||
67 | 14 | # limitations under the License. | ||
68 | 15 | |||
69 | 16 | import cPickle as pickle | ||
70 | 17 | import hashlib | ||
71 | 18 | import os | ||
72 | 19 | from os.path import isdir, join | ||
73 | 20 | |||
74 | 21 | from eventlet import tpool, sleep | ||
75 | 22 | |||
76 | 23 | from swift.common.utils import lock_path, renamer | ||
77 | 24 | |||
78 | 25 | |||
79 | 26 | PICKLE_PROTOCOL = 2 | ||
80 | 27 | ONE_WEEK = 604800 | ||
81 | 28 | HASH_FILE = 'hashes.pkl' | ||
82 | 29 | |||
83 | 30 | |||
84 | 31 | def hash_suffix(path, reclaim_age): | ||
85 | 32 | """ | ||
86 | 33 | Performs reclamation and returns an md5 of all (remaining) files. | ||
87 | 34 | |||
88 | 35 | :param reclaim_age: age in seconds at which to remove tombstones | ||
89 | 36 | """ | ||
90 | 37 | md5 = hashlib.md5() | ||
91 | 38 | for hsh in sorted(os.listdir(path)): | ||
92 | 39 | hsh_path = join(path, hsh) | ||
93 | 40 | files = os.listdir(hsh_path) | ||
94 | 41 | if len(files) == 1: | ||
95 | 42 | if files[0].endswith('.ts'): | ||
96 | 43 | # remove tombstones older than reclaim_age | ||
97 | 44 | ts = files[0].rsplit('.', 1)[0] | ||
98 | 45 | if (time.time() - float(ts)) > reclaim_age: | ||
99 | 46 | os.unlink(join(hsh_path, files[0])) | ||
100 | 47 | files.remove(files[0]) | ||
101 | 48 | elif files: | ||
102 | 49 | files.sort(reverse=True) | ||
103 | 50 | meta = data = tomb = None | ||
104 | 51 | for filename in files: | ||
105 | 52 | if not meta and filename.endswith('.meta'): | ||
106 | 53 | meta = filename | ||
107 | 54 | if not data and filename.endswith('.data'): | ||
108 | 55 | data = filename | ||
109 | 56 | if not tomb and filename.endswith('.ts'): | ||
110 | 57 | tomb = filename | ||
111 | 58 | if (filename < tomb or # any file older than tomb | ||
112 | 59 | filename < data or # any file older than data | ||
113 | 60 | (filename.endswith('.meta') and | ||
114 | 61 | filename < meta)): # old meta | ||
115 | 62 | os.unlink(join(hsh_path, filename)) | ||
116 | 63 | files.remove(filename) | ||
117 | 64 | if not files: | ||
118 | 65 | os.rmdir(hsh_path) | ||
119 | 66 | for filename in files: | ||
120 | 67 | md5.update(filename) | ||
121 | 68 | try: | ||
122 | 69 | os.rmdir(path) | ||
123 | 70 | except OSError: | ||
124 | 71 | pass | ||
125 | 72 | return md5.hexdigest() | ||
126 | 73 | |||
127 | 74 | |||
128 | 75 | def recalculate_hashes(partition_dir, suffixes, reclaim_age=ONE_WEEK): | ||
129 | 76 | """ | ||
130 | 77 | Recalculates hashes for the given suffixes in the partition and updates | ||
131 | 78 | them in the partition's hashes file. | ||
132 | 79 | |||
133 | 80 | :param partition_dir: directory of the partition in which to recalculate | ||
134 | 81 | :param suffixes: list of suffixes to recalculate | ||
135 | 82 | :param reclaim_age: age in seconds at which tombstones should be removed | ||
136 | 83 | """ | ||
137 | 84 | |||
138 | 85 | def tpool_listdir(partition_dir): | ||
139 | 86 | return dict(((suff, None) for suff in os.listdir(partition_dir) | ||
140 | 87 | if len(suff) == 3 and isdir(join(partition_dir, suff)))) | ||
141 | 88 | hashes_file = join(partition_dir, HASH_FILE) | ||
142 | 89 | with lock_path(partition_dir): | ||
143 | 90 | try: | ||
144 | 91 | with open(hashes_file, 'rb') as fp: | ||
145 | 92 | hashes = pickle.load(fp) | ||
146 | 93 | except Exception: | ||
147 | 94 | hashes = tpool.execute(tpool_listdir, partition_dir) | ||
148 | 95 | for suffix in suffixes: | ||
149 | 96 | suffix_dir = join(partition_dir, suffix) | ||
150 | 97 | if os.path.exists(suffix_dir): | ||
151 | 98 | hashes[suffix] = hash_suffix(suffix_dir, reclaim_age) | ||
152 | 99 | elif suffix in hashes: | ||
153 | 100 | del hashes[suffix] | ||
154 | 101 | with open(hashes_file + '.tmp', 'wb') as fp: | ||
155 | 102 | pickle.dump(hashes, fp, PICKLE_PROTOCOL) | ||
156 | 103 | renamer(hashes_file + '.tmp', hashes_file) | ||
157 | 104 | |||
158 | 105 | |||
159 | 106 | def invalidate_hash(suffix_dir): | ||
160 | 107 | """ | ||
161 | 108 | Invalidates the hash for a suffix_dir in the partition's hashes file. | ||
162 | 109 | |||
163 | 110 | :param suffix_dir: absolute path to suffix dir whose hash needs | ||
164 | 111 | invalidating | ||
165 | 112 | """ | ||
166 | 113 | |||
167 | 114 | suffix = os.path.basename(suffix_dir) | ||
168 | 115 | partition_dir = os.path.dirname(suffix_dir) | ||
169 | 116 | hashes_file = join(partition_dir, HASH_FILE) | ||
170 | 117 | with lock_path(partition_dir): | ||
171 | 118 | try: | ||
172 | 119 | with open(hashes_file, 'rb') as fp: | ||
173 | 120 | hashes = pickle.load(fp) | ||
174 | 121 | if suffix in hashes and not hashes[suffix]: | ||
175 | 122 | return | ||
176 | 123 | except Exception: | ||
177 | 124 | return | ||
178 | 125 | hashes[suffix] = None | ||
179 | 126 | with open(hashes_file + '.tmp', 'wb') as fp: | ||
180 | 127 | pickle.dump(hashes, fp, PICKLE_PROTOCOL) | ||
181 | 128 | renamer(hashes_file + '.tmp', hashes_file) | ||
182 | 129 | |||
183 | 130 | |||
184 | 131 | def get_hashes(partition_dir, do_listdir=True, reclaim_age=ONE_WEEK): | ||
185 | 132 | """ | ||
186 | 133 | Get a list of hashes for the suffix dir. do_listdir causes it to mistrust | ||
187 | 134 | the hash cache for suffix existence at the (unexpectedly high) cost of a | ||
188 | 135 | listdir. reclaim_age is just passed on to hash_suffix. | ||
189 | 136 | |||
190 | 137 | :param partition_dir: absolute path of partition to get hashes for | ||
191 | 138 | :param do_listdir: force existence check for all hashes in the partition | ||
192 | 139 | :param reclaim_age: age at which to remove tombstones | ||
193 | 140 | |||
194 | 141 | :returns: tuple of (number of suffix dirs hashed, dictionary of hashes) | ||
195 | 142 | """ | ||
196 | 143 | |||
197 | 144 | def tpool_listdir(hashes, partition_dir): | ||
198 | 145 | return dict(((suff, hashes.get(suff, None)) | ||
199 | 146 | for suff in os.listdir(partition_dir) | ||
200 | 147 | if len(suff) == 3 and isdir(join(partition_dir, suff)))) | ||
201 | 148 | hashed = 0 | ||
202 | 149 | hashes_file = join(partition_dir, HASH_FILE) | ||
203 | 150 | with lock_path(partition_dir): | ||
204 | 151 | modified = False | ||
205 | 152 | hashes = {} | ||
206 | 153 | try: | ||
207 | 154 | with open(hashes_file, 'rb') as fp: | ||
208 | 155 | hashes = pickle.load(fp) | ||
209 | 156 | except Exception: | ||
210 | 157 | do_listdir = True | ||
211 | 158 | if do_listdir: | ||
212 | 159 | hashes = tpool.execute(tpool_listdir, hashes, partition_dir) | ||
213 | 160 | modified = True | ||
214 | 161 | for suffix, hash_ in hashes.items(): | ||
215 | 162 | if not hash_: | ||
216 | 163 | suffix_dir = join(partition_dir, suffix) | ||
217 | 164 | if os.path.exists(suffix_dir): | ||
218 | 165 | try: | ||
219 | 166 | hashes[suffix] = hash_suffix(suffix_dir, reclaim_age) | ||
220 | 167 | hashed += 1 | ||
221 | 168 | except OSError: | ||
222 | 169 | logging.exception('Error hashing suffix') | ||
223 | 170 | hashes[suffix] = None | ||
224 | 171 | else: | ||
225 | 172 | del hashes[suffix] | ||
226 | 173 | modified = True | ||
227 | 174 | sleep() | ||
228 | 175 | if modified: | ||
229 | 176 | with open(hashes_file + '.tmp', 'wb') as fp: | ||
230 | 177 | pickle.dump(hashes, fp, PICKLE_PROTOCOL) | ||
231 | 178 | renamer(hashes_file + '.tmp', hashes_file) | ||
232 | 179 | return hashed, hashes | ||
233 | 0 | 180 | ||
234 | === added file 'swift/obj/hashes.py.moved' | |||
235 | --- swift/obj/hashes.py.moved 1970-01-01 00:00:00 +0000 | |||
236 | +++ swift/obj/hashes.py.moved 2010-10-19 18:43:49 +0000 | |||
237 | @@ -0,0 +1,179 @@ | |||
238 | 1 | # Copyright (c) 2010 OpenStack, LLC. | ||
239 | 2 | # | ||
240 | 3 | # Licensed under the Apache License, Version 2.0 (the "License"); | ||
241 | 4 | # you may not use this file except in compliance with the License. | ||
242 | 5 | # You may obtain a copy of the License at | ||
243 | 6 | # | ||
244 | 7 | # http://www.apache.org/licenses/LICENSE-2.0 | ||
245 | 8 | # | ||
246 | 9 | # Unless required by applicable law or agreed to in writing, software | ||
247 | 10 | # distributed under the License is distributed on an "AS IS" BASIS, | ||
248 | 11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or | ||
249 | 12 | # implied. | ||
250 | 13 | # See the License for the specific language governing permissions and | ||
251 | 14 | # limitations under the License. | ||
252 | 15 | |||
253 | 16 | import cPickle as pickle | ||
254 | 17 | import hashlib | ||
255 | 18 | import os | ||
256 | 19 | from os.path import isdir, join | ||
257 | 20 | |||
258 | 21 | from eventlet import tpool, sleep | ||
259 | 22 | |||
260 | 23 | from swift.common.utils import lock_path, renamer | ||
261 | 24 | |||
262 | 25 | |||
263 | 26 | PICKLE_PROTOCOL = 2 | ||
264 | 27 | ONE_WEEK = 604800 | ||
265 | 28 | HASH_FILE = 'hashes.pkl' | ||
266 | 29 | |||
267 | 30 | |||
268 | 31 | def hash_suffix(path, reclaim_age): | ||
269 | 32 | """ | ||
270 | 33 | Performs reclamation and returns an md5 of all (remaining) files. | ||
271 | 34 | |||
272 | 35 | :param reclaim_age: age in seconds at which to remove tombstones | ||
273 | 36 | """ | ||
274 | 37 | md5 = hashlib.md5() | ||
275 | 38 | for hsh in sorted(os.listdir(path)): | ||
276 | 39 | hsh_path = join(path, hsh) | ||
277 | 40 | files = os.listdir(hsh_path) | ||
278 | 41 | if len(files) == 1: | ||
279 | 42 | if files[0].endswith('.ts'): | ||
280 | 43 | # remove tombstones older than reclaim_age | ||
281 | 44 | ts = files[0].rsplit('.', 1)[0] | ||
282 | 45 | if (time.time() - float(ts)) > reclaim_age: | ||
283 | 46 | os.unlink(join(hsh_path, files[0])) | ||
284 | 47 | files.remove(files[0]) | ||
285 | 48 | elif files: | ||
286 | 49 | files.sort(reverse=True) | ||
287 | 50 | meta = data = tomb = None | ||
288 | 51 | for filename in files: | ||
289 | 52 | if not meta and filename.endswith('.meta'): | ||
290 | 53 | meta = filename | ||
291 | 54 | if not data and filename.endswith('.data'): | ||
292 | 55 | data = filename | ||
293 | 56 | if not tomb and filename.endswith('.ts'): | ||
294 | 57 | tomb = filename | ||
295 | 58 | if (filename < tomb or # any file older than tomb | ||
296 | 59 | filename < data or # any file older than data | ||
297 | 60 | (filename.endswith('.meta') and | ||
298 | 61 | filename < meta)): # old meta | ||
299 | 62 | os.unlink(join(hsh_path, filename)) | ||
300 | 63 | files.remove(filename) | ||
301 | 64 | if not files: | ||
302 | 65 | os.rmdir(hsh_path) | ||
303 | 66 | for filename in files: | ||
304 | 67 | md5.update(filename) | ||
305 | 68 | try: | ||
306 | 69 | os.rmdir(path) | ||
307 | 70 | except OSError: | ||
308 | 71 | pass | ||
309 | 72 | return md5.hexdigest() | ||
310 | 73 | |||
311 | 74 | |||
312 | 75 | def recalculate_hashes(partition_dir, suffixes, reclaim_age=ONE_WEEK): | ||
313 | 76 | """ | ||
314 | 77 | Recalculates hashes for the given suffixes in the partition and updates | ||
315 | 78 | them in the partition's hashes file. | ||
316 | 79 | |||
317 | 80 | :param partition_dir: directory of the partition in which to recalculate | ||
318 | 81 | :param suffixes: list of suffixes to recalculate | ||
319 | 82 | :param reclaim_age: age in seconds at which tombstones should be removed | ||
320 | 83 | """ | ||
321 | 84 | |||
322 | 85 | def tpool_listdir(partition_dir): | ||
323 | 86 | return dict(((suff, None) for suff in os.listdir(partition_dir) | ||
324 | 87 | if len(suff) == 3 and isdir(join(partition_dir, suff)))) | ||
325 | 88 | hashes_file = join(partition_dir, HASH_FILE) | ||
326 | 89 | with lock_path(partition_dir): | ||
327 | 90 | try: | ||
328 | 91 | with open(hashes_file, 'rb') as fp: | ||
329 | 92 | hashes = pickle.load(fp) | ||
330 | 93 | except Exception: | ||
331 | 94 | hashes = tpool.execute(tpool_listdir, partition_dir) | ||
332 | 95 | for suffix in suffixes: | ||
333 | 96 | suffix_dir = join(partition_dir, suffix) | ||
334 | 97 | if os.path.exists(suffix_dir): | ||
335 | 98 | hashes[suffix] = hash_suffix(suffix_dir, reclaim_age) | ||
336 | 99 | elif suffix in hashes: | ||
337 | 100 | del hashes[suffix] | ||
338 | 101 | with open(hashes_file + '.tmp', 'wb') as fp: | ||
339 | 102 | pickle.dump(hashes, fp, PICKLE_PROTOCOL) | ||
340 | 103 | renamer(hashes_file + '.tmp', hashes_file) | ||
341 | 104 | |||
342 | 105 | |||
343 | 106 | def invalidate_hash(suffix_dir): | ||
344 | 107 | """ | ||
345 | 108 | Invalidates the hash for a suffix_dir in the partition's hashes file. | ||
346 | 109 | |||
347 | 110 | :param suffix_dir: absolute path to suffix dir whose hash needs | ||
348 | 111 | invalidating | ||
349 | 112 | """ | ||
350 | 113 | |||
351 | 114 | suffix = os.path.basename(suffix_dir) | ||
352 | 115 | partition_dir = os.path.dirname(suffix_dir) | ||
353 | 116 | hashes_file = join(partition_dir, HASH_FILE) | ||
354 | 117 | with lock_path(partition_dir): | ||
355 | 118 | try: | ||
356 | 119 | with open(hashes_file, 'rb') as fp: | ||
357 | 120 | hashes = pickle.load(fp) | ||
358 | 121 | if suffix in hashes and not hashes[suffix]: | ||
359 | 122 | return | ||
360 | 123 | except Exception: | ||
361 | 124 | return | ||
362 | 125 | hashes[suffix] = None | ||
363 | 126 | with open(hashes_file + '.tmp', 'wb') as fp: | ||
364 | 127 | pickle.dump(hashes, fp, PICKLE_PROTOCOL) | ||
365 | 128 | renamer(hashes_file + '.tmp', hashes_file) | ||
366 | 129 | |||
367 | 130 | |||
368 | 131 | def get_hashes(partition_dir, do_listdir=True, reclaim_age=ONE_WEEK): | ||
369 | 132 | """ | ||
370 | 133 | Get a list of hashes for the suffix dir. do_listdir causes it to mistrust | ||
371 | 134 | the hash cache for suffix existence at the (unexpectedly high) cost of a | ||
372 | 135 | listdir. reclaim_age is just passed on to hash_suffix. | ||
373 | 136 | |||
374 | 137 | :param partition_dir: absolute path of partition to get hashes for | ||
375 | 138 | :param do_listdir: force existence check for all hashes in the partition | ||
376 | 139 | :param reclaim_age: age at which to remove tombstones | ||
377 | 140 | |||
378 | 141 | :returns: tuple of (number of suffix dirs hashed, dictionary of hashes) | ||
379 | 142 | """ | ||
380 | 143 | |||
381 | 144 | def tpool_listdir(hashes, partition_dir): | ||
382 | 145 | return dict(((suff, hashes.get(suff, None)) | ||
383 | 146 | for suff in os.listdir(partition_dir) | ||
384 | 147 | if len(suff) == 3 and isdir(join(partition_dir, suff)))) | ||
385 | 148 | hashed = 0 | ||
386 | 149 | hashes_file = join(partition_dir, HASH_FILE) | ||
387 | 150 | with lock_path(partition_dir): | ||
388 | 151 | modified = False | ||
389 | 152 | hashes = {} | ||
390 | 153 | try: | ||
391 | 154 | with open(hashes_file, 'rb') as fp: | ||
392 | 155 | hashes = pickle.load(fp) | ||
393 | 156 | except Exception: | ||
394 | 157 | do_listdir = True | ||
395 | 158 | if do_listdir: | ||
396 | 159 | hashes = tpool.execute(tpool_listdir, hashes, partition_dir) | ||
397 | 160 | modified = True | ||
398 | 161 | for suffix, hash_ in hashes.items(): | ||
399 | 162 | if not hash_: | ||
400 | 163 | suffix_dir = join(partition_dir, suffix) | ||
401 | 164 | if os.path.exists(suffix_dir): | ||
402 | 165 | try: | ||
403 | 166 | hashes[suffix] = hash_suffix(suffix_dir, reclaim_age) | ||
404 | 167 | hashed += 1 | ||
405 | 168 | except OSError: | ||
406 | 169 | logging.exception('Error hashing suffix') | ||
407 | 170 | hashes[suffix] = None | ||
408 | 171 | else: | ||
409 | 172 | del hashes[suffix] | ||
410 | 173 | modified = True | ||
411 | 174 | sleep() | ||
412 | 175 | if modified: | ||
413 | 176 | with open(hashes_file + '.tmp', 'wb') as fp: | ||
414 | 177 | pickle.dump(hashes, fp, PICKLE_PROTOCOL) | ||
415 | 178 | renamer(hashes_file + '.tmp', hashes_file) | ||
416 | 179 | return hashed, hashes | ||
417 | 0 | 180 | ||
418 | === modified file 'swift/obj/replicator.py' | |||
419 | --- swift/obj/replicator.py 2010-10-19 01:05:54 +0000 | |||
420 | +++ swift/obj/replicator.py 2010-10-19 18:43:49 +0000 | |||
421 | @@ -19,7 +19,6 @@ | |||
422 | 19 | import shutil | 19 | import shutil |
423 | 20 | import time | 20 | import time |
424 | 21 | import logging | 21 | import logging |
425 | 22 | import hashlib | ||
426 | 23 | import itertools | 22 | import itertools |
427 | 24 | import cPickle as pickle | 23 | import cPickle as pickle |
428 | 25 | 24 | ||
429 | @@ -29,168 +28,16 @@ | |||
430 | 29 | from eventlet.support.greenlets import GreenletExit | 28 | from eventlet.support.greenlets import GreenletExit |
431 | 30 | 29 | ||
432 | 31 | from swift.common.ring import Ring | 30 | from swift.common.ring import Ring |
435 | 32 | from swift.common.utils import whataremyips, unlink_older_than, lock_path, \ | 31 | from swift.common.utils import compute_eta, get_logger, unlink_older_than, \ |
436 | 33 | renamer, compute_eta, get_logger | 32 | whataremyips |
437 | 34 | from swift.common.bufferedhttp import http_connect | 33 | from swift.common.bufferedhttp import http_connect |
438 | 35 | from swift.common.daemon import Daemon | 34 | from swift.common.daemon import Daemon |
439 | 35 | from swift.obj.hashes import get_hashes, recalculate_hashes | ||
440 | 36 | from swift.obj.server import DATADIR | ||
441 | 37 | |||
442 | 36 | 38 | ||
443 | 37 | hubs.use_hub('poll') | 39 | hubs.use_hub('poll') |
444 | 38 | 40 | ||
445 | 39 | PICKLE_PROTOCOL = 2 | ||
446 | 40 | ONE_WEEK = 604800 | ||
447 | 41 | HASH_FILE = 'hashes.pkl' | ||
448 | 42 | |||
449 | 43 | |||
450 | 44 | def hash_suffix(path, reclaim_age): | ||
451 | 45 | """ | ||
452 | 46 | Performs reclamation and returns an md5 of all (remaining) files. | ||
453 | 47 | |||
454 | 48 | :param reclaim_age: age in seconds at which to remove tombstones | ||
455 | 49 | """ | ||
456 | 50 | md5 = hashlib.md5() | ||
457 | 51 | for hsh in sorted(os.listdir(path)): | ||
458 | 52 | hsh_path = join(path, hsh) | ||
459 | 53 | files = os.listdir(hsh_path) | ||
460 | 54 | if len(files) == 1: | ||
461 | 55 | if files[0].endswith('.ts'): | ||
462 | 56 | # remove tombstones older than reclaim_age | ||
463 | 57 | ts = files[0].rsplit('.', 1)[0] | ||
464 | 58 | if (time.time() - float(ts)) > reclaim_age: | ||
465 | 59 | os.unlink(join(hsh_path, files[0])) | ||
466 | 60 | files.remove(files[0]) | ||
467 | 61 | elif files: | ||
468 | 62 | files.sort(reverse=True) | ||
469 | 63 | meta = data = tomb = None | ||
470 | 64 | for filename in files: | ||
471 | 65 | if not meta and filename.endswith('.meta'): | ||
472 | 66 | meta = filename | ||
473 | 67 | if not data and filename.endswith('.data'): | ||
474 | 68 | data = filename | ||
475 | 69 | if not tomb and filename.endswith('.ts'): | ||
476 | 70 | tomb = filename | ||
477 | 71 | if (filename < tomb or # any file older than tomb | ||
478 | 72 | filename < data or # any file older than data | ||
479 | 73 | (filename.endswith('.meta') and | ||
480 | 74 | filename < meta)): # old meta | ||
481 | 75 | os.unlink(join(hsh_path, filename)) | ||
482 | 76 | files.remove(filename) | ||
483 | 77 | if not files: | ||
484 | 78 | os.rmdir(hsh_path) | ||
485 | 79 | for filename in files: | ||
486 | 80 | md5.update(filename) | ||
487 | 81 | try: | ||
488 | 82 | os.rmdir(path) | ||
489 | 83 | except OSError: | ||
490 | 84 | pass | ||
491 | 85 | return md5.hexdigest() | ||
492 | 86 | |||
493 | 87 | |||
494 | 88 | def recalculate_hashes(partition_dir, suffixes, reclaim_age=ONE_WEEK): | ||
495 | 89 | """ | ||
496 | 90 | Recalculates hashes for the given suffixes in the partition and updates | ||
497 | 91 | them in the partition's hashes file. | ||
498 | 92 | |||
499 | 93 | :param partition_dir: directory of the partition in which to recalculate | ||
500 | 94 | :param suffixes: list of suffixes to recalculate | ||
501 | 95 | :param reclaim_age: age in seconds at which tombstones should be removed | ||
502 | 96 | """ | ||
503 | 97 | |||
504 | 98 | def tpool_listdir(partition_dir): | ||
505 | 99 | return dict(((suff, None) for suff in os.listdir(partition_dir) | ||
506 | 100 | if len(suff) == 3 and isdir(join(partition_dir, suff)))) | ||
507 | 101 | hashes_file = join(partition_dir, HASH_FILE) | ||
508 | 102 | with lock_path(partition_dir): | ||
509 | 103 | try: | ||
510 | 104 | with open(hashes_file, 'rb') as fp: | ||
511 | 105 | hashes = pickle.load(fp) | ||
512 | 106 | except Exception: | ||
513 | 107 | hashes = tpool.execute(tpool_listdir, partition_dir) | ||
514 | 108 | for suffix in suffixes: | ||
515 | 109 | suffix_dir = join(partition_dir, suffix) | ||
516 | 110 | if os.path.exists(suffix_dir): | ||
517 | 111 | hashes[suffix] = hash_suffix(suffix_dir, reclaim_age) | ||
518 | 112 | elif suffix in hashes: | ||
519 | 113 | del hashes[suffix] | ||
520 | 114 | with open(hashes_file + '.tmp', 'wb') as fp: | ||
521 | 115 | pickle.dump(hashes, fp, PICKLE_PROTOCOL) | ||
522 | 116 | renamer(hashes_file + '.tmp', hashes_file) | ||
523 | 117 | |||
524 | 118 | |||
525 | 119 | def invalidate_hash(suffix_dir): | ||
526 | 120 | """ | ||
527 | 121 | Invalidates the hash for a suffix_dir in the partition's hashes file. | ||
528 | 122 | |||
529 | 123 | :param suffix_dir: absolute path to suffix dir whose hash needs | ||
530 | 124 | invalidating | ||
531 | 125 | """ | ||
532 | 126 | |||
533 | 127 | suffix = os.path.basename(suffix_dir) | ||
534 | 128 | partition_dir = os.path.dirname(suffix_dir) | ||
535 | 129 | hashes_file = join(partition_dir, HASH_FILE) | ||
536 | 130 | with lock_path(partition_dir): | ||
537 | 131 | try: | ||
538 | 132 | with open(hashes_file, 'rb') as fp: | ||
539 | 133 | hashes = pickle.load(fp) | ||
540 | 134 | if suffix in hashes and not hashes[suffix]: | ||
541 | 135 | return | ||
542 | 136 | except Exception: | ||
543 | 137 | return | ||
544 | 138 | hashes[suffix] = None | ||
545 | 139 | with open(hashes_file + '.tmp', 'wb') as fp: | ||
546 | 140 | pickle.dump(hashes, fp, PICKLE_PROTOCOL) | ||
547 | 141 | renamer(hashes_file + '.tmp', hashes_file) | ||
548 | 142 | |||
549 | 143 | |||
550 | 144 | def get_hashes(partition_dir, do_listdir=True, reclaim_age=ONE_WEEK): | ||
551 | 145 | """ | ||
552 | 146 | Get a list of hashes for the suffix dir. do_listdir causes it to mistrust | ||
553 | 147 | the hash cache for suffix existence at the (unexpectedly high) cost of a | ||
554 | 148 | listdir. reclaim_age is just passed on to hash_suffix. | ||
555 | 149 | |||
556 | 150 | :param partition_dir: absolute path of partition to get hashes for | ||
557 | 151 | :param do_listdir: force existence check for all hashes in the partition | ||
558 | 152 | :param reclaim_age: age at which to remove tombstones | ||
559 | 153 | |||
560 | 154 | :returns: tuple of (number of suffix dirs hashed, dictionary of hashes) | ||
561 | 155 | """ | ||
562 | 156 | |||
563 | 157 | def tpool_listdir(hashes, partition_dir): | ||
564 | 158 | return dict(((suff, hashes.get(suff, None)) | ||
565 | 159 | for suff in os.listdir(partition_dir) | ||
566 | 160 | if len(suff) == 3 and isdir(join(partition_dir, suff)))) | ||
567 | 161 | hashed = 0 | ||
568 | 162 | hashes_file = join(partition_dir, HASH_FILE) | ||
569 | 163 | with lock_path(partition_dir): | ||
570 | 164 | modified = False | ||
571 | 165 | hashes = {} | ||
572 | 166 | try: | ||
573 | 167 | with open(hashes_file, 'rb') as fp: | ||
574 | 168 | hashes = pickle.load(fp) | ||
575 | 169 | except Exception: | ||
576 | 170 | do_listdir = True | ||
577 | 171 | if do_listdir: | ||
578 | 172 | hashes = tpool.execute(tpool_listdir, hashes, partition_dir) | ||
579 | 173 | modified = True | ||
580 | 174 | for suffix, hash_ in hashes.items(): | ||
581 | 175 | if not hash_: | ||
582 | 176 | suffix_dir = join(partition_dir, suffix) | ||
583 | 177 | if os.path.exists(suffix_dir): | ||
584 | 178 | try: | ||
585 | 179 | hashes[suffix] = hash_suffix(suffix_dir, reclaim_age) | ||
586 | 180 | hashed += 1 | ||
587 | 181 | except OSError: | ||
588 | 182 | logging.exception('Error hashing suffix') | ||
589 | 183 | hashes[suffix] = None | ||
590 | 184 | else: | ||
591 | 185 | del hashes[suffix] | ||
592 | 186 | modified = True | ||
593 | 187 | sleep() | ||
594 | 188 | if modified: | ||
595 | 189 | with open(hashes_file + '.tmp', 'wb') as fp: | ||
596 | 190 | pickle.dump(hashes, fp, PICKLE_PROTOCOL) | ||
597 | 191 | renamer(hashes_file + '.tmp', hashes_file) | ||
598 | 192 | return hashed, hashes | ||
599 | 193 | |||
600 | 194 | 41 | ||
601 | 195 | class ObjectReplicator(Daemon): | 42 | class ObjectReplicator(Daemon): |
602 | 196 | """ | 43 | """ |
603 | @@ -301,8 +148,9 @@ | |||
604 | 301 | had_any = True | 148 | had_any = True |
605 | 302 | if not had_any: | 149 | if not had_any: |
606 | 303 | return False | 150 | return False |
609 | 304 | args.append(join(rsync_module, node['device'], | 151 | objdir = job.get('segments') and SEGMENTSDIR or DATADIR |
610 | 305 | 'objects', job['partition'])) | 152 | args.append(join(rsync_module, node['device'], objdir, |
611 | 153 | job['partition'])) | ||
612 | 306 | return self._rsync(args) == 0 | 154 | return self._rsync(args) == 0 |
613 | 307 | 155 | ||
614 | 308 | def check_ring(self): | 156 | def check_ring(self): |
615 | @@ -337,12 +185,15 @@ | |||
616 | 337 | for node in job['nodes']: | 185 | for node in job['nodes']: |
617 | 338 | success = self.rsync(node, job, suffixes) | 186 | success = self.rsync(node, job, suffixes) |
618 | 339 | if success: | 187 | if success: |
619 | 188 | headers = {'Content-Length': '0'} | ||
620 | 189 | if job.get('segments'): | ||
621 | 190 | headers['X-Object-Type'] = 'segment' | ||
622 | 340 | with Timeout(self.http_timeout): | 191 | with Timeout(self.http_timeout): |
623 | 341 | http_connect(node['ip'], | 192 | http_connect(node['ip'], |
624 | 342 | node['port'], | 193 | node['port'], |
625 | 343 | node['device'], job['partition'], 'REPLICATE', | 194 | node['device'], job['partition'], 'REPLICATE', |
626 | 344 | '/' + '-'.join(suffixes), | 195 | '/' + '-'.join(suffixes), |
628 | 345 | headers={'Content-Length': '0'}).getresponse().read() | 196 | headers=headers).getresponse().read() |
629 | 346 | responses.append(success) | 197 | responses.append(success) |
630 | 347 | if not suffixes or (len(responses) == \ | 198 | if not suffixes or (len(responses) == \ |
631 | 348 | self.object_ring.replica_count and all(responses)): | 199 | self.object_ring.replica_count and all(responses)): |
632 | @@ -374,10 +225,13 @@ | |||
633 | 374 | node = next(nodes) | 225 | node = next(nodes) |
634 | 375 | attempts_left -= 1 | 226 | attempts_left -= 1 |
635 | 376 | try: | 227 | try: |
636 | 228 | headers = {'Content-Length': '0'} | ||
637 | 229 | if job.get('segments'): | ||
638 | 230 | headers['X-Object-Type'] = 'segment' | ||
639 | 377 | with Timeout(self.http_timeout): | 231 | with Timeout(self.http_timeout): |
640 | 378 | resp = http_connect(node['ip'], node['port'], | 232 | resp = http_connect(node['ip'], node['port'], |
641 | 379 | node['device'], job['partition'], 'REPLICATE', | 233 | node['device'], job['partition'], 'REPLICATE', |
643 | 380 | '', headers={'Content-Length': '0'}).getresponse() | 234 | '', headers=headers).getresponse() |
644 | 381 | if resp.status == 507: | 235 | if resp.status == 507: |
645 | 382 | self.logger.error('%s/%s responded as unmounted' % | 236 | self.logger.error('%s/%s responded as unmounted' % |
646 | 383 | (node['ip'], node['device'])) | 237 | (node['ip'], node['device'])) |
647 | @@ -397,11 +251,14 @@ | |||
648 | 397 | self.rsync(node, job, suffixes) | 251 | self.rsync(node, job, suffixes) |
649 | 398 | recalculate_hashes(job['path'], suffixes, | 252 | recalculate_hashes(job['path'], suffixes, |
650 | 399 | reclaim_age=self.reclaim_age) | 253 | reclaim_age=self.reclaim_age) |
651 | 254 | headers = {'Content-Length': '0'} | ||
652 | 255 | if job.get('segments'): | ||
653 | 256 | headers['X-Object-Type'] = 'segment' | ||
654 | 400 | with Timeout(self.http_timeout): | 257 | with Timeout(self.http_timeout): |
655 | 401 | conn = http_connect(node['ip'], node['port'], | 258 | conn = http_connect(node['ip'], node['port'], |
656 | 402 | node['device'], job['partition'], 'REPLICATE', | 259 | node['device'], job['partition'], 'REPLICATE', |
657 | 403 | '/' + '-'.join(suffixes), | 260 | '/' + '-'.join(suffixes), |
659 | 404 | headers={'Content-Length': '0'}) | 261 | headers=headers) |
660 | 405 | conn.getresponse().read() | 262 | conn.getresponse().read() |
661 | 406 | self.suffix_sync += len(suffixes) | 263 | self.suffix_sync += len(suffixes) |
662 | 407 | except (Exception, Timeout): | 264 | except (Exception, Timeout): |
663 | @@ -489,24 +346,27 @@ | |||
664 | 489 | dev for dev in self.object_ring.devs | 346 | dev for dev in self.object_ring.devs |
665 | 490 | if dev and dev['ip'] in ips and dev['port'] == self.port]: | 347 | if dev and dev['ip'] in ips and dev['port'] == self.port]: |
666 | 491 | dev_path = join(self.devices_dir, local_dev['device']) | 348 | dev_path = join(self.devices_dir, local_dev['device']) |
667 | 492 | obj_path = join(dev_path, 'objects') | ||
668 | 493 | tmp_path = join(dev_path, 'tmp') | ||
669 | 494 | if self.mount_check and not os.path.ismount(dev_path): | 349 | if self.mount_check and not os.path.ismount(dev_path): |
670 | 495 | self.logger.warn('%s is not mounted' % local_dev['device']) | 350 | self.logger.warn('%s is not mounted' % local_dev['device']) |
671 | 496 | continue | 351 | continue |
672 | 352 | tmp_path = join(dev_path, 'tmp') | ||
673 | 497 | unlink_older_than(tmp_path, time.time() - self.reclaim_age) | 353 | unlink_older_than(tmp_path, time.time() - self.reclaim_age) |
686 | 498 | if not os.path.exists(obj_path): | 354 | for objdir in (DATADIR, SEGMENTSDIR): |
687 | 499 | continue | 355 | obj_path = join(dev_path, objdir) |
688 | 500 | for partition in os.listdir(obj_path): | 356 | if os.path.exists(obj_path): |
689 | 501 | try: | 357 | for partition in os.listdir(obj_path): |
690 | 502 | nodes = [node for node in | 358 | try: |
691 | 503 | self.object_ring.get_part_nodes(int(partition)) | 359 | nodes = [node for node in |
692 | 504 | if node['id'] != local_dev['id']] | 360 | self.object_ring.get_part_nodes( |
693 | 505 | jobs.append(dict(path=join(obj_path, partition), | 361 | int(partition)) |
694 | 506 | nodes=nodes, delete=len(nodes) > 2, | 362 | if node['id'] != local_dev['id']] |
695 | 507 | partition=partition)) | 363 | jobs.append(dict( |
696 | 508 | except ValueError: | 364 | path=join(obj_path, partition), |
697 | 509 | continue | 365 | nodes=nodes, delete=len(nodes) > 2, |
698 | 366 | partition=partition, | ||
699 | 367 | segments=(objdir == SEGMENTSDIR))) | ||
700 | 368 | except ValueError: | ||
701 | 369 | continue | ||
702 | 510 | random.shuffle(jobs) | 370 | random.shuffle(jobs) |
703 | 511 | # Partititons that need to be deleted take priority | 371 | # Partititons that need to be deleted take priority |
704 | 512 | jobs.sort(key=lambda job: not job['delete']) | 372 | jobs.sort(key=lambda job: not job['delete']) |
705 | 513 | 373 | ||
706 | === modified file 'swift/obj/server.py' | |||
707 | --- swift/obj/server.py 2010-10-19 01:05:54 +0000 | |||
708 | +++ swift/obj/server.py 2010-10-19 18:43:49 +0000 | |||
709 | @@ -42,12 +42,12 @@ | |||
710 | 42 | from swift.common.constraints import check_object_creation, check_mount, \ | 42 | from swift.common.constraints import check_object_creation, check_mount, \ |
711 | 43 | check_float, check_utf8 | 43 | check_float, check_utf8 |
712 | 44 | from swift.common.exceptions import ConnectionTimeout | 44 | from swift.common.exceptions import ConnectionTimeout |
715 | 45 | from swift.obj.replicator import get_hashes, invalidate_hash, \ | 45 | from swift.obj.hashes import get_hashes, invalidate_hash, recalculate_hashes |
714 | 46 | recalculate_hashes | ||
716 | 47 | 46 | ||
717 | 48 | 47 | ||
718 | 49 | DATADIR = 'objects' | 48 | DATADIR = 'objects' |
720 | 50 | ASYNCDIR = 'async_pending' | 49 | SEGMENTSDIR = 'object_segments' |
721 | 50 | ASYNCDIR = 'object_async' | ||
722 | 51 | PICKLE_PROTOCOL = 2 | 51 | PICKLE_PROTOCOL = 2 |
723 | 52 | METADATA_KEY = 'user.swift.metadata' | 52 | METADATA_KEY = 'user.swift.metadata' |
724 | 53 | MAX_OBJECT_NAME_LENGTH = 1024 | 53 | MAX_OBJECT_NAME_LENGTH = 1024 |
725 | @@ -84,15 +84,31 @@ | |||
726 | 84 | :param obj: object name for the object | 84 | :param obj: object name for the object |
727 | 85 | :param keep_data_fp: if True, don't close the fp, otherwise close it | 85 | :param keep_data_fp: if True, don't close the fp, otherwise close it |
728 | 86 | :param disk_chunk_Size: size of chunks on file reads | 86 | :param disk_chunk_Size: size of chunks on file reads |
729 | 87 | :param segment: If set to not None, indicates which segment of an object | ||
730 | 88 | this file represents | ||
731 | 89 | :param segment_timestamp: X-Timestamp of the object's segments (set on the | ||
732 | 90 | PUT, not changed on POSTs), required if segment | ||
733 | 91 | is set to not None | ||
734 | 87 | """ | 92 | """ |
735 | 88 | 93 | ||
736 | 89 | def __init__(self, path, device, partition, account, container, obj, | 94 | def __init__(self, path, device, partition, account, container, obj, |
738 | 90 | keep_data_fp=False, disk_chunk_size=65536): | 95 | keep_data_fp=False, disk_chunk_size=65536, segment=None, |
739 | 96 | segment_timestamp=None): | ||
740 | 91 | self.disk_chunk_size = disk_chunk_size | 97 | self.disk_chunk_size = disk_chunk_size |
741 | 92 | self.name = '/' + '/'.join((account, container, obj)) | 98 | self.name = '/' + '/'.join((account, container, obj)) |
745 | 93 | name_hash = hash_path(account, container, obj) | 99 | if segment and int(segment): |
746 | 94 | self.datadir = os.path.join(path, device, | 100 | ring_obj = '%s/%s/%s' % (obj, segment_timestamp, segment) |
747 | 95 | storage_directory(DATADIR, partition, name_hash)) | 101 | else: |
748 | 102 | ring_obj = obj | ||
749 | 103 | name_hash = hash_path(account, container, ring_obj) | ||
750 | 104 | if segment is not None: | ||
751 | 105 | self.datadir = os.path.join(path, device, | ||
752 | 106 | storage_directory(SEGMENTSDIR, partition, name_hash)) | ||
753 | 107 | self.no_longer_segment_datadir = os.path.join(path, device, | ||
754 | 108 | storage_directory(DATADIR, partition, name_hash)) | ||
755 | 109 | else: | ||
756 | 110 | self.datadir = os.path.join(path, device, | ||
757 | 111 | storage_directory(DATADIR, partition, name_hash)) | ||
758 | 96 | self.tmpdir = os.path.join(path, device, 'tmp') | 112 | self.tmpdir = os.path.join(path, device, 'tmp') |
759 | 97 | self.metadata = {} | 113 | self.metadata = {} |
760 | 98 | self.meta_file = None | 114 | self.meta_file = None |
761 | @@ -195,7 +211,8 @@ | |||
762 | 195 | except OSError: | 211 | except OSError: |
763 | 196 | pass | 212 | pass |
764 | 197 | 213 | ||
766 | 198 | def put(self, fd, tmppath, metadata, extension='.data'): | 214 | def put(self, fd, tmppath, metadata, extension='.data', |
767 | 215 | no_longer_segment=False): | ||
768 | 199 | """ | 216 | """ |
769 | 200 | Finalize writing the file on disk, and renames it from the temp file to | 217 | Finalize writing the file on disk, and renames it from the temp file to |
770 | 201 | the real location. This should be called after the data has been | 218 | the real location. This should be called after the data has been |
771 | @@ -204,7 +221,10 @@ | |||
772 | 204 | :params fd: file descriptor of the temp file | 221 | :params fd: file descriptor of the temp file |
773 | 205 | :param tmppath: path to the temporary file being used | 222 | :param tmppath: path to the temporary file being used |
774 | 206 | :param metadata: dictionary of metada to be written | 223 | :param metadata: dictionary of metada to be written |
776 | 207 | :param extention: extension to be used when making the file | 224 | :param extension: extension to be used when making the file |
777 | 225 | :param no_longer_segment: Set to True if this was originally an object | ||
778 | 226 | segment but no longer is (case with chunked transfer encoding when | ||
779 | 227 | the object ends up less than the segment size) | ||
780 | 208 | """ | 228 | """ |
781 | 209 | metadata['name'] = self.name | 229 | metadata['name'] = self.name |
782 | 210 | timestamp = normalize_timestamp(metadata['X-Timestamp']) | 230 | timestamp = normalize_timestamp(metadata['X-Timestamp']) |
783 | @@ -217,6 +237,8 @@ | |||
784 | 217 | if 'Content-Length' in metadata: | 237 | if 'Content-Length' in metadata: |
785 | 218 | drop_buffer_cache(fd, 0, int(metadata['Content-Length'])) | 238 | drop_buffer_cache(fd, 0, int(metadata['Content-Length'])) |
786 | 219 | os.fsync(fd) | 239 | os.fsync(fd) |
787 | 240 | if no_longer_segment: | ||
788 | 241 | self.datadir = self.no_longer_segment_datadir | ||
789 | 220 | invalidate_hash(os.path.dirname(self.datadir)) | 242 | invalidate_hash(os.path.dirname(self.datadir)) |
790 | 221 | renamer(tmppath, os.path.join(self.datadir, timestamp + extension)) | 243 | renamer(tmppath, os.path.join(self.datadir, timestamp + extension)) |
791 | 222 | self.metadata = metadata | 244 | self.metadata = metadata |
792 | @@ -355,7 +377,9 @@ | |||
793 | 355 | if error_response: | 377 | if error_response: |
794 | 356 | return error_response | 378 | return error_response |
795 | 357 | file = DiskFile(self.devices, device, partition, account, container, | 379 | file = DiskFile(self.devices, device, partition, account, container, |
797 | 358 | obj, disk_chunk_size=self.disk_chunk_size) | 380 | obj, disk_chunk_size=self.disk_chunk_size, |
798 | 381 | segment=request.headers.get('x-object-segment'), | ||
799 | 382 | segment_timestamp=request.headers['x-timestamp']) | ||
800 | 359 | upload_expiration = time.time() + self.max_upload_time | 383 | upload_expiration = time.time() + self.max_upload_time |
801 | 360 | etag = md5() | 384 | etag = md5() |
802 | 361 | upload_size = 0 | 385 | upload_size = 0 |
803 | @@ -397,17 +421,32 @@ | |||
804 | 397 | if 'content-encoding' in request.headers: | 421 | if 'content-encoding' in request.headers: |
805 | 398 | metadata['Content-Encoding'] = \ | 422 | metadata['Content-Encoding'] = \ |
806 | 399 | request.headers['Content-Encoding'] | 423 | request.headers['Content-Encoding'] |
808 | 400 | file.put(fd, tmppath, metadata) | 424 | if 'x-object-type' in request.headers: |
809 | 425 | metadata['X-Object-Type'] = request.headers['x-object-type'] | ||
810 | 426 | if 'x-object-segment' in request.headers: | ||
811 | 427 | metadata['X-Object-Segment'] = \ | ||
812 | 428 | request.headers['x-object-segment'] | ||
813 | 429 | no_longer_segment = False | ||
814 | 430 | if 'x-object-segment-if-length' in request.headers and \ | ||
815 | 431 | int(request.headers['x-object-segment-if-length']) != \ | ||
816 | 432 | os.fstat(fd).st_size: | ||
817 | 433 | del metadata['X-Object-Type'] | ||
818 | 434 | del metadata['X-Object-Segment'] | ||
819 | 435 | no_longer_segment = True | ||
820 | 436 | file.put(fd, tmppath, metadata, | ||
821 | 437 | no_longer_segment=no_longer_segment) | ||
822 | 401 | file.unlinkold(metadata['X-Timestamp']) | 438 | file.unlinkold(metadata['X-Timestamp']) |
832 | 402 | self.container_update('PUT', account, container, obj, request.headers, | 439 | if 'X-Object-Segment' not in file.metadata: |
833 | 403 | {'x-size': file.metadata['Content-Length'], | 440 | self.container_update('PUT', account, container, obj, |
834 | 404 | 'x-content-type': file.metadata['Content-Type'], | 441 | request.headers, |
835 | 405 | 'x-timestamp': file.metadata['X-Timestamp'], | 442 | {'x-size': request.headers.get('x-object-length', |
836 | 406 | 'x-etag': file.metadata['ETag'], | 443 | file.metadata['Content-Length']), |
837 | 407 | 'x-cf-trans-id': request.headers.get('x-cf-trans-id', '-')}, | 444 | 'x-content-type': file.metadata['Content-Type'], |
838 | 408 | device) | 445 | 'x-timestamp': file.metadata['X-Timestamp'], |
839 | 409 | resp = HTTPCreated(request=request, etag=etag) | 446 | 'x-etag': file.metadata['ETag'], |
840 | 410 | return resp | 447 | 'x-cf-trans-id': request.headers.get('x-cf-trans-id', '-')}, |
841 | 448 | device) | ||
842 | 449 | return HTTPCreated(request=request, etag=etag) | ||
843 | 411 | 450 | ||
844 | 412 | def GET(self, request): | 451 | def GET(self, request): |
845 | 413 | """Handle HTTP GET requests for the Swift Object Server.""" | 452 | """Handle HTTP GET requests for the Swift Object Server.""" |
846 | @@ -420,7 +459,9 @@ | |||
847 | 420 | if self.mount_check and not check_mount(self.devices, device): | 459 | if self.mount_check and not check_mount(self.devices, device): |
848 | 421 | return Response(status='507 %s is not mounted' % device) | 460 | return Response(status='507 %s is not mounted' % device) |
849 | 422 | file = DiskFile(self.devices, device, partition, account, container, | 461 | file = DiskFile(self.devices, device, partition, account, container, |
851 | 423 | obj, keep_data_fp=True, disk_chunk_size=self.disk_chunk_size) | 462 | obj, keep_data_fp=True, disk_chunk_size=self.disk_chunk_size, |
852 | 463 | segment=request.headers.get('x-object-segment'), | ||
853 | 464 | segment_timestamp=request.headers.get('x-object-segment-timestamp')) | ||
854 | 424 | if file.is_deleted(): | 465 | if file.is_deleted(): |
855 | 425 | if request.headers.get('if-match') == '*': | 466 | if request.headers.get('if-match') == '*': |
856 | 426 | return HTTPPreconditionFailed(request=request) | 467 | return HTTPPreconditionFailed(request=request) |
857 | @@ -460,7 +501,8 @@ | |||
858 | 460 | 'application/octet-stream'), app_iter=file, | 501 | 'application/octet-stream'), app_iter=file, |
859 | 461 | request=request, conditional_response=True) | 502 | request=request, conditional_response=True) |
860 | 462 | for key, value in file.metadata.iteritems(): | 503 | for key, value in file.metadata.iteritems(): |
862 | 463 | if key.lower().startswith('x-object-meta-'): | 504 | if key.lower().startswith('x-object-meta-') or \ |
863 | 505 | key.lower() in ('x-object-type', 'x-object-segment'): | ||
864 | 464 | response.headers[key] = value | 506 | response.headers[key] = value |
865 | 465 | response.etag = file.metadata['ETag'] | 507 | response.etag = file.metadata['ETag'] |
866 | 466 | response.last_modified = float(file.metadata['X-Timestamp']) | 508 | response.last_modified = float(file.metadata['X-Timestamp']) |
867 | @@ -482,13 +524,16 @@ | |||
868 | 482 | if self.mount_check and not check_mount(self.devices, device): | 524 | if self.mount_check and not check_mount(self.devices, device): |
869 | 483 | return Response(status='507 %s is not mounted' % device) | 525 | return Response(status='507 %s is not mounted' % device) |
870 | 484 | file = DiskFile(self.devices, device, partition, account, container, | 526 | file = DiskFile(self.devices, device, partition, account, container, |
872 | 485 | obj, disk_chunk_size=self.disk_chunk_size) | 527 | obj, disk_chunk_size=self.disk_chunk_size, |
873 | 528 | segment=request.headers.get('x-object-segment'), | ||
874 | 529 | segment_timestamp=request.headers.get('x-object-segment-timestamp')) | ||
875 | 486 | if file.is_deleted(): | 530 | if file.is_deleted(): |
876 | 487 | return HTTPNotFound(request=request) | 531 | return HTTPNotFound(request=request) |
877 | 488 | response = Response(content_type=file.metadata['Content-Type'], | 532 | response = Response(content_type=file.metadata['Content-Type'], |
878 | 489 | request=request, conditional_response=True) | 533 | request=request, conditional_response=True) |
879 | 490 | for key, value in file.metadata.iteritems(): | 534 | for key, value in file.metadata.iteritems(): |
881 | 491 | if key.lower().startswith('x-object-meta-'): | 535 | if key.lower().startswith('x-object-meta-') or \ |
882 | 536 | key.lower() in ('x-object-type', 'x-object-segment'): | ||
883 | 492 | response.headers[key] = value | 537 | response.headers[key] = value |
884 | 493 | response.etag = file.metadata['ETag'] | 538 | response.etag = file.metadata['ETag'] |
885 | 494 | response.last_modified = float(file.metadata['X-Timestamp']) | 539 | response.last_modified = float(file.metadata['X-Timestamp']) |
886 | @@ -513,7 +558,9 @@ | |||
887 | 513 | return Response(status='507 %s is not mounted' % device) | 558 | return Response(status='507 %s is not mounted' % device) |
888 | 514 | response_class = HTTPNoContent | 559 | response_class = HTTPNoContent |
889 | 515 | file = DiskFile(self.devices, device, partition, account, container, | 560 | file = DiskFile(self.devices, device, partition, account, container, |
891 | 516 | obj, disk_chunk_size=self.disk_chunk_size) | 561 | obj, disk_chunk_size=self.disk_chunk_size, |
892 | 562 | segment=request.headers.get('x-object-segment'), | ||
893 | 563 | segment_timestamp=request.headers.get('x-object-segment-timestamp')) | ||
894 | 517 | if file.is_deleted(): | 564 | if file.is_deleted(): |
895 | 518 | response_class = HTTPNotFound | 565 | response_class = HTTPNotFound |
896 | 519 | metadata = { | 566 | metadata = { |
897 | @@ -522,10 +569,11 @@ | |||
898 | 522 | with file.mkstemp() as (fd, tmppath): | 569 | with file.mkstemp() as (fd, tmppath): |
899 | 523 | file.put(fd, tmppath, metadata, extension='.ts') | 570 | file.put(fd, tmppath, metadata, extension='.ts') |
900 | 524 | file.unlinkold(metadata['X-Timestamp']) | 571 | file.unlinkold(metadata['X-Timestamp']) |
905 | 525 | self.container_update('DELETE', account, container, obj, | 572 | if 'x-object-segment' not in request.headers: |
906 | 526 | request.headers, {'x-timestamp': metadata['X-Timestamp'], | 573 | self.container_update('DELETE', account, container, obj, |
907 | 527 | 'x-cf-trans-id': request.headers.get('x-cf-trans-id', '-')}, | 574 | request.headers, {'x-timestamp': metadata['X-Timestamp'], |
908 | 528 | device) | 575 | 'x-cf-trans-id': request.headers.get('x-cf-trans-id', '-')}, |
909 | 576 | device) | ||
910 | 529 | resp = response_class(request=request) | 577 | resp = response_class(request=request) |
911 | 530 | return resp | 578 | return resp |
912 | 531 | 579 | ||
913 | @@ -538,7 +586,10 @@ | |||
914 | 538 | unquote(request.path), 2, 3, True) | 586 | unquote(request.path), 2, 3, True) |
915 | 539 | if self.mount_check and not check_mount(self.devices, device): | 587 | if self.mount_check and not check_mount(self.devices, device): |
916 | 540 | return Response(status='507 %s is not mounted' % device) | 588 | return Response(status='507 %s is not mounted' % device) |
918 | 541 | path = os.path.join(self.devices, device, DATADIR, partition) | 589 | if request.headers.get('x-object-type') == 'segment': |
919 | 590 | path = os.path.join(self.devices, device, SEGMENTSDIR, partition) | ||
920 | 591 | else: | ||
921 | 592 | path = os.path.join(self.devices, device, DATADIR, partition) | ||
922 | 542 | if not os.path.exists(path): | 593 | if not os.path.exists(path): |
923 | 543 | mkdirs(path) | 594 | mkdirs(path) |
924 | 544 | if suffix: | 595 | if suffix: |
925 | 545 | 596 | ||
926 | === modified file 'swift/proxy/server.py' | |||
927 | --- swift/proxy/server.py 2010-10-15 15:07:19 +0000 | |||
928 | +++ swift/proxy/server.py 2010-10-19 18:43:49 +0000 | |||
929 | @@ -14,21 +14,25 @@ | |||
930 | 14 | # limitations under the License. | 14 | # limitations under the License. |
931 | 15 | 15 | ||
932 | 16 | from __future__ import with_statement | 16 | from __future__ import with_statement |
933 | 17 | try: | ||
934 | 18 | import simplejson as json | ||
935 | 19 | except ImportError: | ||
936 | 20 | import json | ||
937 | 17 | import mimetypes | 21 | import mimetypes |
938 | 18 | import os | 22 | import os |
939 | 19 | import time | 23 | import time |
940 | 20 | import traceback | 24 | import traceback |
941 | 21 | from ConfigParser import ConfigParser | 25 | from ConfigParser import ConfigParser |
942 | 26 | from hashlib import md5 | ||
943 | 22 | from urllib import unquote, quote | 27 | from urllib import unquote, quote |
944 | 23 | import uuid | 28 | import uuid |
945 | 24 | import functools | 29 | import functools |
946 | 25 | 30 | ||
947 | 26 | from eventlet.timeout import Timeout | 31 | from eventlet.timeout import Timeout |
953 | 27 | from webob.exc import HTTPBadRequest, HTTPMethodNotAllowed, \ | 32 | from webob.exc import HTTPBadRequest, HTTPCreated, HTTPInternalServerError, \ |
954 | 28 | HTTPNotFound, HTTPPreconditionFailed, \ | 33 | HTTPMethodNotAllowed, HTTPNotFound, HTTPPreconditionFailed, \ |
955 | 29 | HTTPRequestTimeout, HTTPServiceUnavailable, \ | 34 | HTTPRequestEntityTooLarge, HTTPRequestTimeout, HTTPServerError, \ |
956 | 30 | HTTPUnprocessableEntity, HTTPRequestEntityTooLarge, HTTPServerError, \ | 35 | HTTPServiceUnavailable, HTTPUnprocessableEntity, status_map |
952 | 31 | status_map | ||
957 | 32 | from webob import Request, Response | 36 | from webob import Request, Response |
958 | 33 | 37 | ||
959 | 34 | from swift.common.ring import Ring | 38 | from swift.common.ring import Ring |
960 | @@ -37,7 +41,7 @@ | |||
961 | 37 | from swift.common.bufferedhttp import http_connect | 41 | from swift.common.bufferedhttp import http_connect |
962 | 38 | from swift.common.constraints import check_metadata, check_object_creation, \ | 42 | from swift.common.constraints import check_metadata, check_object_creation, \ |
963 | 39 | check_utf8, MAX_ACCOUNT_NAME_LENGTH, MAX_CONTAINER_NAME_LENGTH, \ | 43 | check_utf8, MAX_ACCOUNT_NAME_LENGTH, MAX_CONTAINER_NAME_LENGTH, \ |
965 | 40 | MAX_FILE_SIZE | 44 | MAX_FILE_SIZE, SEGMENT_SIZE |
966 | 41 | from swift.common.exceptions import ChunkReadTimeout, \ | 45 | from swift.common.exceptions import ChunkReadTimeout, \ |
967 | 42 | ChunkWriteTimeout, ConnectionTimeout | 46 | ChunkWriteTimeout, ConnectionTimeout |
968 | 43 | 47 | ||
969 | @@ -89,6 +93,135 @@ | |||
970 | 89 | return wrapped | 93 | return wrapped |
971 | 90 | 94 | ||
972 | 91 | 95 | ||
973 | 96 | class SegmentedIterable(object): | ||
974 | 97 | """ | ||
975 | 98 | Iterable that returns the object contents for a segmented object in Swift. | ||
976 | 99 | |||
977 | 100 | In addition to these params, you can also set the `response` attr just | ||
978 | 101 | after creating the SegmentedIterable and it will update the response's | ||
979 | 102 | `bytes_transferred` value. Be sure to set the `bytes_transferred` value to | ||
980 | 103 | 0 beforehand. | ||
981 | 104 | |||
982 | 105 | :param controller: The ObjectController instance to work with. | ||
983 | 106 | :param content_length: The total length of the object. | ||
984 | 107 | :param segment_size: The length of each segment (except perhaps the last) | ||
985 | 108 | of the object. | ||
986 | 109 | :param timestamp: The X-Timestamp of the object's segments (set on the PUT, | ||
987 | 110 | not changed on the POSTs). | ||
988 | 111 | """ | ||
989 | 112 | |||
990 | 113 | def __init__(self, controller, content_length, segment_size, timestamp): | ||
991 | 114 | self.controller = controller | ||
992 | 115 | self.content_length = content_length | ||
993 | 116 | self.segment_size = segment_size | ||
994 | 117 | self.timestamp = timestamp | ||
995 | 118 | self.position = 0 | ||
996 | 119 | self.segment = -1 | ||
997 | 120 | self.segment_iter = None | ||
998 | 121 | self.response = None | ||
999 | 122 | |||
1000 | 123 | def load_next_segment(self): | ||
1001 | 124 | """ Loads the self.segment_iter with the next segment's contents. """ | ||
1002 | 125 | self.segment += 1 | ||
1003 | 126 | if self.segment: | ||
1004 | 127 | ring_object_name = '%s/%s/%s' % (self.controller.object_name, | ||
1005 | 128 | self.timestamp, self.segment) | ||
1006 | 129 | else: | ||
1007 | 130 | ring_object_name = self.controller.object_name | ||
1008 | 131 | partition, nodes = self.controller.app.object_ring.get_nodes( | ||
1009 | 132 | self.controller.account_name, self.controller.container_name, | ||
1010 | 133 | ring_object_name) | ||
1011 | 134 | path = '/%s/%s/%s' % (self.controller.account_name, | ||
1012 | 135 | self.controller.container_name, self.controller.object_name) | ||
1013 | 136 | req = Request.blank(path, headers={'X-Object-Segment': self.segment, | ||
1014 | 137 | 'X-Object-Segment-Timestamp': self.timestamp}) | ||
1015 | 138 | resp = self.controller.GETorHEAD_base(req, 'Object', | ||
1016 | 139 | partition, self.controller.iter_nodes(partition, nodes, | ||
1017 | 140 | self.controller.app.object_ring), path, | ||
1018 | 141 | self.controller.app.object_ring.replica_count) | ||
1019 | 142 | if resp.status_int // 100 != 2: | ||
1020 | 143 | raise Exception( | ||
1021 | 144 | 'Could not load segment %s of %s' % (self.segment, path)) | ||
1022 | 145 | self.segment_iter = resp.app_iter | ||
1023 | 146 | |||
1024 | 147 | def __iter__(self): | ||
1025 | 148 | """ Standard iterator function that returns the object's contents. """ | ||
1026 | 149 | while self.position < self.content_length: | ||
1027 | 150 | if not self.segment_iter: | ||
1028 | 151 | self.load_next_segment() | ||
1029 | 152 | while True: | ||
1030 | 153 | with ChunkReadTimeout(self.controller.app.node_timeout): | ||
1031 | 154 | try: | ||
1032 | 155 | chunk = self.segment_iter.next() | ||
1033 | 156 | break | ||
1034 | 157 | except StopIteration: | ||
1035 | 158 | self.load_next_segment() | ||
1036 | 159 | if self.position + len(chunk) > self.content_length: | ||
1037 | 160 | chunk = chunk[:self.content_length - self.position] | ||
1038 | 161 | self.position += len(chunk) | ||
1039 | 162 | if self.response: | ||
1040 | 163 | self.response.bytes_transferred += len(chunk) | ||
1041 | 164 | yield chunk | ||
1042 | 165 | |||
1043 | 166 | def app_iter_range(self, start, stop): | ||
1044 | 167 | """ | ||
1045 | 168 | Non-standard iterator function for use with Webob in serving Range | ||
1046 | 169 | requests more quickly. | ||
1047 | 170 | |||
1048 | 171 | TODO: | ||
1049 | 172 | |||
1050 | 173 | This currently helps on speed by jumping to the proper segment to start | ||
1051 | 174 | with (and ending without reading the trailing segments, but that | ||
1052 | 175 | already happened technically with __iter__). | ||
1053 | 176 | |||
1054 | 177 | But, what it does not do yet is issue a Range request with the first | ||
1055 | 178 | segment to allow the object server to seek to the segment start point. | ||
1056 | 179 | |||
1057 | 180 | Instead, it just reads and throws away all leading segment data. Since | ||
1058 | 181 | segments are 5G by default, it'll have to transfer the whole 5G from | ||
1059 | 182 | the object server to the proxy server even if it only needs the last | ||
1060 | 183 | byte. In practice, this should happen fairly quickly relative to how | ||
1061 | 184 | long requests take for these very large files; but it's still wasteful. | ||
1062 | 185 | |||
1063 | 186 | Anyway, it shouldn't be too hard to implement, I just have other things | ||
1064 | 187 | to work out first. | ||
1065 | 188 | |||
1066 | 189 | :param start: The first byte (zero-based) to return. | ||
1067 | 190 | :param stop: The last byte (zero-based) to return. | ||
1068 | 191 | """ | ||
1069 | 192 | if start: | ||
1070 | 193 | self.segment = (start / self.segment_size) - 1 | ||
1071 | 194 | self.load_next_segment() | ||
1072 | 195 | self.position = self.segment * self.segment_size | ||
1073 | 196 | segment_start = start - (self.segment * self.segment_size) | ||
1074 | 197 | while segment_start: | ||
1075 | 198 | with ChunkReadTimeout(self.controller.app.node_timeout): | ||
1076 | 199 | chunk = self.segment_iter.next() | ||
1077 | 200 | self.position += len(chunk) | ||
1078 | 201 | if len(chunk) > segment_start: | ||
1079 | 202 | chunk = chunk[segment_start:] | ||
1080 | 203 | if self.response: | ||
1081 | 204 | self.response.bytes_transferred += len(chunk) | ||
1082 | 205 | yield chunk | ||
1083 | 206 | segment_start = 0 | ||
1084 | 207 | else: | ||
1085 | 208 | segment_start -= len(chunk) | ||
1086 | 209 | if stop is not None: | ||
1087 | 210 | length = stop - start | ||
1088 | 211 | else: | ||
1089 | 212 | length = None | ||
1090 | 213 | for chunk in self: | ||
1091 | 214 | if length is not None: | ||
1092 | 215 | length -= len(chunk) | ||
1093 | 216 | if length < 0: | ||
1094 | 217 | # Chop off the extra: | ||
1095 | 218 | if self.response: | ||
1096 | 219 | self.response.bytes_transferred -= length | ||
1097 | 220 | yield chunk[:length] | ||
1098 | 221 | break | ||
1099 | 222 | yield chunk | ||
1100 | 223 | |||
1101 | 224 | |||
1102 | 92 | def get_container_memcache_key(account, container): | 225 | def get_container_memcache_key(account, container): |
1103 | 93 | path = '/%s/%s' % (account, container) | 226 | path = '/%s/%s' % (account, container) |
1104 | 94 | return 'container%s' % path | 227 | return 'container%s' % path |
1105 | @@ -518,11 +651,56 @@ | |||
1106 | 518 | aresp = req.environ['swift.authorize'](req) | 651 | aresp = req.environ['swift.authorize'](req) |
1107 | 519 | if aresp: | 652 | if aresp: |
1108 | 520 | return aresp | 653 | return aresp |
1109 | 654 | # This is bit confusing, so an explanation: | ||
1110 | 655 | # * First we attempt the GET/HEAD normally, as this is the usual case. | ||
1111 | 656 | # * If the request was a Range request and gave us a 416 Unsatisfiable | ||
1112 | 657 | # response, we might be trying to do an invalid Range on a manifest | ||
1113 | 658 | # object, so we try again with no Range. | ||
1114 | 659 | # * If it turns out we have a manifest object, and we had a Range | ||
1115 | 660 | # request originally that actually succeeded or we had a HEAD | ||
1116 | 661 | # request, we have to do the request again as a full GET because | ||
1117 | 662 | # we'll need the whole manifest. | ||
1118 | 663 | # * Finally, if we had a manifest object, we pass it and the request | ||
1119 | 664 | # off to GETorHEAD_segmented; otherwise we just return the response. | ||
1120 | 521 | partition, nodes = self.app.object_ring.get_nodes( | 665 | partition, nodes = self.app.object_ring.get_nodes( |
1121 | 522 | self.account_name, self.container_name, self.object_name) | 666 | self.account_name, self.container_name, self.object_name) |
1123 | 523 | return self.GETorHEAD_base(req, 'Object', partition, | 667 | resp = mresp = self.GETorHEAD_base(req, 'Object', partition, |
1124 | 668 | self.iter_nodes(partition, nodes, self.app.object_ring), | ||
1125 | 669 | req.path_info, self.app.object_ring.replica_count) | ||
1126 | 670 | range_value = None | ||
1127 | 671 | if mresp.status_int == 416: | ||
1128 | 672 | range_value = req.range | ||
1129 | 673 | req.range = None | ||
1130 | 674 | mresp = self.GETorHEAD_base(req, 'Object', partition, | ||
1131 | 524 | self.iter_nodes(partition, nodes, self.app.object_ring), | 675 | self.iter_nodes(partition, nodes, self.app.object_ring), |
1132 | 525 | req.path_info, self.app.object_ring.replica_count) | 676 | req.path_info, self.app.object_ring.replica_count) |
1133 | 677 | if mresp.status_int // 100 != 2: | ||
1134 | 678 | return resp | ||
1135 | 679 | if 'x-object-type' in mresp.headers: | ||
1136 | 680 | if mresp.headers['x-object-type'] == 'manifest': | ||
1137 | 681 | if req.method == 'HEAD': | ||
1138 | 682 | req.method = 'GET' | ||
1139 | 683 | mresp = self.GETorHEAD_base(req, 'Object', partition, | ||
1140 | 684 | self.iter_nodes(partition, nodes, | ||
1141 | 685 | self.app.object_ring), req.path_info, | ||
1142 | 686 | self.app.object_ring.replica_count) | ||
1143 | 687 | if mresp.status_int // 100 != 2: | ||
1144 | 688 | return mresp | ||
1145 | 689 | req.method = 'HEAD' | ||
1146 | 690 | elif req.range: | ||
1147 | 691 | range_value = req.range | ||
1148 | 692 | req.range = None | ||
1149 | 693 | mresp = self.GETorHEAD_base(req, 'Object', partition, | ||
1150 | 694 | self.iter_nodes(partition, nodes, | ||
1151 | 695 | self.app.object_ring), req.path_info, | ||
1152 | 696 | self.app.object_ring.replica_count) | ||
1153 | 697 | if mresp.status_int // 100 != 2: | ||
1154 | 698 | return mresp | ||
1155 | 699 | if range_value: | ||
1156 | 700 | req.range = range_value | ||
1157 | 701 | return self.GETorHEAD_segmented(req, mresp) | ||
1158 | 702 | return HTTPNotFound(request=req) | ||
1159 | 703 | return resp | ||
1160 | 526 | 704 | ||
1161 | 527 | @public | 705 | @public |
1162 | 528 | @delay_denial | 706 | @delay_denial |
1163 | @@ -536,6 +714,48 @@ | |||
1164 | 536 | """Handler for HTTP HEAD requests.""" | 714 | """Handler for HTTP HEAD requests.""" |
1165 | 537 | return self.GETorHEAD(req) | 715 | return self.GETorHEAD(req) |
1166 | 538 | 716 | ||
1167 | 717 | def GETorHEAD_segmented(self, req, mresp): | ||
1168 | 718 | """ | ||
1169 | 719 | Performs a GET for a segmented object. | ||
1170 | 720 | |||
1171 | 721 | :param req: The webob.Request to process. | ||
1172 | 722 | :param mresp: The webob.Response for the original manifest request. | ||
1173 | 723 | :returns: webob.Response object. | ||
1174 | 724 | """ | ||
1175 | 725 | manifest = json.loads(''.join(mresp.app_iter)) | ||
1176 | 726 | # Ah, the fun of JSONing strs and getting unicodes back. We | ||
1177 | 727 | # reencode to UTF8 to ensure crap doesn't blow up everywhere | ||
1178 | 728 | # else. | ||
1179 | 729 | keys_to_encode = [] | ||
1180 | 730 | for k, v in manifest.iteritems(): | ||
1181 | 731 | if isinstance(k, unicode): | ||
1182 | 732 | keys_to_encode.append(k) | ||
1183 | 733 | if isinstance(v, unicode): | ||
1184 | 734 | manifest[k] = v.encode('utf8') | ||
1185 | 735 | for k in keys_to_encode: | ||
1186 | 736 | v = manifest[k] | ||
1187 | 737 | del manifest[k] | ||
1188 | 738 | manifest[k.encode('utf8')] = v | ||
1189 | 739 | content_length = int(manifest['content-length']) | ||
1190 | 740 | segment_size = int(manifest['x-segment-size']) | ||
1191 | 741 | headers = dict(mresp.headers) | ||
1192 | 742 | headers.update(manifest) | ||
1193 | 743 | del headers['x-segment-size'] | ||
1194 | 744 | resp = Response(app_iter=SegmentedIterable(self, content_length, | ||
1195 | 745 | segment_size, manifest['x-timestamp']), headers=headers, | ||
1196 | 746 | request=req, conditional_response=True) | ||
1197 | 747 | resp.headers['etag'] = manifest['etag'].strip('"') | ||
1198 | 748 | resp.last_modified = mresp.last_modified | ||
1199 | 749 | resp.content_length = int(manifest['content-length']) | ||
1200 | 750 | resp.content_type = manifest['content-type'] | ||
1201 | 751 | if 'content-encoding' in manifest: | ||
1202 | 752 | resp.content_encoding = manifest['content-encoding'] | ||
1203 | 753 | cresp = req.get_response(resp) | ||
1204 | 754 | # Needed for SegmentedIterable to update bytes_transferred | ||
1205 | 755 | cresp.bytes_transferred = 0 | ||
1206 | 756 | resp.app_iter.response = cresp | ||
1207 | 757 | return cresp | ||
1208 | 758 | |||
1209 | 539 | @public | 759 | @public |
1210 | 540 | @delay_denial | 760 | @delay_denial |
1211 | 541 | def POST(self, req): | 761 | def POST(self, req): |
1212 | @@ -652,11 +872,47 @@ | |||
1213 | 652 | if k.lower().startswith('x-object-meta-'): | 872 | if k.lower().startswith('x-object-meta-'): |
1214 | 653 | new_req.headers[k] = v | 873 | new_req.headers[k] = v |
1215 | 654 | req = new_req | 874 | req = new_req |
1216 | 875 | if req.headers.get('transfer-encoding') == 'chunked' or \ | ||
1217 | 876 | req.content_length > SEGMENT_SIZE: | ||
1218 | 877 | resp = self.PUT_segmented_object(req, data_source, partition, | ||
1219 | 878 | nodes, container_partition, containers) | ||
1220 | 879 | else: | ||
1221 | 880 | resp = self.PUT_whole_object(req, data_source, partition, nodes, | ||
1222 | 881 | container_partition, containers) | ||
1223 | 882 | if 'x-copy-from' in req.headers: | ||
1224 | 883 | resp.headers['X-Copied-From'] = req.headers['x-copy-from'] | ||
1225 | 884 | for k, v in req.headers.items(): | ||
1226 | 885 | if k.lower().startswith('x-object-meta-'): | ||
1227 | 886 | resp.headers[k] = v | ||
1228 | 887 | resp.last_modified = float(req.headers['X-Timestamp']) | ||
1229 | 888 | return resp | ||
1230 | 889 | |||
1231 | 890 | def PUT_whole_object(self, req, data_source, partition, nodes, | ||
1232 | 891 | container_partition=None, containers=None): | ||
1233 | 892 | """ | ||
1234 | 893 | Performs a PUT for a whole object (one with a content-length <= | ||
1235 | 894 | SEGMENT_SIZE). | ||
1236 | 895 | |||
1237 | 896 | :param req: The webob.Request to process. | ||
1238 | 897 | :param data_source: An iterator providing the data to store. | ||
1239 | 898 | :param partition: The object ring partition the object falls on. | ||
1240 | 899 | :param nodes: The object ring nodes the object falls on. | ||
1241 | 900 | :param container_partition: The container ring partition the container | ||
1242 | 901 | for the object falls on, None if the | ||
1243 | 902 | container is not to be updated. | ||
1244 | 903 | :param containers: The container ring nodes the container for the | ||
1245 | 904 | object falls on, None if the container is not to be | ||
1246 | 905 | updated. | ||
1247 | 906 | :returns: webob.Response object. | ||
1248 | 907 | """ | ||
1249 | 908 | conns = [] | ||
1250 | 909 | update_containers = containers is not None | ||
1251 | 655 | for node in self.iter_nodes(partition, nodes, self.app.object_ring): | 910 | for node in self.iter_nodes(partition, nodes, self.app.object_ring): |
1256 | 656 | container = containers.pop() | 911 | if update_containers: |
1257 | 657 | req.headers['X-Container-Host'] = '%(ip)s:%(port)s' % container | 912 | container = containers.pop() |
1258 | 658 | req.headers['X-Container-Partition'] = container_partition | 913 | req.headers['X-Container-Host'] = '%(ip)s:%(port)s' % container |
1259 | 659 | req.headers['X-Container-Device'] = container['device'] | 914 | req.headers['X-Container-Partition'] = container_partition |
1260 | 915 | req.headers['X-Container-Device'] = container['device'] | ||
1261 | 660 | req.headers['Expect'] = '100-continue' | 916 | req.headers['Expect'] = '100-continue' |
1262 | 661 | resp = conn = None | 917 | resp = conn = None |
1263 | 662 | if not self.error_limited(node): | 918 | if not self.error_limited(node): |
1264 | @@ -674,12 +930,14 @@ | |||
1265 | 674 | if conn and resp: | 930 | if conn and resp: |
1266 | 675 | if resp.status == 100: | 931 | if resp.status == 100: |
1267 | 676 | conns.append(conn) | 932 | conns.append(conn) |
1269 | 677 | if not containers: | 933 | if (update_containers and not containers) or \ |
1270 | 934 | len(conns) == len(nodes): | ||
1271 | 678 | break | 935 | break |
1272 | 679 | continue | 936 | continue |
1273 | 680 | elif resp.status == 507: | 937 | elif resp.status == 507: |
1274 | 681 | self.error_limit(node) | 938 | self.error_limit(node) |
1276 | 682 | containers.insert(0, container) | 939 | if update_containers: |
1277 | 940 | containers.insert(0, container) | ||
1278 | 683 | if len(conns) <= len(nodes) / 2: | 941 | if len(conns) <= len(nodes) / 2: |
1279 | 684 | self.app.logger.error( | 942 | self.app.logger.error( |
1280 | 685 | 'Object PUT returning 503, %s/%s required connections, ' | 943 | 'Object PUT returning 503, %s/%s required connections, ' |
1281 | @@ -765,15 +1023,123 @@ | |||
1282 | 765 | statuses.append(503) | 1023 | statuses.append(503) |
1283 | 766 | reasons.append('') | 1024 | reasons.append('') |
1284 | 767 | bodies.append('') | 1025 | bodies.append('') |
1286 | 768 | resp = self.best_response(req, statuses, reasons, bodies, 'Object PUT', | 1026 | return self.best_response(req, statuses, reasons, bodies, 'Object PUT', |
1287 | 769 | etag=etag) | 1027 | etag=etag) |
1295 | 770 | if 'x-copy-from' in req.headers: | 1028 | |
1296 | 771 | resp.headers['X-Copied-From'] = req.headers['x-copy-from'] | 1029 | def PUT_segmented_object(self, req, data_source, partition, nodes, |
1297 | 772 | for k, v in req.headers.items(): | 1030 | container_partition, containers): |
1298 | 773 | if k.lower().startswith('x-object-meta-'): | 1031 | """ |
1299 | 774 | resp.headers[k] = v | 1032 | Performs a PUT for a segmented object (one with a content-length > |
1300 | 775 | resp.last_modified = float(req.headers['X-Timestamp']) | 1033 | SEGMENT_SIZE). |
1301 | 776 | return resp | 1034 | |
1302 | 1035 | :param req: The webob.Request to process. | ||
1303 | 1036 | :param data_source: An iterator providing the data to store. | ||
1304 | 1037 | :param partition: The object ring partition the object falls on. | ||
1305 | 1038 | :param nodes: The object ring nodes the object falls on. | ||
1306 | 1039 | :param container_partition: The container ring partition the container | ||
1307 | 1040 | for the object falls on. | ||
1308 | 1041 | :param containers: The container ring nodes the container for the | ||
1309 | 1042 | object falls on. | ||
1310 | 1043 | :returns: webob.Response object. | ||
1311 | 1044 | """ | ||
1312 | 1045 | req.bytes_transferred = 0 | ||
1313 | 1046 | leftover_chunk = [None] | ||
1314 | 1047 | etag = md5() | ||
1315 | 1048 | def segment_iter(): | ||
1316 | 1049 | amount_given = 0 | ||
1317 | 1050 | while amount_given < SEGMENT_SIZE: | ||
1318 | 1051 | if leftover_chunk[0]: | ||
1319 | 1052 | chunk = leftover_chunk[0] | ||
1320 | 1053 | leftover_chunk[0] = None | ||
1321 | 1054 | else: | ||
1322 | 1055 | with ChunkReadTimeout(self.app.client_timeout): | ||
1323 | 1056 | chunk = data_source.next() | ||
1324 | 1057 | req.bytes_transferred += len(chunk) | ||
1325 | 1058 | etag.update(chunk) | ||
1326 | 1059 | if amount_given + len(chunk) > SEGMENT_SIZE: | ||
1327 | 1060 | yield chunk[:SEGMENT_SIZE - amount_given] | ||
1328 | 1061 | leftover_chunk[0] = chunk[SEGMENT_SIZE - amount_given:] | ||
1329 | 1062 | amount_given = SEGMENT_SIZE | ||
1330 | 1063 | else: | ||
1331 | 1064 | yield chunk | ||
1332 | 1065 | amount_given += len(chunk) | ||
1333 | 1066 | def segment_iter_iter(): | ||
1334 | 1067 | while True: | ||
1335 | 1068 | if not leftover_chunk[0]: | ||
1336 | 1069 | with ChunkReadTimeout(self.app.client_timeout): | ||
1337 | 1070 | leftover_chunk[0] = data_source.next() | ||
1338 | 1071 | req.bytes_transferred += len(leftover_chunk[0]) | ||
1339 | 1072 | etag.update(leftover_chunk[0]) | ||
1340 | 1073 | yield segment_iter() | ||
1341 | 1074 | segment_number = 0 | ||
1342 | 1075 | chunked = req.headers.get('transfer-encoding') == 'chunked' | ||
1343 | 1076 | if not chunked: | ||
1344 | 1077 | amount_left = req.content_length | ||
1345 | 1078 | headers = {'X-Timestamp': req.headers['X-Timestamp'], | ||
1346 | 1079 | 'Content-Type': req.headers['content-type'], | ||
1347 | 1080 | 'X-Object-Type': 'segment'} | ||
1348 | 1081 | for segment_source in segment_iter_iter(): | ||
1349 | 1082 | if chunked: | ||
1350 | 1083 | headers['Transfer-Encoding'] = 'chunked' | ||
1351 | 1084 | if segment_number == 0: | ||
1352 | 1085 | headers['X-Object-Segment-If-Length'] = SEGMENT_SIZE | ||
1353 | 1086 | elif amount_left > SEGMENT_SIZE: | ||
1354 | 1087 | headers['Content-Length'] = SEGMENT_SIZE | ||
1355 | 1088 | else: | ||
1356 | 1089 | headers['Content-Length'] = amount_left | ||
1357 | 1090 | headers['X-Object-Segment'] = segment_number | ||
1358 | 1091 | segment_req = Request.blank(req.path_info, | ||
1359 | 1092 | environ={'REQUEST_METHOD': 'PUT'}, headers=headers) | ||
1360 | 1093 | if 'X-Object-Segment-If-Length' in headers: | ||
1361 | 1094 | del headers['X-Object-Segment-If-Length'] | ||
1362 | 1095 | if segment_number: | ||
1363 | 1096 | ring_object_name = '%s/%s/%s' % (self.object_name, | ||
1364 | 1097 | req.headers['x-timestamp'], segment_number) | ||
1365 | 1098 | else: | ||
1366 | 1099 | ring_object_name = self.object_name | ||
1367 | 1100 | segment_partition, segment_nodes = self.app.object_ring.get_nodes( | ||
1368 | 1101 | self.account_name, self.container_name, ring_object_name) | ||
1369 | 1102 | resp = self.PUT_whole_object(segment_req, segment_source, | ||
1370 | 1103 | segment_partition, segment_nodes) | ||
1371 | 1104 | if resp.status_int // 100 == 4: | ||
1372 | 1105 | return resp | ||
1373 | 1106 | elif resp.status_int // 100 != 2: | ||
1374 | 1107 | return HTTPServiceUnavailable(request=req, | ||
1375 | 1108 | body='Unable to complete very large file operation.') | ||
1376 | 1109 | if segment_number == 0 and req.bytes_transferred < SEGMENT_SIZE: | ||
1377 | 1110 | return HTTPCreated(request=req, etag=etag.hexdigest()) | ||
1378 | 1111 | if not chunked: | ||
1379 | 1112 | amount_left -= SEGMENT_SIZE | ||
1380 | 1113 | segment_number += 1 | ||
1381 | 1114 | etag = etag.hexdigest() | ||
1382 | 1115 | if 'etag' in req.headers and req.headers['etag'].lower() != etag: | ||
1383 | 1116 | return HTTPUnprocessableEntity(request=req) | ||
1384 | 1117 | manifest = {'x-timestamp': req.headers['x-timestamp'], | ||
1385 | 1118 | 'content-length': req.bytes_transferred, | ||
1386 | 1119 | 'content-type': req.headers['content-type'], | ||
1387 | 1120 | 'x-segment-size': SEGMENT_SIZE, | ||
1388 | 1121 | 'etag': etag} | ||
1389 | 1122 | if 'content-encoding' in req.headers: | ||
1390 | 1123 | manifest['content-encoding'] = req.headers['content-encoding'] | ||
1391 | 1124 | manifest = json.dumps(manifest) | ||
1392 | 1125 | headers = {'X-Timestamp': req.headers['X-Timestamp'], | ||
1393 | 1126 | 'Content-Type': req.headers['content-type'], | ||
1394 | 1127 | 'Content-Length': len(manifest), | ||
1395 | 1128 | 'X-Object-Type': 'manifest', | ||
1396 | 1129 | 'X-Object-Length': req.bytes_transferred} | ||
1397 | 1130 | headers.update(i for i in req.headers.iteritems() | ||
1398 | 1131 | if i[0].lower().startswith('x-object-meta-') and len(i[0]) > 14) | ||
1399 | 1132 | manifest_req = Request.blank(req.path_info, | ||
1400 | 1133 | environ={'REQUEST_METHOD': 'PUT'}, body=manifest, headers=headers) | ||
1401 | 1134 | manifest_source = iter(lambda: | ||
1402 | 1135 | manifest_req.body_file.read(self.app.client_chunk_size), '') | ||
1403 | 1136 | resp = self.PUT_whole_object(manifest_req, manifest_source, partition, | ||
1404 | 1137 | nodes, container_partition=container_partition, | ||
1405 | 1138 | containers=containers) | ||
1406 | 1139 | if resp.status_int // 100 != 2: | ||
1407 | 1140 | return HTTPServiceUnavailable(request=req, | ||
1408 | 1141 | body='Unable to complete very large file operation.') | ||
1409 | 1142 | return HTTPCreated(request=req, etag=etag) | ||
1410 | 777 | 1143 | ||
1411 | 778 | @public | 1144 | @public |
1412 | 779 | @delay_denial | 1145 | @delay_denial |
1413 | 780 | 1146 | ||
1414 | === modified file 'test/unit/__init__.py' | |||
1415 | --- test/unit/__init__.py 2010-07-29 20:06:01 +0000 | |||
1416 | +++ test/unit/__init__.py 2010-10-19 18:43:49 +0000 | |||
1417 | @@ -9,6 +9,8 @@ | |||
1418 | 9 | crlfs = 0 | 9 | crlfs = 0 |
1419 | 10 | while crlfs < 2: | 10 | while crlfs < 2: |
1420 | 11 | c = fd.read(1) | 11 | c = fd.read(1) |
1421 | 12 | if not len(c): | ||
1422 | 13 | raise Exception('Never read 2crlfs; got %s' % repr(rv)) | ||
1423 | 12 | rv = rv + c | 14 | rv = rv + c |
1424 | 13 | if c == '\r' and lc != '\n': | 15 | if c == '\r' and lc != '\n': |
1425 | 14 | crlfs = 0 | 16 | crlfs = 0 |
1426 | 15 | 17 | ||
1427 | === added file 'test/unit/obj/test_hashes.py' | |||
1428 | --- test/unit/obj/test_hashes.py 1970-01-01 00:00:00 +0000 | |||
1429 | +++ test/unit/obj/test_hashes.py 2010-10-19 18:43:49 +0000 | |||
1430 | @@ -0,0 +1,28 @@ | |||
1431 | 1 | # Copyright (c) 2010 OpenStack, LLC. | ||
1432 | 2 | # | ||
1433 | 3 | # Licensed under the Apache License, Version 2.0 (the "License"); | ||
1434 | 4 | # you may not use this file except in compliance with the License. | ||
1435 | 5 | # You may obtain a copy of the License at | ||
1436 | 6 | # | ||
1437 | 7 | # http://www.apache.org/licenses/LICENSE-2.0 | ||
1438 | 8 | # | ||
1439 | 9 | # Unless required by applicable law or agreed to in writing, software | ||
1440 | 10 | # distributed under the License is distributed on an "AS IS" BASIS, | ||
1441 | 11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or | ||
1442 | 12 | # implied. | ||
1443 | 13 | # See the License for the specific language governing permissions and | ||
1444 | 14 | # limitations under the License. | ||
1445 | 15 | |||
1446 | 16 | # TODO: Tests | ||
1447 | 17 | |||
1448 | 18 | import unittest | ||
1449 | 19 | from swift.obj import hashes | ||
1450 | 20 | |||
1451 | 21 | class TestHashes(unittest.TestCase): | ||
1452 | 22 | |||
1453 | 23 | def test_placeholder(self): | ||
1454 | 24 | pass | ||
1455 | 25 | |||
1456 | 26 | |||
1457 | 27 | if __name__ == '__main__': | ||
1458 | 28 | unittest.main() | ||
1459 | 0 | 29 | ||
1460 | === added file 'test/unit/obj/test_hashes.py.moved' | |||
1461 | --- test/unit/obj/test_hashes.py.moved 1970-01-01 00:00:00 +0000 | |||
1462 | +++ test/unit/obj/test_hashes.py.moved 2010-10-19 18:43:49 +0000 | |||
1463 | @@ -0,0 +1,28 @@ | |||
1464 | 1 | # Copyright (c) 2010 OpenStack, LLC. | ||
1465 | 2 | # | ||
1466 | 3 | # Licensed under the Apache License, Version 2.0 (the "License"); | ||
1467 | 4 | # you may not use this file except in compliance with the License. | ||
1468 | 5 | # You may obtain a copy of the License at | ||
1469 | 6 | # | ||
1470 | 7 | # http://www.apache.org/licenses/LICENSE-2.0 | ||
1471 | 8 | # | ||
1472 | 9 | # Unless required by applicable law or agreed to in writing, software | ||
1473 | 10 | # distributed under the License is distributed on an "AS IS" BASIS, | ||
1474 | 11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or | ||
1475 | 12 | # implied. | ||
1476 | 13 | # See the License for the specific language governing permissions and | ||
1477 | 14 | # limitations under the License. | ||
1478 | 15 | |||
1479 | 16 | # TODO: Tests | ||
1480 | 17 | |||
1481 | 18 | import unittest | ||
1482 | 19 | from swift.obj import hashes | ||
1483 | 20 | |||
1484 | 21 | class TestHashes(unittest.TestCase): | ||
1485 | 22 | |||
1486 | 23 | def test_placeholder(self): | ||
1487 | 24 | pass | ||
1488 | 25 | |||
1489 | 26 | |||
1490 | 27 | if __name__ == '__main__': | ||
1491 | 28 | unittest.main() |
I'm going to have to redo this puppy. It got to unwieldy so I'm splitting into separate branches.