Merge lp:~fo0bar/swoffsite/generator into lp:swoffsite

[Ryan Finnie]

Set full_listing=False, revert r16, change SWIFT_BATCHSIZE to 1000

- full_listing=False allows the rest of swift_walk's generator
  functionality to work, as full_listing=True internally walks the entire
  container before returning the full data set, causing memory exhaustion
  on large containers.
- Change SWIFT_BATCHSIZE to 1000 to avoid Swift server timeouts. Comments
  updated to point out why >10000 is futile.
- Reverting r16 allows s3_walk to be used as a generator, fixing crashes
  on large S3 buckets.

[Stuart Bishop (stub)]

Yup.

Reducing SWIFT_BATCHSIZE probably does nothing, making an order of magnitude more calls that each return an order of magnitude less information each. I think if pauses cause timeouts between batches, those pauses are due to waiting for large file uploads to complete to S3 (at least on the librarian deploy, it is not uncommon to be blocked on 3 or 4 GB sized files uploading, providing enough disk space to continue, because I was unable to stream them directly from PGP). But we should certainly go with what you have been testing.

review: Approve

[🤖 Canonical IS Merge Bot (canonical-is-mergebot)]

This merge proposal is being monitored by mergebot. Change the status to Approved to merge.

[🤖 Canonical IS Merge Bot (canonical-is-mergebot)]

Change successfully merged at revision 43

=== modified file 'swoffsite/mirror.py'
--- swoffsite/mirror.py	2016-11-18 01:48:41 +0000
+++ swoffsite/mirror.py	2019-05-23 03:25:03 +0000
@@ -37,7 +37,10 @@
 from .streams import SwiftStream, HashStream
 
 
-SWIFT_BATCHSIZE = 10000  # Swift container listing batch size.
+# Swift container listing batch size.
+# Note that setting this to >10000 is futile, as Swift has
+# a hard internal limit of 10000.
+SWIFT_BATCHSIZE = 1000
 
 CHUNK_SIZE = 512*1024
 
@@ -103,7 +106,7 @@
             headers, listing = con.get_container(container,
                                                  marker=marker,
                                                  limit=SWIFT_BATCHSIZE,
-                                                 full_listing=True)
+                                                 full_listing=False)
             if not listing:
                 break
             marker = listing[-1]['name']
@@ -124,12 +127,12 @@
     if b is None:
         con.create_bucket(bucket)
         return
-    for prefix in list(b.list(delimiter='/')):
+    for prefix in b.list(delimiter='/'):
         if not match(include_globs, exclude_globs, prefix.name[:-1]):
             log.debug('Skipping S3 prefix {}'.format(prefix.name))
         else:
             log.info('Listing {} in S3'.format(prefix.name[:-1]))
-            for s3_key in list(b.list(prefix=prefix.name)):
+            for s3_key in b.list(prefix=prefix.name):
                 container, name, _ = parse_s3_name(s3_key.name)
                 if match(include_globs, exclude_globs, container, name):
                     log.debug('S3 {} matched'.format(s3_key.name))