Merge lp:~cjwatson/swoffsite/tz-aware-last-modified-comparison into lp:swoffsite

[Colin Watson]

Do timezone-aware comparisons of last_modified from Swift.

Swift returns last_modified without a timezone indication, but RadosGW returns it with a trailing 'Z', so iso8601.parse_date parses it into a timezone-aware datetime which cannot be compared directly to the naive datetime returned by datetime.utcnow. Handle this difference: we now parse last_modified into an aware datetime in both cases and compare it only with other aware datetimes.

[🤖 Canonical IS Merge Bot (canonical-is-mergebot)]

This merge proposal is being monitored by mergebot. Change the status to Approved to merge.

[Gareth Woolridge (moon127)]

LGTM

review: Approve

[🤖 Canonical IS Merge Bot (canonical-is-mergebot)]

Change successfully merged at revision 47

=== modified file 'swoffsite/mirror.py'
--- swoffsite/mirror.py	2021-07-09 14:45:12 +0000
+++ swoffsite/mirror.py	2021-07-28 09:25:41 +0000
@@ -19,7 +19,7 @@
 __all__ = ['s3_walk', 'swift_walk', 'mirror_swift_to_s3']
 
 from cStringIO import StringIO
-from datetime import datetime
+from datetime import datetime, timedelta, tzinfo
 import fnmatch
 import multiprocessing
 import os.path
@@ -45,6 +45,23 @@
 CHUNK_SIZE = 512*1024
 
 
+# In Python 3 we could just use datetime.timezone.utc, but since this is
+# still Python 2 we have to do it ourselves.
+class UTC(tzinfo):
+
+    def utcoffset(self, dt):
+        return timedelta(0)
+
+    def tzname(self, dt):
+        return 'UTC'
+
+    def dst(self, dt):
+        return timedelta(0)
+
+
+utc = UTC()
+
+
 def get_s3_name(container, name, md5):
     assert '/' not in container, 'Container name must not contain / char'
     return '{}/{}.{}.gpg'.format(container, name, md5)
@@ -467,9 +484,8 @@
             # Object found in Swift that isn't in S3.
             log.debug('{!r} in Swift but not S3'.format(swift_cmp))
             if direction == 's3':
-                lm = iso8601.parse_date(next_swift_obj[1]['last_modified'],
-                                        None)
-                if not recent or datetime.utcnow() > lm + recent:
+                lm = iso8601.parse_date(next_swift_obj[1]['last_modified'])
+                if not recent or datetime.now(utc) > lm + recent:
                     num_new_files += 1
                     pool.do(to_s3_task, dry_run, lock, bucket, gpg_key,
                             next_swift_obj,