Merge lp:~stub/launchpad/feed-end-at into lp:launchpad

[Stuart Bishop]

[stub, r=wgrant][noqa] Add an option to librarian-feed-swift.py allowing us to not process newer files.

Add an option to librarian-feed-swift.py allowing us to not process newer files.

The goal is to ensure that we have at least two copies of Librarian data in two different systems (disk & swift, or swift & offsite backup). By running two jobs in sequence, the first moving all disk files into Swift older than $cutoff, and the second copying all disk files into Swift younger than $cutoff, we create a window where data exists both on disk and in Swift. Our other systems can ensure that the data is mirrored from Swift to offsite during that window.

The alternative approach would be a single run, and providing a new option to not remove files younger than $cutoff. This would be better, but less of a quick and easy fix.

[William Grant (wgrant)]

review: Approve (code)

=== modified file 'cronscripts/librarian-feed-swift.py'
--- cronscripts/librarian-feed-swift.py	2014-12-16 09:20:15 +0000
+++ cronscripts/librarian-feed-swift.py	2015-05-18 13:30:47 +0000
@@ -35,11 +35,16 @@
         self.parser.add_option(
             "--start-since", action="store", dest='start_since',
             default=None, metavar="INTERVAL",
-            help="Migrate files starting from INTERVAL (PostgreSQL syntax)")
+            help="Migrate files older than INTERVAL (PostgreSQL syntax)")
         self.parser.add_option(
             "-e", "--end", action="store", type=int, default=None,
             dest="end", metavar="CONTENT_ID",
             help="Migrate files up to and including CONTENT_ID")
+        self.parser.add_option(
+            "--end-at", action="store", dest='end_at',
+            default=None, metavar="INTERVAL",
+            help="Don't migrate files older than INTERVAL "
+                 "(PostgreSQL syntax)")
 
     def main(self):
         if self.options.rename and self.options.remove:
@@ -58,6 +63,13 @@
                     - CAST(%s AS INTERVAL)
                 """, (unicode(self.options.start_since),)).get_one()[0]
 
+        if self.options.end_at:
+            self.options.end = ISlaveStore(LibraryFileContent).execute("""
+                SELECT MAX(id) FROM LibraryFileContent
+                WHERE datecreated < current_timestamp at time zone 'UTC'
+                    - CAST(%s AS INTERVAL)
+                """, (unicode(self.options.end_at),)).get_one()[0]
+
         if self.options.ids and (self.options.start or self.options.end):
             self.parser.error(
                 "Cannot specify both individual file(s) and range")