Merge lp:~vorlon/ubuntu-archive-tools/python3-process-removals-and-better-error-handling into lp:ubuntu-archive-tools

[Steve Langasek]

ftp-master.debian.org continues to sometimes generate broken date fields in its removals log, and the workaround in process-removals for this doesn't work. Observed output looks like:

[Date: sh: 0: getcwd() failed: No such file or directory
Sat, 06 May 2017 08:45:30 +0000] [ftpmaster: Chris Lamb]

This branch improves handling of this error by grabbing the date after the line break, instead of silently (and unnecessarily) ignoring it.

In an earlier iteration of this branch I found it useful to have a reasonably featureful strptime(), which requires python3; so this also moves the script to python3 in the process.

[Adam Conrad (adconrad)]

LGTM, if it's tested, merge away.

review: Approve

=== modified file 'process-removals'
--- process-removals	2016-11-15 13:40:36 +0000
+++ process-removals	2017-06-15 20:58:12 +0000
@@ -1,4 +1,4 @@
-#!/usr/bin/python
+#!/usr/bin/python3
 
 # Parse removals.txt file
 # Copyright (C) 2004, 2005, 2009, 2010, 2011, 2012  Canonical Ltd.
@@ -69,7 +69,7 @@
     state = "first separator"
     removal = {}
     for line in removals_file:
-        line = line.strip()
+        line = line.strip().decode('UTF-8')
         # skip over spurious empty lines
         if not line and state in ("first separtor", "next separator", "date"):
             continue
@@ -88,24 +88,21 @@
         elif state in ("date", "next separator"):
             try:
                 (date, ftpmaster, unused) = line.split("]")
+                date = date.replace("[Date: ", "")
                 state = "removed from"
-                removal["ftpmaster"] = ftpmaster.replace("[ftpmaster: ", "")
             except:
-                date = line
                 state = "broken date"
-            date = date.replace("[Date: ", "")
-            # Strip the timezone which python's strptime() can't handle
-            # [rock on]
-            date = ' '.join(date.split()[:-1])
-            removal["date"] = time.mktime(time.strptime(
-                date, "%a, %d %b %Y %H:%M:%S"))
-        # ] [ftpmaster:
+        # Sat, 06 May 2017 08:45:30 +0000] [ftpmaster: Chris Lamb]
         elif state == "broken date":
-            (unused1, ftpmaster, unused2) = line.split("]")
-            removal["ftpmaster"] = ftpmaster.replace("] [ftpmaster: ", "")
+            (date, ftpmaster, unused2) = line.split("]")
             state = "removed from"
         # Removed the following packages from unstable:
         elif state == "removed from":
+            # complete processing of the date from the preceding line
+            removal["date"] = time.mktime(time.strptime(
+                date, "%a, %d %b %Y %H:%M:%S %z"))
+            removal["ftpmaster"] = ftpmaster.replace("] [ftpmaster: ", "")
+
             prefix = "Removed the following packages from "
             if not line.startswith(prefix):
                 raise RuntimeError(
@@ -182,7 +179,7 @@
         subp = subprocess.Popen(command, stdout=subprocess.PIPE)
         line = None
         for line in subp.communicate()[0].splitlines():
-            line = line.rstrip("\n")
+            line = line.decode('UTF-8').rstrip("\n")
             if line == "No reverse dependencies found":
                 line = None
             else:
@@ -415,7 +412,7 @@
     read_sources()
 
     if options.filename:
-        removals_file = open(options.filename)
+        removals_file = open(options.filename, "rb")
     else:
         removals_file = fetch_removals_file(options)
     parse_removals_file(options, removals_file)