Merge lp:~vorlon/ubuntu-archive-tools/python3-process-removals-and-better-error-handling into lp:ubuntu-archive-tools

Proposed by Steve Langasek on 2017-06-15
Status: Merged
Merged at revision: 1105
Proposed branch: lp:~vorlon/ubuntu-archive-tools/python3-process-removals-and-better-error-handling
Merge into: lp:ubuntu-archive-tools
Diff against target: 69 lines (+12/-15)
1 file modified
process-removals (+12/-15)
To merge this branch: bzr merge lp:~vorlon/ubuntu-archive-tools/python3-process-removals-and-better-error-handling
Reviewer Review Type Date Requested Status
Adam Conrad 2017-06-15 Approve on 2017-07-06
Review via email: mp+325773@code.launchpad.net

Description of the Change

ftp-master.debian.org continues to sometimes generate broken date fields in its removals log, and the workaround in process-removals for this doesn't work. Observed output looks like:

[Date: sh: 0: getcwd() failed: No such file or directory
Sat, 06 May 2017 08:45:30 +0000] [ftpmaster: Chris Lamb]

This branch improves handling of this error by grabbing the date after the line break, instead of silently (and unnecessarily) ignoring it.

In an earlier iteration of this branch I found it useful to have a reasonably featureful strptime(), which requires python3; so this also moves the script to python3 in the process.

To post a comment you must log in.
Adam Conrad (adconrad) wrote :

LGTM, if it's tested, merge away.

review: Approve

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1=== modified file 'process-removals'
2--- process-removals 2016-11-15 13:40:36 +0000
3+++ process-removals 2017-06-15 20:58:12 +0000
4@@ -1,4 +1,4 @@
5-#!/usr/bin/python
6+#!/usr/bin/python3
7
8 # Parse removals.txt file
9 # Copyright (C) 2004, 2005, 2009, 2010, 2011, 2012 Canonical Ltd.
10@@ -69,7 +69,7 @@
11 state = "first separator"
12 removal = {}
13 for line in removals_file:
14- line = line.strip()
15+ line = line.strip().decode('UTF-8')
16 # skip over spurious empty lines
17 if not line and state in ("first separtor", "next separator", "date"):
18 continue
19@@ -88,24 +88,21 @@
20 elif state in ("date", "next separator"):
21 try:
22 (date, ftpmaster, unused) = line.split("]")
23+ date = date.replace("[Date: ", "")
24 state = "removed from"
25- removal["ftpmaster"] = ftpmaster.replace("[ftpmaster: ", "")
26 except:
27- date = line
28 state = "broken date"
29- date = date.replace("[Date: ", "")
30- # Strip the timezone which python's strptime() can't handle
31- # [rock on]
32- date = ' '.join(date.split()[:-1])
33- removal["date"] = time.mktime(time.strptime(
34- date, "%a, %d %b %Y %H:%M:%S"))
35- # ] [ftpmaster:
36+ # Sat, 06 May 2017 08:45:30 +0000] [ftpmaster: Chris Lamb]
37 elif state == "broken date":
38- (unused1, ftpmaster, unused2) = line.split("]")
39- removal["ftpmaster"] = ftpmaster.replace("] [ftpmaster: ", "")
40+ (date, ftpmaster, unused2) = line.split("]")
41 state = "removed from"
42 # Removed the following packages from unstable:
43 elif state == "removed from":
44+ # complete processing of the date from the preceding line
45+ removal["date"] = time.mktime(time.strptime(
46+ date, "%a, %d %b %Y %H:%M:%S %z"))
47+ removal["ftpmaster"] = ftpmaster.replace("] [ftpmaster: ", "")
48+
49 prefix = "Removed the following packages from "
50 if not line.startswith(prefix):
51 raise RuntimeError(
52@@ -182,7 +179,7 @@
53 subp = subprocess.Popen(command, stdout=subprocess.PIPE)
54 line = None
55 for line in subp.communicate()[0].splitlines():
56- line = line.rstrip("\n")
57+ line = line.decode('UTF-8').rstrip("\n")
58 if line == "No reverse dependencies found":
59 line = None
60 else:
61@@ -415,7 +412,7 @@
62 read_sources()
63
64 if options.filename:
65- removals_file = open(options.filename)
66+ removals_file = open(options.filename, "rb")
67 else:
68 removals_file = fetch_removals_file(options)
69 parse_removals_file(options, removals_file)

Subscribers

People subscribed via source and target branches