Merge lp:~julian-edwards/launchpad/log-parser-bug-680463 into lp:launchpad

[Julian Edwards]

When parsing Apache logs, figure out the length of gzip log files without having to read them in to memory in their entirety which can cause the process to crash when it eats all the available memory.

Figure out the length of gzip log files without having to read them in to memory.

The existing code tries to read the uncompressed contents of a gzip file into memory in their entirety. This makes the PPA log parser blow up quite horribly as the log files are very large.

Use existing test with:
bin/test -cvv test_apachelogparser Test_get_fd_and_file_size

QA Plan
-------

I have got a copy of the production log files that cause the crash on dogfood. Running with the fix allows the processing to continue with no increased memory usage as observed in "top".

[Gavin Panella (allenap)]

review: Approve

=== modified file 'lib/lp/services/apachelogparser/base.py'
--- lib/lp/services/apachelogparser/base.py	2010-11-17 23:20:07 +0000
+++ lib/lp/services/apachelogparser/base.py	2010-11-26 17:47:08 +0000
@@ -64,13 +64,14 @@
     file_path points to a gzipped file.
     """
     if file_path.endswith('.gz'):
+        # The last 4 bytes of the file contains the uncompressed file's
+        # size, modulo 2**32.  This code is somewhat stolen from the gzip
+        # module in Python 2.6.
         fd = gzip.open(file_path)
-        # There doesn't seem to be a better way of figuring out the
-        # uncompressed size of a file, so we'll read the whole file here.
-        file_size = len(fd.read())
-        # Seek back to the beginning of the file as if we had just opened
-        # it.
-        fd.seek(0)
+        fd.fileobj.seek(-4, os.SEEK_END)
+        isize = gzip.read32(fd.fileobj)   # may exceed 2GB
+        file_size = isize & 0xffffffffL
+        fd.fileobj.seek(0)
     else:
         fd = open(file_path)
         file_size = os.path.getsize(file_path)