Launchpad itself

Apache log parser crashes out on the PPA logs

Bug #590766 reported by Julian Edwards
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Launchpad itself
Fix Released
High
Michael Nelson
Launchpad itself 10.07

Bug Description

2010-06-07 12:23:58 ERROR Error (Unable to parse: 0] "GET /ubuntu-langpack/ubuntu/dists/edgy/main/binary-i386/Packages.gz HTTP/1.1" 200 20268 "-" "Debian APT-HTTP/1.3" with the ^(\S*) (\S*) (\S*) (\[[^\]]+\]) \"([^"\\]*(?:\\.[^"\\]*)*)\" (\S*) (\S*) \"([^"\\]*(?:\\.[^"\\]*)*)\" \"([^"\\]*(?:\\.[^"\\]*)*)\"$ regular expression) while parsing "0] "GET /ubuntu-langpack/ubuntu/dists/edgy/main/binary-i386/Packages.gz HTTP/1.1" 200 20268 "-" "Debian APT-HTTP/1.3"

This is terminal, it won't get past this error on subsequent invocations.

Related branches

lp:~michael.nelson/launchpad/588288-log-parser-dont-read-entire-file-qa
Abel Deuring (community): Approve (code)
Diff: 205 lines (+54/-22)
5 files modified
lib/canonical/config/schema-lazr.conf (+2/-2)
lib/canonical/launchpad/scripts/tests/test_librarian_apache_log_parser.py (+7/-3)
lib/lp/services/apachelogparser/base.py (+9/-4)
lib/lp/services/apachelogparser/tests/apache-log-files/launchpadlibrarian.net.access-log (+1/-0)
lib/lp/services/apachelogparser/tests/test_apachelogparser.py (+35/-13)
Revision history for this message
Julian Edwards (julian-edwards) wrote :

I should also say that this is during testing on Dogfood and is 100% reproducible.

Revision history for this message
Julian Edwards (julian-edwards) wrote :

Here's the line that I think it's choking on:

10.211.37.29 - - [02/Jul/2007:18:54:24 +0100] "GET /ubuntu-langpack/ubuntu/dists/edgy/main/binary-i386/Packages.gz HTTP/1.1" 200 50 "-" "Debian APT-HTTP/1.3"

Revision history for this message
Michael Nelson (michael.nelson) wrote :

It may be that the new code in devel (bug 588288), which updates the parse_line function to read one line at a time, is returning the incorrect number of parsed_bytes. I had a brief look and didn't see why this might be the case (as both lines returned by fd.readlines() and fd.readline() include the '\n'.

It certainly looks as though it hasn't got the start of a line (ie. '0] "GET...), but the return values are tested in:
lib/lp/services/apachelogparser/tests/test_apachelogparser.py

Revision history for this message
Gary Poster (gary) wrote :

bigjools: gary_poster: I think it's happened because the offset it's using is out of sync with the file

Changed in launchpad-foundations:
status: New → Triaged
importance: Undecided → High
Michael Nelson (michael.nelson)
Changed in launchpad-foundations:
assignee: nobody → Michael Nelson (michael.nelson)
Revision history for this message
Michael Nelson (michael.nelson) wrote :

I've qa'd the linked branch on dogfood for the librarian apache log file parser and the ppa apache log file parser. By default they will read as many unread lines that exist in each file, but the config option can be used to limit this.

Changed in launchpad-foundations:
status: Triaged → In Progress
Michael Nelson (michael.nelson)
Changed in launchpad-foundations:
status: In Progress → Fix Committed
tags: added: qa-ok
Revision history for this message
Ursula Junque (ursinha) wrote : Bug fixed by a commit

Fixed in stable r11035 <http://bazaar.launchpad.net/~launchpad-pqm/launchpad/stable/revision/11035>.

Changed in launchpad-foundations:
milestone: none → 10.06
tags: added: qa-needstesting
removed: qa-ok
Ursula Junque (ursinha)
tags: added: qa-ok
removed: qa-needstesting
Michael Nelson (michael.nelson)
Changed in launchpad-foundations:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.