Merge lp:~vila/udd/analyze_log into lp:udd

[Vincent Ladeuil]

Handle stdin by allowing a '-' log path.

While working on jubany, I generally have a

 tail -F logs/driver/progress_log

running.

But there is too much noise there.

This patch add support for a '-' log file argument to accept stdin.

It also documents a bit better how the whole script is designed.

[Vincent Ladeuil (vila)]

... sent without the conclusion:

Being able to pipe the tail to 'analyze_log.py -' gives a better view of what is happening on the importer.

Looking to log file itself is still needed in some cases but overall, the filtered view better match my monitoring use case.

[John A Meinel (jameinel)]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 09/05/2011 10:08 AM, Vincent Ladeuil wrote:
> The proposal to merge lp:~vila/udd/analyze_log into lp:udd has been updated.
>
> Commit Message changed to:
>
> Handle stdin by allowing a '-' log path.
>
> For more details, see:
> https://code.launchpad.net/~vila/udd/analyze_log/+merge/74056

Looks reasonable to me.

 merge: approve

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk5l1TIACgkQJdeBCYSNAAONcACgsbM2p8aPGFEXYZ995TWHufq6
D+cAn34u5o7qdHh2izJSSzG2ps9/3BWC
=Ianu
-----END PGP SIGNATURE-----

review: Approve

=== modified file 'analyze_log.py'
--- analyze_log.py	2011-07-16 09:41:49 +0000
+++ analyze_log.py	2011-09-05 07:49:22 +0000
@@ -2,6 +2,14 @@
 """Analyzes the package importer log files collecting various historical data.
 
 This may need to be tweaked as the logger usage evolves.
+
+Roughly speaking, it works by matching lines in a log file, attributing them to
+an ActorParser which see().
+
+There are currently 3 actors all writing to the log: ControllerParser,
+DriverParser and PackageParser.
+
+The Log object defines which actors are expected and which lines they claim.
 """
 
 import datetime
@@ -138,6 +146,11 @@
 
 
 class ControllerParser(ActorParser):
+    """ImportController parser active during one mass_import run.
+
+    It could happen that a log file encompass several runs or starts/ends
+    during a run.
+    """
 
     def __init__(self):
         super(ControllerParser, self).__init__('controller')
@@ -176,6 +189,11 @@
 
 
 class DriverParser(ActorParser):
+    """ImportDriver parser active during one mass_import run.
+
+    It could happen that a log file encompass several runs or starts/ends
+    during a run.
+    """
 
     def __init__(self):
         super(DriverParser, self).__init__('driver')
@@ -221,6 +239,7 @@
 
 
 class PackageParser(ActorParser):
+    """Parser active during one package import for a given package"""
 
     def __init__(self, name):
         super(PackageParser, self).__init__(name)
@@ -244,9 +263,12 @@
 
 class Log(object):
 
-    def __init__(self, path):
+    def __init__(self, path, f=None):
         self.path = path
-        self.stream = iter(open(self.path))
+        if f is None:
+            self.stream = open(self.path)
+        else:
+            self.stream = f
         self.out = sys.stdout
         self.lineno = 0
         self.pushed_back_line = None
@@ -347,7 +369,9 @@
             line = self.pushed_back_line
             self.pushed_back_line = None
         else:
-            line = self.stream.next()
+            line = self.stream.readline()
+            if not line:
+                raise StopIteration
             self.lineno +=1
         return self.lineno, line
 
@@ -466,11 +490,15 @@
         exit(1)
 
     log_path = args[1]
-    if not os.path.exists(log_path):
-        print '%s does not exist' % (log_path,)
-        exit(1)
+    if log_path == '-':
+        f = sys.stdin
+    else:
+        f = None
+        if not os.path.exists(log_path):
+            print '%s does not exist' % (log_path,)
+            exit(1)
 
-    log = Log(log_path)
+    log = Log(log_path, f)
     log.parse()