Merge ~kotodama/container-log-archive-charm:anonymize_web_log-new_options into container-log-archive-charm:master

[Loïc Gomez]

anonymize_web_log.py: allow reading from stdin, add skip_keywords, strip_hashes options

[🤖 Canonical IS Merge Bot (canonical-is-mergebot)]

This merge proposal is being monitored by mergebot. Change the status to Approved to merge.

[Loïc Gomez (kotodama)]

This MP follows a request to access Apache logs with hashes and PII removed.

Example of hashes to obfuscate:
login?id=824a1183f5a22c7c4069...
model_uuid=4372e956-eca1-49fd-bd42-4bee02f05b81

Example of keywords where a skip of the logline would be nice:
/login to ignore GET /identity/login-legacy?did=b
or:
/usso_macaroon/ to ignore GET /identity/login/usso_macaroon/login?...

[Tom Haddon (mthaddon)]

As discussed IRL, would be good to alphabetise the arguments, and preserve the contextmanager for opening the logfile if possible to avoid needing to clean up afterwards.

[Joel Sing (jsing)]

review: Needs Fixing

[Loïc Gomez (kotodama)]

Thanks Joel, I'll push an update soon.

[Tom Haddon (mthaddon)]

A few comments/questions inline, and I see Joel is also reviewing this.

[Loïc Gomez (kotodama)]

Thanks, I'll alphabetize (and update preprocessor too actually) and change the dummy default value.

[Tom Haddon (mthaddon)]

This looks good to me. Will hold off on adding an approval since Joel is also looking at this MP, but fine to merge from my perspective if/when Joel approves.

[🤖 Canonical IS Merge Bot (canonical-is-mergebot)]

Change cannot be self approved, setting status to needs review.

[Loïc Gomez (kotodama)]

Approved by mthaddon by previous comment.

review: Approve (+1)

[🤖 Canonical IS Merge Bot (canonical-is-mergebot)]

Change cannot be self approved, setting status to needs review.

[Tom Haddon (mthaddon)]

LGTM

review: Approve

[🤖 Canonical IS Merge Bot (canonical-is-mergebot)]

Change successfully merged at revision 136c40098c0c573450b524a69dfb2305e16548c1

diff --git a/files/anonymize_web_log.py b/files/anonymize_web_log.py
index 1862847..e2a3712 100755
--- a/files/anonymize_web_log.py
+++ b/files/anonymize_web_log.py
@@ -8,12 +8,15 @@ import re
 import sys
 
 
-def anonymize(new_file, log_file, skip_ips=None, skip_private=False, strip_usernames=False, username_field=2):
+def anonymize(new_file, log_file, skip_ips=None, skip_keywords=None, skip_private=False, strip_hashes=False,
+              strip_usernames=False, username_field=2):
     """
     :param new_file: A file object that will be written out anonymized
     :param log_file: A file object to read from
     :param skip_ips: A netaddr.IPSet of ips to skip
+    :param skip_keywords: A list of keywords to trigger a line skip
     :param skip_private: A boolean, true to skip private ips
+    :param strip_hashes: A boolean, true to strip hashes
     :param strip_usernames: A boolean, true to strip usernames
     :param username_field: An integer specifying the field usernames are indexed by
     :return: (total lines processed, lines skipped)
@@ -21,6 +24,13 @@ def anonymize(new_file, log_file, skip_ips=None, skip_private=False, strip_usern
     ips = {}
     total = 0
     skip = 0
+
+    # Let's consider hashes/uuids are at least 30 char long
+    if strip_hashes:
+        rx_hashes = re.compile(r'[0-9a-fA-F-]{30,}')
+    if skip_keywords:
+        rx_kw = re.compile('({})'.format('|'.join([re.escape(kw) for kw in skip_keywords])))
+
     for line in log_file:
         total += 1
         splits = line.split()
@@ -46,6 +56,23 @@ def anonymize(new_file, log_file, skip_ips=None, skip_private=False, strip_usern
             del splits[username_field]
 
         anonymized_line = "{} {}\n".format(ips[str(ip)], b' '.join(splits[1:]).decode('utf-8'))
+
+        if strip_hashes or skip_keywords:
+            splits = anonymized_line.split('"')
+            if strip_hashes:
+                # anonymize request
+                splits[1] = rx_hashes.sub("<<HASH/UUID_REDACTED>>", splits[1])
+                # anonymize referer
+                splits[3] = rx_hashes.sub("<<HASH/UUID_REDACTED>>", splits[3])
+                # anonymize juju metadata
+                splits[7] = rx_hashes.sub("<<HASH/UUID_REDACTED>>", splits[7])
+                anonymized_line = '"'.join(splits)
+
+            if skip_keywords:
+                if rx_kw.search(splits[1]):
+                    skip += 1
+                    continue
+
         new_file.write(anonymized_line.encode('utf-8'))
 
     return total, skip
@@ -54,16 +81,25 @@ def anonymize(new_file, log_file, skip_ips=None, skip_private=False, strip_usern
 def parse_args(argv):
     # Parse arguments, build skip_set and paths as needed
     parser = argparse.ArgumentParser()
-    parser.add_argument('log_path', help='The path to the log to be anonymized')
-    parser.add_argument('--output', '-o', help='Output file, defaults to input file'
-                                               'with .anonymized extension')
+    input_group = parser.add_mutually_exclusive_group(required=True)
+    input_group.add_argument('log_path', default=None, nargs='?',
+                        help='The path to the log to be anonymized')
+    input_group.add_argument('--stdin', default=False, action='store_true',
+                        help='Read from stdin instead of using log_path (requires: --output.')
+    parser.add_argument('--output', '-o',
+                        help='Output file, defaults to input file with .anonymized extension')
+
     parser.add_argument('--skip', '-s', action='append', help='A cidr notation network for which any'
                                                               'for which any source traffic will be removed'
                                                               'from the final output. Can be specified multiple'
                                                               'times')
+    parser.add_argument('--skip_keywords', '-k', action='append', default=None,
+                        help='If specified remove any request matching one of the keywords.')
     parser.add_argument('--skip_private', action='store_true', default=False,
                         help='If specified remove any private source ips from the final file')
 
+    parser.add_argument('--strip_hashes', action='store_true', default=False,
+                        help='Strip hash-looking strings from log lines.')
     parser.add_argument('--strip_usernames', action='store_true', default=False,
                         help='Strip usernames from log lines.')
     parser.add_argument('--username_field', type=int, default=2,
@@ -80,6 +116,11 @@ def parse_args(argv):
     else:
         args.skip_ips = None
 
+    if args.stdin and not args.output:
+        parser.print_help()
+        print('Error: --stdin requires --output.\n')
+        sys.exit(-1)
+
     return args
 
 
@@ -92,12 +133,19 @@ def main():
 
     # Process the lines of the file
     with open(anonymized_path, 'wb') as new_file:
-        with open(args.log_path, 'rb') as log_file:
+        if args.stdin:
+            log_path = 0
+        else:
+            log_path = args.log_path
+
+        with open(log_path, 'rb') as log_file:
             anonymize_args = (
                 new_file,
                 log_file,
                 args.skip_ips,
+                args.skip_keywords,
                 args.skip_private,
+                args.strip_hashes,
                 args.strip_usernames,
                 args.username_field
             )
diff --git a/files/preprocessor.py b/files/preprocessor.py
index 21dd92f..7048e31 100644
--- a/files/preprocessor.py
+++ b/files/preprocessor.py
@@ -67,7 +67,9 @@ class AnonymizeWebLog(PreProcessor):
         anonymize_web_log.anonymize(new_file,
                                     log_file,
                                     parsed_args.skip_ips,
+                                    parsed_args.skip_keywords,
                                     parsed_args.skip_private,
+                                    parsed_args.strip_hashes,
                                     parsed_args.strip_usernames,
                                     parsed_args.username_field)
         new_file.seek(0)