Merge lp:~harlowja/cloud-init/cmdline-kernel-logging into lp:~cloud-init-dev/cloud-init/trunk

[Joshua Harlow]

Add the ability to override the logging levels via cloud-config/kernel, instead of forcing users to alter the large '05_logging.cfg' blob.

[Server Team CI bot (server-team-bot)]

FAILED: Continuous integration, rev:647
https://server-team-jenkins.canonical.com/job/cloud-init-ci/60/
Executed test runs:
    None: https://server-team-jenkins.canonical.com/job/lp-vote-on-merge/34/console

Click here to trigger a rebuild:
https://server-team-jenkins.canonical.com/job/cloud-init-ci/60/rebuild

review: Needs Fixing (continuous-integration)

[Chad Smith (chad.smith)]

Hello,
Thank you for taking the time to contribute to cloud-init. Cloud-init has moved its revision control system to git. As a result, we are marking all bzr merge proposals as 'rejected'. If you would like to re-submit this proposal for review, please do so by following the current HACKING documentation at http://cloudinit.readthedocs.io/en/latest/topics/hacking.html.

Also, it seems an evolution of this logging configuration has already landed in cloud-init master under the logcfg cloud-config module. If you feel current cloud-init changes don't meet your needs please either file a but against cloud-init or propose an iteration of this branch against master.
https://cloudinit.readthedocs.io/en/latest/topics/logging.html

Unmerged revisions

647. By Joshua Harlow on 2012-09-04

Add log level cloud config example.

646. By Joshua Harlow on 2012-09-04

Instead of using a new kernel cli, allow
the cloud-config yaml (and corresponding kernel text)
to alter the logging level via a simpler manner
then replacing the whole logging blob via a new
log_level key that can provide the desired logging
level to use to override the systems setup level.

645. By Joshua Harlow on 2012-09-04

Add the ability to use a log mode/level from
the command line from the kernel to influence
the applicable logging level (for basic logging
and after logging has been configured).

=== modified file 'bin/cloud-init'
--- bin/cloud-init	2012-08-10 03:48:01 +0000
+++ bin/cloud-init	2012-09-04 20:51:20 +0000
@@ -490,7 +490,7 @@
     args = parser.parse_args()
 
     # Setup basic logging to start (until reinitialized)
-    # iff in debug mode...
+    # iff in debug mode (or cmdline provides a level)...
     if args.debug:
         logging.setupBasicLogging()
 

=== modified file 'cloudinit/log.py'
--- cloudinit/log.py	2012-08-22 18:12:32 +0000
+++ cloudinit/log.py	2012-09-04 20:51:20 +0000
@@ -40,17 +40,34 @@
 DEBUG = logging.DEBUG
 NOTSET = logging.NOTSET
 
+# String -> log level mapping
+LEVEL_NAMES = {
+    'CRITICAL': CRITICAL,
+    'C': CRITICAL,
+    'FATAL': FATAL,
+    'F': FATAL,
+    'ERROR': ERROR,
+    'E': ERROR,
+    'WARNING': WARNING,
+    'W': WARNING,
+    'WARN': WARNING,
+    'INFO': INFO,
+    'I': INFO,
+    'DEBUG': DEBUG,
+    'D': DEBUG,
+}
+
 # Default basic format
 DEF_CON_FORMAT = '%(asctime)s - %(filename)s[%(levelname)s]: %(message)s'
 
 
-def setupBasicLogging():
+def setupBasicLogging(log_level=DEBUG):
     root = logging.getLogger()
     console = logging.StreamHandler(sys.stderr)
     console.setFormatter(logging.Formatter(DEF_CON_FORMAT))
-    console.setLevel(DEBUG)
+    console.setLevel(log_level)
     root.addHandler(console)
-    root.setLevel(DEBUG)
+    root.setLevel(log_level)
 
 
 def setupLogging(cfg=None):
@@ -76,6 +93,7 @@
 
     # See if any of them actually load...
     am_tried = 0
+    am_worked = 0
     for log_cfg in log_cfgs:
         try:
             am_tried += 1
@@ -90,7 +108,8 @@
             # Attempt to load its config
             logging.config.fileConfig(log_cfg)
             # The first one to work wins!
-            return
+            am_worked += 1
+            break
         except Exception:
             # We do not write any logs of this here, because the default
             # configuration includes an attempt at using /dev/log, followed
@@ -99,13 +118,40 @@
             pass
 
     # If it didn't work, at least setup a basic logger (if desired)
-    basic_enabled = cfg.get('log_basic', True)
-
-    sys.stderr.write(("WARN: no logging configured!"
-                      " (tried %s configs)\n") % (am_tried))
+    basic_enabled = False
+    if not am_worked:
+        basic_enabled = cfg.get('log_basic', True)
+        sys.stderr.write(("WARN: no logging configured!"
+                          " (tried %s configs)\n") % (am_tried))
     if basic_enabled:
         sys.stderr.write("Setting up basic logging...\n")
         setupBasicLogging()
+    if 'log_level' in cfg:
+        override_level = translateLevels([cfg['log_level']])
+        if override_level:
+            # Adjust the root and the cloud-init logger
+            # to be the level desired, possibly overriding
+            # any configured levels...
+            for logger in [logging.getLogger(), getLogger()]:
+                logger.setLevel(override_level)
+
+
+def translateLevels(levels, quiet=True):
+    # First filter to known levels
+    known_levels = []
+    for level_name in levels:
+        level_name = level_name.strip().upper()
+        if level_name not in LEVEL_NAMES:
+            if quiet:
+                continue
+            else:
+                raise ValueError("Untranslatable log level %s" % (level_name))
+        known_levels.append(LEVEL_NAMES[level_name])
+    if not known_levels:
+        return None
+    # Pick the lowest level
+    known_levels.sort()
+    return known_levels[0]
 
 
 def getLogger(name='cloudinit'):

=== modified file 'cloudinit/util.py'
--- cloudinit/util.py	2012-08-28 03:51:00 +0000
+++ cloudinit/util.py	2012-09-04 20:51:20 +0000
@@ -700,34 +700,37 @@
     return mergedict(confd_cfg, cfg)
 
 
-def read_cc_from_cmdline(cmdline=None):
-    # this should support reading cloud-config information from
-    # the kernel command line.  It is intended to support content of the
-    # format:
-    #  cc: <yaml content here> [end_cc]
-    # this would include:
-    # cc: ssh_import_id: [smoser, kirkland]\\n
-    # cc: ssh_import_id: [smoser, bob]\\nruncmd: [ [ ls, -l ], echo hi ] end_cc
-    # cc:ssh_import_id: [smoser] end_cc cc:runcmd: [ [ ls, -l ] ] end_cc
+def read_tokens_from_cmdline(tok_begin, tok_end, cmdline=None):
     if cmdline is None:
         cmdline = get_cmdline()
-
-    tag_begin = "cc:"
-    tag_end = "end_cc"
-    begin_l = len(tag_begin)
-    end_l = len(tag_end)
+    tokens = []
+    begin_l = len(tok_begin)
+    end_l = len(tok_end)
     clen = len(cmdline)
-    tokens = []
-    begin = cmdline.find(tag_begin)
+    begin = cmdline.find(tok_begin)
     while begin >= 0:
-        end = cmdline.find(tag_end, begin + begin_l)
+        end = cmdline.find(tok_end, begin + begin_l)
         if end < 0:
             end = clen
         tokens.append(cmdline[begin + begin_l:end].lstrip().replace("\\n",
                                                                     "\n"))
-
-        begin = cmdline.find(tag_begin, end + end_l)
-
+        begin = cmdline.find(tok_begin, end + end_l)
+    return tokens
+
+
+def read_cc_from_cmdline(cmdline=None):
+    # This should support reading cloud-config information from
+    # the kernel command line.
+    # It is intended to support content of the format:
+    #
+    # cc: <yaml content here> [end_cc]
+    #
+    # For example:
+    #
+    # cc: ssh_import_id: [smoser, kirkland]\\n
+    # cc: ssh_import_id: [smoser, bob]\\nruncmd: [ [ ls, -l ], echo hi ] end_cc
+    # cc: ssh_import_id: [smoser] end_cc cc:runcmd: [ [ ls, -l ] ] end_cc
+    tokens = read_tokens_from_cmdline("cc:", "end_cc", cmdline)
     return '\n'.join(tokens)
 
 

=== added file 'doc/examples/cloud-config-log-level.txt'
--- doc/examples/cloud-config-log-level.txt	1970-01-01 00:00:00 +0000
+++ doc/examples/cloud-config-log-level.txt	2012-09-04 20:51:20 +0000
@@ -0,0 +1,28 @@
+#cloud-config
+# vim: syntax=yaml
+#
+# This is the configuration syntax that can be used to alter the logging
+# level of the cloud-init python logging subsystem. It is typically easier
+# to use this new setting than overriding the full logging config that is
+# setup by cloud-init. This syntax can also be used in the cc: <yaml> end_cc
+# style of specifying this configuration via the kernel command line.
+#
+log_level: DEBUG
+# This will override the system logging and make it use DEBUG level instead
+# of the typical WARN level.
+#
+# Other valid options here are the following:
+# C => 'CRITICAL'
+# E => 'ERROR'
+# D => 'DEBUG'
+# F => 'FATAL'
+# I => 'INFO'
+# WARN => 'WARNING'
+# W => 'WARNING'
+# ERROR
+# DEBUG
+# FATAL
+# INFO
+# WARNING
+# CRITICAL
+