Merge lp:~raharper/curtin/trunk.fix-missing-install-log into lp:~curtin-dev/curtin/trunk

[Ryan Harper]

reporting: default webhook handler level to DEBUG, no filtering

The webhook handler in curtin reporting included a default level
of INFO and included a event filter which would not report events
that were of lower importance than what was set. The result was
for unconfigured WebHook handlers, some of the expected messages
including post_files used by maas to collect the curtin install
log were no longer sent.

This patch sets WebHookHandler level to DEBUG, the same level set for
LogHandler and PrintHandler, further we remove the event filtering.
The events themselves include the level value and the event collector
can apply their own filtering as needed. Along with the changes we
clean up some comments and include a new unittest which specifically
tests that a FinishReportingEvent with post_files is posted by the
WebHookHandler.

[lp:~raharper/curtin/trunk.fix-missing-install-log]

391. By Ryan Harper on 2016-06-14

merge from trunk

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:390
https://server-team-jenkins.canonical.com/job/curtin-ci/256/
Executed test runs:
    None: https://server-team-jenkins.canonical.com/job/generic-update-mp/253/console

Click here to trigger a rebuild:
https://server-team-jenkins.canonical.com/job/curtin-ci/256/rebuild

review: Approve (continuous-integration)

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:391
https://server-team-jenkins.canonical.com/job/curtin-ci/257/
Executed test runs:
    None: https://server-team-jenkins.canonical.com/job/generic-update-mp/254/console

Click here to trigger a rebuild:
https://server-team-jenkins.canonical.com/job/curtin-ci/257/rebuild

review: Approve (continuous-integration)

[Scott Moser (smoser)]

I think this is fine. there is some cleanup to do there, where we'd like to avoid passing strings around explicitly (rather than level.DEBUG), but this is fine and will un-break some things.

feel free to do more if you want.

[Ryan Harper (raharper)]

vmtest passed on diglett, merging to trunk.

----------------------------------------------------------------------
Ran 610 tests in 9712.856s

OK
Tue, 14 Jun 2016 17:15:22 -0500: vmtest end [0] in 9713s

=== modified file 'curtin/reporter/handlers.py'
--- curtin/reporter/handlers.py	2016-04-04 19:09:47 +0000
+++ curtin/reporter/handlers.py	2016-06-14 14:07:09 +0000
@@ -23,7 +23,7 @@
 
 
 class LogHandler(ReportingHandler):
-    """Publishes events to the cloud-init log at the ``INFO`` log level."""
+    """Publishes events to the curtin log at the ``DEBUG`` log level."""
 
     def __init__(self, level="DEBUG"):
         super(LogHandler, self).__init__()
@@ -39,9 +39,9 @@
         self.level = level
 
     def publish_event(self, event):
-        """Publish an event to the ``INFO`` log level."""
+        """Publish an event to the ``DEBUG`` log level."""
         logger = logging.getLogger(
-            '.'.join(['cloudinit', 'reporting', event.event_type, event.name]))
+            '.'.join(['curtin', 'reporting', event.event_type, event.name]))
         logger.log(self.level, event.as_string())
 
 
@@ -55,7 +55,7 @@
 class WebHookHandler(ReportingHandler):
     def __init__(self, endpoint, consumer_key=None, token_key=None,
                  token_secret=None, consumer_secret=None, timeout=None,
-                 retries=None, level="INFO"):
+                 retries=None, level="DEBUG"):
         super(WebHookHandler, self).__init__()
 
         self.oauth_helper = url_helper.OauthUrlHelper(
@@ -72,15 +72,6 @@
         self.headers = {'Content-Type': 'application/json'}
 
     def publish_event(self, event):
-        if isinstance(event.level, int):
-            ev_level = event.level
-        else:
-            try:
-                ev_level = getattr(logging, event.level.upper())
-            except:
-                ev_level = logging.INFO
-        if ev_level < self.level:
-            return
         try:
             return self.oauth_helper.geturl(
                 url=self.endpoint, data=event.as_dict(),

=== modified file 'tests/unittests/test_reporter.py'
--- tests/unittests/test_reporter.py	2016-04-04 19:09:47 +0000
+++ tests/unittests/test_reporter.py	2016-06-14 14:07:09 +0000
@@ -128,7 +128,8 @@
     def test_webhook_handler(self, mock_url_helper):
         event = events.ReportingEvent(events.START_EVENT_TYPE, 'test_event',
                                       'test event', level='INFO')
-        webhook_handler = handlers.WebHookHandler('127.0.0.1:8000')
+        webhook_handler = handlers.WebHookHandler('127.0.0.1:8000',
+                                                  level='INFO')
         webhook_handler.publish_event(event)
         webhook_handler.oauth_helper.geturl.assert_called_with(
             url='127.0.0.1:8000', data=event.as_dict(),
@@ -136,7 +137,6 @@
         event.level = 'DEBUG'
         webhook_handler.oauth_helper.geturl.called = False
         webhook_handler.publish_event(event)
-        self.assertFalse(webhook_handler.oauth_helper.geturl.called)
         webhook_handler = handlers.WebHookHandler('127.0.0.1:8000',
                                                   level="INVALID")
         self.assertEquals(webhook_handler.level, 30)
@@ -194,3 +194,24 @@
                              base64.b64encode(test_data).decode())
         finally:
             os.remove(tmp[1])
+
+    @patch('curtin.url_helper.OauthUrlHelper')
+    def test_webhook_handler_post_files(self, mock_url_helper):
+        test_data = b'abcdefg'
+        tmp = tempfile.mkstemp()
+        tmpfname = tmp[1]
+        try:
+            with open(tmpfname, 'wb') as fp:
+                fp.write(test_data)
+            event = events.FinishReportingEvent('test_event_name',
+                                                'test event description',
+                                                post_files=[tmpfname],
+                                                level='INFO')
+            webhook_handler = handlers.WebHookHandler('127.0.0.1:8000',
+                                                      level='INFO')
+            webhook_handler.publish_event(event)
+            webhook_handler.oauth_helper.geturl.assert_called_with(
+                url='127.0.0.1:8000', data=event.as_dict(),
+                headers=webhook_handler.headers, retries=None)
+        finally:
+            os.remove(tmpfname)