Merge lp:~thomir-deactivatedaccount/adt-image-mapper/trunk-fix-logging into lp:adt-image-mapper

[Thomi Richards]

Enable logstash, configure logging properly.

Enable logstash, configure logging properly.

[Celso Providelo (cprov)]

Thomi,

Just a minor point inline and one question.

Assuming ./logs/ exists for the apps (the upstart template creates it), gunicorn.log and access.log will only contain gunicorn specific messages, application INFO, WARN and ERROR will go to app.log, correct ?

It's odd because we can't really avoid looking in all 3 files almost always to completely figure out problems. Maybe with some time live we find alternatives to glue these streams together, perhaps in LS.

I was expecting more from gunicorn, µWSGI has a lot more logging options, but not logstash, http://uwsgi-docs.readthedocs.org/en/latest/Logging.html

review: Approve

=== modified file 'README.rst'
--- README.rst	2015-03-16 21:00:16 +0000
+++ README.rst	2015-03-19 22:31:03 +0000
@@ -26,6 +26,13 @@
 	os_auth_url = http://172.20.161.138:5000/v2.0/
 	os_region_name = bot-prototype
 
+Optionally, add a `[logstash]` section, which will enable the logstash logging handler::
+
+  [logstash]
+  host = localhost
+  port = 5959
+  version = 1
+  
 Run the servce::
 
 	$ gunicorn adt_image_mapper:app

=== modified file 'adt-service.conf'
--- adt-service.conf	2015-03-16 21:00:16 +0000
+++ adt-service.conf	2015-03-19 22:31:03 +0000
@@ -6,3 +6,8 @@
 os_password = <redacted>
 os_auth_url = http://172.20.161.138:5000/v2.0/
 os_region_name = bot-prototype
+
+[logstash]
+host = localhost
+port = 5959
+version = 1

=== modified file 'adt_image_mapper/__init__.py'
--- adt_image_mapper/__init__.py	2015-03-18 19:43:42 +0000
+++ adt_image_mapper/__init__.py	2015-03-19 22:31:03 +0000
@@ -17,9 +17,11 @@
 
 import configparser
 import logging
+import os
 import sys
 
 import flask
+import logstash
 
 from adt_image_mapper import (
     cloud,
@@ -29,15 +31,12 @@
 
 
 def create_flask_app(config=None):
-    logging.basicConfig(stream=sys.stdout, level=logging.INFO)
-    # Uncomment this next line if you want detailed debugging information about
-    # why candidate images are rejected. Shouldn't be needed.
-    # logging.getLogger(__name__).setLevel(logging.DEBUG)
-
     if config is None:
         config = configparser.ConfigParser()
         config.read("adt-service.conf")
 
+    configure_logging(config)
+
     app = flask.Flask('adt_image_mapper')
 
     app.add_url_rule(
@@ -69,4 +68,38 @@
     return app
 
 
+
+def configure_logging(config):
+    root_logger = logging.getLogger()
+    root_logger.setLevel(logging.INFO)
+
+    # silence requests logging, which is created by keystone & glance:
+    requests_logger = logging.getLogger('requests')
+    requests_logger.setLevel(logging.WARNING)
+
+    log_path = os.path.abspath(os.path.join(__file__, '../../logs/app.log'))
+    log_dir = os.path.dirname(log_path)
+    handler = None
+    if os.path.exists(log_dir):
+        handler = logging.FileHandler(log_path)
+    else:
+        print("'logs' directory '{}' does not exist, using stderr for app log.".format(log_dir))
+        handler = logging.StreamHandler()
+    handler.setFormatter(
+        logging.Formatter(
+            '%(asctime)s  %(name)s %(levelname)s: %(message)s'
+        )
+    )
+    root_logger.addHandler(handler)
+
+    if 'logstash' in config:
+        root_logger.addHandler(
+            logstash.LogstashHandler(
+                config['logstash']['host'],
+                int(config['logstash']['port']),
+                int(config['logstash']['version'])
+            )
+        )
+
+
 app = create_flask_app()

=== modified file 'adt_image_mapper/cloud.py'
--- adt_image_mapper/cloud.py	2015-03-19 02:55:47 +0000
+++ adt_image_mapper/cloud.py	2015-03-19 22:31:03 +0000
@@ -64,7 +64,7 @@
             glance_client = self.get_glance_client(refresh=True)
             candidate_image = self._get_candidate_image(
                 glance_client, series_name, architecture)
-        logger.info("Search finished, found %r", candidate_image)
+        logger.info("Search finished, found image %s", candidate_image.name)
         return candidate_image.name if candidate_image else None
 
     def get_glance_client(self, refresh=False):

=== modified file 'adt_image_mapper/v1.py'
--- adt_image_mapper/v1.py	2015-03-13 01:08:05 +0000
+++ adt_image_mapper/v1.py	2015-03-19 22:31:03 +0000
@@ -14,6 +14,7 @@
 # You should have received a copy of the GNU General Public License
 # along with this program.  If not, see <http://www.gnu.org/licenses/>.
 #
+import logging
 
 from flask import (
     current_app,
@@ -26,11 +27,13 @@
     MissingRequestParameters,
 )
 
+logger = logging.getLogger(__name__)
 
 def get_image():
     required = { 'series', 'architecture' }
     missing = required.difference(request.args.keys())
     if missing:
+        logger.error("Got request '%r' with missing arguments %r", request.args, missing)
         raise MissingRequestParameters(
             missing,
             'Must specify both series and architecture name'

=== modified file 'requirements.txt'
--- requirements.txt	2015-03-19 14:14:03 +0000
+++ requirements.txt	2015-03-19 22:31:03 +0000
@@ -8,3 +8,4 @@
 gunicorn==19.3.0
 python-glanceclient==0.16.1
 python-keystoneclient==1.2.0
+python-logstash==0.4.2