Merge lp:~james-w/canonical-identity-provider/django-timeline into lp:canonical-identity-provider/release

[James Westby]

Add timeline-django to record sql queries in oopses.

Hi,

This adds SQL queries to the generated oopses, using django-timeline.

The majority of the change is adding the new dependencies and wiring it up.
The rest is configuring the code to redact queries against the tables that
contain sensitive information.

Thanks,

James

[Ricardo Kirkner (ricardokirkner)]

For tracking dependencies in a way that is auditable we prefer to track branches owned by ~ubuntuone-pqm-team. Please use the following branches instead of trunk:

lp:~ubuntuone-pqm-team/python-oops-timeline/stable
lp:~ubuntuone-pqm-team/python-timeline/stable
lp:~ubuntuone-pqm-team/python-timeline-django/stable

[Natalia Bidart (nataliabidart)]

Looks good!

review: Approve

[Ubuntu One Auto Pilot (otto-pilot)]

The attempt to merge lp:~james-w/canonical-identity-provider/django-timeline into lp:canonical-identity-provider failed. Below is the output from the failed tests.

Updating download cache at dir /mnt/tarmac/cache/canonical-identity-provider/isd-download-cache
Using saved parent location: bzr+ssh://bazaar.launchpad.net/~canonical-isd-hackers/+junk/download-cache/
No revisions or tags to pull.
[localhost] local: which virtualenv
[localhost] local: /usr/bin/python /usr/bin/virtualenv --distribute --clear --system-site-packages .env
Not deleting .env/bin
New python executable in .env/bin/python
Installing distribute.............................................................................................................................................................................................done.
Installing pip...............done.
[localhost] local: dpkg -l libpq-dev 2> /dev/null | grep '^ii' | wc -l
[localhost] local: dpkg -l libxml2-dev 2> /dev/null | grep '^ii' | wc -l
[localhost] local: dpkg -l libxslt1-dev 2> /dev/null | grep '^ii' | wc -l
[localhost] local: dpkg -l memcached 2> /dev/null | grep '^ii' | wc -l
[localhost] local: dpkg -l postgresql-plpython-9.1 2> /dev/null | grep '^ii' | wc -l
[localhost] local: dpkg -l python-dev 2> /dev/null | grep '^ii' | wc -l
[localhost] local: dpkg -l python-m2crypto 2> /dev/null | grep '^ii' | wc -l
[localhost] local: dpkg -l swig 2> /dev/null | grep '^ii' | wc -l
[localhost] local: dpkg -l config-manager 2> /dev/null | grep '^ii' | wc -l
[localhost] local: dpkg -l python-egenix-mx-base-dev 2> /dev/null | grep '^ii' | wc -l
[localhost] local: /usr/lib/config-manager/cm.py update /tmp/tmpO1bi_a
UnknownErrorFromSmartServer(Server sent an unexpected error: ('error', 'GhostRevisionsHaveNoRevno', 'Could not determine revno for {10} because its ancestry shows a ghost at {<email address hidden>}')) Server sent an unexpected error: ('error', 'GhostRevisionsHaveNoRevno', 'Could not determine revno for {10} because its ancestry shows a ghost at {<email address hidden>}')

Traceback (most recent call last):
  File "/usr/lib/config-manager/cm.py", line 30, in <module>
    main(sys.argv)
  File "/usr/lib/pymodules/python2.7/config_manager/__init__.py", line 384, in main
    config.update(os.path.abspath(os.curdir))
  File "/usr/lib/pymodules/python2.7/config_manager/__init__.py", line 91, in update
    entry.update(dir)
  File "/usr/lib/pymodules/python2.7/config_manager/__init__.py", line 306, in update
    return self.build(path)
  File "/usr/lib/pymodules/python2.7/config_manager/__init__.py", line 280, in build
    raise ValueError("unknown url type '%s'" % self.url)
ValueError: unknown url type 'bzr+ssh://bazaar.launchpad.net/~ubuntuone-pqm-team/python-timeline/stable;revno=10'

Fatal error: local() encountered an error (return code 1) while executing '/usr/lib/config-manager/cm.py update /tmp/tmpO1bi_a'

Aborting.

=== modified file 'config-manager.txt'
--- config-manager.txt	2013-08-05 15:26:51 +0000
+++ config-manager.txt	2013-08-07 15:56:25 +0000
@@ -34,6 +34,7 @@
 canonical-identity-provider/branches/oops-amqp             bzr+ssh://bazaar.isd/~ubuntuone-pqm-team/python-oops-amqp/stable;revno=0.0.7
 canonical-identity-provider/branches/oops-datedir-repo     bzr+ssh://bazaar.isd/~ubuntuone-pqm-team/python-oops-datedir-repo/stable;revno=0.0.20
 canonical-identity-provider/branches/oops-dictconfig       bzr+ssh://bazaar.isd/~ubuntuone-pqm-team/python-oops-dictconfig/stable;revno=0.0.4
+canonical-identity-provider/branches/oops-timeline         bzr+ssh://bazaar.isd/~ubuntuone-pqm-team/python-oops-timeline/stable;revno=3
 canonical-identity-provider/branches/oops-wsgi             bzr+ssh://bazaar.isd/~ubuntuone-pqm-team/python-oops-wsgi/stable;revno=36
 canonical-identity-provider/branches/pystatsd              bzr+ssh://bazaar.isd/~ubuntuone-pqm-team/pystatsd/stable;revno=53
 canonical-identity-provider/branches/python-oath           bzr+ssh://bazaar.isd/~ubuntuone-pqm-team/python-oath/stable;revno=53
@@ -42,4 +43,6 @@
 canonical-identity-provider/branches/requests              bzr+ssh://bazaar.isd/~ubuntuone-pqm-team/requests/stable;revno=v1.1.0
 canonical-identity-provider/branches/requests-oauthlib     bzr+ssh://bazaar.isd/~ubuntuone-pqm-team/requests-oauthlib/stable;revno=16
 canonical-identity-provider/branches/ssoclient             bzr+ssh://bazaar.isd/~canonical-isd-hackers/canonical-identity-provider/ssoclient;revno=7
+canonical-identity-provider/branches/timeline              bzr+ssh://bazaar.isd/~ubuntuone-pqm-team/python-timeline/stable;revno=10
+canonical-identity-provider/branches/timeline-django       bzr+ssh://bazaar.isd/~ubuntuone-pqm-team/python-timeline-django/stable;revno=17
 canonical-identity-provider/branches/yui-3.10.0            bzr+ssh://bazaar.isd/~ubuntuone-pqm-team/yui/stable;revno=v3.10.0

=== modified file 'django_project/wsgi.py'
--- django_project/wsgi.py	2013-08-02 16:09:22 +0000
+++ django_project/wsgi.py	2013-08-07 15:56:25 +0000
@@ -1,5 +1,4 @@
 import os
-import platform
 import sys
 
 # before doing anything else, patch stdout to avoid breakage in production
@@ -16,7 +15,6 @@
 paths.setup_paths()
 
 from django.conf import settings
-from django.core.handlers.wsgi import WSGIHandler
 
 os.environ['PGCONNECT_TIMEOUT'] = str(settings.PGCONNECT_TIMEOUT)
 os.environ['PGSSLMODE'] = 'allow'
@@ -27,13 +25,27 @@
 
 # Wrap the application in the Oops wsgi app to catch unhandled exceptions
 # and create oops for them.
-#
-# First we create the config that defines what to do with the oopses.
+
+from identityprovider.oops_config import install_timeline_hooks
 import oops_dictconfig
 from oops_wsgi import make_app, install_hooks
+from timeline_django import setup as timeline_django_setup
+from timeline_django.wsgi import make_app as timeline_django_make_app
+from timeline.wsgi import make_app as timeline_make_app
 
+# First we create the config that defines what to do with the oopses.
 config = oops_dictconfig.config_from_dict(settings.OOPSES)
 install_hooks(config)
 
+# Install the hooks to record the sql queries
+timeline_django_setup.setup_for_requests()
+
+# Set up the timeline in the django and wsgi envs
+app = timeline_django_make_app(app)
+app = timeline_make_app(app)
+
+# Add the hooks to collect and sanitise the timeline
+install_timeline_hooks()
+
 # Then we wrap the django app in the oops one
 application = make_app(app, config, oops_on_status=['500'])

=== added symlink 'lib/oops_timeline'
=== target is u'../branches/oops-timeline/oops_timeline/'
=== added symlink 'lib/timeline'
=== target is u'../branches/timeline/timeline'
=== added symlink 'lib/timeline_django'
=== target is u'../branches/timeline-django/timeline_django/'
=== added file 'src/identityprovider/oops_config.py'
--- src/identityprovider/oops_config.py	1970-01-01 00:00:00 +0000
+++ src/identityprovider/oops_config.py	2013-08-07 15:56:25 +0000
@@ -0,0 +1,10 @@
+import oops_timeline
+import timeline_django.filters
+
+from identityprovider import timeline_filters
+
+
+def install_timeline_hooks(config):
+    oops_timeline.install_hooks(config)
+    timeline_django.filters.install_hooks(config)
+    timeline_filters.install_hooks(config)

=== added file 'src/identityprovider/tests/test_oops.py'
--- src/identityprovider/tests/test_oops.py	1970-01-01 00:00:00 +0000
+++ src/identityprovider/tests/test_oops.py	2013-08-07 15:56:25 +0000
@@ -0,0 +1,26 @@
+import oops
+import timeline as mod_timeline
+from unittest import TestCase
+
+from identityprovider import oops_config
+
+
+class OopsTests(TestCase):
+
+    def test_timeline_filters_sensitive(self):
+        reports = []
+
+        def capture(report):
+            reports.append(report)
+            return []
+        config = oops.Config()
+        oops_config.install_timeline_hooks(config)
+        config.publisher = capture
+        timeline = mod_timeline.Timeline()
+        timeline.start('SQL-master', 'SELECT * from accountpassword').finish()
+        report = config.create(context=dict(timeline=timeline))
+        config.publish(report)
+        self.assertEqual(1, len(reports))
+        report = reports[0]
+        self.assertEqual('SELECT <accountpassword query redacted>',
+                         report['timeline'][0][3])

=== added file 'src/identityprovider/timeline_filters.py'
--- src/identityprovider/timeline_filters.py	1970-01-01 00:00:00 +0000
+++ src/identityprovider/timeline_filters.py	2013-08-07 15:56:25 +0000
@@ -0,0 +1,36 @@
+# Copyright 2013 Canonical Ltd.  This software is licensed under the
+# GNU Affero General Public License version 3 (see the file LICENSE).
+
+from timeline_django import filters
+
+
+def filter_table(table_name, reason):
+    def do_filter(query):
+        return filters._filter_table(query, table_name, reason)
+
+    def filter_query(report, context):
+        return filters._filter_query(report, do_filter)
+    return filter_query
+
+
+def install_hooks(config):
+    # Filter queries for tables that contain sensitive information
+    config.on_create.append(
+        filter_table('accountpassword', 'accountpassword'))
+    config.on_create.append(
+        filter_table('api_user', 'api_user'))
+    config.on_create.append(
+        filter_table('authtoken', 'authtoken'))
+    config.on_create.append(
+        filter_table('identityprovider_authenticationdevice',
+                     'authenticationdevice'))
+    config.on_create.append(
+        filter_table('oauth_consumer', 'oauth_consumer'))
+    config.on_create.append(
+        filter_table('oauth_token', 'oauth_token'))
+    config.on_create.append(
+        filter_table('openidauthorization', 'openidauthorization'))
+    config.on_create.append(
+        filter_table('piston_consumer', 'piston_consumer'))
+    config.on_create.append(
+        filter_table('piston_token', 'piston_token'))