Merge lp:~james-w/canonical-identity-provider/oops2 into lp:canonical-identity-provider/release

[James Westby]

Prepare for moving to v2 of the oops stack.

Hi,

This is the prep that is needed for moving to v2 of the oops stack.

It's not perfect yet, as a limitation in the oops stack means we currently
have to choose between showing the user the id of their oops, and having reliable
delivery if amqp is down. I'm working on fixing that limitation, and we can
adopt that when it is ready so that we don't have to choose.

Some comments:

  * There are extra requirements that this brings. They should all be
    packaged in CAT already, so I think it should be a case of adding them
    to the dependencies packaage and getting them installed?

  * It requires some config being set to get the oopses delivered, but the
    code won't have a problem if it isn't set. How is the config delivered
    to production for SSO?

  * I didn't make use of settings.BRAND for the new oopses. I'm not sure what
    it is for. However we can adjust the reporter or the template to include
    extra information if we like. If the info is dynamic and important to have
    then I can add the code for it.

Thanks,

James

[James Westby (james-w)]

Oh, I renamed the wsgi symlink so that I could use it with runserver, as it needs to
import it. I can undo that if it's not correct.

Thanks,

James

[James Westby (james-w)]

nessita points out that sourcedeps might be preferred to pip now. Also sso is on lucid, and
the packages we have will be for precise.

There's also a question of whether requirements.txt should list all the new deps (e.g. the
amqp library that oops_amqp will use), or just the top-level ones.

[Ricardo Kirkner (ricardokirkner)]

Approving, but noting some extra things we need to add in later

- add requirements as sourcedeps
- add tests for wsgi_handler.py
- add a way for testing wsgi.py

review: Approve

[Ubuntu One Auto Pilot (otto-pilot)]

The attempt to merge lp:~james-w/canonical-identity-provider/oops2 into lp:canonical-identity-provider failed. Below is the output from the failed tests.

Updating download cache at dir /mnt/tarmac/cache/isd-download-cache
Using saved parent location: bzr+ssh://bazaar.launchpad.net/~canonical-isd-hackers/+junk/download-cache/
No revisions or tags to pull.
[localhost] local: which virtualenv
[localhost] local: /usr/bin/python /usr/bin/virtualenv --version
[localhost] local: /usr/bin/python /usr/bin/virtualenv --distribute --clear .env
Not deleting .env/bin
New python executable in .env/bin/python
Installing distribute.............................................................................................................................................................................................done.
Installing pip...............done.
[localhost] local: dpkg -l libpq-dev 2> /dev/null | grep '^ii' | wc -l
[localhost] local: dpkg -l libxml2-dev 2> /dev/null | grep '^ii' | wc -l
[localhost] local: dpkg -l libxslt1-dev 2> /dev/null | grep '^ii' | wc -l
[localhost] local: dpkg -l memcached 2> /dev/null | grep '^ii' | wc -l
[localhost] local: dpkg -l postgresql-plpython-9.1 2> /dev/null | grep '^ii' | wc -l
[localhost] local: dpkg -l python-dev 2> /dev/null | grep '^ii' | wc -l
[localhost] local: dpkg -l python-m2crypto 2> /dev/null | grep '^ii' | wc -l
[localhost] local: dpkg -l swig 2> /dev/null | grep '^ii' | wc -l
[localhost] local: dpkg -l config-manager 2> /dev/null | grep '^ii' | wc -l
[localhost] local: dpkg -l python-egenix-mx-base-dev 2> /dev/null | grep '^ii' | wc -l
[localhost] local: rm -rf M2Crypto*
[localhost] local: ln -s /usr/lib/python2.7/dist-packages/M2Crypto* .
[localhost] local: /usr/lib/config-manager/cm.py update /tmp/tmpHHZDAd
[localhost] local: . /mnt/tarmac/cache/canonical-identity-provider/trunk/.env/bin/activate && make install PACKAGES="-r /mnt/tarmac/cache/canonical-identity-provider/trunk/requirements.txt"
pip install --find-links=file:///mnt/tarmac/cache/isd-download-cache --no-index pip==dev
Ignoring indexes: http://pypi.python.org/simple/
Downloading/unpacking pip==dev
  Running setup.py egg_info for package pip

    warning: no files found matching '*.html' under directory 'docs'
    warning: no previously-included files matching '*.txt' found under directory 'docs/_build'
    no previously-included directories found matching 'docs/_build/_sources'
Installing collected packages: pip
  Found existing installation: pip 1.1
    Uninstalling pip:
      Successfully uninstalled pip
  Running setup.py install for pip

    warning: no files found matching '*.html' under directory 'docs'
    warning: no previously-included files matching '*.txt' found under directory 'docs/_build'
    no previously-included directories found matching 'docs/_build/_sources'
    Installing pip script to /mnt/tarmac/cache/canonical-identity-provider/trunk/.env/bin
    Installing pip-2.7 script to /mnt/tarmac/cache/canonical-identity-provider/trunk/.env/bin
Successfully installed pip
Cleaning up...
pip install --find-links=. --no-index -r /mnt/tarmac/cache/canonical-identity-provider/trunk/requirements.txt
Ignoring indexes: ht...

[Ubuntu One Auto Pilot (otto-pilot)]

The attempt to merge lp:~james-w/canonical-identity-provider/oops2 into lp:canonical-identity-provider failed. Below is the output from the failed tests.

Updating download cache at dir /mnt/tarmac/cache/isd-download-cache
Using saved parent location: bzr+ssh://bazaar.launchpad.net/~canonical-isd-hackers/+junk/download-cache/
No revisions or tags to pull.
[localhost] local: which virtualenv
[localhost] local: /usr/bin/python /usr/bin/virtualenv --version
[localhost] local: /usr/bin/python /usr/bin/virtualenv --distribute --clear .env
Not deleting .env/bin
New python executable in .env/bin/python
Installing distribute.............................................................................................................................................................................................done.
Installing pip...............done.
[localhost] local: dpkg -l libpq-dev 2> /dev/null | grep '^ii' | wc -l
[localhost] local: dpkg -l libxml2-dev 2> /dev/null | grep '^ii' | wc -l
[localhost] local: dpkg -l libxslt1-dev 2> /dev/null | grep '^ii' | wc -l
[localhost] local: dpkg -l memcached 2> /dev/null | grep '^ii' | wc -l
[localhost] local: dpkg -l postgresql-plpython-9.1 2> /dev/null | grep '^ii' | wc -l
[localhost] local: dpkg -l python-dev 2> /dev/null | grep '^ii' | wc -l
[localhost] local: dpkg -l python-m2crypto 2> /dev/null | grep '^ii' | wc -l
[localhost] local: dpkg -l swig 2> /dev/null | grep '^ii' | wc -l
[localhost] local: dpkg -l config-manager 2> /dev/null | grep '^ii' | wc -l
[localhost] local: dpkg -l python-egenix-mx-base-dev 2> /dev/null | grep '^ii' | wc -l
[localhost] local: rm -rf M2Crypto*
[localhost] local: ln -s /usr/lib/python2.7/dist-packages/M2Crypto* .
[localhost] local: /usr/lib/config-manager/cm.py update /tmp/tmpU44EE3
[localhost] local: . /mnt/tarmac/cache/canonical-identity-provider/trunk/.env/bin/activate && make install PACKAGES="-r /mnt/tarmac/cache/canonical-identity-provider/trunk/requirements.txt"
pip install --find-links=file:///mnt/tarmac/cache/isd-download-cache --no-index pip==dev
Ignoring indexes: http://pypi.python.org/simple/
Downloading/unpacking pip==dev
  Running setup.py egg_info for package pip

    warning: no files found matching '*.html' under directory 'docs'
    warning: no previously-included files matching '*.txt' found under directory 'docs/_build'
    no previously-included directories found matching 'docs/_build/_sources'
Installing collected packages: pip
  Found existing installation: pip 1.1
    Uninstalling pip:
      Successfully uninstalled pip
  Running setup.py install for pip

    warning: no files found matching '*.html' under directory 'docs'
    warning: no previously-included files matching '*.txt' found under directory 'docs/_build'
    no previously-included directories found matching 'docs/_build/_sources'
    Installing pip script to /mnt/tarmac/cache/canonical-identity-provider/trunk/.env/bin
    Installing pip-2.7 script to /mnt/tarmac/cache/canonical-identity-provider/trunk/.env/bin
Successfully installed pip
Cleaning up...
pip install --find-links=. --no-index -r /mnt/tarmac/cache/canonical-identity-provider/trunk/requirements.txt
Ignoring indexes: ht...

[Ubuntu One Auto Pilot (otto-pilot)]

The attempt to merge lp:~james-w/canonical-identity-provider/oops2 into lp:canonical-identity-provider failed. Below is the output from the failed tests.

Updating download cache at dir /mnt/tarmac/cache/isd-download-cache
Using saved parent location: bzr+ssh://bazaar.launchpad.net/~canonical-isd-hackers/+junk/download-cache/
Now on revision 16.
[localhost] local: which virtualenv
[localhost] local: /usr/bin/python /usr/bin/virtualenv --version
[localhost] local: /usr/bin/python /usr/bin/virtualenv --distribute --clear .env
Not deleting .env/bin
New python executable in .env/bin/python
Installing distribute.............................................................................................................................................................................................done.
Installing pip...............done.
[localhost] local: dpkg -l libpq-dev 2> /dev/null | grep '^ii' | wc -l
[localhost] local: dpkg -l libxml2-dev 2> /dev/null | grep '^ii' | wc -l
[localhost] local: dpkg -l libxslt1-dev 2> /dev/null | grep '^ii' | wc -l
[localhost] local: dpkg -l memcached 2> /dev/null | grep '^ii' | wc -l
[localhost] local: dpkg -l postgresql-plpython-9.1 2> /dev/null | grep '^ii' | wc -l
[localhost] local: dpkg -l python-dev 2> /dev/null | grep '^ii' | wc -l
[localhost] local: dpkg -l python-m2crypto 2> /dev/null | grep '^ii' | wc -l
[localhost] local: dpkg -l swig 2> /dev/null | grep '^ii' | wc -l
[localhost] local: dpkg -l config-manager 2> /dev/null | grep '^ii' | wc -l
[localhost] local: dpkg -l python-egenix-mx-base-dev 2> /dev/null | grep '^ii' | wc -l
[localhost] local: rm -rf M2Crypto*
[localhost] local: ln -s /usr/lib/python2.7/dist-packages/M2Crypto* .
[localhost] local: /usr/lib/config-manager/cm.py update /tmp/tmpilY6JQ
[localhost] local: . /mnt/tarmac/cache/canonical-identity-provider/trunk/.env/bin/activate && make install PACKAGES="-r /mnt/tarmac/cache/canonical-identity-provider/trunk/requirements.txt"
pip install --find-links=file:///mnt/tarmac/cache/isd-download-cache --no-index pip==dev
Ignoring indexes: http://pypi.python.org/simple/
Downloading/unpacking pip==dev
  Running setup.py egg_info for package pip

    warning: no files found matching '*.html' under directory 'docs'
    warning: no previously-included files matching '*.txt' found under directory 'docs/_build'
    no previously-included directories found matching 'docs/_build/_sources'
Installing collected packages: pip
  Found existing installation: pip 1.1
    Uninstalling pip:
      Successfully uninstalled pip
  Running setup.py install for pip

    warning: no files found matching '*.html' under directory 'docs'
    warning: no previously-included files matching '*.txt' found under directory 'docs/_build'
    no previously-included directories found matching 'docs/_build/_sources'
    Installing pip script to /mnt/tarmac/cache/canonical-identity-provider/trunk/.env/bin
    Installing pip-2.7 script to /mnt/tarmac/cache/canonical-identity-provider/trunk/.env/bin
Successfully installed pip
Cleaning up...
pip install --find-links=. --no-index -r /mnt/tarmac/cache/canonical-identity-provider/trunk/requirements.txt
Ignoring indexes: http://pypi....

=== modified file '.bzrignore'
--- .bzrignore	2013-03-05 13:51:14 +0000
+++ .bzrignore	2013-03-19 21:11:19 +0000
@@ -28,4 +28,5 @@
 branches/*
 django_project/config_dev/config/local.cfg
 scripts/local.cfg-
+./oopses/*
 

=== modified file 'django_project/config_dev/config/devel.cfg'
--- django_project/config_dev/config/devel.cfg	2013-03-12 19:14:34 +0000
+++ django_project/config_dev/config/devel.cfg	2013-03-19 21:11:19 +0000
@@ -196,3 +196,19 @@
 
 [api]
 api_host = http://%(hostname)s
+
+[oopses]
+oopses = oops_config
+
+[oops_config]
+template = oops_template
+publishers = oops_dev_publisher
+
+[oops_template]
+reporter = SSO-DEV
+
+[oops_dev_publisher]
+type = datedir
+error_dir = oopses
+inherit_id = true
+only_new = false

=== modified file 'django_project/config_dev/django.wsgi'
--- django_project/config_dev/django.wsgi	2013-02-22 12:16:44 +0000
+++ django_project/config_dev/django.wsgi	2013-03-19 21:11:19 +0000
@@ -31,16 +31,19 @@
 
 os.environ['PGCONNECT_TIMEOUT'] = str(settings.PGCONNECT_TIMEOUT)
 
-default_id = ''.join(x for x in platform.node() if x.isalpha())
-appserver_id = getattr(settings, 'APPSERVER_ID', default_id)
-appserver_id += settings.BRAND
-
-app = WSGIHandler()
-
-
-def application(environ, start_response):
-    oops_dir = getattr(settings, 'OOPS_DIR', '/tmp')
-    serializer = 'canonical.oops.serializer.OOPSRFC822Serializer'
-    oops = OopsWare(app, oops_dir=oops_dir, key=appserver_id, hide_meta=True,
-        serializer_factory_name=serializer)
-    return oops(environ, start_response)
+from identityprovider.wsgi_handler import OOPSWSGIHandler
+
+app = OOPSWSGIHandler()
+
+# Wrap the application in the Oops wsgi app to catch unhandled exceptions
+# and create oops for them.
+#
+# First we create the config that defines what to do with the oopses.
+import oops_dictconfig
+from oops_wsgi import make_app, install_hooks
+
+config = oops_dictconfig.config_from_dict(settings.OOPSES)
+install_hooks(config)
+
+# Then we wrap the django app in the oops one
+application = make_app(app, config, oops_on_status=['500'])

=== renamed symlink 'django_project/django.wsgi' => 'django_project/wsgi.py'
=== modified file 'identityprovider/schema.py'
--- identityprovider/schema.py	2013-03-12 19:14:34 +0000
+++ identityprovider/schema.py	2013-03-19 21:11:19 +0000
@@ -18,6 +18,7 @@
     merge,
 )
 from django_configglue.schema import schemas
+from oops_dictconfig.configglue_options import OopsOption
 
 from ubuntu_sso_saml.schema import Saml2IdpSchema
 
@@ -278,6 +279,9 @@
         session_cookie_secure = BoolOption(default=True)
         session_cookie_httponly = BoolOption(default=True)
 
+    class oopses(Section):
+        oopses = OopsOption()
+
 
 # merge all contrib schemas into the base schema
 # order matters

=== added file 'identityprovider/wsgi_handler.py'
--- identityprovider/wsgi_handler.py	1970-01-01 00:00:00 +0000
+++ identityprovider/wsgi_handler.py	2013-03-19 21:11:19 +0000
@@ -0,0 +1,32 @@
+import uuid
+
+from oops_wsgi import django as oops_wsgi_django
+
+
+class OOPSWSGIHandler(oops_wsgi_django.OOPSWSGIHandler):
+    """
+    Custom WSGI Handler that generates oops ids when an error is encountered
+    so the ids can be reported to the user.
+    """
+
+    def handle_uncaught_exception(self, request, resolver, exc_info):
+        # Generate OOPS id early so that we can render the error page
+        # right
+        # away and not depend on passing thread locals around.
+        if 'oops.report' in request.environ:
+            unique_id = uuid.uuid4().hex
+            request.environ['oops.report']['id'] = "OOPS-%s" % unique_id
+        return super(OOPSWSGIHandler, self).handle_uncaught_exception(
+            request, resolver, exc_info)
+
+    def __call__(self, environ, start_response):
+        def start_response_with_exc_info(status, headers, exc_info=None):
+            """Custom start_response callback for wsgi."""
+            # This will pass the exception information back to oops_wsgi.  It
+            # should not be necessary once
+            # https://code.djangoproject.com/ticket/16674 has been merged.
+            if exc_info is None:
+                exc_info = environ['oops.context'].get('exc_info', None)
+            return start_response(status, headers, exc_info)
+        return super(OOPSWSGIHandler, self).__call__(
+            environ, start_response_with_exc_info)

=== added directory 'oopses'
=== modified file 'requirements/install.txt'
--- requirements/install.txt	2013-03-05 15:52:39 +0000
+++ requirements/install.txt	2013-03-19 21:11:19 +0000
@@ -14,6 +14,10 @@
 M2Crypto==0.21.1
 oath==1.0
 oauth==1.0.1
+oops_amqp==0.0.7
+oops_datedir_repo==0.0.20
+oops_dictconfig==0.0.4
+oops_wsgi==0.0.10
 psycopg2==2.4.1
 pystatsd==0.1.6
 python-memcached==1.44

=== modified file 'webui/views/errors.py'
--- webui/views/errors.py	2012-12-04 14:17:22 +0000
+++ webui/views/errors.py	2013-03-19 21:11:19 +0000
@@ -13,7 +13,7 @@
 
     def __call__(self, request):
         template = loader.get_template(self.template_name)
-        oopsid = request.META.get('OOPSID')
+        oopsid = request.environ.get('oops.report', {}).get('id', None)
         atts = {
             'request': request,
             'oopsid': oopsid,