Merge lp:~michael.nelson/canonical-identity-provider/add-convoy-combo into lp:canonical-identity-provider/release

[Michael Nelson]

WIP for discussion: An initial combo loader service which assumes a structure enabling multiple versions of various projects to be loaded.

Depending on the needs of the page, the loader can be use to load resources for a single project (ie. '?widget/widget-base.js&anim/anim-base.js' both from YUI), or across projects (ie. '?yui/3.3.0/widget-widget-base.js&ubuntuone/js/ie/html5shiv.js'). See the tests below.

The combo_url setting is added and used so that later we can switch to an external combo loader on a separate subdomain with a config change.

Questions
=========
 * Location? (in webui, but static media is in identityprovider?)

TODO
====
 * Enable revision in request for local sso media
 *

[lp:~michael.nelson/canonical-identity-provider/add-convoy-combo]

776. By Michael Nelson on 2013-04-10

GREEN: combo template tag uses static_url.

777. By Michael Nelson on 2013-04-10

REFACTOR: move combo template tag to webui app.

778. By Michael Nelson on 2013-04-10

REFACTOR: combo template tag uses combo when debug is false.

779. By Michael Nelson on 2013-04-10

REFACTOR: add prefx option to combo tag.

780. By Michael Nelson on 2013-04-11

Convert to django 1.3 valid tag :/

781. By Michael Nelson on 2013-04-11

Use combo tag on ubuntuone base template.

782. By Michael Nelson on 2013-04-11

Use combo for js loading on login.

783. By Michael Nelson on 2013-04-11

Merged trunk from prev.

[Natalia Bidart (nataliabidart)]

Hi! I'm doing some cleanup on SSO MPs, I assume this is no longer valid, will set as Rejected.

Please shange the global status if this is still current. Thanks!

Unmerged revisions

783. By Michael Nelson on 2013-04-11

Merged trunk from prev.

782. By Michael Nelson on 2013-04-11

Use combo for js loading on login.

781. By Michael Nelson on 2013-04-11

Use combo tag on ubuntuone base template.

780. By Michael Nelson on 2013-04-11

Convert to django 1.3 valid tag :/

779. By Michael Nelson on 2013-04-10

REFACTOR: add prefx option to combo tag.

778. By Michael Nelson on 2013-04-10

REFACTOR: combo template tag uses combo when debug is false.

777. By Michael Nelson on 2013-04-10

REFACTOR: move combo template tag to webui app.

776. By Michael Nelson on 2013-04-10

GREEN: combo template tag uses static_url.

775. By Michael Nelson on 2013-04-10

Merged fix_static_url

774. By Michael Nelson on 2013-04-10

RED: combo template tag uses static_url

=== modified file '.bzrignore'
--- .bzrignore	2013-03-18 21:08:47 +0000
+++ .bzrignore	2013-04-11 12:24:22 +0000
@@ -30,3 +30,5 @@
 scripts/local.cfg-
 ./oopses/*
 
+# Collected static files.
+django_project/static/*

=== modified file 'django_project/config_dev/config/devel.cfg'
--- django_project/config_dev/config/devel.cfg	2013-04-03 20:34:24 +0000
+++ django_project/config_dev/config/devel.cfg	2013-04-11 12:24:22 +0000
@@ -22,6 +22,7 @@
     django.contrib.messages
     django.contrib.sessions
     django.contrib.sites
+    django.contrib.staticfiles
     django.contrib.admin
     django_openid_auth
     django_configglue

=== modified file 'django_project/config_dev/config/main.cfg'
--- django_project/config_dev/config/main.cfg	2013-04-11 12:24:22 +0000
+++ django_project/config_dev/config/main.cfg	2013-04-11 12:24:22 +0000
@@ -37,6 +37,7 @@
     django.contrib.messages
     django.contrib.sessions
     django.contrib.sites
+    django.contrib.staticfiles
     django.contrib.admin
     django_openid_auth
     django_configglue
@@ -80,8 +81,10 @@
 server_email = noreply@%(domain)s
 session_cookie_secure = true
 session_engine = django.contrib.sessions.backends.db
-static_root = media
+static_root = django_project/static
 static_url = /assets/
+staticfiles_dirs = 
+    django_project/external_staticfiles
 template_context_processors =
     django.contrib.messages.context_processors.messages
     django.contrib.auth.context_processors.auth

=== added directory 'django_project/external_staticfiles'
=== added directory 'django_project/external_staticfiles/yui'
=== added symlink 'django_project/external_staticfiles/yui/3.3.0'
=== target is u'../../../identityprovider/media/src-js/lazrjs/yui'
=== added directory 'django_project/static'
=== added directory 'identityprovider/media/yui'
=== added symlink 'identityprovider/media/yui/3.3.0'
=== target is u'../src-js/lazrjs/yui/'
=== modified file 'identityprovider/schema.py'
--- identityprovider/schema.py	2013-03-28 13:08:29 +0000
+++ identityprovider/schema.py	2013-04-11 12:24:22 +0000
@@ -169,6 +169,7 @@
             help="URL to POST when account data has changed")
         embedded_trust_root = StringOption()
         max_password_reset_tokens = IntOption(default=5)
+        static_combo_url = StringOption(default="/combo/")
 
     class twofactor(Section):
         hotp_drift = IntOption()

=== added directory 'identityprovider/static'
=== added directory 'identityprovider/static/identityprovider'
=== added symlink 'identityprovider/static/identityprovider/ubuntu'
=== target is u'../../media/ubuntu'
=== added symlink 'identityprovider/static/identityprovider/ubuntuone'
=== target is u'../../media/ubuntuone'
=== modified file 'requirements/install.txt'
--- requirements/install.txt	2013-04-10 18:03:19 +0000
+++ requirements/install.txt	2013-04-11 12:24:22 +0000
@@ -1,6 +1,7 @@
 argparse==1.2.1
 BeautifulSoup==3.2.1
 configglue==1.0.3
+convoy==0.2.4
 Django==1.3
 django-adminaudit==0.3.3
 django-configglue==0.6.1

=== modified file 'webui/templates/ubuntuone/base.html'
--- webui/templates/ubuntuone/base.html	2013-04-11 12:24:22 +0000
+++ webui/templates/ubuntuone/base.html	2013-04-11 12:24:22 +0000
@@ -1,4 +1,4 @@
-{% load i18n static_url %}<!DOCTYPE html>
+{% load i18n static_url combo %}<!DOCTYPE html>
 <!--[if IE 7 ]><html class="ie7" lang="en" dir="ltr"><![endif]-->
 <!--[if IE 8 ]><html class="ie8" lang="en" dir="ltr"><![endif]-->
 <!--[if (gte IE 9)|!(IE)]><!--><html lang="en" dir="ltr" {% block html_extra %}{% endblock %}><!--<![endif]-->
@@ -14,11 +14,7 @@
     <link rel="icon" type="image/vnd.microsoft.icon" href="{{ STATIC_URL }}identityprovider/ubuntu/favicon.ico">
 
     {% block corecss %}
-    <link href="{{STATIC_URL}}identityprovider/ubuntuone/css/grids-min.css" rel="stylesheet" type="text/css">
-    <link href="{{STATIC_URL}}identityprovider/ubuntuone/css/cssgrids-responsive-min.css" rel="stylesheet" type="text/css">
-    <link href="{{STATIC_URL}}identityprovider/ubuntuone/css/base.css" rel="stylesheet" type="text/css">
-    <link href="{{STATIC_URL}}identityprovider/ubuntuone/css/ubuntuone.css" rel="stylesheet" type="text/css">
-    <link href="{{STATIC_URL}}identityprovider/ubuntuone/css/footer-base.css" rel="stylesheet" type="text/css">
+    {% combo prefix='identityprovider/ubuntuone/css' 'grids-min.css' 'cssgrids-responsive-min.css' 'base.css' 'ubuntuone.css' 'footer-base.css' %}
 
     <link href='http://fonts.googleapis.com/css?family=Ubuntu:400,300,700' rel='stylesheet' type='text/css'>
 
@@ -85,7 +81,7 @@
 {% block general_js %}
 
     {% comment %}To be replaced by u1 js{% endcomment %}
-    <script src="{{ STATIC_URL }}identityprovider/lazr-js/yui/yui-min.js"></script>
+    {% combo prefix='yui/3.3.0' 'yui/yui-min.js' 'oop/oop-min.js' 'loader/loader-min.js' 'event-custom/event-custom-base-min.js' 'event/event-base-min.js' 'pluginhost/pluginhost-min.js' 'dom/dom-min.js' 'node/node-min.js' 'event/event-delegate-min.js' %}
 
 {% endblock %}
 

=== added directory 'webui/templatetags'
=== added file 'webui/templatetags/__init__.py'
=== added file 'webui/templatetags/combo.py'
--- webui/templatetags/combo.py	1970-01-01 00:00:00 +0000
+++ webui/templatetags/combo.py	2013-04-11 12:24:22 +0000
@@ -0,0 +1,71 @@
+# Copyright 2013 Canonical Ltd.  This software is licensed under the
+# GNU Affero General Public License version 3 (see the file LICENSE).
+
+from django import template
+from django.conf import settings
+from django.template import TemplateSyntaxError
+from django.template.base import Node
+from django.template.defaulttags import kwarg_re
+
+register = template.Library()
+
+
+css_template = ('<link href="{0}" rel="stylesheet" type="text/css" '
+                'media="screen" />')
+
+js_template = '<script type="text/javascript" src="{0}"></script>'
+
+# XXX Once we move beyond django 1.3 we can just use a simple_tag
+# which will handle *args and **kwargs with half the code.
+
[e-mail]
+def combo(parser, token):
+    """Parse the args and kwargs - based on django's url tag."""
+    bits = token.split_contents()
+    tagname = bits[0]
+    if len(bits) < 2:
+        raise TemplateSyntaxError(
+            "'%s' takes at least one argument." % tagname)
+
+    filenames = []
+    options = {}
+    bits = bits[1:]
+    for bit in bits:
+        match = kwarg_re.match(bit)
+        if not match:
+            raise TemplateSyntaxError(
+                "Malformed arguments to '{0}' tag.".format(tagname))
+        name, value = match.groups()
+        if name:
+            options[name] = parser.compile_filter(value)
+        else:
+            filenames.append(parser.compile_filter(value))
+
+    return ComboNode(filenames, options)
+
+
+class ComboNode(Node):
+
+    def __init__(self, filenames, options):
+        self.filenames = filenames
+        self.options = options
+
+    def render(self, context):
+        filenames = [f.resolve(context) for f in self.filenames]
+        options = dict([(k, v.resolve(context))
+                       for k, v in self.options.items()])
+        prefix = options.get('prefix', '').strip('/')
+        template = css_template
+        if filenames[0].endswith('.js'):
+            template = js_template
+
+        filenames = [f.strip('/') for f in filenames]
+        if prefix:
+            filenames = ['/'.join([prefix, filename]) for filename in filenames]
+
+        if settings.DEBUG or len(filenames) == 1:
+            return "\n".join([
+                template.format(settings.STATIC_URL + f) for f in filenames])
+
+        url = settings.STATIC_COMBO_URL + '?' + '&'.join(filenames)
+        return template.format(url)

=== added file 'webui/tests/test_templatetags.py'
--- webui/tests/test_templatetags.py	1970-01-01 00:00:00 +0000
+++ webui/tests/test_templatetags.py	2013-04-11 12:24:22 +0000
@@ -0,0 +1,75 @@
+# Copyright 2013 Canonical Ltd.  This software is licensed under the
+# GNU Affero General Public License version 3 (see the file LICENSE).
+
+from unittest import TestCase
+
+from django.conf import settings
+from django.template import (
+    Context,
+    Template,
+)
+from mock import patch
+
+
+class ComboTestCase(TestCase):
+
+    def test_debug(self):
+        cases = ((
+            "'a/b/foo.css' 'd/e/goo.css'",
+            '<link href="/static/a/b/foo.css" rel="stylesheet"'
+            ' type="text/css" media="screen" />\n'
+            '<link href="/static/d/e/goo.css" rel="stylesheet"'
+            ' type="text/css" media="screen" />',
+        ), (
+            "'a/b/foo.css'",
+            '<link href="/static/a/b/foo.css" rel="stylesheet"'
+            ' type="text/css" media="screen" />'
+        ), (
+            "'a/b/foo.js' 'd/e/goo.js'",
+            '<script type="text/javascript" src="/static/a/b/foo.js">'
+            '</script>\n'
+            '<script type="text/javascript" src="/static/d/e/goo.js">'
+            '</script>'
+        ), (
+            "'foo.js' 'goo.js' prefix='a/b'",
+            '<script type="text/javascript" src="/static/a/b/foo.js">'
+            '</script>\n'
+            '<script type="text/javascript" src="/static/a/b/goo.js">'
+            '</script>'
+        ))
+
+        with patch.multiple(settings, DEBUG=True, STATIC_URL='/static/'):
+            for (tag_input, expected) in cases:
+                result = Template(
+                    "{% load combo %}"
+                    "{% combo " + tag_input + " %}").render(Context())
+
+                self.assertEqual(expected, result)
+
+    def test_non_debug(self):
+        cases = ((
+            "'a/b/foo.css' 'd/e/goo.css'",
+            '<link href="/combo/?a/b/foo.css&d/e/goo.css" rel="stylesheet"'
+            ' type="text/css" media="screen" />',
+        ), (
+            "'a/b/foo.css'",
+            '<link href="/static/a/b/foo.css" rel="stylesheet"'
+            ' type="text/css" media="screen" />'
+        ), (
+            "'a/b/foo.js' 'd/e/goo.js'",
+            '<script type="text/javascript"'
+            ' src="/combo/?a/b/foo.js&d/e/goo.js"></script>',
+        ), (
+            "'foo.js' 'goo.js' prefix='a/b'",
+            '<script type="text/javascript"'
+            ' src="/combo/?a/b/foo.js&a/b/goo.js"></script>',
+        ))
+
+        with patch.multiple(settings, DEBUG=False, STATIC_COMBO_URL='/combo/',
+                STATIC_URL='/static/'):
+            for (tag_input, expected) in cases:
+                result = Template(
+                    "{% load combo %}"
+                    "{% combo " + tag_input + " %}").render(Context())
+
+                self.assertEqual(expected, result)

=== modified file 'webui/tests/test_views_ui.py'
--- webui/tests/test_views_ui.py	2013-04-09 13:39:29 +0000
+++ webui/tests/test_views_ui.py	2013-04-11 12:24:22 +0000
@@ -16,6 +16,7 @@
 from django.conf import settings
 from django.contrib.sessions.models import Session
 from django.core import mail
+from django.core.management import call_command
 from django.core.urlresolvers import reverse
 from django.http import QueryDict, HttpResponse
 
@@ -1528,3 +1529,23 @@
         self.assertTrue(ui.authenticate_device(account, '123456'))
         self.assertTrue(devices[0].authenticate.called)
         self.assertTrue(devices[1].authenticate.called)
+
+
+class ComboViewTestCase(SSOBaseTestCase):
+
+    @classmethod
+    def setUpClass(cls):
+        """Ensure the static files are ready for combo loading."""
+        call_command('collectstatic', verbosity=0, interactive=False)
+
+    def test_combine_files_multiple_projects(self):
+        response = self.client.get(settings.STATIC_COMBO_URL +
+            '?yui/3.3.0/widget/widget-base.js'
+            '&identityprovider/ubuntuone/js/ie/html5shiv.js')
+
+        self.assertEqual(response['content-type'], 'text/javascript')
+        self.assertNotContains(response, '[missing]')
+        self.assertContains(response, '/* yui/3.3.0/widget/widget-base.js */')
+        self.assertContains(
+            response,
+            '/* identityprovider/ubuntuone/js/ie/html5shiv.js */')

=== modified file 'webui/urls.py'
--- webui/urls.py	2013-04-05 05:55:33 +0000
+++ webui/urls.py	2013-04-11 12:24:22 +0000
@@ -73,7 +73,7 @@
     url(r'^\+faq$', 'static_page', {'page_name': 'faq'}, name='faq'),
     url(r'^\+ubuntuone-account$', 'static_page',
         {'page_name': 'ubuntuone-account'}, name='ubuntuone-account'),
-
+    url(r'^combo/$', 'combo_view', name='combo-view'),
 )
 
 urlpatterns += patterns(

=== modified file 'webui/views/ui.py'
--- webui/views/ui.py	2013-04-02 08:38:51 +0000
+++ webui/views/ui.py	2013-04-11 12:24:22 +0000
@@ -3,7 +3,12 @@
 # LICENSE).
 
 import logging
+import os
 
+from convoy.combo import (
+    combine_files,
+    parse_qs,
+)
 from django import forms
 from django.conf import settings
 from django.contrib import auth, messages
@@ -12,6 +17,7 @@
 from django.db.models import F
 from django.http import (
     Http404,
+    HttpResponse,
     HttpResponseNotAllowed,
     HttpResponseRedirect,
     urlencode,
@@ -970,3 +976,26 @@
 
     return render_to_response('static/%s.html' % page_name,
                               RequestContext(request))
+
+
+def combo_view(request):
+    """Handle a request for combining a set of files."""
+    fnames = parse_qs(request.META.get("QUERY_STRING", ""))
+    content_type = "text/plain"
+
+    if fnames:
+        if fnames[0].endswith(".js"):
+            content_type = "text/javascript"
+        elif fnames[0].endswith(".css"):
+            content_type = "text/css"
+
+        # XXX Check what resource_prefix is used for.
+        content = combine_files(
+            fnames, os.path.abspath(settings.STATIC_ROOT),
+            resource_prefix=settings.STATIC_URL, rewrite_urls=True)
+        # We're turning the generator returned by combine_files into a string
+        # here since GZipMiddleware would consume it if not. See Bug #822888.
+        return HttpResponse(
+            content_type=content_type, status=200,
+            content="".join(content))
+    return HttpResponse(content_type=content_type, status=404)