Merge lp:~elachuni/ubuntu-webcatalog/celery into lp:ubuntu-webcatalog

[Anthony Lenton]

Added a view to schedule asynchronous tasks via Celery.

Overview
========
This branch adds a view to schedule asynchronous tasks via Celery. Only one basic task is provided to import exhibits (plus one that always fails, for testing), the full set of tasks will be in a separate MP, now that the groundwork is in place. A brief video demo can be found on http://people.canonical.com/~anthony/celery-uwc.ogv

Details
=======
The current branch counts on using CELERY_RESULT_BACKEND='database', as it fetches the result for each task from the DB. Any BROKER_BACKEND can be used, I specifically tested with djkombu and amqp locally.

The TaskMeta model was registered in the admin so that results can be browsed and searched in the admin UI after a task has run. The ModelAdmin class was improved a bit after grabbing the video, you can see the final list view on http://people.canonical.com/~anthony/TaskMetaAdmin.png

The list of tasks available is put together by introspecting webcatalog.tasks, so adding new tasks available will be picked up and added automatically.

The task list template uses YUI.io to retrieve the task's status, but submits a POST form and reloads the whole page when you schedule a new task; I considered doing everything with ajax, but POST via ajax is a bit tricky, specially when csrf tokens are involved, so I went for this simpler solution that seems to be responsive enough.

[James Westby (james-w)]

30 -openid_launchpad_staff_teams = canonical-ca-hackers
31 +openid_launchpad_staff_teams = canonical-isd-hackers

That looks backwards?

332 + if (message.hasClass('success')) {

That looks as if it will loop forever on successful tasks?

Most of this branch looks pretty good to me.

Thanks,

James

[lp:~elachuni/ubuntu-webcatalog/celery]

152. By Anthony Lenton on 2012-06-28

Added a few tasks.

153. By Anthony Lenton on 2012-06-28

Merged in latest changes from trunk.

154. By Anthony Lenton on 2012-06-29

Query TaskState model instead of TaskMeta.

[Anthony Lenton (elachuni)]

> 30 -openid_launchpad_staff_teams = canonical-ca-hackers
> 31 +openid_launchpad_staff_teams = canonical-isd-hackers
>
> That looks backwards?

True, for some reason staging sso *still* doesn't report me as being part of canonical-ca-hackers, so I use canonical-isd-hackers instead during dev, I've put it back now, thanks!

> 332 + if (message.hasClass('success')) {
>
> That looks as if it will loop forever on successful tasks?

Hm it shouldn't, that call is outside of check_status so it's just to trigger the first call. Maybe moving this into an function called init() or onload() would make it clearer.

[lp:~elachuni/ubuntu-webcatalog/celery]

155. By Anthony Lenton on 2012-07-02

Pulled back config change per code review.

156. By Anthony Lenton on 2012-07-02

Merged in latest changes from trunk. Made all tests pass.

[Danny Tamez (zematynnad)]

line 111 - we don't need this any longer as we're now on 2.6
looks like line 115 is not needed
at 158, that looks like that could have stayed static but I guess it makes more sense to make it a a regular method as that's how the class will now be used...
same for 183

screencast was great! Is there any way to have something more useful than the task id to help find the task you want to look at? I'm not sure of the use cases for searching for a task so that id may be the only way that makes sense...

Approved - feel free to act on or ignore the comments.

review: Approve

[lp:~elachuni/ubuntu-webcatalog/celery]

157. By Anthony Lenton on 2012-07-02

Removed pointless imported with_statement

=== modified file 'django_project/config/main.cfg'
--- django_project/config/main.cfg	2012-06-08 08:24:11 +0000
+++ django_project/config/main.cfg	2012-07-02 21:27:20 +0000
@@ -25,6 +25,7 @@
                  south
                  preflight
                  pgtools
+                 djcelery
 login_url = /openid/login/
 managers = %(admins)s
 middleware_classes = django.middleware.cache.UpdateCacheMiddleware
@@ -55,12 +56,12 @@
 static_root = ./django_project/static/
 static_url = /assets/
 admin_media_prefix = /assets/admin/
+test_runner = djcelery.contrib.test_runner.CeleryTestSuiteRunner
 
 # Django-1.1 backwards compatibility
 database_engine = sqlite3
 database_name = webcatalog.db
 
-
 [django_databases]
 default = default_database
 
@@ -103,6 +104,7 @@
 # Structure here is:   LP Team = Django Group
 canonical-isd-hackers = admin
 canonical-losas = admin
+canonical-ca-hackers = developers
 
 [oops]
 oops_dir = /srv/%(hostname)s/staging-logs/www-oops
@@ -134,3 +136,10 @@
 sso_api_auth_username = insert-your-sso-api-username-here
 sso_api_auth_password = insert-your-sso-api-password-here
 sso_auth_mode_no_ubuntu_sso_plaintext_only = True
+
+[celery]
+broker_backend = amqp
+broker_url = amqp://guest:guest@localhost:5672//
+celery_imports = webcatalog.tasks
+celery_result_backend = database
+celerybeat_scheduler = djcelery.schedulers.DatabaseScheduler
 \ No newline at end of file
 
=== modified file 'django_project/settings.py'
--- django_project/settings.py	2012-03-27 20:43:10 +0000
+++ django_project/settings.py	2012-07-02 21:27:20 +0000
@@ -13,4 +13,7 @@
                     'local.cfg'])
                 if os.path.exists(path)]
 
-configglue(WebCatalogSchema, config_files, __name__)
 \ No newline at end of file
+configglue(WebCatalogSchema, config_files, __name__)
+
+import djcelery
+djcelery.setup_loader()
 
=== modified file 'setup.py'
--- setup.py	2012-06-28 14:28:26 +0000
+++ setup.py	2012-07-02 21:27:20 +0000
@@ -50,6 +50,8 @@
         'ssoclient==1.0',
         'pep8',
         'PIL',
+        'celery',
+        'django-celery',
         ],
     package_data = find_packages_data('src'),
     dependency_links = [
 
=== modified file 'src/webcatalog/admin.py'
--- src/webcatalog/admin.py	2012-06-28 13:29:29 +0000
+++ src/webcatalog/admin.py	2012-07-02 21:27:20 +0000
@@ -17,10 +17,8 @@
 
 """Admin classes for the Apps Directory."""
 
-from __future__ import (
-    absolute_import,
-    with_statement,
-)
+from __future__ import absolute_import
+
 from django.contrib import admin
 from webcatalog.models import (
     Application,
 
=== added file 'src/webcatalog/decorators.py'
--- src/webcatalog/decorators.py	1970-01-01 00:00:00 +0000
+++ src/webcatalog/decorators.py	2012-07-02 21:27:20 +0000
@@ -0,0 +1,32 @@
+#   -*- coding: utf-8 -*-
+#   This file is part of the Apps Directory
+#   Copyright (C) 2011 Canonical Ltd.
+#
+#   This program is free software: you can redistribute it and/or modify
+#   it under the terms of the GNU Affero General Public License as
+#   published by the Free Software Foundation, either version 3 of the
+#   License, or (at your option) any later version.
+#
+#   This program is distributed in the hope that it will be useful,
+#   but WITHOUT ANY WARRANTY; without even the implied warranty of
+#   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+#   GNU Affero General Public License for more details.
+#
+#   You should have received a copy of the GNU Affero General Public License
+#   along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+"""Decorators for the webcatalog application."""
+
+from __future__ import absolute_import
+
+from django.contrib.auth.decorators import permission_required
+
+
+__metaclass__ = type
+__all__ = [
+    'scheduler_required',
+]
+
+
+scheduler_required = permission_required('webcatalog.schedule_tasks',
+                                         login_url='/cat/forbidden/')
 
=== modified file 'src/webcatalog/department_filters.py'
--- src/webcatalog/department_filters.py	2012-06-06 17:42:04 +0000
+++ src/webcatalog/department_filters.py	2012-07-02 21:27:20 +0000
@@ -17,10 +17,7 @@
 
 """Department filters."""
 
-from __future__ import (
-    absolute_import,
-    with_statement,
-)
+from __future__ import absolute_import
 
 import re
 
 
=== modified file 'src/webcatalog/forms.py'
--- src/webcatalog/forms.py	2012-06-28 14:25:19 +0000
+++ src/webcatalog/forms.py	2012-07-02 21:27:20 +0000
@@ -17,10 +17,7 @@
 
 """Forms used by the Apps Directory."""
 
-from __future__ import (
-    absolute_import,
-    with_statement,
-)
+from __future__ import absolute_import
 
 import apt
 import json
 
=== modified file 'src/webcatalog/management/commands/cleanup.py'
--- src/webcatalog/management/commands/cleanup.py	2012-06-06 16:38:11 +0000
+++ src/webcatalog/management/commands/cleanup.py	2012-07-02 21:27:20 +0000
@@ -27,16 +27,17 @@
 
 try:
     import psycopg2
-    from psycopg2.extensions import ISOLATION_LEVEL_AUTOCOMMIT
     psycopg2_available = True
 except ImportError:
     psycopg2_available = False
 
 
 class DjangoSessionCleaner(object):
-    @staticmethod
-    def declare_expired_keys_cursor(cur):
-        print "Opening cursor"
+    def __init__(self, cmd):
+        self.cmd = cmd
+
+    def declare_expired_keys_cursor(self, cur):
+        self.cmd.output("Opening cursor\n", 1)
         # We order by expire_date to force the index to be used.
         cur.execute("CLOSE ALL")
         cur.execute("""
@@ -46,8 +47,7 @@
             ORDER BY expire_date
             """)
 
-    @staticmethod
-    def remove_batch(cur, batch_size):
+    def remove_batch(self, cur, batch_size):
         cur.execute("FETCH %s FROM _django_session_clean", [batch_size])
         session_keys = [session_key for session_key, in cur.fetchall()]
         if not session_keys:
@@ -66,9 +66,11 @@
 
 
 class OAuthNonceCleaner(object):
-    @staticmethod
-    def declare_expired_keys_cursor(cur):
-        print "Opening cursor"
+    def __init__(self, cmd):
+        self.cmd = cmd
+
+    def declare_expired_keys_cursor(self, cur):
+        self.cmd.output("Opening cursor\n", 1)
         # We order by created_at to force the index to be used.
         # Use a 5 hour clock skew as defined in openid.store.nonce
         cur.execute("CLOSE ALL")
@@ -79,8 +81,7 @@
             ORDER BY created_at
             """)
 
-    @staticmethod
-    def remove_batch(cur, batch_size):
+    def remove_batch(self, cur, batch_size):
         cur.execute("FETCH %s FROM _oauth_nonce_clean", [batch_size])
         nonce_ids = [nonce_id for nonce_id, in cur.fetchall()]
         if not nonce_ids:
@@ -117,6 +118,7 @@
     )
 
     def handle_label(self, table, **options):
+        self.verbosity = int(options['verbosity'])
         if not psycopg2_available:
             raise CommandError('This command requires psycopg2')
 
@@ -128,8 +130,7 @@
             msg = ("Invalid cleaner.\nNo cleaner found for table: %s\n"
                    "Supported tables are: %s" % (table, CLEANERS.keys()))
             raise CommandError(msg)
-        cleaner = cleaner_cls()
-        connection.isolation_level = ISOLATION_LEVEL_AUTOCOMMIT
+        cleaner = cleaner_cls(self)
         cur = connection.cursor()
 
         removed = -1
@@ -149,12 +150,12 @@
             removed = cleaner.remove_batch(cur, batch_size)
             actual_batch_time = time.time() - batch_start
             total_removed += removed
-            print "Removed %d rows (%d total removed). Batch size %d" % (
-                removed, total_removed, batch_size)
+            message = "Removed %d rows (%d total removed). Batch size %d\n"
+            self.output(message % (removed, total_removed, batch_size), 1)
 
             # Done
             if removed == 0:
-                print "All done."
+                self.output("All done.\n", 1)
                 break
 
             # Increase or decrease the batch size by 10%, minimum 1.
@@ -163,3 +164,10 @@
                 batch_size -= batch_size_wobble
             elif actual_batch_time < target_batch_time * 0.9:
                 batch_size += batch_size_wobble
+
+    def output(self, message, level=None, flush=False):
+        if hasattr(self, 'stdout'):
+            if level is None or self.verbosity >= level:
+                self.stdout.write(message)
+            if flush:
+                self.stdout.flush()
 
=== modified file 'src/webcatalog/management/commands/import_all_app_install_data.py'
--- src/webcatalog/management/commands/import_all_app_install_data.py	2012-06-28 13:13:15 +0000
+++ src/webcatalog/management/commands/import_all_app_install_data.py	2012-07-02 21:27:20 +0000
@@ -17,10 +17,7 @@
 
 """Management command to import app install data for all distroseries."""
 
-from __future__ import (
-    absolute_import,
-    with_statement,
-)
+from __future__ import absolute_import
 
 __metaclass__ = type
 __all__ = []
@@ -37,7 +34,8 @@
         for distroseries in settings.UBUNTU_SERIES_FOR_VERSIONS.values():
             self.output("Importing app-install-data for {0}\n".format(
                 distroseries), 1)
-            call_command('import_app_install_data', distroseries)
+            call_command('import_app_install_data', distroseries,
+                         verbosity=self.verbosity)
         self.output("Running check_all_latest across all distroseries.\n", 1)
         call_command('check_all_latest')
 
 
=== modified file 'src/webcatalog/management/commands/import_all_ratings_stats.py'
--- src/webcatalog/management/commands/import_all_ratings_stats.py	2012-06-21 20:18:44 +0000
+++ src/webcatalog/management/commands/import_all_ratings_stats.py	2012-07-02 21:27:20 +0000
@@ -17,10 +17,7 @@
 
 """Management command to import review statistics for all distroseries."""
 
-from __future__ import (
-    absolute_import,
-    with_statement,
-)
+from __future__ import absolute_import
 
 __metaclass__ = type
 __all__ = []
@@ -37,7 +34,8 @@
         for distroseries in settings.UBUNTU_SERIES_FOR_VERSIONS.values():
             self.output("Importing ratings stats for {0}\n".format(
                 distroseries), 1)
-            call_command('import_ratings_stats', distroseries)
+            call_command('import_ratings_stats', distroseries,
+                         verbosity=self.verbosity)
 
     def output(self, message, level=None, flush=False):
         if hasattr(self, 'stdout'):
 
=== modified file 'src/webcatalog/management/commands/import_app_install_data.py'
--- src/webcatalog/management/commands/import_app_install_data.py	2012-06-28 14:46:12 +0000
+++ src/webcatalog/management/commands/import_app_install_data.py	2012-07-02 21:27:20 +0000
@@ -17,10 +17,7 @@
 
 """Management command to import app install data for a distroseries."""
 
-from __future__ import (
-    absolute_import,
-    with_statement,
-)
+from __future__ import absolute_import
 
 import os
 import re
 
=== modified file 'src/webcatalog/management/commands/import_sca_apps.py'
--- src/webcatalog/management/commands/import_sca_apps.py	2012-06-28 14:25:19 +0000
+++ src/webcatalog/management/commands/import_sca_apps.py	2012-07-02 21:27:20 +0000
@@ -19,10 +19,7 @@
    Center.
 """
 
-from __future__ import (
-    absolute_import,
-    with_statement,
-)
+from __future__ import absolute_import
 
 import json
 import os
 
=== modified file 'src/webcatalog/managers.py'
--- src/webcatalog/managers.py	2012-06-14 09:23:32 +0000
+++ src/webcatalog/managers.py	2012-07-02 21:27:20 +0000
@@ -17,11 +17,7 @@
 
 """Django object managers."""
 
-from __future__ import (
-    absolute_import,
-    with_statement,
-)
-
+from __future__ import absolute_import
 
 __metaclass__ = type
 __all__ = [
 
=== modified file 'src/webcatalog/models/applications.py'
--- src/webcatalog/models/applications.py	2012-06-28 12:27:22 +0000
+++ src/webcatalog/models/applications.py	2012-07-02 21:27:20 +0000
@@ -17,19 +17,22 @@
 
 """Models for the package application."""
 
-from __future__ import (
-    absolute_import,
-    with_statement,
-)
+from __future__ import absolute_import
 
 import logging
 from datetime import datetime
 
 from django.conf import settings
-from django.contrib.auth.models import User
+from django.contrib.auth.models import (
+    Group,
+    Permission,
+    User,
+)
+from django.contrib.contenttypes.models import ContentType
 from django.core.urlresolvers import reverse
 from django.db import models
 from django.template.defaultfilters import slugify
+from django.db.models.signals import post_syncdb
 
 from webcatalog.department_filters import department_filters
 from webcatalog.managers import ApplicationManager, DistroSeriesManager
@@ -290,3 +293,20 @@
 
         pkgs = '&'.join('pkg_name=%s' % pkg_name for pkg_name in pkg_names)
         return reverse('wc-package-list') + '?' + pkgs
+
+
+# GroupPermissions can't go in fixtures because Permissions are stored as
+# model metadata, so our Permission's primary key can change without warning
+def post_syncdb_handler(sender, **kwargs):
+    ct = ContentType.objects.get_for_model(Application)
+    group, _ = Group.objects.get_or_create(name='developers')
+    perm, _ = Permission.objects.get_or_create(
+        codename='schedule_tasks',
+        name="Can schedule asynchronous tasks via the web",
+        content_type=ct,
+    )
+    if not perm in group.permissions.all():
+        group.permissions.add(perm)
+        group.save()
+
+post_syncdb.connect(post_syncdb_handler)
 
=== modified file 'src/webcatalog/preflight.py'
--- src/webcatalog/preflight.py	2012-06-06 17:42:04 +0000
+++ src/webcatalog/preflight.py	2012-07-02 21:27:20 +0000
@@ -17,10 +17,7 @@
 
 """Preflight check for webcatalog."""
 
-from __future__ import (
-    absolute_import,
-    with_statement,
-)
+from __future__ import absolute_import
 
 from django import db
 from django.conf import settings
 
=== modified file 'src/webcatalog/schema.py'
--- src/webcatalog/schema.py	2012-06-14 23:40:45 +0000
+++ src/webcatalog/schema.py	2012-07-02 21:27:20 +0000
@@ -98,3 +98,14 @@
         rec_service_root = schema.StringOption(
             default="http://rec.ubuntu.com/api/1.0")
         num_recommended_apps = schema.IntOption(default=4)
+
+    class celery(schema.Section):
+        broker_backend = schema.StringOption(default='memory')
+        celery_result_backend = schema.StringOption(default='memory')
+        celery_always_eager = schema.BoolOption()
+        celery_eager_propagates_exceptions = schema.BoolOption()
+        broker_url = schema.StringOption()
+        celery_imports = schema.ListOption(item=schema.StringOption())
+        celery_ignore_result = schema.BoolOption()
+        celery_disable_rate_limits = schema.BoolOption(default=True)
+        celerybeat_scheduler = schema.StringOption()
 
=== modified file 'src/webcatalog/static/css/webcatalog.css'
--- src/webcatalog/static/css/webcatalog.css	2012-05-29 13:28:54 +0000
+++ src/webcatalog/static/css/webcatalog.css	2012-07-02 21:27:20 +0000
@@ -361,7 +361,7 @@
     margin-left: 7px;
 }
 p.error {
-    background-color: #f2cfce;
+    background-color: #df382b;
 }
 .emaillinkportlet {
     margin-top: 64px;
 
=== added file 'src/webcatalog/tasks.py'
--- src/webcatalog/tasks.py	1970-01-01 00:00:00 +0000
+++ src/webcatalog/tasks.py	2012-07-02 21:27:20 +0000
@@ -0,0 +1,54 @@
+from time import sleep
+
+from celery.task import task
+from django.conf import settings
+from django.core.management import call_command
+
+
+@task(name="webcatalog.tasks.import_exhibits")
+def import_exhibits():
+    """Import all exhibits"""
+    call_command('import_exhibits')
+
+
+@task(name="webcatalog.tasks.import_sca")
+def import_sca():
+    """Import all data from the Software Center Agent (MyApps)"""
+    call_command('import_sca_apps')
+
+
+@task(name="webcatalog.tasks.import_app_install_data")
+def import_app_install_data():
+    """Import all data from app-install-data and apt-cache"""
+    call_command('import_all_app_install_data', verbosity=0)
+
+
+@task(name="webcatalog.tasks.import_ratings_stats")
+def import_ratings_stats():
+    """Import all ratings and reviews data"""
+    call_command('import_all_ratings_stats', verbosity=0)
+
+
+@task(name="webcatalog.tasks.check_all_latest")
+def check_all_latest():
+    """Update the 'is_latest' bit on all Applications"""
+    call_command('check_all_latest')
+
+
+@task(name="webcatalog.tasks.cleanup_sessions")
+def cleanup_sessions():
+    """Remove stale sessions from the DB"""
+    call_command('cleanup', 'django_session', verbosity=0)
+
+
+@task(name="webcatalog.tasks.cleanup_nonces")
+def cleanup_nonces():
+    """Remove stale OAuth nonces from the DB"""
+    call_command('cleanup', 'webcatalog_nonce', verbosity=0)
+
+
+@task(name="webcatalog.tasks.fail")
+def fail():
+    """Sleep for 5 seconds, then raise an error"""
+    sleep(5)
+    raise ZeroDivisionError('Intentionally failed')
 
=== added file 'src/webcatalog/templates/forbidden.html'
--- src/webcatalog/templates/forbidden.html	1970-01-01 00:00:00 +0000
+++ src/webcatalog/templates/forbidden.html	2012-07-02 21:27:20 +0000
@@ -0,0 +1,23 @@
+{% extends "webcatalog/base.html" %}
+{% load i18n %}
+
+{% block title %}
+  {% trans "Access forbidden" %}
+{% endblock %}
+
+{% block header %}
+  {% trans "Access forbidden" %}
+{% endblock %}
+
+{% block content %}
+    <p> You just tried to access a feature which you don't have permission
+        to use.</p>
+    {% if extra_message %}
+    <p>{{ extra_message }}</p>
+    {% endif %}
+    {% if next %}
+    <p> If you signed in with the wrong account you might want to try
+    <a href="{{next}}">logging in again</a>.
+    </p>
+    {% endif %}
+{% endblock %}
 
=== added file 'src/webcatalog/templates/webcatalog/task_list.html'
--- src/webcatalog/templates/webcatalog/task_list.html	1970-01-01 00:00:00 +0000
+++ src/webcatalog/templates/webcatalog/task_list.html	2012-07-02 21:27:20 +0000
@@ -0,0 +1,84 @@
+{% extends "webcatalog/base.html" %}
+{% load i18n %}
+
+{% block title %}{% trans "Ubuntu Apps Directory" %} &mdash; {% trans "Schedule tasks" %}{% endblock %}
+{% block header %}{% trans "Schedule asynchronous tasks" %}{% endblock %}
+{% block head_extra %}
+    {{ block.super }}
+<script src="{% url wc-combo %}?yui/3.4.0/build/yui/yui-min.js"></script>
+<style type="text/css">
+div.task button {
+    width: 200px;
+    height: 30px;
+}
+
+div#messages {
+    min-height: 40px;
+}
+div.traceback {
+    background: #ddd;
+    padding: 8px;
+    margin-bottom: 16px;
+}
+</style>
+<script>
+YUI({combine: true, comboBase: '{% url wc-combo %}?', root: 'yui/3.4.0/build/'}).use('io-base', 'node-base', function (Y) {
+    var check_status = function(task_id, message) {
+        var check_uri = "/cat/task_status/" + task_id + '/';
+        var UNREADY_STATES = ["PENDING", "RECEIVED", "STARTED", "RETRY"];
+
+        Y.io(check_uri, {
+            on: {
+                complete: function(id, response){
+                    if (response.status == 200) {
+                        var result = JSON.parse(response.responseText);
+                        var status = result['status'];
+                        if (UNREADY_STATES.indexOf(status) >= 0) {
+                            message.append('. . .&nbsp;');
+                            setTimeout(function() {check_status(task_id, message)}, 3000);
+                        }
+                        else {
+                            if (status == 'FAILURE') {
+                                message.removeClass('success');
+                                message.addClass('error');
+                                Y.one('#messages').append(
+                                    '<div class="traceback"><pre>' +
+                                    result['traceback'] + '</pre></div>');
+                            }
+                            message.append(status);
+                            Y.all('button').removeAttribute('disabled');
+                        }
+                    }
+                }
+            }
+        });
+    }
+    Y.all('p.message').each(function (message) {
+        if (message.hasClass('success')) {
+            var task_id = message.one('span').getContent();
+            Y.all('button').setAttribute('disabled', 'disabled')
+            check_status(task_id, message);
+        }
+    });
+});
+</script>
+{% endblock %}
+{% block content %}
+
+<div id="messages">
+{% if messages %}
+    {% for message in messages %}
+    <p class="message{% if message.tags %} {{ message.tags }}{% endif %}">{{ message|safe }}</p>
+    {% endfor %}
+{% endif %}
+</div>
+{% for task in task_list %}
+<div class="task">
+<form method="post" action=".">{% csrf_token %}
+<input type="hidden" name="task" value="{{ task.name }}"/>
+<button type="submit">{{ task.name }}</button> {{ task.doc }}
+</form>
+</div>
+{% endfor %}
+
+{% endblock %}
 
=== modified file 'src/webcatalog/templatetags/webcatalog.py'
--- src/webcatalog/templatetags/webcatalog.py	2012-06-06 16:38:11 +0000
+++ src/webcatalog/templatetags/webcatalog.py	2012-07-02 21:27:20 +0000
@@ -17,10 +17,7 @@
 
 """Custom template tags for the Apps Directory."""
 
-from __future__ import (
-    absolute_import,
-    with_statement,
-)
+from __future__ import absolute_import
 
 __metaclass__ = type
 __all__ = [
 
=== modified file 'src/webcatalog/tests/__init__.py'
--- src/webcatalog/tests/__init__.py	2012-06-28 09:49:11 +0000
+++ src/webcatalog/tests/__init__.py	2012-07-02 21:27:20 +0000
@@ -27,6 +27,7 @@
 from .test_migrations import *
 from .test_pep8 import *
 from .test_preflight import *
+from .test_tasks import *
 from .test_templatetags import *
 from .test_utilities import *
 from .test_views import *
 
=== modified file 'src/webcatalog/tests/factory.py'
--- src/webcatalog/tests/factory.py	2012-06-28 09:49:11 +0000
+++ src/webcatalog/tests/factory.py	2012-07-02 21:27:20 +0000
@@ -17,17 +17,19 @@
 
 """A TestCase class with a built in object factory."""
 
-from __future__ import (
-    absolute_import,
-    with_statement,
-)
+from __future__ import absolute_import
+
 import os
 from datetime import (
     datetime,
     timedelta,
 )
 from itertools import count
-from django.contrib.auth.models import User
+from django.contrib.auth.models import (
+    Group,
+    Permission,
+    User,
+)
 from django.contrib.sessions.models import Session
 from django.test import (
     TestCase,
@@ -70,7 +72,8 @@
 
     def make_user(self, username=None, email=None, password='test',
                   first_name=None, last_name=None, open_id=None,
-                  is_admin=False):
+                  is_admin=False, permissions=None, groups=None,
+                  log_in=False):
         if username is None:
             username = self.get_unique_string(prefix='username')
         if email is None:
@@ -98,6 +101,15 @@
         useropenid = UserOpenID.objects.create(
             user=user, claimed_id=open_id, display_id=open_id)
 
+        if permissions:
+            for permission in permissions:
+                user.user_permissions.add(
+                    Permission.objects.get(codename=permission))
+
+        if groups:
+            for group_name in groups:
+                group, created = Group.objects.get_or_create(name=group_name)
+                user.groups.add(group)
         return user
 
     def make_application(self, package_name=None, name=None,
@@ -241,6 +253,15 @@
         super(TestCaseWithFactory, self).setUp()
         self.factory = WebCatalogObjectFactory()
 
+    def make_user(self, *args, **kwargs):
+        """Invokes self.factory.make_user(), but can also log the user in"""
+        logged_in = kwargs.pop('logged_in', False)
+        password = kwargs.get('password', 'test')
+        result = self.factory.make_user(*args, **kwargs)
+        if logged_in:
+            self.client.login(username=result.username, password=password)
+        return result
+
 
 class TxTestCaseWithFactory(TransactionTestCase):
 
 
=== modified file 'src/webcatalog/tests/helpers.py'
--- src/webcatalog/tests/helpers.py	2012-06-28 09:34:49 +0000
+++ src/webcatalog/tests/helpers.py	2012-07-02 21:27:20 +0000
@@ -17,10 +17,7 @@
 
 """Helpers for testing the Apps Directory."""
 
-from __future__ import (
-    absolute_import,
-    with_statement,
-)
+from __future__ import absolute_import
 
 __metaclass__ = type
 __all__ = [
 
=== modified file 'src/webcatalog/tests/test_commands.py'
--- src/webcatalog/tests/test_commands.py	2012-06-28 14:14:23 +0000
+++ src/webcatalog/tests/test_commands.py	2012-07-02 21:27:20 +0000
@@ -17,10 +17,8 @@
 
 """Test cases for the Apps Directory management commands."""
 
-from __future__ import (
-    absolute_import,
-    with_statement,
-)
+from __future__ import absolute_import
+
 import apt
 import json
 import os
@@ -716,7 +714,7 @@
     def test_updates_last_import_record(self):
         onion = self.factory.make_distroseries(code_name='onion')
         orig_timestamp = datetime(2011, 07, 18, 14, 43)
-        import_record = ReviewStatsImport.objects.create(
+        ReviewStatsImport.objects.create(
             distroseries=onion, last_import=orig_timestamp)
 
         call_command('import_ratings_stats', 'onion')
@@ -728,7 +726,7 @@
     def test_download_review_stats_no_previous(self):
         # If there have been no previous imports, the complete stats
         # will be retrieved.
-        onion = self.factory.make_distroseries(code_name='onion')
+        self.factory.make_distroseries(code_name='onion')
 
         call_command('import_ratings_stats', 'onion')
 
 
=== modified file 'src/webcatalog/tests/test_department_filters.py'
--- src/webcatalog/tests/test_department_filters.py	2012-06-06 18:00:26 +0000
+++ src/webcatalog/tests/test_department_filters.py	2012-07-02 21:27:20 +0000
@@ -17,11 +17,7 @@
 
 """Test cases for department filters."""
 
-from __future__ import (
-    absolute_import,
-    with_statement,
-)
-
+from __future__ import absolute_import
 
 from webcatalog.tests.factory import TestCaseWithFactory
 from webcatalog.department_filters import (
 
=== modified file 'src/webcatalog/tests/test_forms.py'
--- src/webcatalog/tests/test_forms.py	2012-06-28 14:25:19 +0000
+++ src/webcatalog/tests/test_forms.py	2012-07-02 21:27:20 +0000
@@ -17,10 +17,7 @@
 
 """Test cases for the Apps Directory forms."""
 
-from __future__ import (
-    absolute_import,
-    with_statement,
-)
+from __future__ import absolute_import
 
 from django.forms import ValidationError
 from django.test import TestCase
 
=== modified file 'src/webcatalog/tests/test_managers.py'
--- src/webcatalog/tests/test_managers.py	2012-06-26 12:40:57 +0000
+++ src/webcatalog/tests/test_managers.py	2012-07-02 21:27:20 +0000
@@ -17,10 +17,7 @@
 
 """Test cases for object managers."""
 
-from __future__ import (
-    absolute_import,
-    with_statement,
-)
+from __future__ import absolute_import
 
 from webcatalog.tests.factory import TestCaseWithFactory
 from webcatalog.models import Application
 
=== modified file 'src/webcatalog/tests/test_models.py'
--- src/webcatalog/tests/test_models.py	2012-06-06 19:48:08 +0000
+++ src/webcatalog/tests/test_models.py	2012-07-02 21:27:20 +0000
@@ -17,10 +17,7 @@
 
 """Test cases for models."""
 
-from __future__ import (
-    absolute_import,
-    with_statement,
-)
+from __future__ import absolute_import
 
 from django.core.urlresolvers import reverse
 from django.core.files.images import ImageFile
 
=== modified file 'src/webcatalog/tests/test_preflight.py'
--- src/webcatalog/tests/test_preflight.py	2012-06-21 20:18:44 +0000
+++ src/webcatalog/tests/test_preflight.py	2012-07-02 21:27:20 +0000
@@ -28,25 +28,13 @@
 
 
 class TestPreflight(TestCaseWithFactory):
-
-    def login(self, add_to_group=True, user=None):
-        if not user:
-            user = User.objects.create_user(username='test',
-                                            email='[e-mail]',
-                                            password='test')
-        if add_to_group:
-            group = Group.objects.create(name='preflight')
-            user.groups.add(group)
-            user.save()
-        self.client.login(username='test', password='test')
-
     @patch('webcatalog.utilities.WebServices.identity_provider')
     @patch('webcatalog.utilities.WebServices.get_screenshots_for_package')
     @patch('webcatalog.utilities.WebServices.recommender_api')
     @patch('webcatalog.utilities.WebServices.rnr_api')
     def test_login_needed_for_preflight(self, mock_rnr, mock_recommender,
                                         mock_screenshots, mock_sso):
-        self.login()
+        self.make_user(groups=['preflight'], logged_in=True)
 
         with patch_settings(PREFLIGHT_GROUPS=['preflight']):
             response = self.client.get('/preflight/')
@@ -60,8 +48,7 @@
         self.assertEqual(404, response.status_code)
 
     def test_login_but_wrong_group_means_preflight_not_found(self):
-        with patch_settings(PREFLIGHT_GROUPS=['preflight']):
-            self.login(add_to_group=False)
+        self.make_user(logged_in=True)
 
         response = self.client.get('/preflight/')
 
@@ -77,7 +64,7 @@
         mock_recommender.server_status.return_value = 'ok'
         mock_screenshots.return_value = ['some_screenshot.jpg']
         mock_sso.validate_token.return_value = False
-        self.login()
+        self.make_user(groups=['preflight'], logged_in=True)
 
         with patch_settings(PREFLIGHT_GROUPS=['preflight']):
             response = self.client.get('/preflight/')
 
=== added file 'src/webcatalog/tests/test_tasks.py'
--- src/webcatalog/tests/test_tasks.py	1970-01-01 00:00:00 +0000
+++ src/webcatalog/tests/test_tasks.py	2012-07-02 21:27:20 +0000
@@ -0,0 +1,42 @@
+from django.conf import settings
+from django.test import TestCase
+from mock import patch
+
+from webcatalog.models import Exhibit
+from webcatalog.tasks import (
+    fail,
+    import_exhibits,
+)
+
+
+class ImportExhbitsTaskTestCase(TestCase):
+    @patch('webcatalog.management.commands.import_exhibits.urllib.urlopen')
+    def test_success(self, mock_urlopen):
+        """Test that the ``import_exhibits`` task runs with no errors"""
+        fd = mock_urlopen.return_value
+        fd.code = 200
+        fd.read.return_value = """[{
+        "package_names": "foobar",
+        "banner_url": "http://example.com/exhibits/foobar.png",
+        "distroseries": [{"version": "12.04", "code_name": "precise"}],
+        "html": "<div><div>Hello world!</div></div>",
+        "published": true,
+        "date_created": "2012-06-11 17:22:45",
+        "id": 2
+        }]"""
+        expected_url = '%sexhibits/en/' % settings.SCA_API_URL
+
+        result = import_exhibits.delay()
+
+        self.assertIsNone(result.get())
+        self.assertTrue(result.successful())
+        mock_urlopen.assertCalledWith(expected_url)
+        self.assertEqual(1, Exhibit.objects.all().count())
+
+
+class FailTaskTestCase(TestCase):
+    @patch('webcatalog.tasks.sleep')
+    def test_delays_and_fails_as_expected(self, mock_sleep):
+        """Test that the ``fail`` task raises an Exception as expected"""
+        self.assertRaises(ZeroDivisionError, fail.delay)
+        mock_sleep.assertCalledWith(5)
 
=== modified file 'src/webcatalog/tests/test_templatetags.py'
--- src/webcatalog/tests/test_templatetags.py	2012-06-14 13:57:03 +0000
+++ src/webcatalog/tests/test_templatetags.py	2012-07-02 21:27:20 +0000
@@ -17,13 +17,9 @@
 
 """Tests for the Apps Directory template tags."""
 
-from __future__ import (
-    absolute_import,
-    with_statement,
-)
+from __future__ import absolute_import
 
 import unittest
-from decimal import Decimal
 
 from django.core.urlresolvers import reverse
 from django.template import Context
 
=== modified file 'src/webcatalog/tests/test_views.py'
--- src/webcatalog/tests/test_views.py	2012-06-28 19:06:37 +0000
+++ src/webcatalog/tests/test_views.py	2012-07-02 21:27:20 +0000
@@ -17,10 +17,7 @@
 
 """Test cases for the Apps Directory views."""
 
-from __future__ import (
-    absolute_import,
-    with_statement,
-)
+from __future__ import absolute_import
 
 import json
 import re
@@ -53,6 +50,8 @@
     'ApplicationReviewsTestCase',
     'ApplicationScreenshotsTestCase',
     'IndexTestCase',
+    'TasksViewTestCase',
+    'TaskStatusViewTestCase',
     'TermsOfServiceTestCase',
     'OverviewTestCase',
     'SearchTestCase',
@@ -1313,3 +1312,116 @@
         expected = '/* js/foo.css */\n/* [missing] */\n'
         self.assertEqual('text/css', response['Content-Type'])
         self.assertEqual(expected, response.content)
+
+
+class TasksViewTestCase(TestCaseWithFactory):
+    """Tests for the task list"""
+    def test_not_allowed_if_not_logged_in(self):
+        url = reverse('wc-tasks')
+
+        response = self.client.get(url)
+
+        self.assertEqual(302, response.status_code)
+        expected = reverse('wc-forbidden') + '?next=' + url
+        self.assertTrue(response['location'].endswith(expected))
+
+    def test_scheduler_perm_is_required(self):
+        self.make_user(logged_in=True)
+        url = reverse('wc-tasks')
+
+        response = self.client.get(url)
+
+        self.assertEqual(302, response.status_code)
+        expected = reverse('wc-forbidden') + '?next=' + url
+        self.assertTrue(response['location'].endswith(expected))
+
+    def test_includes_all_tasks(self):
+        self.maxDiff = None
+        self.make_user(permissions=['schedule_tasks'], logged_in=True)
+        url = reverse('wc-tasks')
+
+        response = self.client.get(url)
+
+        tasks = response.context[0]['task_list']
+        expected = [
+            {'doc': 'Import all exhibits', 'name': 'import_exhibits'},
+            {
+                'doc': 'Remove stale OAuth nonces from the DB',
+                'name': 'cleanup_nonces',
+            },
+            {
+                'doc': 'Remove stale sessions from the DB',
+                'name': 'cleanup_sessions',
+            },
+            {
+                'doc': 'Sleep for 5 seconds, then raise an error',
+                'name': 'fail',
+            },
+            {
+                'doc': 'Import all data from app-install-data and apt-cache',
+                'name': 'import_app_install_data',
+            },
+            {
+                'doc': ('Import all data from the Software '
+                        'Center Agent (MyApps)'),
+                'name': 'import_sca',
+            },
+            {
+                'doc': 'Import all ratings and reviews data',
+                'name': 'import_ratings_stats',
+            },
+            {
+                'doc': "Update the 'is_latest' bit on all Applications",
+                'name': 'check_all_latest',
+            },
+        ]
+        self.assertEqual(sorted(expected), sorted(tasks))
+
+    @patch('webcatalog.management.commands.import_exhibits.urllib.urlopen')
+    def test_post_schedules_task(self, mock_urlopen):
+        fd = mock_urlopen.return_value
+        fd.code = 200
+        fd.read.return_value = "[]"
+        self.make_user(permissions=['schedule_tasks'], logged_in=True)
+        url = reverse('wc-tasks')
+        data = {'task': 'import_exhibits'}
+
+        response = self.client.post(url, data=data, follow=True)
+
+        msg_pat = r'Task id <span>[-\da-f]+</span> submitted'
+        match = re.search(msg_pat, response.content)
+        self.assertFalse(match is None)
+        self.assertEqual(1, mock_urlopen.call_count)
+
+
+class TaskStatusViewTestCase(TestCaseWithFactory):
+    """Tests for the task status getter"""
+    def test_not_allowed_if_not_logged_in(self):
+        url = reverse('wc-task-status', args=['cafe'])
+
+        response = self.client.get(url)
+
+        self.assertEqual(302, response.status_code)
+        expected = reverse('wc-forbidden') + '?next=' + url
+        self.assertTrue(response['location'].endswith(expected))
+
+    def test_scheduler_perm_is_required(self):
+        self.make_user(logged_in=True)
+        url = reverse('wc-task-status', args=['cafe'])
+
+        response = self.client.get(url)
+
+        self.assertEqual(302, response.status_code)
+        expected = reverse('wc-forbidden') + '?next=' + url
+        self.assertTrue(response['location'].endswith(expected))
+
+    def test_unknown_task_is_reported_pending(self):
+        self.make_user(permissions=['schedule_tasks'], logged_in=True)
+        url = reverse('wc-task-status', args=['cafe'])
+
+        response = self.client.get(url)
+
+        data = json.loads(response.content)
+        expected = {"status": "PENDING", "traceback": None, "id": "cafe"}
+
+        self.assertEqual(expected, data)
 
=== modified file 'src/webcatalog/urls.py'
--- src/webcatalog/urls.py	2012-06-06 17:42:04 +0000
+++ src/webcatalog/urls.py	2012-07-02 21:27:20 +0000
@@ -17,10 +17,8 @@
 
 """Url configuration for the Apps Directory."""
 
-from __future__ import (
-    absolute_import,
-    with_statement,
-)
+from __future__ import absolute_import
+
 from django.conf.urls.defaults import patterns, include, url
 from django.views.generic import TemplateView
 
@@ -59,6 +57,10 @@
     url(r'^tos/plain/$', name="wc-tos-plain",
         view=TemplateView.as_view(template_name="webcatalog/tos_plain.html")),
     url(r'^combo/$', 'combo_view', name='wc-combo'),
+    url(r'^forbidden/$', 'forbidden', name='wc-forbidden'),
+    url(r'^tasks/$', 'tasks', name='wc-tasks'),
+    url(r'^task_status/(?P<task_id>[-\da-f]+)/$', 'task_status',
+        name='wc-task-status'),
 
     (r'^api/', include('webcatalog.api.urls')),
 )
 
=== modified file 'src/webcatalog/views.py'
--- src/webcatalog/views.py	2012-06-28 14:46:12 +0000
+++ src/webcatalog/views.py	2012-07-02 21:27:20 +0000
@@ -17,15 +17,14 @@
 
 """Views for the Apps Directory app."""
 
-from __future__ import (
-    absolute_import,
-    with_statement,
-)
+from __future__ import absolute_import
 
 import json
 import operator
 import os
+from inspect import getmembers
 
+from celery.task import Task
 from convoy.combo import combine_files, parse_qs
 from django.conf import settings
 from django.contrib import messages
@@ -33,17 +32,23 @@
 from django.core.urlresolvers import reverse
 from django.db.models import Q
 from django.http import (
+    HttpResponse,
+    HttpResponseForbidden,
+    HttpResponseNotFound,
     HttpResponseRedirect,
-    HttpResponse,
 )
 from django.shortcuts import (
-    get_list_or_404,
     get_object_or_404,
     render_to_response,
 )
 from django.template import RequestContext
+from django.template.loader import render_to_string
 from django.utils.translation import ugettext as _
+from django.views.decorators.cache import never_cache
+from django.views.decorators.http import require_GET
+from djcelery.models import TaskState
 
+import webcatalog.tasks
 from webcatalog.forms import EmailDownloadLinkForm
 from webcatalog.models import (
     Application,
@@ -51,6 +56,7 @@
     DistroSeries,
     Exhibit,
 )
+from webcatalog.decorators import scheduler_required
 from webcatalog.utilities import WebServices
 
 
@@ -59,8 +65,10 @@
     'application_detail',
     'application_recommends',
     'application_reviews',
+    'department_overview',
+    'forbidden',
     'index',
-    'department_overview',
+    'schedule_task',
     'search',
 ]
 
@@ -295,3 +303,49 @@
             content_type=content_type, status=200,
             content="".join(content))
     return HttpResponse(content_type=content_type, status=404)
+
+
+@scheduler_required
+@never_cache
+def tasks(request):
+    if request.method == 'POST':
+        task_name = request.POST.get('task', '')
+        task = getattr(webcatalog.tasks, task_name, None)
+        if not task:
+            messages.error(request, 'Invalid task name "%s"' % task_name)
+            return HttpResponseRedirect(reverse('wc-tasks'))
+        result = task.delay()
+        messages.success(request,
+                         'Task id <span>%s</span> submitted' % result.task_id)
+        return HttpResponseRedirect(reverse('wc-tasks'))
+    is_task = lambda x: isinstance(x, Task)
+    task_list = [
+        {'name': key, 'doc': val.__doc__}
+        for key, val in getmembers(webcatalog.tasks, is_task)
+    ]
+    context = {
+        'task_list': sorted(task_list),
+    }
+    return render_to_response('webcatalog/task_list.html',
+                              RequestContext(request, context))
+
+
+@scheduler_required
+@require_GET
+@never_cache
+def task_status(request, task_id):
+    try:
+        task = TaskState.objects.get(task_id=task_id)
+        data = {
+            'id': task.task_id,
+            'status': task.state,
+            'traceback': task.traceback,
+        }
+    except TaskState.DoesNotExist:
+        data = {'id': task_id, 'status': 'PENDING', 'traceback': None}
+    return HttpResponse(json.dumps(data), mimetype='application/json')
+
+
+def forbidden(request):
+    return HttpResponseForbidden(
+        render_to_string('forbidden.html', RequestContext(request)))