Merge lp:~elachuni/ubuntu-webcatalog/celery into lp:ubuntu-webcatalog
- celery
- Merge into trunk
Status: | Merged |
---|---|
Approved by: | Anthony Lenton |
Approved revision: | 157 |
Merged at revision: | 156 |
Proposed branch: | lp:~elachuni/ubuntu-webcatalog/celery |
Merge into: | lp:ubuntu-webcatalog |
Diff against target: |
1262 lines (+551/-136) 35 files modified
django_project/config/main.cfg (+10/-1) django_project/settings.py (+4/-1) setup.py (+2/-0) src/webcatalog/admin.py (+2/-4) src/webcatalog/decorators.py (+32/-0) src/webcatalog/department_filters.py (+1/-4) src/webcatalog/forms.py (+1/-4) src/webcatalog/management/commands/cleanup.py (+24/-16) src/webcatalog/management/commands/import_all_app_install_data.py (+3/-5) src/webcatalog/management/commands/import_all_ratings_stats.py (+3/-5) src/webcatalog/management/commands/import_app_install_data.py (+1/-4) src/webcatalog/management/commands/import_sca_apps.py (+1/-4) src/webcatalog/managers.py (+1/-5) src/webcatalog/models/applications.py (+25/-5) src/webcatalog/preflight.py (+1/-4) src/webcatalog/schema.py (+11/-0) src/webcatalog/static/css/webcatalog.css (+1/-1) src/webcatalog/tasks.py (+54/-0) src/webcatalog/templates/forbidden.html (+23/-0) src/webcatalog/templates/webcatalog/task_list.html (+84/-0) src/webcatalog/templatetags/webcatalog.py (+1/-4) src/webcatalog/tests/__init__.py (+1/-0) src/webcatalog/tests/factory.py (+27/-6) src/webcatalog/tests/helpers.py (+1/-4) src/webcatalog/tests/test_commands.py (+4/-6) src/webcatalog/tests/test_department_filters.py (+1/-5) src/webcatalog/tests/test_forms.py (+1/-4) src/webcatalog/tests/test_managers.py (+1/-4) src/webcatalog/tests/test_models.py (+1/-4) src/webcatalog/tests/test_preflight.py (+3/-16) src/webcatalog/tests/test_tasks.py (+42/-0) src/webcatalog/tests/test_templatetags.py (+1/-5) src/webcatalog/tests/test_views.py (+116/-4) src/webcatalog/urls.py (+6/-4) src/webcatalog/views.py (+61/-7) |
To merge this branch: | bzr merge lp:~elachuni/ubuntu-webcatalog/celery |
Related bugs: |
Reviewer | Review Type | Date Requested | Status |
---|---|---|---|
Danny Tamez (community) | Approve | ||
Review via email: mp+112398@code.launchpad.net |
Commit message
Added a view to schedule asynchronous tasks via Celery.
Description of the change
Overview
========
This branch adds a view to schedule asynchronous tasks via Celery. Only one basic task is provided to import exhibits (plus one that always fails, for testing), the full set of tasks will be in a separate MP, now that the groundwork is in place. A brief video demo can be found on http://
Details
=======
The current branch counts on using CELERY_
The TaskMeta model was registered in the admin so that results can be browsed and searched in the admin UI after a task has run. The ModelAdmin class was improved a bit after grabbing the video, you can see the final list view on http://
The list of tasks available is put together by introspecting webcatalog.tasks, so adding new tasks available will be picked up and added automatically.
The task list template uses YUI.io to retrieve the task's status, but submits a POST form and reloads the whole page when you schedule a new task; I considered doing everything with ajax, but POST via ajax is a bit tricky, specially when csrf tokens are involved, so I went for this simpler solution that seems to be responsive enough.
James Westby (james-w) wrote : | # |
- 152. By Anthony Lenton
-
Added a few tasks.
- 153. By Anthony Lenton
-
Merged in latest changes from trunk.
- 154. By Anthony Lenton
-
Query TaskState model instead of TaskMeta.
Anthony Lenton (elachuni) wrote : | # |
> 30 -openid_
> 31 +openid_
>
> That looks backwards?
True, for some reason staging sso *still* doesn't report me as being part of canonical-
> 332 + if (message.
>
> That looks as if it will loop forever on successful tasks?
Hm it shouldn't, that call is outside of check_status so it's just to trigger the first call. Maybe moving this into an function called init() or onload() would make it clearer.
- 155. By Anthony Lenton
-
Pulled back config change per code review.
- 156. By Anthony Lenton
-
Merged in latest changes from trunk. Made all tests pass.
Danny Tamez (zematynnad) wrote : | # |
line 111 - we don't need this any longer as we're now on 2.6
looks like line 115 is not needed
at 158, that looks like that could have stayed static but I guess it makes more sense to make it a a regular method as that's how the class will now be used...
same for 183
screencast was great! Is there any way to have something more useful than the task id to help find the task you want to look at? I'm not sure of the use cases for searching for a task so that id may be the only way that makes sense...
Approved - feel free to act on or ignore the comments.
- 157. By Anthony Lenton
-
Removed pointless imported with_statement
Preview Diff
1 | === modified file 'django_project/config/main.cfg' |
2 | --- django_project/config/main.cfg 2012-06-08 08:24:11 +0000 |
3 | +++ django_project/config/main.cfg 2012-07-02 21:27:20 +0000 |
4 | @@ -25,6 +25,7 @@ |
5 | south |
6 | preflight |
7 | pgtools |
8 | + djcelery |
9 | login_url = /openid/login/ |
10 | managers = %(admins)s |
11 | middleware_classes = django.middleware.cache.UpdateCacheMiddleware |
12 | @@ -55,12 +56,12 @@ |
13 | static_root = ./django_project/static/ |
14 | static_url = /assets/ |
15 | admin_media_prefix = /assets/admin/ |
16 | +test_runner = djcelery.contrib.test_runner.CeleryTestSuiteRunner |
17 | |
18 | # Django-1.1 backwards compatibility |
19 | database_engine = sqlite3 |
20 | database_name = webcatalog.db |
21 | |
22 | - |
23 | [django_databases] |
24 | default = default_database |
25 | |
26 | @@ -103,6 +104,7 @@ |
27 | # Structure here is: LP Team = Django Group |
28 | canonical-isd-hackers = admin |
29 | canonical-losas = admin |
30 | +canonical-ca-hackers = developers |
31 | |
32 | [oops] |
33 | oops_dir = /srv/%(hostname)s/staging-logs/www-oops |
34 | @@ -134,3 +136,10 @@ |
35 | sso_api_auth_username = insert-your-sso-api-username-here |
36 | sso_api_auth_password = insert-your-sso-api-password-here |
37 | sso_auth_mode_no_ubuntu_sso_plaintext_only = True |
38 | + |
39 | +[celery] |
40 | +broker_backend = amqp |
41 | +broker_url = amqp://guest:guest@localhost:5672// |
42 | +celery_imports = webcatalog.tasks |
43 | +celery_result_backend = database |
44 | +celerybeat_scheduler = djcelery.schedulers.DatabaseScheduler |
45 | \ No newline at end of file |
46 | |
47 | === modified file 'django_project/settings.py' |
48 | --- django_project/settings.py 2012-03-27 20:43:10 +0000 |
49 | +++ django_project/settings.py 2012-07-02 21:27:20 +0000 |
50 | @@ -13,4 +13,7 @@ |
51 | 'local.cfg']) |
52 | if os.path.exists(path)] |
53 | |
54 | -configglue(WebCatalogSchema, config_files, __name__) |
55 | \ No newline at end of file |
56 | +configglue(WebCatalogSchema, config_files, __name__) |
57 | + |
58 | +import djcelery |
59 | +djcelery.setup_loader() |
60 | |
61 | === modified file 'setup.py' |
62 | --- setup.py 2012-06-28 14:28:26 +0000 |
63 | +++ setup.py 2012-07-02 21:27:20 +0000 |
64 | @@ -50,6 +50,8 @@ |
65 | 'ssoclient==1.0', |
66 | 'pep8', |
67 | 'PIL', |
68 | + 'celery', |
69 | + 'django-celery', |
70 | ], |
71 | package_data = find_packages_data('src'), |
72 | dependency_links = [ |
73 | |
74 | === modified file 'src/webcatalog/admin.py' |
75 | --- src/webcatalog/admin.py 2012-06-28 13:29:29 +0000 |
76 | +++ src/webcatalog/admin.py 2012-07-02 21:27:20 +0000 |
77 | @@ -17,10 +17,8 @@ |
78 | |
79 | """Admin classes for the Apps Directory.""" |
80 | |
81 | -from __future__ import ( |
82 | - absolute_import, |
83 | - with_statement, |
84 | -) |
85 | +from __future__ import absolute_import |
86 | + |
87 | from django.contrib import admin |
88 | from webcatalog.models import ( |
89 | Application, |
90 | |
91 | === added file 'src/webcatalog/decorators.py' |
92 | --- src/webcatalog/decorators.py 1970-01-01 00:00:00 +0000 |
93 | +++ src/webcatalog/decorators.py 2012-07-02 21:27:20 +0000 |
94 | @@ -0,0 +1,32 @@ |
95 | +# -*- coding: utf-8 -*- |
96 | +# This file is part of the Apps Directory |
97 | +# Copyright (C) 2011 Canonical Ltd. |
98 | +# |
99 | +# This program is free software: you can redistribute it and/or modify |
100 | +# it under the terms of the GNU Affero General Public License as |
101 | +# published by the Free Software Foundation, either version 3 of the |
102 | +# License, or (at your option) any later version. |
103 | +# |
104 | +# This program is distributed in the hope that it will be useful, |
105 | +# but WITHOUT ANY WARRANTY; without even the implied warranty of |
106 | +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the |
107 | +# GNU Affero General Public License for more details. |
108 | +# |
109 | +# You should have received a copy of the GNU Affero General Public License |
110 | +# along with this program. If not, see <http://www.gnu.org/licenses/>. |
111 | + |
112 | +"""Decorators for the webcatalog application.""" |
113 | + |
114 | +from __future__ import absolute_import |
115 | + |
116 | +from django.contrib.auth.decorators import permission_required |
117 | + |
118 | + |
119 | +__metaclass__ = type |
120 | +__all__ = [ |
121 | + 'scheduler_required', |
122 | +] |
123 | + |
124 | + |
125 | +scheduler_required = permission_required('webcatalog.schedule_tasks', |
126 | + login_url='/cat/forbidden/') |
127 | |
128 | === modified file 'src/webcatalog/department_filters.py' |
129 | --- src/webcatalog/department_filters.py 2012-06-06 17:42:04 +0000 |
130 | +++ src/webcatalog/department_filters.py 2012-07-02 21:27:20 +0000 |
131 | @@ -17,10 +17,7 @@ |
132 | |
133 | """Department filters.""" |
134 | |
135 | -from __future__ import ( |
136 | - absolute_import, |
137 | - with_statement, |
138 | -) |
139 | +from __future__ import absolute_import |
140 | |
141 | import re |
142 | |
143 | |
144 | === modified file 'src/webcatalog/forms.py' |
145 | --- src/webcatalog/forms.py 2012-06-28 14:25:19 +0000 |
146 | +++ src/webcatalog/forms.py 2012-07-02 21:27:20 +0000 |
147 | @@ -17,10 +17,7 @@ |
148 | |
149 | """Forms used by the Apps Directory.""" |
150 | |
151 | -from __future__ import ( |
152 | - absolute_import, |
153 | - with_statement, |
154 | -) |
155 | +from __future__ import absolute_import |
156 | |
157 | import apt |
158 | import json |
159 | |
160 | === modified file 'src/webcatalog/management/commands/cleanup.py' |
161 | --- src/webcatalog/management/commands/cleanup.py 2012-06-06 16:38:11 +0000 |
162 | +++ src/webcatalog/management/commands/cleanup.py 2012-07-02 21:27:20 +0000 |
163 | @@ -27,16 +27,17 @@ |
164 | |
165 | try: |
166 | import psycopg2 |
167 | - from psycopg2.extensions import ISOLATION_LEVEL_AUTOCOMMIT |
168 | psycopg2_available = True |
169 | except ImportError: |
170 | psycopg2_available = False |
171 | |
172 | |
173 | class DjangoSessionCleaner(object): |
174 | - @staticmethod |
175 | - def declare_expired_keys_cursor(cur): |
176 | - print "Opening cursor" |
177 | + def __init__(self, cmd): |
178 | + self.cmd = cmd |
179 | + |
180 | + def declare_expired_keys_cursor(self, cur): |
181 | + self.cmd.output("Opening cursor\n", 1) |
182 | # We order by expire_date to force the index to be used. |
183 | cur.execute("CLOSE ALL") |
184 | cur.execute(""" |
185 | @@ -46,8 +47,7 @@ |
186 | ORDER BY expire_date |
187 | """) |
188 | |
189 | - @staticmethod |
190 | - def remove_batch(cur, batch_size): |
191 | + def remove_batch(self, cur, batch_size): |
192 | cur.execute("FETCH %s FROM _django_session_clean", [batch_size]) |
193 | session_keys = [session_key for session_key, in cur.fetchall()] |
194 | if not session_keys: |
195 | @@ -66,9 +66,11 @@ |
196 | |
197 | |
198 | class OAuthNonceCleaner(object): |
199 | - @staticmethod |
200 | - def declare_expired_keys_cursor(cur): |
201 | - print "Opening cursor" |
202 | + def __init__(self, cmd): |
203 | + self.cmd = cmd |
204 | + |
205 | + def declare_expired_keys_cursor(self, cur): |
206 | + self.cmd.output("Opening cursor\n", 1) |
207 | # We order by created_at to force the index to be used. |
208 | # Use a 5 hour clock skew as defined in openid.store.nonce |
209 | cur.execute("CLOSE ALL") |
210 | @@ -79,8 +81,7 @@ |
211 | ORDER BY created_at |
212 | """) |
213 | |
214 | - @staticmethod |
215 | - def remove_batch(cur, batch_size): |
216 | + def remove_batch(self, cur, batch_size): |
217 | cur.execute("FETCH %s FROM _oauth_nonce_clean", [batch_size]) |
218 | nonce_ids = [nonce_id for nonce_id, in cur.fetchall()] |
219 | if not nonce_ids: |
220 | @@ -117,6 +118,7 @@ |
221 | ) |
222 | |
223 | def handle_label(self, table, **options): |
224 | + self.verbosity = int(options['verbosity']) |
225 | if not psycopg2_available: |
226 | raise CommandError('This command requires psycopg2') |
227 | |
228 | @@ -128,8 +130,7 @@ |
229 | msg = ("Invalid cleaner.\nNo cleaner found for table: %s\n" |
230 | "Supported tables are: %s" % (table, CLEANERS.keys())) |
231 | raise CommandError(msg) |
232 | - cleaner = cleaner_cls() |
233 | - connection.isolation_level = ISOLATION_LEVEL_AUTOCOMMIT |
234 | + cleaner = cleaner_cls(self) |
235 | cur = connection.cursor() |
236 | |
237 | removed = -1 |
238 | @@ -149,12 +150,12 @@ |
239 | removed = cleaner.remove_batch(cur, batch_size) |
240 | actual_batch_time = time.time() - batch_start |
241 | total_removed += removed |
242 | - print "Removed %d rows (%d total removed). Batch size %d" % ( |
243 | - removed, total_removed, batch_size) |
244 | + message = "Removed %d rows (%d total removed). Batch size %d\n" |
245 | + self.output(message % (removed, total_removed, batch_size), 1) |
246 | |
247 | # Done |
248 | if removed == 0: |
249 | - print "All done." |
250 | + self.output("All done.\n", 1) |
251 | break |
252 | |
253 | # Increase or decrease the batch size by 10%, minimum 1. |
254 | @@ -163,3 +164,10 @@ |
255 | batch_size -= batch_size_wobble |
256 | elif actual_batch_time < target_batch_time * 0.9: |
257 | batch_size += batch_size_wobble |
258 | + |
259 | + def output(self, message, level=None, flush=False): |
260 | + if hasattr(self, 'stdout'): |
261 | + if level is None or self.verbosity >= level: |
262 | + self.stdout.write(message) |
263 | + if flush: |
264 | + self.stdout.flush() |
265 | |
266 | === modified file 'src/webcatalog/management/commands/import_all_app_install_data.py' |
267 | --- src/webcatalog/management/commands/import_all_app_install_data.py 2012-06-28 13:13:15 +0000 |
268 | +++ src/webcatalog/management/commands/import_all_app_install_data.py 2012-07-02 21:27:20 +0000 |
269 | @@ -17,10 +17,7 @@ |
270 | |
271 | """Management command to import app install data for all distroseries.""" |
272 | |
273 | -from __future__ import ( |
274 | - absolute_import, |
275 | - with_statement, |
276 | -) |
277 | +from __future__ import absolute_import |
278 | |
279 | __metaclass__ = type |
280 | __all__ = [] |
281 | @@ -37,7 +34,8 @@ |
282 | for distroseries in settings.UBUNTU_SERIES_FOR_VERSIONS.values(): |
283 | self.output("Importing app-install-data for {0}\n".format( |
284 | distroseries), 1) |
285 | - call_command('import_app_install_data', distroseries) |
286 | + call_command('import_app_install_data', distroseries, |
287 | + verbosity=self.verbosity) |
288 | self.output("Running check_all_latest across all distroseries.\n", 1) |
289 | call_command('check_all_latest') |
290 | |
291 | |
292 | === modified file 'src/webcatalog/management/commands/import_all_ratings_stats.py' |
293 | --- src/webcatalog/management/commands/import_all_ratings_stats.py 2012-06-21 20:18:44 +0000 |
294 | +++ src/webcatalog/management/commands/import_all_ratings_stats.py 2012-07-02 21:27:20 +0000 |
295 | @@ -17,10 +17,7 @@ |
296 | |
297 | """Management command to import review statistics for all distroseries.""" |
298 | |
299 | -from __future__ import ( |
300 | - absolute_import, |
301 | - with_statement, |
302 | -) |
303 | +from __future__ import absolute_import |
304 | |
305 | __metaclass__ = type |
306 | __all__ = [] |
307 | @@ -37,7 +34,8 @@ |
308 | for distroseries in settings.UBUNTU_SERIES_FOR_VERSIONS.values(): |
309 | self.output("Importing ratings stats for {0}\n".format( |
310 | distroseries), 1) |
311 | - call_command('import_ratings_stats', distroseries) |
312 | + call_command('import_ratings_stats', distroseries, |
313 | + verbosity=self.verbosity) |
314 | |
315 | def output(self, message, level=None, flush=False): |
316 | if hasattr(self, 'stdout'): |
317 | |
318 | === modified file 'src/webcatalog/management/commands/import_app_install_data.py' |
319 | --- src/webcatalog/management/commands/import_app_install_data.py 2012-06-28 14:46:12 +0000 |
320 | +++ src/webcatalog/management/commands/import_app_install_data.py 2012-07-02 21:27:20 +0000 |
321 | @@ -17,10 +17,7 @@ |
322 | |
323 | """Management command to import app install data for a distroseries.""" |
324 | |
325 | -from __future__ import ( |
326 | - absolute_import, |
327 | - with_statement, |
328 | -) |
329 | +from __future__ import absolute_import |
330 | |
331 | import os |
332 | import re |
333 | |
334 | === modified file 'src/webcatalog/management/commands/import_sca_apps.py' |
335 | --- src/webcatalog/management/commands/import_sca_apps.py 2012-06-28 14:25:19 +0000 |
336 | +++ src/webcatalog/management/commands/import_sca_apps.py 2012-07-02 21:27:20 +0000 |
337 | @@ -19,10 +19,7 @@ |
338 | Center. |
339 | """ |
340 | |
341 | -from __future__ import ( |
342 | - absolute_import, |
343 | - with_statement, |
344 | -) |
345 | +from __future__ import absolute_import |
346 | |
347 | import json |
348 | import os |
349 | |
350 | === modified file 'src/webcatalog/managers.py' |
351 | --- src/webcatalog/managers.py 2012-06-14 09:23:32 +0000 |
352 | +++ src/webcatalog/managers.py 2012-07-02 21:27:20 +0000 |
353 | @@ -17,11 +17,7 @@ |
354 | |
355 | """Django object managers.""" |
356 | |
357 | -from __future__ import ( |
358 | - absolute_import, |
359 | - with_statement, |
360 | -) |
361 | - |
362 | +from __future__ import absolute_import |
363 | |
364 | __metaclass__ = type |
365 | __all__ = [ |
366 | |
367 | === modified file 'src/webcatalog/models/applications.py' |
368 | --- src/webcatalog/models/applications.py 2012-06-28 12:27:22 +0000 |
369 | +++ src/webcatalog/models/applications.py 2012-07-02 21:27:20 +0000 |
370 | @@ -17,19 +17,22 @@ |
371 | |
372 | """Models for the package application.""" |
373 | |
374 | -from __future__ import ( |
375 | - absolute_import, |
376 | - with_statement, |
377 | -) |
378 | +from __future__ import absolute_import |
379 | |
380 | import logging |
381 | from datetime import datetime |
382 | |
383 | from django.conf import settings |
384 | -from django.contrib.auth.models import User |
385 | +from django.contrib.auth.models import ( |
386 | + Group, |
387 | + Permission, |
388 | + User, |
389 | +) |
390 | +from django.contrib.contenttypes.models import ContentType |
391 | from django.core.urlresolvers import reverse |
392 | from django.db import models |
393 | from django.template.defaultfilters import slugify |
394 | +from django.db.models.signals import post_syncdb |
395 | |
396 | from webcatalog.department_filters import department_filters |
397 | from webcatalog.managers import ApplicationManager, DistroSeriesManager |
398 | @@ -290,3 +293,20 @@ |
399 | |
400 | pkgs = '&'.join('pkg_name=%s' % pkg_name for pkg_name in pkg_names) |
401 | return reverse('wc-package-list') + '?' + pkgs |
402 | + |
403 | + |
404 | +# GroupPermissions can't go in fixtures because Permissions are stored as |
405 | +# model metadata, so our Permission's primary key can change without warning |
406 | +def post_syncdb_handler(sender, **kwargs): |
407 | + ct = ContentType.objects.get_for_model(Application) |
408 | + group, _ = Group.objects.get_or_create(name='developers') |
409 | + perm, _ = Permission.objects.get_or_create( |
410 | + codename='schedule_tasks', |
411 | + name="Can schedule asynchronous tasks via the web", |
412 | + content_type=ct, |
413 | + ) |
414 | + if not perm in group.permissions.all(): |
415 | + group.permissions.add(perm) |
416 | + group.save() |
417 | + |
418 | +post_syncdb.connect(post_syncdb_handler) |
419 | |
420 | === modified file 'src/webcatalog/preflight.py' |
421 | --- src/webcatalog/preflight.py 2012-06-06 17:42:04 +0000 |
422 | +++ src/webcatalog/preflight.py 2012-07-02 21:27:20 +0000 |
423 | @@ -17,10 +17,7 @@ |
424 | |
425 | """Preflight check for webcatalog.""" |
426 | |
427 | -from __future__ import ( |
428 | - absolute_import, |
429 | - with_statement, |
430 | -) |
431 | +from __future__ import absolute_import |
432 | |
433 | from django import db |
434 | from django.conf import settings |
435 | |
436 | === modified file 'src/webcatalog/schema.py' |
437 | --- src/webcatalog/schema.py 2012-06-14 23:40:45 +0000 |
438 | +++ src/webcatalog/schema.py 2012-07-02 21:27:20 +0000 |
439 | @@ -98,3 +98,14 @@ |
440 | rec_service_root = schema.StringOption( |
441 | default="http://rec.ubuntu.com/api/1.0") |
442 | num_recommended_apps = schema.IntOption(default=4) |
443 | + |
444 | + class celery(schema.Section): |
445 | + broker_backend = schema.StringOption(default='memory') |
446 | + celery_result_backend = schema.StringOption(default='memory') |
447 | + celery_always_eager = schema.BoolOption() |
448 | + celery_eager_propagates_exceptions = schema.BoolOption() |
449 | + broker_url = schema.StringOption() |
450 | + celery_imports = schema.ListOption(item=schema.StringOption()) |
451 | + celery_ignore_result = schema.BoolOption() |
452 | + celery_disable_rate_limits = schema.BoolOption(default=True) |
453 | + celerybeat_scheduler = schema.StringOption() |
454 | |
455 | === modified file 'src/webcatalog/static/css/webcatalog.css' |
456 | --- src/webcatalog/static/css/webcatalog.css 2012-05-29 13:28:54 +0000 |
457 | +++ src/webcatalog/static/css/webcatalog.css 2012-07-02 21:27:20 +0000 |
458 | @@ -361,7 +361,7 @@ |
459 | margin-left: 7px; |
460 | } |
461 | p.error { |
462 | - background-color: #f2cfce; |
463 | + background-color: #df382b; |
464 | } |
465 | .emaillinkportlet { |
466 | margin-top: 64px; |
467 | |
468 | === added file 'src/webcatalog/tasks.py' |
469 | --- src/webcatalog/tasks.py 1970-01-01 00:00:00 +0000 |
470 | +++ src/webcatalog/tasks.py 2012-07-02 21:27:20 +0000 |
471 | @@ -0,0 +1,54 @@ |
472 | +from time import sleep |
473 | + |
474 | +from celery.task import task |
475 | +from django.conf import settings |
476 | +from django.core.management import call_command |
477 | + |
478 | + |
479 | +@task(name="webcatalog.tasks.import_exhibits") |
480 | +def import_exhibits(): |
481 | + """Import all exhibits""" |
482 | + call_command('import_exhibits') |
483 | + |
484 | + |
485 | +@task(name="webcatalog.tasks.import_sca") |
486 | +def import_sca(): |
487 | + """Import all data from the Software Center Agent (MyApps)""" |
488 | + call_command('import_sca_apps') |
489 | + |
490 | + |
491 | +@task(name="webcatalog.tasks.import_app_install_data") |
492 | +def import_app_install_data(): |
493 | + """Import all data from app-install-data and apt-cache""" |
494 | + call_command('import_all_app_install_data', verbosity=0) |
495 | + |
496 | + |
497 | +@task(name="webcatalog.tasks.import_ratings_stats") |
498 | +def import_ratings_stats(): |
499 | + """Import all ratings and reviews data""" |
500 | + call_command('import_all_ratings_stats', verbosity=0) |
501 | + |
502 | + |
503 | +@task(name="webcatalog.tasks.check_all_latest") |
504 | +def check_all_latest(): |
505 | + """Update the 'is_latest' bit on all Applications""" |
506 | + call_command('check_all_latest') |
507 | + |
508 | + |
509 | +@task(name="webcatalog.tasks.cleanup_sessions") |
510 | +def cleanup_sessions(): |
511 | + """Remove stale sessions from the DB""" |
512 | + call_command('cleanup', 'django_session', verbosity=0) |
513 | + |
514 | + |
515 | +@task(name="webcatalog.tasks.cleanup_nonces") |
516 | +def cleanup_nonces(): |
517 | + """Remove stale OAuth nonces from the DB""" |
518 | + call_command('cleanup', 'webcatalog_nonce', verbosity=0) |
519 | + |
520 | + |
521 | +@task(name="webcatalog.tasks.fail") |
522 | +def fail(): |
523 | + """Sleep for 5 seconds, then raise an error""" |
524 | + sleep(5) |
525 | + raise ZeroDivisionError('Intentionally failed') |
526 | |
527 | === added file 'src/webcatalog/templates/forbidden.html' |
528 | --- src/webcatalog/templates/forbidden.html 1970-01-01 00:00:00 +0000 |
529 | +++ src/webcatalog/templates/forbidden.html 2012-07-02 21:27:20 +0000 |
530 | @@ -0,0 +1,23 @@ |
531 | +{% extends "webcatalog/base.html" %} |
532 | +{% load i18n %} |
533 | + |
534 | +{% block title %} |
535 | + {% trans "Access forbidden" %} |
536 | +{% endblock %} |
537 | + |
538 | +{% block header %} |
539 | + {% trans "Access forbidden" %} |
540 | +{% endblock %} |
541 | + |
542 | +{% block content %} |
543 | + <p> You just tried to access a feature which you don't have permission |
544 | + to use.</p> |
545 | + {% if extra_message %} |
546 | + <p>{{ extra_message }}</p> |
547 | + {% endif %} |
548 | + {% if next %} |
549 | + <p> If you signed in with the wrong account you might want to try |
550 | + <a href="{{next}}">logging in again</a>. |
551 | + </p> |
552 | + {% endif %} |
553 | +{% endblock %} |
554 | |
555 | === added file 'src/webcatalog/templates/webcatalog/task_list.html' |
556 | --- src/webcatalog/templates/webcatalog/task_list.html 1970-01-01 00:00:00 +0000 |
557 | +++ src/webcatalog/templates/webcatalog/task_list.html 2012-07-02 21:27:20 +0000 |
558 | @@ -0,0 +1,84 @@ |
559 | +{% extends "webcatalog/base.html" %} |
560 | +{% load i18n %} |
561 | + |
562 | +{% block title %}{% trans "Ubuntu Apps Directory" %} — {% trans "Schedule tasks" %}{% endblock %} |
563 | +{% block header %}{% trans "Schedule asynchronous tasks" %}{% endblock %} |
564 | +{% block head_extra %} |
565 | + {{ block.super }} |
566 | +<script src="{% url wc-combo %}?yui/3.4.0/build/yui/yui-min.js"></script> |
567 | +<style type="text/css"> |
568 | +div.task button { |
569 | + width: 200px; |
570 | + height: 30px; |
571 | +} |
572 | + |
573 | +div#messages { |
574 | + min-height: 40px; |
575 | +} |
576 | +div.traceback { |
577 | + background: #ddd; |
578 | + padding: 8px; |
579 | + margin-bottom: 16px; |
580 | +} |
581 | +</style> |
582 | +<script> |
583 | +YUI({combine: true, comboBase: '{% url wc-combo %}?', root: 'yui/3.4.0/build/'}).use('io-base', 'node-base', function (Y) { |
584 | + var check_status = function(task_id, message) { |
585 | + var check_uri = "/cat/task_status/" + task_id + '/'; |
586 | + var UNREADY_STATES = ["PENDING", "RECEIVED", "STARTED", "RETRY"]; |
587 | + |
588 | + Y.io(check_uri, { |
589 | + on: { |
590 | + complete: function(id, response){ |
591 | + if (response.status == 200) { |
592 | + var result = JSON.parse(response.responseText); |
593 | + var status = result['status']; |
594 | + if (UNREADY_STATES.indexOf(status) >= 0) { |
595 | + message.append('. . . '); |
596 | + setTimeout(function() {check_status(task_id, message)}, 3000); |
597 | + } |
598 | + else { |
599 | + if (status == 'FAILURE') { |
600 | + message.removeClass('success'); |
601 | + message.addClass('error'); |
602 | + Y.one('#messages').append( |
603 | + '<div class="traceback"><pre>' + |
604 | + result['traceback'] + '</pre></div>'); |
605 | + } |
606 | + message.append(status); |
607 | + Y.all('button').removeAttribute('disabled'); |
608 | + } |
609 | + } |
610 | + } |
611 | + } |
612 | + }); |
613 | + } |
614 | + Y.all('p.message').each(function (message) { |
615 | + if (message.hasClass('success')) { |
616 | + var task_id = message.one('span').getContent(); |
617 | + Y.all('button').setAttribute('disabled', 'disabled') |
618 | + check_status(task_id, message); |
619 | + } |
620 | + }); |
621 | +}); |
622 | +</script> |
623 | +{% endblock %} |
624 | +{% block content %} |
625 | + |
626 | +<div id="messages"> |
627 | +{% if messages %} |
628 | + {% for message in messages %} |
629 | + <p class="message{% if message.tags %} {{ message.tags }}{% endif %}">{{ message|safe }}</p> |
630 | + {% endfor %} |
631 | +{% endif %} |
632 | +</div> |
633 | +{% for task in task_list %} |
634 | +<div class="task"> |
635 | +<form method="post" action=".">{% csrf_token %} |
636 | +<input type="hidden" name="task" value="{{ task.name }}"/> |
637 | +<button type="submit">{{ task.name }}</button> {{ task.doc }} |
638 | +</form> |
639 | +</div> |
640 | +{% endfor %} |
641 | + |
642 | +{% endblock %} |
643 | |
644 | === modified file 'src/webcatalog/templatetags/webcatalog.py' |
645 | --- src/webcatalog/templatetags/webcatalog.py 2012-06-06 16:38:11 +0000 |
646 | +++ src/webcatalog/templatetags/webcatalog.py 2012-07-02 21:27:20 +0000 |
647 | @@ -17,10 +17,7 @@ |
648 | |
649 | """Custom template tags for the Apps Directory.""" |
650 | |
651 | -from __future__ import ( |
652 | - absolute_import, |
653 | - with_statement, |
654 | -) |
655 | +from __future__ import absolute_import |
656 | |
657 | __metaclass__ = type |
658 | __all__ = [ |
659 | |
660 | === modified file 'src/webcatalog/tests/__init__.py' |
661 | --- src/webcatalog/tests/__init__.py 2012-06-28 09:49:11 +0000 |
662 | +++ src/webcatalog/tests/__init__.py 2012-07-02 21:27:20 +0000 |
663 | @@ -27,6 +27,7 @@ |
664 | from .test_migrations import * |
665 | from .test_pep8 import * |
666 | from .test_preflight import * |
667 | +from .test_tasks import * |
668 | from .test_templatetags import * |
669 | from .test_utilities import * |
670 | from .test_views import * |
671 | |
672 | === modified file 'src/webcatalog/tests/factory.py' |
673 | --- src/webcatalog/tests/factory.py 2012-06-28 09:49:11 +0000 |
674 | +++ src/webcatalog/tests/factory.py 2012-07-02 21:27:20 +0000 |
675 | @@ -17,17 +17,19 @@ |
676 | |
677 | """A TestCase class with a built in object factory.""" |
678 | |
679 | -from __future__ import ( |
680 | - absolute_import, |
681 | - with_statement, |
682 | -) |
683 | +from __future__ import absolute_import |
684 | + |
685 | import os |
686 | from datetime import ( |
687 | datetime, |
688 | timedelta, |
689 | ) |
690 | from itertools import count |
691 | -from django.contrib.auth.models import User |
692 | +from django.contrib.auth.models import ( |
693 | + Group, |
694 | + Permission, |
695 | + User, |
696 | +) |
697 | from django.contrib.sessions.models import Session |
698 | from django.test import ( |
699 | TestCase, |
700 | @@ -70,7 +72,8 @@ |
701 | |
702 | def make_user(self, username=None, email=None, password='test', |
703 | first_name=None, last_name=None, open_id=None, |
704 | - is_admin=False): |
705 | + is_admin=False, permissions=None, groups=None, |
706 | + log_in=False): |
707 | if username is None: |
708 | username = self.get_unique_string(prefix='username') |
709 | if email is None: |
710 | @@ -98,6 +101,15 @@ |
711 | useropenid = UserOpenID.objects.create( |
712 | user=user, claimed_id=open_id, display_id=open_id) |
713 | |
714 | + if permissions: |
715 | + for permission in permissions: |
716 | + user.user_permissions.add( |
717 | + Permission.objects.get(codename=permission)) |
718 | + |
719 | + if groups: |
720 | + for group_name in groups: |
721 | + group, created = Group.objects.get_or_create(name=group_name) |
722 | + user.groups.add(group) |
723 | return user |
724 | |
725 | def make_application(self, package_name=None, name=None, |
726 | @@ -241,6 +253,15 @@ |
727 | super(TestCaseWithFactory, self).setUp() |
728 | self.factory = WebCatalogObjectFactory() |
729 | |
730 | + def make_user(self, *args, **kwargs): |
731 | + """Invokes self.factory.make_user(), but can also log the user in""" |
732 | + logged_in = kwargs.pop('logged_in', False) |
733 | + password = kwargs.get('password', 'test') |
734 | + result = self.factory.make_user(*args, **kwargs) |
735 | + if logged_in: |
736 | + self.client.login(username=result.username, password=password) |
737 | + return result |
738 | + |
739 | |
740 | class TxTestCaseWithFactory(TransactionTestCase): |
741 | |
742 | |
743 | === modified file 'src/webcatalog/tests/helpers.py' |
744 | --- src/webcatalog/tests/helpers.py 2012-06-28 09:34:49 +0000 |
745 | +++ src/webcatalog/tests/helpers.py 2012-07-02 21:27:20 +0000 |
746 | @@ -17,10 +17,7 @@ |
747 | |
748 | """Helpers for testing the Apps Directory.""" |
749 | |
750 | -from __future__ import ( |
751 | - absolute_import, |
752 | - with_statement, |
753 | -) |
754 | +from __future__ import absolute_import |
755 | |
756 | __metaclass__ = type |
757 | __all__ = [ |
758 | |
759 | === modified file 'src/webcatalog/tests/test_commands.py' |
760 | --- src/webcatalog/tests/test_commands.py 2012-06-28 14:14:23 +0000 |
761 | +++ src/webcatalog/tests/test_commands.py 2012-07-02 21:27:20 +0000 |
762 | @@ -17,10 +17,8 @@ |
763 | |
764 | """Test cases for the Apps Directory management commands.""" |
765 | |
766 | -from __future__ import ( |
767 | - absolute_import, |
768 | - with_statement, |
769 | -) |
770 | +from __future__ import absolute_import |
771 | + |
772 | import apt |
773 | import json |
774 | import os |
775 | @@ -716,7 +714,7 @@ |
776 | def test_updates_last_import_record(self): |
777 | onion = self.factory.make_distroseries(code_name='onion') |
778 | orig_timestamp = datetime(2011, 07, 18, 14, 43) |
779 | - import_record = ReviewStatsImport.objects.create( |
780 | + ReviewStatsImport.objects.create( |
781 | distroseries=onion, last_import=orig_timestamp) |
782 | |
783 | call_command('import_ratings_stats', 'onion') |
784 | @@ -728,7 +726,7 @@ |
785 | def test_download_review_stats_no_previous(self): |
786 | # If there have been no previous imports, the complete stats |
787 | # will be retrieved. |
788 | - onion = self.factory.make_distroseries(code_name='onion') |
789 | + self.factory.make_distroseries(code_name='onion') |
790 | |
791 | call_command('import_ratings_stats', 'onion') |
792 | |
793 | |
794 | === modified file 'src/webcatalog/tests/test_department_filters.py' |
795 | --- src/webcatalog/tests/test_department_filters.py 2012-06-06 18:00:26 +0000 |
796 | +++ src/webcatalog/tests/test_department_filters.py 2012-07-02 21:27:20 +0000 |
797 | @@ -17,11 +17,7 @@ |
798 | |
799 | """Test cases for department filters.""" |
800 | |
801 | -from __future__ import ( |
802 | - absolute_import, |
803 | - with_statement, |
804 | -) |
805 | - |
806 | +from __future__ import absolute_import |
807 | |
808 | from webcatalog.tests.factory import TestCaseWithFactory |
809 | from webcatalog.department_filters import ( |
810 | |
811 | === modified file 'src/webcatalog/tests/test_forms.py' |
812 | --- src/webcatalog/tests/test_forms.py 2012-06-28 14:25:19 +0000 |
813 | +++ src/webcatalog/tests/test_forms.py 2012-07-02 21:27:20 +0000 |
814 | @@ -17,10 +17,7 @@ |
815 | |
816 | """Test cases for the Apps Directory forms.""" |
817 | |
818 | -from __future__ import ( |
819 | - absolute_import, |
820 | - with_statement, |
821 | -) |
822 | +from __future__ import absolute_import |
823 | |
824 | from django.forms import ValidationError |
825 | from django.test import TestCase |
826 | |
827 | === modified file 'src/webcatalog/tests/test_managers.py' |
828 | --- src/webcatalog/tests/test_managers.py 2012-06-26 12:40:57 +0000 |
829 | +++ src/webcatalog/tests/test_managers.py 2012-07-02 21:27:20 +0000 |
830 | @@ -17,10 +17,7 @@ |
831 | |
832 | """Test cases for object managers.""" |
833 | |
834 | -from __future__ import ( |
835 | - absolute_import, |
836 | - with_statement, |
837 | -) |
838 | +from __future__ import absolute_import |
839 | |
840 | from webcatalog.tests.factory import TestCaseWithFactory |
841 | from webcatalog.models import Application |
842 | |
843 | === modified file 'src/webcatalog/tests/test_models.py' |
844 | --- src/webcatalog/tests/test_models.py 2012-06-06 19:48:08 +0000 |
845 | +++ src/webcatalog/tests/test_models.py 2012-07-02 21:27:20 +0000 |
846 | @@ -17,10 +17,7 @@ |
847 | |
848 | """Test cases for models.""" |
849 | |
850 | -from __future__ import ( |
851 | - absolute_import, |
852 | - with_statement, |
853 | -) |
854 | +from __future__ import absolute_import |
855 | |
856 | from django.core.urlresolvers import reverse |
857 | from django.core.files.images import ImageFile |
858 | |
859 | === modified file 'src/webcatalog/tests/test_preflight.py' |
860 | --- src/webcatalog/tests/test_preflight.py 2012-06-21 20:18:44 +0000 |
861 | +++ src/webcatalog/tests/test_preflight.py 2012-07-02 21:27:20 +0000 |
862 | @@ -28,25 +28,13 @@ |
863 | |
864 | |
865 | class TestPreflight(TestCaseWithFactory): |
866 | - |
867 | - def login(self, add_to_group=True, user=None): |
868 | - if not user: |
869 | - user = User.objects.create_user(username='test', |
870 | - email='test@test.com', |
871 | - password='test') |
872 | - if add_to_group: |
873 | - group = Group.objects.create(name='preflight') |
874 | - user.groups.add(group) |
875 | - user.save() |
876 | - self.client.login(username='test', password='test') |
877 | - |
878 | @patch('webcatalog.utilities.WebServices.identity_provider') |
879 | @patch('webcatalog.utilities.WebServices.get_screenshots_for_package') |
880 | @patch('webcatalog.utilities.WebServices.recommender_api') |
881 | @patch('webcatalog.utilities.WebServices.rnr_api') |
882 | def test_login_needed_for_preflight(self, mock_rnr, mock_recommender, |
883 | mock_screenshots, mock_sso): |
884 | - self.login() |
885 | + self.make_user(groups=['preflight'], logged_in=True) |
886 | |
887 | with patch_settings(PREFLIGHT_GROUPS=['preflight']): |
888 | response = self.client.get('/preflight/') |
889 | @@ -60,8 +48,7 @@ |
890 | self.assertEqual(404, response.status_code) |
891 | |
892 | def test_login_but_wrong_group_means_preflight_not_found(self): |
893 | - with patch_settings(PREFLIGHT_GROUPS=['preflight']): |
894 | - self.login(add_to_group=False) |
895 | + self.make_user(logged_in=True) |
896 | |
897 | response = self.client.get('/preflight/') |
898 | |
899 | @@ -77,7 +64,7 @@ |
900 | mock_recommender.server_status.return_value = 'ok' |
901 | mock_screenshots.return_value = ['some_screenshot.jpg'] |
902 | mock_sso.validate_token.return_value = False |
903 | - self.login() |
904 | + self.make_user(groups=['preflight'], logged_in=True) |
905 | |
906 | with patch_settings(PREFLIGHT_GROUPS=['preflight']): |
907 | response = self.client.get('/preflight/') |
908 | |
909 | === added file 'src/webcatalog/tests/test_tasks.py' |
910 | --- src/webcatalog/tests/test_tasks.py 1970-01-01 00:00:00 +0000 |
911 | +++ src/webcatalog/tests/test_tasks.py 2012-07-02 21:27:20 +0000 |
912 | @@ -0,0 +1,42 @@ |
913 | +from django.conf import settings |
914 | +from django.test import TestCase |
915 | +from mock import patch |
916 | + |
917 | +from webcatalog.models import Exhibit |
918 | +from webcatalog.tasks import ( |
919 | + fail, |
920 | + import_exhibits, |
921 | +) |
922 | + |
923 | + |
924 | +class ImportExhbitsTaskTestCase(TestCase): |
925 | + @patch('webcatalog.management.commands.import_exhibits.urllib.urlopen') |
926 | + def test_success(self, mock_urlopen): |
927 | + """Test that the ``import_exhibits`` task runs with no errors""" |
928 | + fd = mock_urlopen.return_value |
929 | + fd.code = 200 |
930 | + fd.read.return_value = """[{ |
931 | + "package_names": "foobar", |
932 | + "banner_url": "http://example.com/exhibits/foobar.png", |
933 | + "distroseries": [{"version": "12.04", "code_name": "precise"}], |
934 | + "html": "<div><div>Hello world!</div></div>", |
935 | + "published": true, |
936 | + "date_created": "2012-06-11 17:22:45", |
937 | + "id": 2 |
938 | + }]""" |
939 | + expected_url = '%sexhibits/en/' % settings.SCA_API_URL |
940 | + |
941 | + result = import_exhibits.delay() |
942 | + |
943 | + self.assertIsNone(result.get()) |
944 | + self.assertTrue(result.successful()) |
945 | + mock_urlopen.assertCalledWith(expected_url) |
946 | + self.assertEqual(1, Exhibit.objects.all().count()) |
947 | + |
948 | + |
949 | +class FailTaskTestCase(TestCase): |
950 | + @patch('webcatalog.tasks.sleep') |
951 | + def test_delays_and_fails_as_expected(self, mock_sleep): |
952 | + """Test that the ``fail`` task raises an Exception as expected""" |
953 | + self.assertRaises(ZeroDivisionError, fail.delay) |
954 | + mock_sleep.assertCalledWith(5) |
955 | |
956 | === modified file 'src/webcatalog/tests/test_templatetags.py' |
957 | --- src/webcatalog/tests/test_templatetags.py 2012-06-14 13:57:03 +0000 |
958 | +++ src/webcatalog/tests/test_templatetags.py 2012-07-02 21:27:20 +0000 |
959 | @@ -17,13 +17,9 @@ |
960 | |
961 | """Tests for the Apps Directory template tags.""" |
962 | |
963 | -from __future__ import ( |
964 | - absolute_import, |
965 | - with_statement, |
966 | -) |
967 | +from __future__ import absolute_import |
968 | |
969 | import unittest |
970 | -from decimal import Decimal |
971 | |
972 | from django.core.urlresolvers import reverse |
973 | from django.template import Context |
974 | |
975 | === modified file 'src/webcatalog/tests/test_views.py' |
976 | --- src/webcatalog/tests/test_views.py 2012-06-28 19:06:37 +0000 |
977 | +++ src/webcatalog/tests/test_views.py 2012-07-02 21:27:20 +0000 |
978 | @@ -17,10 +17,7 @@ |
979 | |
980 | """Test cases for the Apps Directory views.""" |
981 | |
982 | -from __future__ import ( |
983 | - absolute_import, |
984 | - with_statement, |
985 | -) |
986 | +from __future__ import absolute_import |
987 | |
988 | import json |
989 | import re |
990 | @@ -53,6 +50,8 @@ |
991 | 'ApplicationReviewsTestCase', |
992 | 'ApplicationScreenshotsTestCase', |
993 | 'IndexTestCase', |
994 | + 'TasksViewTestCase', |
995 | + 'TaskStatusViewTestCase', |
996 | 'TermsOfServiceTestCase', |
997 | 'OverviewTestCase', |
998 | 'SearchTestCase', |
999 | @@ -1313,3 +1312,116 @@ |
1000 | expected = '/* js/foo.css */\n/* [missing] */\n' |
1001 | self.assertEqual('text/css', response['Content-Type']) |
1002 | self.assertEqual(expected, response.content) |
1003 | + |
1004 | + |
1005 | +class TasksViewTestCase(TestCaseWithFactory): |
1006 | + """Tests for the task list""" |
1007 | + def test_not_allowed_if_not_logged_in(self): |
1008 | + url = reverse('wc-tasks') |
1009 | + |
1010 | + response = self.client.get(url) |
1011 | + |
1012 | + self.assertEqual(302, response.status_code) |
1013 | + expected = reverse('wc-forbidden') + '?next=' + url |
1014 | + self.assertTrue(response['location'].endswith(expected)) |
1015 | + |
1016 | + def test_scheduler_perm_is_required(self): |
1017 | + self.make_user(logged_in=True) |
1018 | + url = reverse('wc-tasks') |
1019 | + |
1020 | + response = self.client.get(url) |
1021 | + |
1022 | + self.assertEqual(302, response.status_code) |
1023 | + expected = reverse('wc-forbidden') + '?next=' + url |
1024 | + self.assertTrue(response['location'].endswith(expected)) |
1025 | + |
1026 | + def test_includes_all_tasks(self): |
1027 | + self.maxDiff = None |
1028 | + self.make_user(permissions=['schedule_tasks'], logged_in=True) |
1029 | + url = reverse('wc-tasks') |
1030 | + |
1031 | + response = self.client.get(url) |
1032 | + |
1033 | + tasks = response.context[0]['task_list'] |
1034 | + expected = [ |
1035 | + {'doc': 'Import all exhibits', 'name': 'import_exhibits'}, |
1036 | + { |
1037 | + 'doc': 'Remove stale OAuth nonces from the DB', |
1038 | + 'name': 'cleanup_nonces', |
1039 | + }, |
1040 | + { |
1041 | + 'doc': 'Remove stale sessions from the DB', |
1042 | + 'name': 'cleanup_sessions', |
1043 | + }, |
1044 | + { |
1045 | + 'doc': 'Sleep for 5 seconds, then raise an error', |
1046 | + 'name': 'fail', |
1047 | + }, |
1048 | + { |
1049 | + 'doc': 'Import all data from app-install-data and apt-cache', |
1050 | + 'name': 'import_app_install_data', |
1051 | + }, |
1052 | + { |
1053 | + 'doc': ('Import all data from the Software ' |
1054 | + 'Center Agent (MyApps)'), |
1055 | + 'name': 'import_sca', |
1056 | + }, |
1057 | + { |
1058 | + 'doc': 'Import all ratings and reviews data', |
1059 | + 'name': 'import_ratings_stats', |
1060 | + }, |
1061 | + { |
1062 | + 'doc': "Update the 'is_latest' bit on all Applications", |
1063 | + 'name': 'check_all_latest', |
1064 | + }, |
1065 | + ] |
1066 | + self.assertEqual(sorted(expected), sorted(tasks)) |
1067 | + |
1068 | + @patch('webcatalog.management.commands.import_exhibits.urllib.urlopen') |
1069 | + def test_post_schedules_task(self, mock_urlopen): |
1070 | + fd = mock_urlopen.return_value |
1071 | + fd.code = 200 |
1072 | + fd.read.return_value = "[]" |
1073 | + self.make_user(permissions=['schedule_tasks'], logged_in=True) |
1074 | + url = reverse('wc-tasks') |
1075 | + data = {'task': 'import_exhibits'} |
1076 | + |
1077 | + response = self.client.post(url, data=data, follow=True) |
1078 | + |
1079 | + msg_pat = r'Task id <span>[-\da-f]+</span> submitted' |
1080 | + match = re.search(msg_pat, response.content) |
1081 | + self.assertFalse(match is None) |
1082 | + self.assertEqual(1, mock_urlopen.call_count) |
1083 | + |
1084 | + |
1085 | +class TaskStatusViewTestCase(TestCaseWithFactory): |
1086 | + """Tests for the task status getter""" |
1087 | + def test_not_allowed_if_not_logged_in(self): |
1088 | + url = reverse('wc-task-status', args=['cafe']) |
1089 | + |
1090 | + response = self.client.get(url) |
1091 | + |
1092 | + self.assertEqual(302, response.status_code) |
1093 | + expected = reverse('wc-forbidden') + '?next=' + url |
1094 | + self.assertTrue(response['location'].endswith(expected)) |
1095 | + |
1096 | + def test_scheduler_perm_is_required(self): |
1097 | + self.make_user(logged_in=True) |
1098 | + url = reverse('wc-task-status', args=['cafe']) |
1099 | + |
1100 | + response = self.client.get(url) |
1101 | + |
1102 | + self.assertEqual(302, response.status_code) |
1103 | + expected = reverse('wc-forbidden') + '?next=' + url |
1104 | + self.assertTrue(response['location'].endswith(expected)) |
1105 | + |
1106 | + def test_unknown_task_is_reported_pending(self): |
1107 | + self.make_user(permissions=['schedule_tasks'], logged_in=True) |
1108 | + url = reverse('wc-task-status', args=['cafe']) |
1109 | + |
1110 | + response = self.client.get(url) |
1111 | + |
1112 | + data = json.loads(response.content) |
1113 | + expected = {"status": "PENDING", "traceback": None, "id": "cafe"} |
1114 | + |
1115 | + self.assertEqual(expected, data) |
1116 | |
1117 | === modified file 'src/webcatalog/urls.py' |
1118 | --- src/webcatalog/urls.py 2012-06-06 17:42:04 +0000 |
1119 | +++ src/webcatalog/urls.py 2012-07-02 21:27:20 +0000 |
1120 | @@ -17,10 +17,8 @@ |
1121 | |
1122 | """Url configuration for the Apps Directory.""" |
1123 | |
1124 | -from __future__ import ( |
1125 | - absolute_import, |
1126 | - with_statement, |
1127 | -) |
1128 | +from __future__ import absolute_import |
1129 | + |
1130 | from django.conf.urls.defaults import patterns, include, url |
1131 | from django.views.generic import TemplateView |
1132 | |
1133 | @@ -59,6 +57,10 @@ |
1134 | url(r'^tos/plain/$', name="wc-tos-plain", |
1135 | view=TemplateView.as_view(template_name="webcatalog/tos_plain.html")), |
1136 | url(r'^combo/$', 'combo_view', name='wc-combo'), |
1137 | + url(r'^forbidden/$', 'forbidden', name='wc-forbidden'), |
1138 | + url(r'^tasks/$', 'tasks', name='wc-tasks'), |
1139 | + url(r'^task_status/(?P<task_id>[-\da-f]+)/$', 'task_status', |
1140 | + name='wc-task-status'), |
1141 | |
1142 | (r'^api/', include('webcatalog.api.urls')), |
1143 | ) |
1144 | |
1145 | === modified file 'src/webcatalog/views.py' |
1146 | --- src/webcatalog/views.py 2012-06-28 14:46:12 +0000 |
1147 | +++ src/webcatalog/views.py 2012-07-02 21:27:20 +0000 |
1148 | @@ -17,15 +17,14 @@ |
1149 | |
1150 | """Views for the Apps Directory app.""" |
1151 | |
1152 | -from __future__ import ( |
1153 | - absolute_import, |
1154 | - with_statement, |
1155 | -) |
1156 | +from __future__ import absolute_import |
1157 | |
1158 | import json |
1159 | import operator |
1160 | import os |
1161 | +from inspect import getmembers |
1162 | |
1163 | +from celery.task import Task |
1164 | from convoy.combo import combine_files, parse_qs |
1165 | from django.conf import settings |
1166 | from django.contrib import messages |
1167 | @@ -33,17 +32,23 @@ |
1168 | from django.core.urlresolvers import reverse |
1169 | from django.db.models import Q |
1170 | from django.http import ( |
1171 | + HttpResponse, |
1172 | + HttpResponseForbidden, |
1173 | + HttpResponseNotFound, |
1174 | HttpResponseRedirect, |
1175 | - HttpResponse, |
1176 | ) |
1177 | from django.shortcuts import ( |
1178 | - get_list_or_404, |
1179 | get_object_or_404, |
1180 | render_to_response, |
1181 | ) |
1182 | from django.template import RequestContext |
1183 | +from django.template.loader import render_to_string |
1184 | from django.utils.translation import ugettext as _ |
1185 | +from django.views.decorators.cache import never_cache |
1186 | +from django.views.decorators.http import require_GET |
1187 | +from djcelery.models import TaskState |
1188 | |
1189 | +import webcatalog.tasks |
1190 | from webcatalog.forms import EmailDownloadLinkForm |
1191 | from webcatalog.models import ( |
1192 | Application, |
1193 | @@ -51,6 +56,7 @@ |
1194 | DistroSeries, |
1195 | Exhibit, |
1196 | ) |
1197 | +from webcatalog.decorators import scheduler_required |
1198 | from webcatalog.utilities import WebServices |
1199 | |
1200 | |
1201 | @@ -59,8 +65,10 @@ |
1202 | 'application_detail', |
1203 | 'application_recommends', |
1204 | 'application_reviews', |
1205 | + 'department_overview', |
1206 | + 'forbidden', |
1207 | 'index', |
1208 | - 'department_overview', |
1209 | + 'schedule_task', |
1210 | 'search', |
1211 | ] |
1212 | |
1213 | @@ -295,3 +303,49 @@ |
1214 | content_type=content_type, status=200, |
1215 | content="".join(content)) |
1216 | return HttpResponse(content_type=content_type, status=404) |
1217 | + |
1218 | + |
1219 | +@scheduler_required |
1220 | +@never_cache |
1221 | +def tasks(request): |
1222 | + if request.method == 'POST': |
1223 | + task_name = request.POST.get('task', '') |
1224 | + task = getattr(webcatalog.tasks, task_name, None) |
1225 | + if not task: |
1226 | + messages.error(request, 'Invalid task name "%s"' % task_name) |
1227 | + return HttpResponseRedirect(reverse('wc-tasks')) |
1228 | + result = task.delay() |
1229 | + messages.success(request, |
1230 | + 'Task id <span>%s</span> submitted' % result.task_id) |
1231 | + return HttpResponseRedirect(reverse('wc-tasks')) |
1232 | + is_task = lambda x: isinstance(x, Task) |
1233 | + task_list = [ |
1234 | + {'name': key, 'doc': val.__doc__} |
1235 | + for key, val in getmembers(webcatalog.tasks, is_task) |
1236 | + ] |
1237 | + context = { |
1238 | + 'task_list': sorted(task_list), |
1239 | + } |
1240 | + return render_to_response('webcatalog/task_list.html', |
1241 | + RequestContext(request, context)) |
1242 | + |
1243 | + |
1244 | +@scheduler_required |
1245 | +@require_GET |
1246 | +@never_cache |
1247 | +def task_status(request, task_id): |
1248 | + try: |
1249 | + task = TaskState.objects.get(task_id=task_id) |
1250 | + data = { |
1251 | + 'id': task.task_id, |
1252 | + 'status': task.state, |
1253 | + 'traceback': task.traceback, |
1254 | + } |
1255 | + except TaskState.DoesNotExist: |
1256 | + data = {'id': task_id, 'status': 'PENDING', 'traceback': None} |
1257 | + return HttpResponse(json.dumps(data), mimetype='application/json') |
1258 | + |
1259 | + |
1260 | +def forbidden(request): |
1261 | + return HttpResponseForbidden( |
1262 | + render_to_string('forbidden.html', RequestContext(request))) |
30 -openid_ launchpad_ staff_teams = canonical- ca-hackers launchpad_ staff_teams = canonical- isd-hackers
31 +openid_
That looks backwards?
332 + if (message. hasClass( 'success' )) {
That looks as if it will loop forever on successful tasks?
Most of this branch looks pretty good to me.
Thanks,
James