Merge lp:~doanac/lava-scheduler/failure-reporting into lp:lava-scheduler

[Andy Doan]

This change is an attempt to rid us of the google doc:

 https://docs.google.com/a/linaro.org/spreadsheet/ccc?key=0AnxpY5uv-BlNdG9zYTdDLWZWRVFGaWFxQzRLNWtaNmc

Since this updates views, I thought I'd walk you throught the basic usage with some screenshots:

1) Adding "failure tags". This can be used for common issues like "downloading root.tgz"
 http://people.linaro.org/~doanac/failure-pics/admin_panel.png

2) Drill-down
The main way I see this working is through our current "reports" view. I've added hyper-links to each date range, that will link to what you see in item 3:

 http://people.linaro.org/~doanac/failure-pics/reports-main.png

3) Reports view:
When you first look at this type of view, it might not have failures annotated. it would look like:

 http://people.linaro.org/~doanac/failure-pics/failure-view-empty.png

This page is pretty dynamic and allows narrowing down results via some GET parameters (documented in commit message)

4) "Annotating" a failure
When you look at a job, if its INCOMPLETE, CANCELED, or CANCELING, you'll be given an "annotate failure" link (assuming you are a super user):

 http://people.linaro.org/~doanac/failure-pics/failure-action.png

This page is then pretty simple:

 http://people.linaro.org/~doanac/failure-pics/failure-annotate.png

The styling is ugly but it should work for us

5) After annotating some failures the reports view will now be a little nicer:
 http://people.linaro.org/~doanac/failure-pics/failure-view-data.png

[Michael Hudson-Doyle (mwhudson)]

Andy Doan <email address hidden> writes:

> Andy Doan has proposed merging lp:~doanac/lava-scheduler/failure-reporting into lp:lava-scheduler.
>
> Requested reviews:
> Linaro Validation Team (linaro-validation)
>
> For more details, see:
> https://code.launchpad.net/~doanac/lava-scheduler/failure-reporting/+merge/141679
>
> This change is an attempt to rid us of the google doc:
>
> https://docs.google.com/a/linaro.org/spreadsheet/ccc?key=0AnxpY5uv-BlNdG9zYTdDLWZWRVFGaWFxQzRLNWtaNmc
>
> Since this updates views, I thought I'd walk you throught the basic usage with some screenshots:
>
> 1) Adding "failure tags". This can be used for common issues like "downloading root.tgz"
> http://people.linaro.org/~doanac/failure-pics/admin_panel.png
>
> 2) Drill-down
> The main way I see this working is through our current "reports" view. I've added hyper-links to each date range, that will link to what you see in item 3:
>
> http://people.linaro.org/~doanac/failure-pics/reports-main.png
>
> 3) Reports view:
> When you first look at this type of view, it might not have failures annotated. it would look like:
>
> http://people.linaro.org/~doanac/failure-pics/failure-view-empty.png
>
> This page is pretty dynamic and allows narrowing down results via some GET parameters (documented in commit message)
>
> 4) "Annotating" a failure
> When you look at a job, if its INCOMPLETE, CANCELED, or CANCELING, you'll be given an "annotate failure" link (assuming you are a super user):
>
> http://people.linaro.org/~doanac/failure-pics/failure-action.png
>
> This page is then pretty simple:
>
> http://people.linaro.org/~doanac/failure-pics/failure-annotate.png
>
> The styling is ugly but it should work for us
>
> 5) After annotating some failures the reports view will now be a little nicer:
> http://people.linaro.org/~doanac/failure-pics/failure-view-data.png

This is a super awesome surprise! A few comments below, but mostly I'm
a massive +1.

>
> === added file 'lava_scheduler_app/templates/lava_scheduler_app/failure_report.html'
> --- lava_scheduler_app/templates/lava_scheduler_app/failure_report.html 1970-01-01 00:00:00 +0000
> +++ lava_scheduler_app/templates/lava_scheduler_app/failure_report.html 2013-01-02 22:49:23 +0000
> @@ -0,0 +1,12 @@
> +{% extends "lava_scheduler_app/_content.html" %}
> +
> +{% load django_tables2 %}
> +
> +{% block content %}
> +<h2>Failure Report</h2>
> +
> +{% block content_columns %}

I don't think you need this.

> +{% render_table failed_job_table %}
> +{% endblock %}
> +
> +{% endblock %}
>
> === added file 'lava_scheduler_app/templates/lava_scheduler_app/job_annotate_failure.html'
> --- lava_scheduler_app/templates/lava_scheduler_app/job_annotate_failure.html 1970-01-01 00:00:00 +0000
> +++ lava_scheduler_app/templates/lava_scheduler_app/job_annotate_failure.html 2013-01-02 22:49:23 +0000
> @@ -0,0 +1,16 @@
> +{% extends "lava_scheduler_app/job_sidebar.html" %}
> +
> +{% block content %}
> +<h2>Annotate Job Failure - {{ job.id }} </h2>
> +
> +{% if form.errors %}
> +<h3>Errors found in submission</h3>
> +{{ form.errors }}
> +{% endif %}
> +
> +<form action="{% url lava.scheduler.job.annotate_failure job.pk %}" method="post"...

[lp:~doanac/lava-scheduler/failure-reporting]

236. By Andy Doan on 2013-01-03

address code review issues

237. By Andy Doan on 2013-01-03

one more code review fix

The code works, but just because of some django coersion. This makes
it explicit

[Andy Doan (doanac)]

All comments addressed in latest push.

On 01/02/2013 07:46 PM, Michael Hudson-Doyle wrote:
>> + dt = request.GET.get('device_type', None)
>> >+ if dt:
>> >+ jobs = jobs.filter(actual_device__device_type=dt)
> Does this work? I'd have thought that dt would need to be an actual
> DeviceType object here. Or that you'd have to say
> actual_device__device_type__name=dt or something.

It did actually work, but I went ahead and fixed it. I guess django did
some coercion for me, but making this explicit is best.

>> >+ device = request.GET.get('device', None)
>> >+ if device:
>> >+ jobs = jobs.filter(actual_device__hostname=device)
>> >+
>> >+ start = request.GET.get('start', None)
>> >+ if start:
>> >+ now = datetime.datetime.now()
>> >+ start = now + datetime.timedelta(int(start))
>> >+
>> >+ end = request.GET.get('end', None)
>> >+ if end:
>> >+ end = now + datetime.timedelta(int(end))
>> >+ jobs = jobs.filter(start_time__range=(start, end))
> So if you specify start but not end, there is no filtering? I guess
> that's OK.

Yeah - that sucks. I thought about making the parameter be "range", but
then you wind up with some odd syntax requirement in a URL.

[Michael Hudson-Doyle (mwhudson)]

Cool, looks good to go. By "not using as_p" I meant doing the form rendering more manually, but never mind that for now :-)

=== modified file 'lava_scheduler_app/admin.py'
--- lava_scheduler_app/admin.py	2012-10-30 08:27:20 +0000
+++ lava_scheduler_app/admin.py	2013-01-03 16:18:24 +0000
@@ -1,6 +1,6 @@
 from django.contrib import admin
 from lava_scheduler_app.models import (
-    Device, DeviceStateTransition, DeviceType, TestJob, Tag,
+    Device, DeviceStateTransition, DeviceType, TestJob, Tag, JobFailureTag,
     )
 
 # XXX These actions should really go to another screen that asks for a reason.
@@ -55,3 +55,4 @@
 admin.site.register(DeviceType)
 admin.site.register(TestJob, TestJobAdmin)
 admin.site.register(Tag)
+admin.site.register(JobFailureTag)

=== added file 'lava_scheduler_app/migrations/0029_auto__add_jobfailuretag__add_field_testjob_failure_comment.py'
--- lava_scheduler_app/migrations/0029_auto__add_jobfailuretag__add_field_testjob_failure_comment.py	1970-01-01 00:00:00 +0000
+++ lava_scheduler_app/migrations/0029_auto__add_jobfailuretag__add_field_testjob_failure_comment.py	2013-01-03 16:18:24 +0000
@@ -0,0 +1,181 @@
+# -*- coding: utf-8 -*-
+import datetime
+from south.db import db
+from south.v2 import SchemaMigration
+from django.db import models
+
+
+class Migration(SchemaMigration):
+
+    def forwards(self, orm):
+        # Adding model 'JobFailureTag'
+        db.create_table('lava_scheduler_app_jobfailuretag', (
+            ('id', self.gf('django.db.models.fields.AutoField')(primary_key=True)),
+            ('name', self.gf('django.db.models.fields.CharField')(unique=True, max_length=256)),
+            ('description', self.gf('django.db.models.fields.TextField')(null=True, blank=True)),
+        ))
+        db.send_create_signal('lava_scheduler_app', ['JobFailureTag'])
+
+        # Adding field 'TestJob.failure_comment'
+        db.add_column('lava_scheduler_app_testjob', 'failure_comment',
+                      self.gf('django.db.models.fields.TextField')(null=True, blank=True),
+                      keep_default=False)
+
+        # Adding M2M table for field failure_tags on 'TestJob'
+        db.create_table('lava_scheduler_app_testjob_failure_tags', (
+            ('id', models.AutoField(verbose_name='ID', primary_key=True, auto_created=True)),
+            ('testjob', models.ForeignKey(orm['lava_scheduler_app.testjob'], null=False)),
+            ('jobfailuretag', models.ForeignKey(orm['lava_scheduler_app.jobfailuretag'], null=False))
+        ))
+        db.create_unique('lava_scheduler_app_testjob_failure_tags', ['testjob_id', 'jobfailuretag_id'])
+
+
+    def backwards(self, orm):
+        # Deleting model 'JobFailureTag'
+        db.delete_table('lava_scheduler_app_jobfailuretag')
+
+        # Deleting field 'TestJob.failure_comment'
+        db.delete_column('lava_scheduler_app_testjob', 'failure_comment')
+
+        # Removing M2M table for field failure_tags on 'TestJob'
+        db.delete_table('lava_scheduler_app_testjob_failure_tags')
+
+
+    models = {
+        'auth.group': {
+            'Meta': {'object_name': 'Group'},
+            'id': ('django.db.models.fields.AutoField', [], {'primary_key': 'True'}),
+            'name': ('django.db.models.fields.CharField', [], {'unique': 'True', 'max_length': '80'}),
+            'permissions': ('django.db.models.fields.related.ManyToManyField', [], {'to': "orm['auth.Permission']", 'symmetrical': 'False', 'blank': 'True'})
+        },
+        'auth.permission': {
+            'Meta': {'ordering': "('content_type__app_label', 'content_type__model', 'codename')", 'unique_together': "(('content_type', 'codename'),)", 'object_name': 'Permission'},
+            'codename': ('django.db.models.fields.CharField', [], {'max_length': '100'}),
+            'content_type': ('django.db.models.fields.related.ForeignKey', [], {'to': "orm['contenttypes.ContentType']"}),
+            'id': ('django.db.models.fields.AutoField', [], {'primary_key': 'True'}),
+            'name': ('django.db.models.fields.CharField', [], {'max_length': '50'})
+        },
+        'auth.user': {
+            'Meta': {'object_name': 'User'},
+            'date_joined': ('django.db.models.fields.DateTimeField', [], {'default': 'datetime.datetime.now'}),
+            'email': ('django.db.models.fields.EmailField', [], {'max_length': '75', 'blank': 'True'}),
+            'first_name': ('django.db.models.fields.CharField', [], {'max_length': '30', 'blank': 'True'}),
+            'groups': ('django.db.models.fields.related.ManyToManyField', [], {'to': "orm['auth.Group']", 'symmetrical': 'False', 'blank': 'True'}),
+            'id': ('django.db.models.fields.AutoField', [], {'primary_key': 'True'}),
+            'is_active': ('django.db.models.fields.BooleanField', [], {'default': 'True'}),
+            'is_staff': ('django.db.models.fields.BooleanField', [], {'default': 'False'}),
+            'is_superuser': ('django.db.models.fields.BooleanField', [], {'default': 'False'}),
+            'last_login': ('django.db.models.fields.DateTimeField', [], {'default': 'datetime.datetime.now'}),
+            'last_name': ('django.db.models.fields.CharField', [], {'max_length': '30', 'blank': 'True'}),
+            'password': ('django.db.models.fields.CharField', [], {'max_length': '128'}),
+            'user_permissions': ('django.db.models.fields.related.ManyToManyField', [], {'to': "orm['auth.Permission']", 'symmetrical': 'False', 'blank': 'True'}),
+            'username': ('django.db.models.fields.CharField', [], {'unique': 'True', 'max_length': '30'})
+        },
+        'contenttypes.contenttype': {
+            'Meta': {'ordering': "('name',)", 'unique_together': "(('app_label', 'model'),)", 'object_name': 'ContentType', 'db_table': "'django_content_type'"},
+            'app_label': ('django.db.models.fields.CharField', [], {'max_length': '100'}),
+            'id': ('django.db.models.fields.AutoField', [], {'primary_key': 'True'}),
+            'model': ('django.db.models.fields.CharField', [], {'max_length': '100'}),
+            'name': ('django.db.models.fields.CharField', [], {'max_length': '100'})
+        },
+        'dashboard_app.bundle': {
+            'Meta': {'ordering': "['-uploaded_on']", 'object_name': 'Bundle'},
+            '_gz_content': ('django.db.models.fields.files.FileField', [], {'max_length': '100', 'null': 'True', 'db_column': "'gz_content'"}),
+            '_raw_content': ('django.db.models.fields.files.FileField', [], {'max_length': '100', 'null': 'True', 'db_column': "'content'"}),
+            'bundle_stream': ('django.db.models.fields.related.ForeignKey', [], {'related_name': "'bundles'", 'to': "orm['dashboard_app.BundleStream']"}),
+            'content_filename': ('django.db.models.fields.CharField', [], {'max_length': '256'}),
+            'content_sha1': ('django.db.models.fields.CharField', [], {'max_length': '40', 'unique': 'True', 'null': 'True'}),
+            'id': ('django.db.models.fields.AutoField', [], {'primary_key': 'True'}),
+            'is_deserialized': ('django.db.models.fields.BooleanField', [], {'default': 'False'}),
+            'uploaded_by': ('django.db.models.fields.related.ForeignKey', [], {'blank': 'True', 'related_name': "'uploaded_bundles'", 'null': 'True', 'to': "orm['auth.User']"}),
+            'uploaded_on': ('django.db.models.fields.DateTimeField', [], {'default': 'datetime.datetime.utcnow'})
+        },
+        'dashboard_app.bundlestream': {
+            'Meta': {'object_name': 'BundleStream'},
+            'group': ('django.db.models.fields.related.ForeignKey', [], {'to': "orm['auth.Group']", 'null': 'True', 'blank': 'True'}),
+            'id': ('django.db.models.fields.AutoField', [], {'primary_key': 'True'}),
+            'is_anonymous': ('django.db.models.fields.BooleanField', [], {'default': 'False'}),
+            'is_public': ('django.db.models.fields.BooleanField', [], {'default': 'False'}),
+            'name': ('django.db.models.fields.CharField', [], {'max_length': '64', 'blank': 'True'}),
+            'pathname': ('django.db.models.fields.CharField', [], {'unique': 'True', 'max_length': '128'}),
+            'slug': ('django.db.models.fields.CharField', [], {'max_length': '64', 'blank': 'True'}),
+            'user': ('django.db.models.fields.related.ForeignKey', [], {'to': "orm['auth.User']", 'null': 'True', 'blank': 'True'})
+        },
+        'lava_scheduler_app.device': {
+            'Meta': {'object_name': 'Device'},
+            'current_job': ('django.db.models.fields.related.ForeignKey', [], {'related_name': "'+'", 'null': 'True', 'on_delete': 'models.SET_NULL', 'to': "orm['lava_scheduler_app.TestJob']", 'blank': 'True', 'unique': 'True'}),
+            'device_type': ('django.db.models.fields.related.ForeignKey', [], {'to': "orm['lava_scheduler_app.DeviceType']"}),
+            'device_version': ('django.db.models.fields.CharField', [], {'default': 'None', 'max_length': '200', 'null': 'True', 'blank': 'True'}),
+            'health_status': ('django.db.models.fields.IntegerField', [], {'default': '0'}),
+            'hostname': ('django.db.models.fields.CharField', [], {'max_length': '200', 'primary_key': 'True'}),
+            'last_health_report_job': ('django.db.models.fields.related.ForeignKey', [], {'related_name': "'+'", 'null': 'True', 'on_delete': 'models.SET_NULL', 'to': "orm['lava_scheduler_app.TestJob']", 'blank': 'True', 'unique': 'True'}),
+            'status': ('django.db.models.fields.IntegerField', [], {'default': '1'}),
+            'tags': ('django.db.models.fields.related.ManyToManyField', [], {'to': "orm['lava_scheduler_app.Tag']", 'symmetrical': 'False', 'blank': 'True'})
+        },
+        'lava_scheduler_app.devicestatetransition': {
+            'Meta': {'object_name': 'DeviceStateTransition'},
+            'created_by': ('django.db.models.fields.related.ForeignKey', [], {'to': "orm['auth.User']", 'null': 'True', 'on_delete': 'models.SET_NULL', 'blank': 'True'}),
+            'created_on': ('django.db.models.fields.DateTimeField', [], {'auto_now_add': 'True', 'blank': 'True'}),
+            'device': ('django.db.models.fields.related.ForeignKey', [], {'related_name': "'transitions'", 'to': "orm['lava_scheduler_app.Device']"}),
+            'id': ('django.db.models.fields.AutoField', [], {'primary_key': 'True'}),
+            'job': ('django.db.models.fields.related.ForeignKey', [], {'to': "orm['lava_scheduler_app.TestJob']", 'null': 'True', 'on_delete': 'models.SET_NULL', 'blank': 'True'}),
+            'message': ('django.db.models.fields.TextField', [], {'null': 'True', 'blank': 'True'}),
+            'new_state': ('django.db.models.fields.IntegerField', [], {}),
+            'old_state': ('django.db.models.fields.IntegerField', [], {})
+        },
+        'lava_scheduler_app.devicetype': {
+            'Meta': {'object_name': 'DeviceType'},
+            'display': ('django.db.models.fields.BooleanField', [], {'default': 'True'}),
+            'health_check_job': ('django.db.models.fields.TextField', [], {'default': 'None', 'null': 'True', 'blank': 'True'}),
+            'name': ('django.db.models.fields.SlugField', [], {'max_length': '50', 'primary_key': 'True'})
+        },
+        'lava_scheduler_app.jobfailuretag': {
+            'Meta': {'object_name': 'JobFailureTag'},
+            'description': ('django.db.models.fields.TextField', [], {'null': 'True', 'blank': 'True'}),
+            'id': ('django.db.models.fields.AutoField', [], {'primary_key': 'True'}),
+            'name': ('django.db.models.fields.CharField', [], {'unique': 'True', 'max_length': '256'})
+        },
+        'lava_scheduler_app.tag': {
+            'Meta': {'object_name': 'Tag'},
+            'description': ('django.db.models.fields.TextField', [], {'null': 'True', 'blank': 'True'}),
+            'id': ('django.db.models.fields.AutoField', [], {'primary_key': 'True'}),
+            'name': ('django.db.models.fields.SlugField', [], {'unique': 'True', 'max_length': '50'})
+        },
+        'lava_scheduler_app.testjob': {
+            'Meta': {'object_name': 'TestJob'},
+            '_results_bundle': ('django.db.models.fields.related.OneToOneField', [], {'null': 'True', 'db_column': "'results_bundle_id'", 'on_delete': 'models.SET_NULL', 'to': "orm['dashboard_app.Bundle']", 'blank': 'True', 'unique': 'True'}),
+            '_results_link': ('django.db.models.fields.CharField', [], {'default': 'None', 'max_length': '400', 'null': 'True', 'db_column': "'results_link'", 'blank': 'True'}),
+            'actual_device': ('django.db.models.fields.related.ForeignKey', [], {'default': 'None', 'related_name': "'+'", 'null': 'True', 'blank': 'True', 'to': "orm['lava_scheduler_app.Device']"}),
+            'definition': ('django.db.models.fields.TextField', [], {}),
+            'description': ('django.db.models.fields.CharField', [], {'default': 'None', 'max_length': '200', 'null': 'True', 'blank': 'True'}),
+            'end_time': ('django.db.models.fields.DateTimeField', [], {'null': 'True', 'blank': 'True'}),
+            'failure_comment': ('django.db.models.fields.TextField', [], {'null': 'True', 'blank': 'True'}),
+            'failure_tags': ('django.db.models.fields.related.ManyToManyField', [], {'symmetrical': 'False', 'related_name': "'failure_tags'", 'blank': 'True', 'to': "orm['lava_scheduler_app.JobFailureTag']"}),
+            'group': ('django.db.models.fields.related.ForeignKey', [], {'to': "orm['auth.Group']", 'null': 'True', 'blank': 'True'}),
+            'health_check': ('django.db.models.fields.BooleanField', [], {'default': 'False'}),
+            'id': ('django.db.models.fields.AutoField', [], {'primary_key': 'True'}),
+            'is_public': ('django.db.models.fields.BooleanField', [], {'default': 'False'}),
+            'log_file': ('django.db.models.fields.files.FileField', [], {'default': 'None', 'max_length': '100', 'null': 'True', 'blank': 'True'}),
+            'priority': ('django.db.models.fields.IntegerField', [], {'default': '50'}),
+            'requested_device': ('django.db.models.fields.related.ForeignKey', [], {'default': 'None', 'related_name': "'+'", 'null': 'True', 'blank': 'True', 'to': "orm['lava_scheduler_app.Device']"}),
+            'requested_device_type': ('django.db.models.fields.related.ForeignKey', [], {'default': 'None', 'related_name': "'+'", 'null': 'True', 'blank': 'True', 'to': "orm['lava_scheduler_app.DeviceType']"}),
+            'start_time': ('django.db.models.fields.DateTimeField', [], {'null': 'True', 'blank': 'True'}),
+            'status': ('django.db.models.fields.IntegerField', [], {'default': '0'}),
+            'submit_time': ('django.db.models.fields.DateTimeField', [], {'auto_now_add': 'True', 'blank': 'True'}),
+            'submit_token': ('django.db.models.fields.related.ForeignKey', [], {'to': "orm['linaro_django_xmlrpc.AuthToken']", 'null': 'True', 'on_delete': 'models.SET_NULL', 'blank': 'True'}),
+            'submitter': ('django.db.models.fields.related.ForeignKey', [], {'related_name': "'+'", 'to': "orm['auth.User']"}),
+            'tags': ('django.db.models.fields.related.ManyToManyField', [], {'to': "orm['lava_scheduler_app.Tag']", 'symmetrical': 'False', 'blank': 'True'}),
+            'user': ('django.db.models.fields.related.ForeignKey', [], {'to': "orm['auth.User']", 'null': 'True', 'blank': 'True'})
+        },
+        'linaro_django_xmlrpc.authtoken': {
+            'Meta': {'object_name': 'AuthToken'},
+            'created_on': ('django.db.models.fields.DateTimeField', [], {'auto_now': 'True', 'blank': 'True'}),
+            'description': ('django.db.models.fields.TextField', [], {'default': "''", 'blank': 'True'}),
+            'id': ('django.db.models.fields.AutoField', [], {'primary_key': 'True'}),
+            'last_used_on': ('django.db.models.fields.DateTimeField', [], {'null': 'True'}),
+            'secret': ('django.db.models.fields.CharField', [], {'default': "'gz3f80buhio70b6ptm90c0bly6640oiylkimx0t3okbuq5ckezltlfyiz0ndcmgd8osaqu9h9mc8224108zatlq2hs8drzq0cgbqc22ia6f4lf7bg98r0i12nhti33yj'", 'unique': 'True', 'max_length': '128'}),
+            'user': ('django.db.models.fields.related.ForeignKey', [], {'related_name': "'auth_tokens'", 'to': "orm['auth.User']"})
+        }
+    }
+
+    complete_apps = ['lava_scheduler_app']
\ No newline at end of file

=== modified file 'lava_scheduler_app/models.py'
--- lava_scheduler_app/models.py	2012-12-03 05:12:52 +0000
+++ lava_scheduler_app/models.py	2013-01-03 16:18:24 +0000
@@ -201,6 +201,18 @@
     #    return device_type.device_set.all()
 
 
+class JobFailureTag(models.Model):
+    """
+    Allows us to maintain a set of common ways jobs fail. These can then be
+    associated with a TestJob so we can do easy data mining
+    """
+    name = models.CharField(unique=True, max_length=256)
+
+    description = models.TextField(null=True, blank=True)
+
+    def __unicode__(self):
+        return self.name
+
 
 class TestJob(RestrictedResource):
     """
@@ -313,6 +325,10 @@
     log_file = models.FileField(
         upload_to='lava-logs', default=None, null=True, blank=True)
 
+    failure_tags = models.ManyToManyField(
+        JobFailureTag, blank=True, related_name='failure_tags')
+    failure_comment = models.TextField(null=True, blank=True)
+
     _results_link = models.CharField(
         max_length=400, default=None, null=True, blank=True, db_column="results_link")
 
@@ -436,8 +452,20 @@
             job.tags.add(tag)
         return job
 
+    def _can_admin(self, user):
+        """ used to check for things like if the user can cancel or annotate
+        a job failure
+        """
+        return user.is_superuser or user == self.submitter
+
+    def can_annotate(self, user):
+        """
+        Permission required for user to add failure information to a job
+        """
+        return self._can_admin(user)
+
     def can_cancel(self, user):
-        return user.is_superuser or user == self.submitter
+        return self._can_admin(user)
 
     def cancel(self):
         if self.status == TestJob.RUNNING:

=== added file 'lava_scheduler_app/templates/lava_scheduler_app/failure_report.html'
--- lava_scheduler_app/templates/lava_scheduler_app/failure_report.html	1970-01-01 00:00:00 +0000
+++ lava_scheduler_app/templates/lava_scheduler_app/failure_report.html	2013-01-03 16:18:24 +0000
@@ -0,0 +1,9 @@
+{% extends "lava_scheduler_app/_content.html" %}
+
+{% load django_tables2 %}
+
+{% block content %}
+<h2>Failure Report</h2>
+{% render_table failed_job_table %}
+
+{% endblock %}

=== added file 'lava_scheduler_app/templates/lava_scheduler_app/job_annotate_failure.html'
--- lava_scheduler_app/templates/lava_scheduler_app/job_annotate_failure.html	1970-01-01 00:00:00 +0000
+++ lava_scheduler_app/templates/lava_scheduler_app/job_annotate_failure.html	2013-01-03 16:18:24 +0000
@@ -0,0 +1,16 @@
+{% extends "lava_scheduler_app/job_sidebar.html" %}
+
+{% block content %}
+<h2>Annotate Job Failure - {{ job.id }} </h2>
+
+{% if form.errors %}
+<h3>Errors found in submission</h3>
+{{ form.errors }}
+{% endif %}
+
+<form action="{% url lava.scheduler.job.annotate_failure job.pk %}" method="post">
+{% csrf_token %}
+{{ form }}
+<input type="submit" value="Submit" />
+</form>
+{% endblock %}

=== modified file 'lava_scheduler_app/templates/lava_scheduler_app/job_sidebar.html'
--- lava_scheduler_app/templates/lava_scheduler_app/job_sidebar.html	2012-11-13 20:44:35 +0000
+++ lava_scheduler_app/templates/lava_scheduler_app/job_sidebar.html	2013-01-03 16:18:24 +0000
@@ -83,14 +83,21 @@
     {% endif %}
 
 </ul>
+{% if show_cancel or show_failure %}
+<h2>Actions</h2>
 {% if show_cancel %}
-<h2>Actions</h2>
 <form method="POST"
       action="{% url lava.scheduler.job.cancel job.pk %}">
   {% csrf_token %}
   <button id="cancel-button">Cancel Job</button>
 </form>
 {% endif %}
+{% if show_failure %}
+<ul>
+ <li><a href="{% url lava.scheduler.job.annotate_failure job.pk %}">Annotate Failure</a></li>
+</ul>
+{% endif %}
+{% endif %}
 
 {% endblock %}
 

=== modified file 'lava_scheduler_app/templates/lava_scheduler_app/reports.html'
--- lava_scheduler_app/templates/lava_scheduler_app/reports.html	2012-07-20 20:29:27 +0000
+++ lava_scheduler_app/templates/lava_scheduler_app/reports.html	2013-01-03 16:18:24 +0000
@@ -13,7 +13,7 @@
 var ddates= [];
 {% for day in health_day_report %}
   dpass.push([{{forloop.counter0}}, 100*{{day.pass}}/({{day.pass}}+{{day.fail}})]);
-  ddates.push([{{forloop.counter0}}, "{{day.date}}<br/>Pass: {{day.pass}}<br/>Fail: {{day.fail}}"]);
+  ddates.push([{{forloop.counter0}}, "<a href='{{day.failure_url}}'>{{day.date}}</a><br/>Pass: {{day.pass}}<br/>Fail: {{day.fail}}"]);
 {% endfor %}
 
   var ddata = [
@@ -46,7 +46,7 @@
 var wdates= [];
 {% for week in health_week_report %}
   wpass.push([{{forloop.counter0}}, 100*{{week.pass}}/({{week.pass}}+{{week.fail}})]);
-  wdates.push([{{forloop.counter0}}, "{{week.date}}<br/>Pass: {{week.pass}}<br/>Fail: {{week.fail}}"]);
+  wdates.push([{{forloop.counter0}}, "<a href='{{week.failure_url}}'>{{week.date}}</a><br/>Pass: {{week.pass}}<br/>Fail: {{week.fail}}"]);
 {% endfor %}
 
   var wdata = [
@@ -80,7 +80,7 @@
 var jddates= [];
 {% for day in job_day_report %}
   jdpass.push([{{forloop.counter0}}, 100*{{day.pass}}/({{day.pass}}+{{day.fail}})]);
-  jddates.push([{{forloop.counter0}}, "{{day.date}}<br/>Pass: {{day.pass}}<br/>Fail: {{day.fail}}"]);
+  jddates.push([{{forloop.counter0}}, "<a href='{{day.failure_url}}'>{{day.date}}</a><br/>Pass: {{day.pass}}<br/>Fail: {{day.fail}}"]);
 {% endfor %}
 
   var jddata = [
@@ -113,7 +113,7 @@
 var jwdates= [];
 {% for week in job_week_report %}
   jwpass.push([{{forloop.counter0}}, 100*{{week.pass}}/({{week.pass}}+{{week.fail}})]);
-  jwdates.push([{{forloop.counter0}}, "{{week.date}}<br/>Pass: {{week.pass}}<br/>Fail: {{week.fail}}"]);
+  jwdates.push([{{forloop.counter0}}, "<a href='{{week.failure_url}}'>{{week.date}}</a><br/>Pass: {{week.pass}}<br/>Fail: {{week.fail}}"]);
 {% endfor %}
 
   var jwdata = [

=== modified file 'lava_scheduler_app/urls.py'
--- lava_scheduler_app/urls.py	2012-06-16 03:04:57 +0000
+++ lava_scheduler_app/urls.py	2013-01-03 16:18:24 +0000
@@ -9,6 +9,12 @@
     url(r'^reports$',
         'reports',
         name='lava.scheduler.reports'),
+    url(r'^reports/failures$',
+        'failure_report',
+        name='lava.scheduler.failure_report'),
+    url(r'^reports/failures_json$',
+        'failed_jobs_json',
+        name='lava.scheduler.failed_jobs_json'),
     url(r'^active_jobs_json$',
         'index_active_jobs_json',
         name='lava.scheduler.active_jobs_json'),
@@ -81,6 +87,9 @@
     url(r'^job/(?P<pk>[0-9]+)/cancel$',
         'job_cancel',
         name='lava.scheduler.job.cancel'),
+    url(r'^job/(?P<pk>[0-9]+)/annotate_failure$',
+        'job_annotate_failure',
+        name='lava.scheduler.job.annotate_failure'),
     url(r'^job/(?P<pk>[0-9]+)/json$',
         'job_json',
         name='lava.scheduler.job.json'),

=== modified file 'lava_scheduler_app/views.py'
--- lava_scheduler_app/views.py	2012-11-22 03:18:31 +0000
+++ lava_scheduler_app/views.py	2013-01-03 16:18:24 +0000
@@ -6,6 +6,8 @@
 import datetime
 from dateutil.relativedelta import relativedelta
 
+from django import forms
+
 from django.conf import settings
 from django.core.exceptions import PermissionDenied
 from django.core.urlresolvers import reverse
@@ -48,6 +50,7 @@
     Device,
     DeviceType,
     DeviceStateTransition,
+    JobFailureTag,
     TestJob,
     )
 
@@ -215,10 +218,13 @@
         ).values(
             'status'
         )
+    url = reverse('lava.scheduler.failure_report')
+    params = 'start=%s&end=%s&health_check=%d' % (start_day, end_day, health_check)
     return {
         'pass': res.filter(status=TestJob.COMPLETE).count(),
         'fail': res.exclude(status=TestJob.COMPLETE).count(),
         'date': start_date.strftime('%m-%d'),
+        'failure_url': '%s?%s' % (url, params),
     }
 
 @BreadCrumb("Reports", parent=lava_index)
@@ -250,6 +256,71 @@
         },
         RequestContext(request))
 
+
+class TagsColumn(Column):
+
+    def render(self, value):
+        return ', '.join([x.name for x in value.all()])
+
+
+class FailedJobTable(JobTable):
+    failure_tags = TagsColumn()
+    failure_comment = Column()
+
+    def get_queryset(self, request):
+        failures = [TestJob.INCOMPLETE, TestJob.CANCELED, TestJob.CANCELING]
+        jobs = TestJob.objects.filter(status__in=failures)
+
+        health = request.GET.get('health_check', None)
+        if health:
+            jobs = jobs.filter(health_check=_str_to_bool(health))
+
+        dt = request.GET.get('device_type', None)
+        if dt:
+            jobs = jobs.filter(actual_device__device_type__name=dt)
+
+        device = request.GET.get('device', None)
+        if device:
+            jobs = jobs.filter(actual_device__hostname=device)
+
+        start = request.GET.get('start', None)
+        if start:
+            now = datetime.datetime.now()
+            start = now + datetime.timedelta(int(start))
+
+            end = request.GET.get('end', None)
+            if end:
+                end = now + datetime.timedelta(int(end))
+                jobs = jobs.filter(start_time__range=(start, end))
+        return jobs
+
+    class Meta:
+        exclude = ('status', 'submitter', 'end_time', 'priority', 'description')
+
+
+def failed_jobs_json(request):
+    return FailedJobTable.json(request, params=(request,))
+
+
+def _str_to_bool(str):
+    return str.lower() in ['1', 'true', 'yes']
+
+
+@BreadCrumb("Failure Report", parent=reports)
+def failure_report(request):
+    return render_to_response(
+        "lava_scheduler_app/failure_report.html",
+        {
+            'failed_job_table': FailedJobTable(
+                'failure_report',
+                reverse(failed_jobs_json),
+                params=(request,)
+            ),
+            'bread_crumb_trail': BreadCrumbTrail.leading_to(reports),
+        },
+        RequestContext(request))
+
+
 @BreadCrumb("All Devices", parent=index)
 def device_list(request):
     return render_to_response(
@@ -490,8 +561,9 @@
     data = {
         'job': job,
         'show_cancel': job.status <= TestJob.RUNNING and job.can_cancel(request.user),
+        'show_failure': job.status > TestJob.COMPLETE and job.can_annotate(request.user),
         'bread_crumb_trail': BreadCrumbTrail.leading_to(job_detail, pk=pk),
-        'show_reload_page' : job.status <= TestJob.RUNNING,
+        'show_reload_page': job.status <= TestJob.RUNNING,
     }
 
     log_file = job.log_file
@@ -656,6 +728,34 @@
             "you cannot cancel this job", content_type="text/plain")
 
 
+class FailureForm(forms.ModelForm):
+    class Meta:
+        model = TestJob
+        fields = ('failure_tags', 'failure_comment')
+
+
+def job_annotate_failure(request, pk):
+    job = get_restricted_job(request.user, pk)
+    if not job.can_annotate(request.user):
+        raise PermissionDenied()
+
+    if request.method == 'POST':
+        form = FailureForm(request.POST, instance=job)
+        if form.is_valid():
+            form.save()
+            return redirect(job)
+    else:
+        form = FailureForm(instance=job)
+
+    return render_to_response(
+        "lava_scheduler_app/job_annotate_failure.html",
+        {
+            'form': form,
+            'job': job,
+        },
+        RequestContext(request))
+
+
 def job_json(request, pk):
     job = get_restricted_job(request.user, pk)
     json_text = simplejson.dumps({