Code review comment for lp:~pfalcon/linaro-ci-dashboard/kernel_chain

Milo:

Here's my story: I didn't do work on ci-dashboard for a while, so when I restarted with work on these tested, I was greeted with the need to upgrade jenkins.war. I did, and then on first "python dashboard/manage.py test" run I got errors similar to those you quoted, except that in my case there were more errors "JenkinsException: create[testjob_integration] failed" and no "JenkinsException: build[2] does not exist for job test". I almost prepared to scratch my head and blame spurious jenkins upgrades that we have so far, but then I decided just to rerun the testsuite. Those errors were gone. There was another one though:

====
FAIL: test_login_teams (django_openid_auth.tests.test_views.RelyingPartyTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/pymodules/python2.7/django_openid_auth/tests/test_views.py", line 359, in test_login_teams
    self.assertTrue(group in user.groups.all())
AssertionError: False is not true
====

But that's known by all of us (?) heizenbug, except that it has much lower oscillation frequency (i.e. I have it stable, including with the trunk now even with several reruns, tomorrow it may be gone, day after appear again).

So, answering your question: it's "normal" (in quotes), in the sense that it's matter of fact that it does happen for more than one developer. It's not normal in other senses of this word, and we should take measures to investigate and fix those non-deterministic failures. First measure would be to use well-known jenkins version, instead of non-deterministic one - I proposed that some time ago, but didn't make that change at that time, so now it seems time has come.