multiple StateNotFoundError failures in image 206

I tried to revert AP and the toolkit, to previous versions known to create that issue, The StateNotFoundError still happens though. This isn't dialer-app specific, weather-app and a lot of others are impacted as well

One example (100% reproducible on runnning the whole weather-app testsuite on mako, having between 3 and 5 failures)

Traceback (most recent call last):
File "/home/phablet/autopilot/ubuntu_weather_app/tests/test_settings.py", line 193, in test_switch_scale_cancel
self._check_units('metric')
File "/home/phablet/autopilot/ubuntu_weather_app/tests/test_settings.py", line 50, in _check_units
'QQuickItem', focus=True)
File "/usr/lib/python2.7/dist-packages/autopilot/introspection/dbus.py", line 322, in select_single
raise StateNotFoundError(type_name, **kwargs)
StateNotFoundError: State not found for class 'QQuickItem' and filters {'focus': True}.

I spoke with Bill briefly about the dialer app failures, and what I wrote on the terminal app bug applies here as well.

https://bugs.launchpad.net/ubuntu-terminal-app/+bug/1284594/comments/4

The TLDR, is tests need to be improved to fail well; that is it should be obvious what failed when a failure occurs.

A statenotfound error from Autopilot simply means the object in question didn't exist when you went to look for it. They often pop-up because of timing issues. Likely the test failed before this point and you didn't realize it. For instance a view didn't appear (app issue or platform /qmlscene issue), a click didn't register (possible app bug or platform bug, or potentially autopilot bug if click events aren't working), a UI element wasn't found (app bug, test bug, or platform bug), or a UI element wasn't able to be introspected (autopilot bug).

In the scenarios I mentioned above the issue is not not usually autopilot. And indeed I would venture the bugs we see are app, platform, or testcase bugs. Most certainly all of them have some sort of testcase bug as the test is likely not failing well as mentioned above, leaving this ambiguity on what went wrong and what the issue is.

So these issues are 1) Almost certainly not autopilot bugs, 2) Definitely are testcase bugs to the extent the tests aren't failing well, 3) Potentially the application, the test, or platform bugs.

The QA team spoke about this at length. The take-home points are:

1) This is almost certainly a bug in the test cases, rather than in autopilot itself. The fact that the issue exists independant of the autopilot version seems to confirm this. Unless anyone objects, I'm going to remove the bug task against autopilot, and add one for the actual applications that are failing.

2) The root cause here is that the tests are written poorly. Specifically, they don't fail well. Many tests are writtenn such that they have a bunch of actions, and then assertions at the end of the test case. This is best practise for *unit* tests, but does not work well in an acceptance test where the test code and applicaiton code are completely asynchronous of each other.

3) Leo is slowly converting existing acceptance test suites to the page object pattern (http://code.google.com/p/selenium/wiki/PageObjects) which allows a greater level of abstraction, and should allow better failure messages. This process will likely help situations like this in the future.

4) balloons is going to blog about the page object conversion process, so hopefully it won't be just Leo doing the conversion, which should speed things up a bit.

I ran all the dialer app tests on my mako, rev 207 and got no failures. Also, it seems to be working nicely here:
http://ci.ubuntu.com/smokeng/trusty/touch/mako/207:20140225:20140224/6813/dialer_app/

Marking as incomplete there, waiting for more information. Bill or Didier, are you still seeing the errors on the dialer?

Hey Leo, as told, the errors are flaky, so please don't take one local run as the source of truth. See as well my comments:
"One example (100% reproducible on runnning the whole weather-app testsuite on mako, having between 3 and 5 failures)". Did you try on weather-app?

You can look as well as the errors on http://ci.ubuntu.com/smokeng/trusty/touch/.

So, if I take last image:
http://ci.ubuntu.com/smokeng/trusty/touch/mako/208:20140226:20140224/6828/

I can see the state no found on:
http://ci.ubuntu.com/smokeng/trusty/touch/mako/208:20140226:20140224/6828/ubuntu_weather_app/817461/ Only there.

On image 207 (which has NO change from 208):
clock_app:
http://ci.ubuntu.com/smokeng/trusty/touch/mako/207:20140225:20140224/6813/ubuntu_clock_app/814911/

on image 206 (which has NO change for the sdk, or autopilot or on the application itself):
dialer_app:
http://ci.ubuntu.com/smokeng/trusty/touch/mako/206:20140224:20140224/6796/dialer_app/813408/
http://ci.ubuntu.com/smokeng/trusty/touch/mako/206:20140224:20140224/6796/dialer_app/813409/
http://ci.ubuntu.com/smokeng/trusty/touch/mako/206:20140224:20140224/6796/dialer_app/813412/
http://ci.ubuntu.com/smokeng/trusty/touch/mako/206:20140224:20140224/6796/dialer_app/813414/

weather_app:
http://ci.ubuntu.com/smokeng/trusty/touch/mako/206:20140224:20140224/6796/ubuntu_weather_app/813047/

So, that would mean that all those tests are not well written? Can you have a look at those?

Regarding the failing weather app tests: the flaky tests are caused by not considering the newly added convergent layout. But a MP is already in the making and should (hopefully) land today.

Just FYI, Omer and Chris Gagnon are actively investigating this.

I was not able to reproduce locally, I notice that the system settle tests are also failing.

https://jenkins.qa.ubuntu.com/job/trusty-touch-mako-smoke-daily/86/testReport/(root)/dialer_app/settle_before/
https://jenkins.qa.ubuntu.com/job/trusty-touch-mako-smoke-daily/81/testReport/(root)/ubuntuuitoolkit/settle_before/

Omer found running the dialer_app.tests.test_calls.TestCalls.test_outgoing_answer_remote_hangup test repeatedly in a loop would reproduce the failure.

It currently looks like the phone being under load is exposing flaky tests. I am working to better reproduce this now. The dialer-app definitely has a poorly written test.

I have seen tests fail when under load -- clicks have a tendency to be missed somewhere (app, platform?) when under load.

Perhaps the bug I just filed against sudoku is a dupe of this? bug 1285279 - easily reproducible with r161 of sudoku.

I have this MP that fixes a number of flaky tests in the dialer app https://code.launchpad.net/~chris.gagnon/dialer-app/wait-for-historydelegate0/+merge/208449

This bug was fixed in the package dialer-app - 0.1+14.04.20140226-0ubuntu1

---------------
dialer-app (0.1+14.04.20140226-0ubuntu1) trusty; urgency=low

  [ chris.gagnon ]
  * Use wait_select_single when getting HistoryDelegate0 instead of
    select_single, to fix a couple flaky tests (LP: #1284316)
 -- Ubuntu daily release <email address hidden> Wed, 26 Feb 2014 21:27:58 +0000

@Chris: the system settle test failures are false positive that we discussed at length on the phone mailing list. To be clear, the phone isn't under heavy load.

Fix committed into lp:ubuntu-weather-app at revision 210, scheduled for release in ubuntu-weather-app, milestone alpha-1