Comment 3 for bug 1284316

I spoke with Bill briefly about the dialer app failures, and what I wrote on the terminal app bug applies here as well.

https://bugs.launchpad.net/ubuntu-terminal-app/+bug/1284594/comments/4

The TLDR, is tests need to be improved to fail well; that is it should be obvious what failed when a failure occurs.

A statenotfound error from Autopilot simply means the object in question didn't exist when you went to look for it. They often pop-up because of timing issues. Likely the test failed before this point and you didn't realize it. For instance a view didn't appear (app issue or platform /qmlscene issue), a click didn't register (possible app bug or platform bug, or potentially autopilot bug if click events aren't working), a UI element wasn't found (app bug, test bug, or platform bug), or a UI element wasn't able to be introspected (autopilot bug).

In the scenarios I mentioned above the issue is not not usually autopilot. And indeed I would venture the bugs we see are app, platform, or testcase bugs. Most certainly all of them have some sort of testcase bug as the test is likely not failing well as mentioned above, leaving this ambiguity on what went wrong and what the issue is.

So these issues are 1) Almost certainly not autopilot bugs, 2) Definitely are testcase bugs to the extent the tests aren't failing well, 3) Potentially the application, the test, or platform bugs.