Ubuntu
webbrowser-app package

Intermittent unit test failure with Qt 5.6 in yakkety

Bug #1611403 reported by Olivier Tilloy
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
webbrowser-app (Ubuntu)
Fix Released
Critical
Olivier Tilloy

Bug Description

I’m seeing tst_FaviconFetcherTests intermittently fail in CI runs on yakkety. Those test runs are carried out in sbuild environments with -proposed enabled, where Qt 5.6.1 recently appeared.

I’m able to reproduce the failure locally in a yakkety chroot:

FAIL! : FaviconFetcherTests::shouldCancelRequests() Compared values are not the same
   Actual (serverSpy->count() + errorSpy->count()): 101
   Expected (requests) : 100
   Loc: [/tmp/webbrowser-app/tests/unittests/favicon-fetcher/tst_FaviconFetcherTests.cpp(297)]

I’m unable to reproduce in the same chroot with Qt 5.5.1.

Related branches

lp:~osomon/webbrowser-app/relax-test-expectation-1611403
system-apps-ci-bot: Needs Fixing (continuous-integration)
Ubuntu Phablet Team: Pending requested
Diff: 12 lines (+1/-1)
1 file modified
tests/unittests/favicon-fetcher/tst_FaviconFetcherTests.cpp (+1/-1)
Olivier Tilloy (osomon)
Changed in webbrowser-app (Ubuntu):
status: New → In Progress
importance: Undecided → Critical
assignee: nobody → Olivier Tilloy (osomon)
Revision history for this message
Olivier Tilloy (osomon) wrote :

I’ve been running that one failing test in a loop and most of the failures exhibit an unexpected value of 101, but occasionally some fail with a greater value (observed values of 102, 103, and even 113 and 124).

In all those cases, it appears the value of serverSpy->count() is always 100, and the value of errorSpy->count() is ≥ 0.

I suspect a behaviour change in Qt. We now have to support three different versions:
 - 5.4.1 in vivid+overlay
 - 5.5.1 in xenial+overlay
 - 5.6.1 in yakkety

Given these constraints, it looks like the test expectations will need to be made laxer.

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package webbrowser-app - 0.23+16.10.20160825-0ubuntu1

---------------
webbrowser-app (0.23+16.10.20160825-0ubuntu1) yakkety; urgency=medium

  [ Florian Boucault ]
  * Load images needed at startup asynchronously.

  [ Michael Terry ]
  * Use less hard-coded colors in favor of theme colors. (LP: #1581047)

  [ Olivier Tilloy ]
  * Relax one unit test’s expectations to cater for different behaviours
    in different Qt versions (LP: #1611403)
  * Revert the workaround for bug #1585942 now that the bug has been
    properly fixed. (LP: #1585942)
  * Skip webapp container autopilot tests that exercise keyboard
    shortcuts when not run on desktop. (LP: #1611832)
  * Add missing string to translation template. (LP: #1607713)
  * Fix a handful of harmless warnings in QML unit tests.
  * Add user agent overrides for Google recaptcha. (LP: #1599146)

 -- Olivier Tilloy <email address hidden> Thu, 25 Aug 2016 09:53:50 +0000

Changed in webbrowser-app (Ubuntu):
status: In Progress → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.