Error rate calculations inappropriately include Autopilot/LRT errors
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Daisy |
Fix Released
|
High
|
Brian Murray | ||
lrt |
Fix Released
|
Undecided
|
Unassigned |
Bug Description
When Autopilot or LRT triggers a crash, the error is submitted to errors.ubuntu.com just as it would be if the error was experienced by a machine in normal use.
The error is then, as usual, included in the error rate calculations and the occurrence counts. But both of these are problems.
The calculated error rate is naturally understood (or would be, if the axis was labelled) as the number of errors per day experienced by a machine in normal use. But the point of automated tests is that they encounter errors much more quickly than normal use will. This is an example of Campbell's Law: reducing the error rate is a good thing, but we could reduce the measured error rate by not running the automated tests any more, which would be a bad thing.
Including fuzzer errors in the occurrence counts is also a bad thing, because it may lead to poor prioritization of fixes. For example, imagine there was a crash whenever you switched apps less than half a second after revealing the Launcher. Humans would seldom encounter errors like this, but a fuzzer often would, and would submit it many times. So it would rank highly in the occurrences table, misleading developers into thinking that it was more important than errors humans are encountering more often.
If the automated error reports are a drop in the bucket, neither of these things matter, so this bug can be marked Won't Fix.
Otherwise, either Autopilot and LRT should override Apport's usual behavior, and report bugs directly to Launchpad rather than submitting errors; or Errors should ignore Autopilot/LRT reports when calculating error rates and counting occurrences. The latter would be more complicated, but would have the advantage that the automated tools might sometimes provide the only evidence that a bug remains unfixed in a new package version.
description: | updated |
summary: |
- Error rate calculations inappropriately include fuzzer errors + Error rate calculations inappropriately include Autopilot/LRT errors |
no longer affects: | autopilot |
affects: | errors → daisy |
Changed in daisy: | |
status: | New → In Progress |
assignee: | nobody → Brian Murray (brian-murray) |
importance: | Undecided → High |
Changed in daisy: | |
status: | In Progress → Fix Released |
Brian points out that the same problem affects Autopilot. It's reporting errors to errors.ubuntu.com that inflate the measured error rate.