I can't tell how to run the Media card tests

Bug #970857 reported by Glen Ditchfield
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Checkbox
Fix Released
High
Jeff Lane 
checkbox (Ubuntu)
Fix Released
High
Jeff Lane 

Bug Description

The media card tests are unclear, and I don't know what I am supposed to do.

The test description says '(1) Click "Test" and insert an SD card into the reader ... (?) The verification of this test is automated. Do not change the automated selected result."

When I click "Test" the "in progress" widget appears and begins sliding left and right. When I insert an SD card, the widget disappears and a directory browser opens to display the card's contents, and nothing else happens on screen.

Excellent. Now what? Is the test over? Is this the expected result? Was Checkbox supposed to automatically advance to the next test? Am I supposed to click "Yes" or "No", and what do those buttons mean in this test? It would help if the "question" at the end of the description was actually a yes/no question.

Step 2 is "Do not unplug the device after the test." I did not plug in a device, and was not asked to. Did you mean "Do not eject the card after the test"? Probably not, because I have to eject the SD card to perform the SDHC test. But when is it safe to eject the card?

ProblemType: Bug
DistroRelease: Ubuntu 12.04
Package: checkbox 0.13.5
ProcVersionSignature: Ubuntu 3.2.0-20.33-generic-pae 3.2.12
Uname: Linux 3.2.0-20-generic-pae i686
ApportVersion: 1.95-0ubuntu1
Architecture: i386
Date: Sun Apr 1 09:27:52 2012
InstallationMedia: Ubuntu 12.04 LTS "Precise Pangolin" - Beta i386 (20120328)
ProcEnviron:
 LANGUAGE=en_CA:en
 TERM=xterm
 LANG=en_CA.UTF-8
 SHELL=/bin/bash
SourcePackage: checkbox
UpgradeStatus: No upgrade log present (probably fresh install)

Related branches

Revision history for this message
Glen Ditchfield (gjditchfield) wrote :
Revision history for this message
Jeff Lane  (bladernr) wrote :

Hi,

Thank you for taking the time to report this. Based on your description, and my memory of the test text as it stands, the tests could use a little revision. I think part of the confusion is the window that opens (sometimes) when the SD card is mounted.

And the use of the term "unplug" later on, which could be better stated as "do not REMOVE the card" rather than "do not UNPLUG the card"

In any case, I'll confirm this and get to work on it in the morning. Thanks again for reporting this bug.

Changed in checkbox (Ubuntu):
status: New → Confirmed
importance: Undecided → High
assignee: nobody → Jeff Lane (bladernr)
Revision history for this message
Daniel Manrique (roadmr) wrote :

Marking as Triaged, as we know what needs to be done.

Also, the "do not remove the card" in the insert test message is correct; if the storage-transfer test for the card passes, then the next test is a remove test that will ask you to remove the card, thus leaving the socket empty for the SDHC test. If either the insert or storage-transfer tests fail for the SD card, there *will* be an SD card when the user goes to try to insert an SDHC card in the socket. Should we maybe update the test description to add a first step saying "remove any existing cards from the card reader, then press test and insert a SD/SDHC card" ?

Changed in checkbox (Ubuntu):
status: Confirmed → Triaged
Revision history for this message
Jeff Lane  (bladernr) wrote :

No... Not IMO at least... what REALLY needs to happen is for us to reliably order tests and not have to rely on brittle dependency satisfaction.

For example, with all the removable media tests we follow the following:

Insert DEVICE
DEVICE Storage (depends on Insert)
Remove DEVICE (depends on DEVICE storage)

DEVICE Storage rightly depends on Insert because you can't run the storage tests if the insert test fails.
Remove DEVICE, however, should STILL run even IF Remove DEVICE fails.

IN all honesty, I'd rather see niggling things like this fixed properly than with bandaid workarounds like changing description text or cheating with depends: or doing what OEM used to do by alphabetizing the test names (like: 0_testA, 1_testB, 2_testC)

Fixing the problem with ordering jobs would fix this and so many other issues... and yeah, I know it's certainly NOT a trivial fix, unfortunately.

I would love it if we could do something like these:

1: something similar to Try, Except, Finally... like Try: Insert, Then: Storage, Finally: Remove with Finally: acting like the cleanup test that runs regardless of whether Storage passes or fails.

2: Confirmed that whitelist ordering really does work. Removing the depends from the multimedia card and USB tests would be a great way to test this (and perhaps the fixes to whitelist ordering actually have resolved this problem already, in which case we just need to go through the jobs and remove the unnecessary dependencies.

3: A way to arbitrarily create strings of jobs via one metajob, perhaps like this:

plugin: SOME PLUGIN
name: SDHC_Test
depends: device.type == 'SDHC' (or whatever)
jobs: multimedia/sdhc_insert, multimedia/sdhc_storage, multimedia/sdhc_remove
description: Compound job that will run all three MMC tests against an SDHC card

Anyway, those are just my thoughts. The main point is that I'd really rather see this fixed overall than just patched up because I think fixing the underlying issue in the test process here will fix a lot of similar issues/complexities in other jobs as well.

Revision history for this message
Jeff Lane  (bladernr) wrote :

OK, part of this was resolved more cleanly by fixing job ordering a while back. I've just pushed a branch to take care of the actual test instructions now that should address the rest of this.

Changed in checkbox (Ubuntu):
status: Triaged → In Progress
status: In Progress → Fix Committed
Jeff Lane  (bladernr)
Changed in checkbox:
status: New → Fix Committed
importance: Undecided → High
assignee: nobody → Jeff Lane (bladernr)
Revision history for this message
Launchpad Janitor (janitor) wrote :
Download full text (7.8 KiB)

This bug was fixed in the package checkbox - 0.16

---------------
checkbox (0.16) saucy; urgency=low

  * New upstream release (LP: #1178403):

  * Launchpad automated translation updates

  [ Jeff Lane ]
  * scripts/cpu_offlining: Modified script to no longer offline cpu0 to resolve
    a bug on ARM. Modified output so most of it is redirected to stderr for
    fail cases, we don't need that much for success cases. (LP: #1078897)
  * jobs/mediacard.txt.in: Modified test instructions to be less confusing
    (LP: #970857)
  * scripts/cpu_topology: define the cpuinfo nested dicts on creation rather
    than define elements during parsing of /proc/cpuinfo (LP: #1111878)
  * scripts/lsmod_info: Corrected error handling for the check_output() call to
    trap the correct error. (LP: #1103647)
  * jobs/camera.txt.in: removed an extraneous requres line for gir1.2
    scripts/camera_test: added code to determine what version of gst we're
    using and set video type and plugin accordingly. (LP: #1100594)
  * scripts/network_check: added ability to specify custom target URL for
    debugging failures (LP: #1128017)
  * scripts/sleep_test_log_check: added new script to parse fwts logs for
    errors
    jobs/hibernate.txt.in, jobs/stress.txt.in, jobs/suspend.txt.in: added jobs
    to use the sleep_test_log_check script after s3/s4 tests. Modified current
    s3/s4 tests to use the new None fail level for fwts_test
    scripts/fwts_test: no longer fails on fwts errors if "-f none" is chosen.
    Now it should only fail by manual indication if the actual sleep action
    fails. (LP: #1169922)
  * scripts/bluetooth_test: test no longer generates odd Broken Pipe error when
    the dpkg bit verifies obexftp is installed (LP: #1169488)
  * jobs/networking.txt.in: fixed networking/info jobs that were not being run
    because $output was being resolved too early (LP: #1065983)
  * scripts/memory_compare: cleaned up the output to be more explanatory and
    easier to visually parse (LP: #1072666)
  * jobs/miscellanea.txt.in: Added jobs for manual verification of PXE boot and
    remote IPMI to improve server test coverage.
  * jobs/led.txt.in: Modified Jobs: led/power, led/wlan, led/wlan-disabled. New
    jobs: led/power-blink-suspend, led/suspend, led/mute
    jobs/keys.txt.in: modified keys/wireless
  * plugins/lock_prompt.py: added a lock release to cleanly clear lock on stop.
    plugins/persist_prompt.py: promoted save to run before lock release.
    checkbox/lib/fifo.py: trap OSError exception at close when the input/output
    fifo fds disappear before fifo.close() can get to them. (LP: #115561)
  * jobs/info.txt.in: fixed bad driver name in audio_codecs job (LP: #1165215)
  * jobs/mediacard.txt.in: reduce test file size for MMC to 64MB (LP: #1167214)
  * jobs/led.txt.in, jobs/suspend.txt.in: removed bits about WLAN LED flashing
    from the WLAN jobs as that no longer happens on data tx/rx (LP: #1157606)
  * jobs/power-management.txt.in: added manual reboot and shutdown tests for
    server and Xen testing. Took the opportunity to correct punctuation in the
    descriptions or other manual jobs.
    jobs/virtualization.txt.in: added virtualization/x...

Read more...

Changed in checkbox (Ubuntu):
status: Fix Committed → Fix Released
Changed in checkbox:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.