Merge lp:~sylvain-pineau/checkbox/usb_version_check into lp:checkbox

[Sylvain Pineau]

This MR modifies the existing block_device_resource job to add usb3 compliance information.
A new read performance test has been added to usb.txt.in specific for usb3 devices.

For relevant results, USB3 devices must be plugged in before checkbox starts.

[Daniel Manrique (roadmr)]

I tested this by merging and building a test checkbox package, adding the tests to whitelists, and doing a checkbox run with a USB 3.0 device inserted.

The test passed and the device (a hard disk) returned a throughput of 84 MB/s, just above the required threshold of 80.

I even looked at the very nice checkbox report with this information.

I have the following comments/requests for information:

1- On the test selection screen, the "Verify USB3 external storage" tests created by the usb/usb3_read_performance job don't appear below the usb category, which looks ugly :( can this be fixed somehow?

2- The test ran on /dev/sda (internal harddisk), even though it had block_device.state=='internal' and block_device.usb3=='unsupported'. I think this relates to bug 1027849, which is being worked on, so we may end up having to wait for that to get fixed.

3- Is it possible to determine whether the motherboard/controller supports USB3 without plugging in a USB3 device? This test will mainly be used for certification, but if we need to plug in a USB3 device into systems before the tests start, this means we can only test as many systems as we have USB3 devices. Compare this with how the existing usb tests work: if the system supports usb2, then you get prompted to insert a compatible device to test, if you have no device you can skip the test.

One possible way of detecting this is looking for xhci initialization in the kernel log:

[ 1.150592] calling xhci_hcd_init+0x0/0x29 @ 1
[ 1.150608] xhci_hcd 0000:06:00.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
[ 1.150622] xhci_hcd 0000:06:00.0: setting latency timer to 64
[ 1.150626] xhci_hcd 0000:06:00.0: xHCI Host Controller
[ 1.150658] xhci_hcd 0000:06:00.0: new USB bus registered, assigned bus number 3
[ 1.150818] xhci_hcd 0000:06:00.0: irq 16, io mem 0xca100000
[ 1.150878] xhci_hcd 0000:06:00.0: irq 42 for MSI/MSI-X
[ 1.150882] xhci_hcd 0000:06:00.0: irq 43 for MSI/MSI-X
[ 1.150885] xhci_hcd 0000:06:00.0: irq 44 for MSI/MSI-X
[ 1.150889] xhci_hcd 0000:06:00.0: irq 45 for MSI/MSI-X
[ 1.150892] xhci_hcd 0000:06:00.0: irq 46 for MSI/MSI-X
[ 1.151030] xHCI xhci_add_endpoint called for root hub
[ 1.151032] xHCI xhci_check_bandwidth called for root hub
[ 1.151101] xhci_hcd 0000:06:00.0: xHCI Host Controller
[ 1.151130] xhci_hcd 0000:06:00.0: new USB bus registered, assigned bus number 4
[ 1.154277] xHCI xhci_add_endpoint called for root hub
[ 1.154279] xHCI xhci_check_bandwidth called for root hub
[ 1.198284] initcall xhci_hcd_init+0x0/0x29 returned 0 after 46671 usecs

Compare to what happens on a non-USB3.0 system:
[ 0.577014] calling xhci_hcd_init+0x0/0x49 @ 1
[ 0.577031] initcall xhci_hcd_init+0x0/0x49 returned 0 after 13 usecs

review: Needs Information

[Daniel Manrique (roadmr)]

On a system with USB 3.0 I went to /sys/bus/usb/devices. There's a device for each of the USB controllers (4 in all). The USB 3.0 controller seems to be distinguished by:

cat bDeviceClass
09 (09 means Hub)
cat speed
5000 (as opposed to 480 for USB 2.0 controllers).
cat version
3.0
cat product
xHCI Host Controller (as opposed to EHCI Host Controller or even OHCI for USB 1.0).

I think this could be used to determine whether a system supports USB 3.0 even if an actual device is not currently plugged in.

Unfortunately not a lot of this information seems present in udevadm output :(

[Daniel Manrique (roadmr)]

OK, I've been looking into this, and I think spineau's research is quite sound; the problem is that we can't know if a specific disk device supports usb3 if it's not inserted beforehand. This is needed because we're running the disk_performance test, not the usb_storage_transfer test like we do for manual USB testing. So even though the user experience may not be ideal, this seems to be the only way to know which disks to run the tests against... :(

[Chris Gagnon (chris.gagnon)]

I've done some usb3 tests in the sutton project

You can look at lsusb and grep for 'Linux Foundation 3.0 root hub' or 1d6b:0003 to see if usb 3.0 is supported, if the device is on the same bus you can tell if it's being seen as a USB 3.0 device or a USB 2.0 Device.

Example

$lsusb
<stuff here>
Bus 006 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
<more stuff here>
Bus 006 Device 002: ID <idof>:<usbkey>

if the device is on bus 006 it's a usb 3.0 device on the 3.0 root hub.

If it's a 3.0 device on a 2.0 port it will show up as on the same bus as a 1d6b:0002 Linux Foundation 2.0 root hub...

[Daniel Manrique (roadmr)]

OK, so the issue here is not USB3 device detection, but test creation for each connected drive. That's where the need to have the device plugged in beforehand comes from. Based on that, I have no objections and I'll merge this now. Thanks!

review: Approve

=== modified file 'debian/changelog'
--- debian/changelog	2012-07-19 15:32:45 +0000
+++ debian/changelog	2012-07-20 07:45:25 +0000
@@ -2,6 +2,12 @@
 
   [Daniel Manrique]
   * New version 0.14.3 for Quantal Quetzal development.
+  
+    [Sylvain Pineau]
+  * jobs/usb.txt.in, scripts/disk_read_performance_test: Add a USB3 read
+    performance test.
+    scripts/block_device_resource: Add the maximum usb specification supported
+    by both a block device and the corresponding root hub port.
 
   [Jeff Marcom]
   * Added timeout to job call for disk smart test. 

=== modified file 'jobs/usb.txt.in'
--- jobs/usb.txt.in	2012-07-11 19:01:24 +0000
+++ jobs/usb.txt.in	2012-07-20 07:45:25 +0000
@@ -140,3 +140,21 @@
      5. Repeat with each external USB slot.
  VERIFICATION:
      Do all USB slots work with the device?
+
+plugin: local
+name: usb/usb3_read_performance
+requires:
+ device.category == 'DISK'
+_description: Verify USB3 external storage performs at or above baseline performance
+command:
+ cat <<'EOF' | run_templates -t -s 'udev_resource | filter_templates -w "category=DISK"'
+ plugin: shell
+ name: usb/usb3_read_performance_`ls /sys$path/block`
+ requires:
+  device.path == "$path"
+  block_device.name == "`ls /sys$path/block`" and block_device.state == 'removable' and block_device.usb3 == 'supported'
+  package.name == 'linux'
+ description: USB3 read performance test for $product
+ user: root
+ command: disk_read_performance_test `ls /sys$path/block | sed 's|!|/|'`
+ EOF

=== modified file 'scripts/block_device_resource'
--- scripts/block_device_resource	2012-06-29 19:47:58 +0000
+++ scripts/block_device_resource	2012-07-20 07:45:25 +0000
@@ -4,6 +4,7 @@
 import re
 from glob import glob
 
+rootdir_pattern = re.compile('^.*?/devices')
 
 def device_state(name):
     """
@@ -13,7 +14,6 @@
         if f.read(1) == '1':
             return 'removable'
 
-    rootdir_pattern = re.compile('^.*?/devices')
     path = rootdir_pattern.sub('', os.readlink('/sys/block/%s' % name))
     hotplug_buses = ("usb", "ieee1394", "mmc", "pcmcia", "firewire")
     for bus in hotplug_buses:
@@ -27,10 +27,41 @@
     return 'internal'
 
 
+def usb_support(name, version):
+    """
+    Check the USB specification number for both hub port and device
+    """
+    path = rootdir_pattern.sub('', os.readlink('/sys/block/%s' % name))
+
+    # Remove the usb config.interface part of the path
+    m = re.match('((.*usb\d+).*\/)\d-[\d\.:\-]+\/.*', path)
+    if m:
+        device_path = m.group(1)
+        hub_port_path = m.group(2)
+
+        # Check the highest version of USB the device supports
+        with open('/sys/devices/%s/version' %device_path) as f:
+            if float(f.readline()) < version:
+                return 'unsupported'
+
+        # Check the highest version of USB the hub supports
+        with open('/sys/devices/%s/version' %hub_port_path) as f:
+            if float(f.readline()) < version:
+                return 'unsupported'
+
+        return 'supported'
+
+    return 'unsupported'
+
+
 for path in glob('/sys/block/*/device'):
     name = re.sub('.*/(.*?)/device', '\g<1>', path)
     state = device_state(name)
+    usb2 = usb_support(name, 2.00)
+    usb3 = usb_support(name, 3.00)
     print("""\
 name: %s
 state: %s
-""" % (name, state))
+usb2: %s
+usb3: %s
+""" % (name, state, usb2, usb3))

=== modified file 'scripts/disk_read_performance_test'
--- scripts/disk_read_performance_test	2012-07-11 02:54:40 +0000
+++ scripts/disk_read_performance_test	2012-07-20 07:45:25 +0000
@@ -12,10 +12,23 @@
   echo "---------------------------------------------------"
 
   disk_type=`udevadm info --name /dev/$disk --query property | grep "ID_BUS" | awk '{gsub(/ID_BUS=/," ")}{printf $1}'`
+  dev_path=`udevadm info --name /dev/$disk --query property | grep "DEVPATH" | awk '{gsub(/DEVPATH=/," ")}{printf $1}'`
   echo "INFO: $disk type is $disk_type"
 
   case $disk_type in
-    "usb" ) MIN_BUF_READ=7;; #Custom metrics are guesstimates for now...
+    "usb" ) 
+            #Custom metrics are guesstimates for now...
+            MIN_BUF_READ=7
+            
+            # Increase MIN_BUF_READ if a USB3 device is plugged in a USB3 hub port
+            if  [[ $dev_path =~ ((.*usb[0-9]+).*\/)[0-9]-[0-9\.:\-]+\/.* ]]; then
+                device_version=`cat '/sys/'${BASH_REMATCH[1]}'/version'`
+                hub_port_version=`cat '/sys/'${BASH_REMATCH[2]}'/version'`
+                if [ $(echo "$device_version >= 3.00"|bc -l) -eq 1 -a $(echo "$hub_port_version >= 3.00"|bc -l) -eq 1 ]; then
+                    MIN_BUF_READ=80
+                fi
+            fi
+            ;;
     "ide" ) MIN_BUF_READ=40;;
     *     ) MIN_BUF_READ=$DEFAULT_BUF_READ;;
   esac