Merge lp:~rodsmith/checkbox/smart-for-dmraid into lp:checkbox

[Rod Smith]

Add support for several hardware RAID types to SMART test. The changes should not affect tests on non-RAID systems; but on supported RAID systems, the test will now run and pass if all disks on a Linux device (/dev/sda, etc.) pass. If any one RAID disk fails, the entire device will fail. See https://certification.canonical.com/hardware/200905-2781/submission/107752/test/148/result/7917277/ for an example output on a RAID disk.

[Jeff Lane  (bladernr)]

Thanks

review: Approve

[Sylvain Pineau (sylvain-pineau)]

The attempt to merge lp:~rodsmith/checkbox/smart-for-dmraid into lp:checkbox failed. Below is the output from the failed tests.

[precise] starting container
[precise] Unable to start ephemeral container!
[precise] stdout:
[precise] stderr: http://paste.ubuntu.com/14505819/
[precise] NOTE: unable to execute tests, marked as failed
[precise] Destroying failed container to reclaim resources
[trusty] starting container
[trusty] (timing) 0.07user 0.08system 0:05.32elapsed 2%CPU (0avgtext+0avgdata 10248maxresident)k
[trusty] (timing) 0inputs+32outputs (0major+8692minor)pagefaults 0swaps
[trusty] provisioning container
[trusty] (timing) 53.95user 15.48system 1:18.15elapsed 88%CPU (0avgtext+0avgdata 61780maxresident)k
[trusty] (timing) 0inputs+18672outputs (0major+2396549minor)pagefaults 0swaps
[trusty-testing] Starting tests...
Found a test script: ./checkbox-ng/requirements/container-tests-checkbox-ng-unit
[trusty-testing] container-tests-checkbox-ng-unit: PASS
[trusty-testing] (timing) 0.75user 0.15system 0:00.92elapsed 98%CPU (0avgtext+0avgdata 31932maxresident)k
[trusty-testing] (timing) 0inputs+1088outputs (0major+23160minor)pagefaults 0swaps
Found a test script: ./checkbox-support/requirements/container-tests-checkbox-support
[trusty-testing] container-tests-checkbox-support: PASS
[trusty-testing] (timing) 25.20user 0.18system 0:25.44elapsed 99%CPU (0avgtext+0avgdata 106572maxresident)k
[trusty-testing] (timing) 0inputs+1072outputs (0major+38377minor)pagefaults 0swaps
Found a test script: ./checkbox-touch/requirements/container-tests-touch-unit-tests
[trusty-testing] container-tests-touch-unit-tests: PASS
[trusty-testing] (timing) 0.01user 0.01system 0:00.03elapsed 78%CPU (0avgtext+0avgdata 2180maxresident)k
[trusty-testing] (timing) 0inputs+0outputs (0major+2384minor)pagefaults 0swaps
Found a test script: ./plainbox/plainbox/impl/providers/categories/requirements/container-tests-provider-categories
[trusty-testing] container-tests-provider-categories: PASS
[trusty-testing] (timing) 0.96user 0.06system 0:01.04elapsed 98%CPU (0avgtext+0avgdata 29420maxresident)k
[trusty-testing] (timing) 0inputs+48outputs (0major+10225minor)pagefaults 0swaps
Found a test script: ./plainbox/requirements/001-container-tests-plainbox-egg-info
[trusty-testing] 001-container-tests-plainbox-egg-info: PASS
[trusty-testing] (timing) 0.42user 0.06system 0:00.49elapsed 97%CPU (0avgtext+0avgdata 19096maxresident)k
[trusty-testing] (timing) 0inputs+96outputs (0major+13247minor)pagefaults 0swaps
Found a test script: ./plainbox/requirements/container-tests-plainbox
[trusty-testing] container-tests-plainbox: PASS
[trusty-testing] (timing) 50.91user 1.01system 0:52.15elapsed 99%CPU (0avgtext+0avgdata 95620maxresident)k
[trusty-testing] (timing) 0inputs+2712outputs (0major+126340minor)pagefaults 0swaps
Found a test script: ./plainbox/requirements/container-tests-plainbox-documentation
[trusty-testing] container-tests-plainbox-documentation: PASS
[trusty-testing] (timing) 125.20user 0.40system 2:05.82elapsed 99%CPU (0avgtext+0avgdata 115680maxresident)k
[trusty-testing] (timing) 0...

[Jeff Lane  (bladernr)]

Ok. lets try again...

review: Approve

[Sylvain Pineau (sylvain-pineau)]

The attempt to merge lp:~rodsmith/checkbox/smart-for-dmraid into lp:checkbox failed. Below is the output from the failed tests.

[precise] starting container
[precise] (timing) 0.08user 0.08system 0:05.53elapsed 3%CPU (0avgtext+0avgdata 10256maxresident)k
[precise] (timing) 0inputs+32outputs (0major+8746minor)pagefaults 0swaps
[precise] provisioning container
[precise] (timing) 42.31user 12.01system 1:02.83elapsed 86%CPU (0avgtext+0avgdata 95724maxresident)k
[precise] (timing) 0inputs+19088outputs (0major+1856017minor)pagefaults 0swaps
[precise-testing] Starting tests...
Found a test script: ./checkbox-ng/requirements/container-tests-checkbox-ng-unit
[precise-testing] container-tests-checkbox-ng-unit: PASS
[precise-testing] (timing) 0.69user 0.11system 0:00.81elapsed 98%CPU (0avgtext+0avgdata 47532maxresident)k
[precise-testing] (timing) 0inputs+1336outputs (0major+22398minor)pagefaults 0swaps
Found a test script: ./checkbox-support/requirements/container-tests-checkbox-support
[precise-testing] container-tests-checkbox-support: PASS
[precise-testing] (timing) 23.83user 0.19system 0:24.13elapsed 99%CPU (0avgtext+0avgdata 150596maxresident)k
[precise-testing] (timing) 0inputs+1328outputs (0major+40310minor)pagefaults 0swaps
Found a test script: ./checkbox-touch/requirements/container-tests-touch-unit-tests
[precise-testing] container-tests-touch-unit-tests: PASS
[precise-testing] (timing) 0.01user 0.01system 0:00.03elapsed 84%CPU (0avgtext+0avgdata 2180maxresident)k
[precise-testing] (timing) 0inputs+8outputs (0major+2378minor)pagefaults 0swaps
Found a test script: ./plainbox/plainbox/impl/providers/categories/requirements/container-tests-provider-categories
[precise-testing] container-tests-provider-categories: PASS
[precise-testing] (timing) 0.99user 0.12system 0:01.13elapsed 98%CPU (0avgtext+0avgdata 46960maxresident)k
[precise-testing] (timing) 0inputs+64outputs (0major+14400minor)pagefaults 0swaps
Found a test script: ./plainbox/requirements/001-container-tests-plainbox-egg-info
[precise-testing] 001-container-tests-plainbox-egg-info: PASS
[precise-testing] (timing) 0.28user 0.09system 0:00.39elapsed 97%CPU (0avgtext+0avgdata 20240maxresident)k
[precise-testing] (timing) 0inputs+96outputs (0major+11873minor)pagefaults 0swaps
Found a test script: ./plainbox/requirements/container-tests-plainbox
[precise-testing] container-tests-plainbox: PASS
[precise-testing] (timing) 55.59user 1.44system 0:57.21elapsed 99%CPU (0avgtext+0avgdata 195980maxresident)k
[precise-testing] (timing) 0inputs+3304outputs (0major+198548minor)pagefaults 0swaps
Found a test script: ./plainbox/requirements/container-tests-plainbox-documentation
[precise-testing] container-tests-plainbox-documentation: PASS
[precise-testing] (timing) 141.13user 0.61system 2:22.04elapsed 99%CPU (0avgtext+0avgdata 179064maxresident)k
[precise-testing] (timing) 0inputs+41232outputs (0major+54659minor)pagefaults 0swaps
Found a test script: ./plainbox/requirements/container-tests-plainbox-integration
[precise-testing] container-tests-plainbox-integration: PASS
[precise-testing] (ti...

[Jeff Lane  (bladernr)]

Try again... Tarmac is on a roll

=== modified file 'providers/plainbox-provider-checkbox/bin/disk_smart'
--- providers/plainbox-provider-checkbox/bin/disk_smart	2016-01-11 21:50:37 +0000
+++ providers/plainbox-provider-checkbox/bin/disk_smart	2016-01-13 16:02:11 +0000
@@ -77,6 +77,9 @@
 from subprocess import CalledProcessError
 from argparse import ArgumentParser
 
+# NOTE: If raid_types changes, also change it in block_device_resource script!
+raid_types = ["megaraid", "cciss", "3ware", "areca"]
+
 
 class ListHandler(logging.StreamHandler):
 
@@ -97,38 +100,100 @@
             logging.StreamHandler.emit(self, record)
 
 
-def enable_smart(disk):
+def enable_smart(disk, raid_element, raid_type):
     """Log data and, if necessary, enable SMART on the specified disk.
 
     See also smart_support() in block_device_resource script.
     :param disk:
         disk device filename (e.g., /dev/sda)
+    :param raid_element:
+        element number to enable in RAID array; undefined if not a RAID device
+    :param raid_type:
+        type of raid device (none, megaraid, etc.)
     :returns:
         True if enabling smart was successful, False otherwise
     """
     # Check with smartctl to record basic SMART data on the disk
-    command = 'smartctl -i %s' % disk
+    if raid_type == 'none':
+        command = 'smartctl -i {}'.format(disk)
+        logging.debug('SMART Info for disk {}'.format(disk))
+    else:
+        command = 'smartctl -i {} -d {},{}'.format(disk, raid_type,
+                                                   raid_element)
+        logging.debug('SMART Info for disk {}, element {}'.
+                      format(disk, raid_element))
     diskinfo_bytes = (Popen(command, stdout=PIPE, shell=True)
                       .communicate()[0])
     diskinfo = (diskinfo_bytes.decode(encoding='utf-8', errors='ignore')
                 .splitlines())
-    logging.debug('SMART Info for disk %s', disk)
     logging.debug(diskinfo)
     if len(diskinfo) > 2 and not any("SMART support is" in s and "Enabled"
                                      in s for s in diskinfo):
         logging.debug('SMART disabled; attempting to enable it.')
-        command = 'smartctl -s on %s' % disk
+        if raid_type == 'none':
+            command = 'smartctl -s on {}'.format(disk)
+        else:
+            command = ('smartctl -s on {} -d {},{}'.
+                       format(disk, raid_type, raid_element))
         try:
             check_call(shlex.split(command))
             return True
         except CalledProcessError:
+            if raid_type == 'none':
+                logging.warning('SMART could not be enabled on {}'.
+                                format(disk))
+            else:
+                logging.warning('SMART could not be enabled on {}, element '
+                                '{}'.format(disk, raid_element))
             return False
     return True
 
 
-def run_smart_test(disk, type='short'):
-    ctl_command = 'smartctl -t %s %s' % (type, disk)
-    logging.debug('Beginning test with %s', ctl_command)
+def count_raid_disks(disk):
+    """Count the disks in a RAID array.
+
+    :param disk:
+        Disk device filename (e.g., /dev/sda)
+    :returns:
+        Number of disks in array (0 for non-RAID disk)
+        Type of RAID (none, megaraid, 3ware, areca, or cciss; note that only
+                      none and megaraid are tested, as of Jan. 2016)
+    """
+    raid_element = 0
+    raid_type = 'none'
+    command = 'smartctl -i {}'.format(disk)
+    diskinfo_bytes = (Popen(command, stdout=PIPE, shell=True)
+                      .communicate()[0])
+    diskinfo = (diskinfo_bytes.decode(encoding='utf-8', errors='ignore')
+                .splitlines())
+    for type in raid_types:
+        if any("-d {},N".format(type) in s for s in diskinfo):
+            logging.info('Found RAID controller of type {}'.format(type))
+            raid_type = type
+            break
+    if raid_type != 'none':
+        # This is a hardware RAID controller, so count individual disks....
+        disk_exists = True
+        while disk_exists:
+            command = ('smartctl -i {} -d {},{}'.
+                       format(disk, raid_type, raid_element))
+            try:
+                check_output(shlex.split(command))
+                raid_element += 1
+            except CalledProcessError:
+                disk_exists = False
+        logging.info("Counted {} RAID disks on {}\n".
+                     format(raid_element, disk))
+    return raid_element, raid_type
+
+
+def initiate_smart_test(disk, raid_element, raid_type, type='short'):
+    if raid_type == 'none':
+        ctl_command = 'smartctl -t {} {}'.format(type, disk)
+    else:
+        ctl_command = ('smartctl -t {} {} -d {},{}'.
+                       format(type, disk, raid_type, raid_element))
+    logging.debug('Beginning test with {}'.format(ctl_command))
 
     smart_proc = Popen(ctl_command, stderr=PIPE, stdout=PIPE,
                        universal_newlines=True, shell=True)
@@ -139,11 +204,17 @@
     return smart_proc.returncode
 
 
-def get_smart_entries(disk, type='selftest'):
+def get_smart_entries(disk, raid_element, raid_type, type='selftest'):
     entries = []
     try:
-        stdout = check_output(['smartctl', '-l', type, disk],
-                              universal_newlines=True)
+        if raid_type == 'none':
+            stdout = check_output(['smartctl', '-l', type, disk],
+                                  universal_newlines=True)
+        else:
+            stdout = check_output(['smartctl', '-l', type, disk,
+                                   '-d', '{},{}'.
+                                   format(raid_type, raid_element)],
+                                  universal_newlines=True)
         returncode = 0
     except CalledProcessError as err:
         stdout = err.output
@@ -185,10 +256,14 @@
     return entries, returncode
 
 
-# Returns True if an "in-progress" message is found in the smartctl
-# output, False if such a message is not found. In the former case,
-# the in-progress message entries are logged.
 def in_progress(current_entries):
+    """Check to see if the test is in progress.
+
+    :param current_entries:
+        Output of smartctl command to be checked for status indicator.
+    :returns:
+        True if an "in-progress" message is found, False otherwise
+    """
     statuses = [entry for entry in current_entries
                 if isinstance(entry, dict)
                 and 'status' in entry
@@ -205,13 +280,28 @@
         return False
 
 
-# Wait for SMART test to complete; return status and return code.
-# Note that different disks return different types of values.
-# Some return no status reports while a test is ongoing; others
-# show a status line at the START of the list of tests, and
-# others show a status line at the END of the list of tests
-# (and then move it to the top once the tests are done).
-def poll_for_status(args, disk, previous_entries):
+def poll_for_status(args, disk, raid_element, raid_type, previous_entries):
+    """Poll a disk for its SMART status.
+
+    Wait for SMART test to complete; return status and return code.
+    Note that different disks return different types of values.
+    Some return no status reports while a test is ongoing; others
+    show a status line at the START of the list of tests, and
+    others show a status line at the END of the list of tests
+    (and then move it to the top once the tests are done).
+    :param args:
+        Script's command-line arguments
+    :param disk:
+        Disk device (e.g., /dev/sda)
+    :param raid_element:
+        RAID disk number (undefined for non-RAID disk)
+    :param raid_type:
+        Type of RAID device (megaraid, etc.)
+    :param previous_entries:
+        Previous SMART output; used to spot a change
+    :returns:
+        Current output and return code
+    """
     # Priming read... this is here in case our test is finished or fails
     # immediate after it beginsAccording to.
     logging.debug('Polling selftest.log for status')
@@ -221,7 +311,8 @@
         # Poll every sleep seconds until test is complete$
         time.sleep(args.sleep)
 
-        current_entries, returncode = get_smart_entries(disk)
+        current_entries, returncode = get_smart_entries(disk, raid_element,
+                                                        raid_type)
         if current_entries != previous_entries:
             if not in_progress(current_entries):
                 keep_going = False
@@ -239,8 +330,69 @@
         return current_entries[0]['status'], returncode
 
 
+def run_smart_test(args, disk, raid_element, raid_type):
+    """Run a test on a single disk device (possibly multiple RAID elements).
+
+    :param args:
+        Command-line arguments passed to script
+    :param disk:
+        Disk device filename (e.g., /dev/sda)
+    :param raid_element:
+        Number of RAID array element or undefined for non-RAID disk
+    :param raid_type:
+        Type of RAID device (e.g., megaraid)
+    :returns:
+        True for success, False for failure
+    """
+    previous_entries, returncode = get_smart_entries(disk, raid_element,
+                                                     raid_type)
+    if raid_type == 'none':
+        logging.info("Starting SMART self-test on {}".format(disk))
+    else:
+        logging.info("Starting SMART self-test on {}, element {}".
+                     format(disk, raid_element))
+    if initiate_smart_test(disk, raid_element, raid_type) != 0:
+        logging.error("Error reported during smartctl test")
+        return False
+
+    if len(previous_entries) > 20:
+        # Abort the previous instance
+        # so that polling can identify the difference
+        initiate_smart_test(disk, raid_element, raid_type)
+        previous_entries, returncode = get_smart_entries(disk, raid_element,
+                                                         raid_type)
+
+    status, returncode = poll_for_status(args, disk, raid_element, raid_type,
+                                         previous_entries)
+
+    if returncode != 0:
+        log, returncode = get_smart_entries(disk, raid_element, raid_type)
+        if raid_type == 'none':
+            logging.error("FAIL: SMART Self-Test appears to have failed "
+                          "for some reason. Run 'sudo smartctl -l selftest "
+                          "{}' to see the SMART log".format(disk))
+        else:
+            logging.error("FAIL: SMART Self-Test appears to have failed "
+                          "for some reason. Run 'sudo smartctl -l selftest "
+                          "{} -d {},{}' to see the SMART log".
+                          format(disk, raid_type, raid_element))
+        logging.debug("Last smartctl return code: %d", returncode)
+        logging.debug("Last smartctl run status: %s", status)
+        return False
+    else:
+        if raid_type == 'none':
+            logging.info("PASS: SMART Self-Test on {} completed without error".
+                         format(disk))
+        else:
+            logging.info("PASS: SMART Self-Test on {}, element {} completed "
+                         "without error\n".format(disk, raid_element))
+        return True
+
+
 def main():
-    description = 'Tests that SMART capabilities on disks that support SMART function.'
+    """Test SMART capabilities on disks that support SMART functions."""
+    description = ('Tests SMART capabilities on disks that support '
+                   'SMART functions.')
     parser = ArgumentParser(description=description)
     parser.add_argument('-b', '--block-dev',
                         metavar='DISK',
@@ -278,35 +430,21 @@
         parser.error("You must be root to run this program")
 
     disk = args.block_dev
-    if not enable_smart(disk):
-        logging.warning('SMART could not be enabled on %s' % disk)
-        return 1
-
-    # Initiate a self test and start polling until the test is done
-    previous_entries, returncode = get_smart_entries(disk)
-    logging.info("Starting SMART self-test on %s", disk)
-    if run_smart_test(disk) != 0:
-        logging.error("Error reported during smartctl test")
-        return 1
-
-    if len(previous_entries) > 20:
-        # Abort the previous instance
-        # so that polling can identify the difference
-        run_smart_test(disk)
-        previous_entries, returncode = get_smart_entries(disk)
-
-    status, returncode = poll_for_status(args, disk, previous_entries)
-
-    if returncode != 0:
-        log, returncode = get_smart_entries(disk)
-        logging.error("FAIL: SMART Self-Test appears to have failed for some reason. "
-                      "Run 'sudo smartctl -l selftest %s' to see the SMART log",
-                      disk)
-        logging.debug("Last smartctl return code: %d", returncode)
-        logging.debug("Last smartctl run status: %s", status)
-        return 1
-    else:
-        logging.info("PASS: SMART Self-Test completed without error")
+    num_disks, raid_type = count_raid_disks(disk)
+    if num_disks == 0:
+        success = enable_smart(disk, -1, raid_type)
+        success = success and run_smart_test(args, disk, -1, raid_type)
+    else:
+        success = True
+        for raid_element in range(0, num_disks):
+            if enable_smart(disk, raid_element, raid_type):
+                success = (run_smart_test(args, disk, raid_element, raid_type)
+                           and success)
+            else:
+                success = False
+    if success is False:
+        return 1
+    else:
         return 0
 
 

=== modified file 'providers/plainbox-provider-resource-generic/bin/block_device_resource'
--- providers/plainbox-provider-resource-generic/bin/block_device_resource	2016-01-11 23:52:28 +0000
+++ providers/plainbox-provider-resource-generic/bin/block_device_resource	2016-01-13 16:02:11 +0000
@@ -2,17 +2,18 @@
 
 import os
 import re
+import shlex
 from glob import glob
-from subprocess import Popen, PIPE
+from subprocess import Popen, PIPE, check_output, CalledProcessError
 
 rootdir_pattern = re.compile('^.*?/devices')
 
+# NOTE: If raid_types changes, also change it in disk_smart script!
+raid_types = ["megaraid", "cciss", "3ware", "areca"]
+
 
 def device_state(name):
-    """
-    Follow pmount policy to determine whether a device is removable
-    or internal.
-    """
+    """Follow pmount policy to find if a device is removable or internal."""
     with open('/sys/block/%s/device/block/%s/removable' % (name, name),
               "rt") as f:
         if f.read(1) == '1':
@@ -69,10 +70,49 @@
     return 'no'
 
 
+def smart_support_raid(name, raid_type):
+    """Check for availability of SMART support in a RAID device.
+
+    See also count_raid_disks() in disk_smart script.
+    :param name:
+        disk device filename within /dev (e.g., sda)
+    :param raid_type:
+        type of raid device (e.g., megaraid, 3ware, etc. -- codes used
+                             by smartctl)
+    :returns:
+        'True' or 'False' as string (for return to Checkbox)
+    """
+    supported = 'False'
+    disk_num = 0
+    disk_exists = True
+    # Loop through all disks in array to verify that SMART is available on
+    # at least one of them. Note that if there's a mix of supported and
+    # unsupported, this test returns 'True', which will result in a failure
+    # of disk_smart. This is by design, since such a mix is likely an assembly
+    # error by the manufacturer.
+    while disk_exists:
+        command = 'smartctl -i /dev/{} -d {},{}'.format(name, raid_type,
+                                                        disk_num)
+        # Return 'True' if the output (diskinfo) includes
+        # "SMART support is.*Available", and terminate check when a failure
+        # is found or when number of disks rises above level supported by
+        # smartctl (which likely indicates a bug).
+        try:
+            diskinfo = (check_output(shlex.split(command)).decode('utf-8').
+                        splitlines())
+            if any("SMART support is" in s and "Available" in s for
+                   s in diskinfo):
+                supported = 'True'
+                disk_num += 1
+        except CalledProcessError:
+            disk_exists = False
+    return supported
+
+
 def smart_support(name):
     """Check for availability of SMART support in the device.
 
-    See also enable_smart() in disk_smart script.
+    See also count_raid_disks() in disk_smart script.
     :param name:
         disk device filename within /dev (e.g., sda)
     :returns:
@@ -92,6 +132,11 @@
         if any("SMART support is" in s and "Available" in s
                for s in diskinfo):
             supported = 'True'
+        else:
+            for type in raid_types:
+                if any("-d {},N".format(type) in s for s in diskinfo):
+                    supported = smart_support_raid(name, type)
+                    break
     return supported
 
 for path in glob('/sys/block/*/device'):