Merge ~newell-jensen/maas:fio-test into maas:master

[Andres Rodriguez]

Add builtin script fio test for hardware testing.

[Lee Trager (ltrager)]

Thanks for working on this. Currently this doesn't work as fio needs to be installed. I'm also a little concerned we're dropping STDERR. Can you see if fio outputs anything to STDERR?

review: Needs Fixing

[Newell Jensen (newell-jensen)]

> Thanks for working on this. Currently this doesn't work as fio needs to be
> installed. I'm also a little concerned we're dropping STDERR. Can you see if
> fio outputs anything to STDERR?

Go ahead and have another look at it. Hopefully changes in the spec will not demand too many changes with this :)

[Newell Jensen (newell-jensen)]

Setting to WIP since things have been changed in the spec after meeting.

[Lee Trager (ltrager)]

Updated review, we still need to add the MAAS metadata embedded YAML and modify MAAS to load the script in the database(I'm reworking that code now).

[Lee Trager (ltrager)]

Looks good! One small thing inline and you have a merge conflict but I won't block you.

review: Approve

[Andres Rodriguez (andreserl)]

This branch has been reverted because it is dependent on https://code.launchpad.net/~ltrager/maas/+git/maas/+merge/329983 , which has also been reverted.

Unmerged commits

9d6cb80... by Newell Jensen on 2017-09-01

Review fixes.

c9b16c7... by Newell Jensen on 2017-08-29

Fix lint.

7fcca1f... by Newell Jensen on 2017-08-29

Decode bytes from regex matching.

390f153... by Newell Jensen on 2017-08-29

Update embedded YAML and add fio to builtin scripts.

1a27095... by Newell Jensen on 2017-08-25

Update results dictionary in embedded YAML and change dictionary keys used in results file to match.

1db6b27... by Newell Jensen on 2017-08-25

Update embedded YAML.

4e46174... by Newell Jensen on 2017-08-23

Delete YAML metadata title's curly brackets to make it more readable.

667f9d0... by Newell Jensen on 2017-08-22

Add YAML metadata to fio test.

027849c... by Newell Jensen on 2017-08-17

Only print stdout if it is not None.

86eaea0... by Newell Jensen on 2017-08-16

Review fixes.

diff --git a/src/metadataserver/builtin_scripts/__init__.py b/src/metadataserver/builtin_scripts/__init__.py
index c47f1a2..bfade7a 100644
--- a/src/metadataserver/builtin_scripts/__init__.py
+++ b/src/metadataserver/builtin_scripts/__init__.py
@@ -131,6 +131,10 @@ BUILTIN_SCRIPTS = [
         name='7z',
         filename='seven_z.py',
         ),
+    BuiltinScript(
+        name='fio',
+        filename='fio.py',
+        ),
     ]
 
 
diff --git a/src/metadataserver/builtin_scripts/fio.py b/src/metadataserver/builtin_scripts/fio.py
new file mode 100644
index 0000000..0cb5204
--- /dev/null
+++ b/src/metadataserver/builtin_scripts/fio.py
@@ -0,0 +1,161 @@
+#!/usr/bin/env python3
+#
+# fio - Run fio on supplied drive.
+#
+# Author: Newell Jensen <[e-mail]>
+#
+# Copyright (C) 2017 Canonical
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as
+# published by the Free Software Foundation, either version 3 of the
+# License, or (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+# --- Start MAAS 1.0 script metadata ---
+# name: fio
+# title: Storage benchmark
+# description: Run Fio benchmarking against selected storage devices.
+# tags: storage
+# script_type: testing
+# hardware_type: storage
+# parallel: instance
+# results:
+#  random_read:
+#   title: Random read
+#   description: Read speed when reading randomly from the disk.
+#  random_read_iops:
+#   title: Random read IOPS
+#   description: IOPS when reading randomly from the disk.
+#  sequential_read:
+#   title: Sequential read
+#   description: Read speed when reading sequentialy from the disk.
+#  sequential_read_iops:
+#   title: Sequential read IOPS
+#   description: IOPS when reading sequentialy from the disk.
+#  random_write:
+#   title: Random write
+#   description: Write speed when reading randomly from the disk.
+#  random_write_iops:
+#   title: Random write IOPS
+#   description: IOPS when reading randomly from the disk.
+#  sequential_write:
+#   title: Sequential write
+#   description: Write speed when reading sequentialy from the disk.
+#  sequential_write_iops:
+#   title: Sequential write IOPS
+#   description: IOPS when reading sequentialy from the disk.
+# parameters:
+#  storage: {type: storage}
+# packages: {apt: fio}
+# destructive: true
+# --- End MAAS 1.0 script metadata ---
+
+
+import argparse
+from copy import deepcopy
+import os
+import re
+from subprocess import (
+    PIPE,
+    Popen,
+    STDOUT,
+)
+import sys
+
+import yaml
+
+
+CMD = [
+    'sudo', '-n', 'fio', '--randrepeat=1', '--ioengine=libaio',
+    '--direct=1', '--gtod_reduce=1', '--name=fio_test', '--bs=4k',
+    '--iodepth=64', '--size=4G'
+]
+
+REGEX = b"bw=([0-9]+[a-zA-Z]+/s), iops=([0-9]+)"
+
+
+def run_cmd(cmd):
+    """Execute `cmd` and return output or exit if error."""
+    proc = Popen(cmd, stdout=PIPE, stderr=STDOUT)
+    # Currently, we are piping stderr to STDOUT.
+    stdout, _ = proc.communicate()
+
+    # Print stdout to the console.
+    if stdout is not None:
+        print('Running command: %s\n' % ' '.join(cmd))
+        print(stdout.decode())
+        print('-' * 80)
+    if proc.returncode == 0:
+        return stdout, proc.returncode
+    sys.exit(proc.returncode)
+
+
+def run_fio_test(readwrite, result_path):
+    """Run fio for the given type of test specified by `cmd`."""
+    cmd = deepcopy(CMD)
+    cmd.append('--readwrite=%s' % readwrite)
+    stdout, returncode = run_cmd(cmd)
+    if result_path is not None:
+        # Parse the results for the desired information and
+        # then wrtie this to the results file.
+        match = re.search(REGEX, stdout)
+        if match is None:
+            print("Warning: results could not be found.")
+        return match
+
+
+def run_fio(storage):
+    """Execute fio tests for supplied storage device.
+
+    Performs random and sequential read and write tests.
+    """
+    result_path = os.environ.get("RESULT_PATH")
+    CMD.append('--filename=%s' % storage)
+    random_read_match = run_fio_test("randread", result_path)
+    sequential_read_match = run_fio_test("read", result_path)
+    random_write_match = run_fio_test("randwrite", result_path)
+    sequential_write_match = run_fio_test("write", result_path)
+
+    # Write out YAML file if RESULT_PATH is set.
+    # The status is hardcoded at the moment because there isn't a
+    # degraded state yet for fio.  This most likely will change in
+    # the future when there is an agreed upon crtteria for fio to
+    # mark a storage device in the degraded state based on one of
+    # its tests.
+    if all(var is not None for var in [
+            result_path, random_read_match, sequential_read_match,
+            random_write_match, sequential_write_match]):
+        results = {
+            'status': "passed",
+            'results': {
+                'random_read': random_read_match.group(1).decode(),
+                'random_read_iops': random_read_match.group(2).decode(),
+                'sequential_read': sequential_read_match.group(1).decode(),
+                'sequential_read_iops': (
+                    sequential_read_match.group(2).decode()),
+                'random_write': random_write_match.group(1).decode(),
+                'random_write_iops': random_write_match.group(2).decode(),
+                'sequential_write': sequential_write_match.group(1).decode(),
+                'sequential_write_iops': (
+                    sequential_write_match.group(2).decode()),
+            }
+        }
+        with open(result_path, 'w') as results_file:
+            yaml.safe_dump(results, results_file)
+
+
+if __name__ == '__main__':
+    parser = argparse.ArgumentParser(description='Fio Hardware Testing.')
+    parser.add_argument(
+        '--storage', dest='storage',
+        help='path to storage device you want to test. e.g. /dev/sda')
+    args = parser.parse_args()
+    sys.exit(run_fio(args.storage))
diff --git a/src/metadataserver/builtin_scripts/tests/test_fio.py b/src/metadataserver/builtin_scripts/tests/test_fio.py
new file mode 100644
index 0000000..1e39a9b
--- /dev/null
+++ b/src/metadataserver/builtin_scripts/tests/test_fio.py
@@ -0,0 +1,137 @@
+# Copyright 2017 Canonical Ltd.  This software is licensed under the
+# GNU Affero General Public License version 3 (see the file LICENSE).
+
+"""Test builtin_script fio."""
+
+__all__ = []
+
+from copy import deepcopy
+import io
+import os
+import re
+from subprocess import (
+    PIPE,
+    Popen,
+    STDOUT,
+)
+from textwrap import dedent
+from unittest.mock import ANY
+
+from maastesting.factory import factory
+from maastesting.matchers import MockCalledOnceWith
+from maastesting.testcase import MAASTestCase
+from metadataserver.builtin_scripts import fio
+import yaml
+
+
+FIO_READ_OUTPUT = dedent("""
+    ...
+    Starting 1 process
+    Jobs: 1 (f=1): [r] [100.0% done] [62135K/0K /s] [15.6K/0 iops]
+    test: (groupid=0, jobs=1): err= 0: pid=31181: Fri May 9 15:38:57 2014
+      read : io=1024.0MB, bw=62748KB/s, iops=15686 , runt= 16711msec
+      ...
+    """)
+
+FIO_WRITE_OUTPUT = dedent("""
+    ...
+    Starting 1 process
+    Jobs: 1 (f=1): [w] [100.0% done] [0K/26326K /s] [0 /6581 iops]
+    test: (groupid=0, jobs=1): err= 0: pid=31235: Fri May 9 16:16:21 2014
+      write: io=1024.0MB, bw=29195KB/s, iops=7298 , runt= 35916msec
+      ...
+    """)
+
+
+class TestFioTest(MAASTestCase):
+
+    def test_run_cmd_runs_cmd_and_returns_output(self):
+        cmd = factory.make_string()
+        output = factory.make_string()
+        mock_popen = self.patch(fio, "Popen")
+        mock_popen.return_value = Popen(
+            ['echo', '-n', output], stdout=PIPE, stderr=PIPE)
+
+        cmd_output = fio.run_cmd(cmd)
+
+        self.assertEquals((output.encode(), 0), cmd_output)
+        self.assertThat(mock_popen, MockCalledOnceWith(
+            cmd, stdout=PIPE, stderr=STDOUT))
+
+    def test_run_cmd_runs_cmd_and_exits_on_error(self):
+        cmd = factory.make_string()
+        mock_popen = self.patch(fio, "Popen")
+        proc = mock_popen.return_value
+        proc.communicate.return_value = (b"Output", None)
+        proc.returncode = 1
+
+        self.assertRaises(SystemExit, fio.run_cmd, cmd)
+        self.assertThat(mock_popen, MockCalledOnceWith(
+            cmd, stdout=PIPE, stderr=STDOUT))
+
+    def test_run_fio_test_runs_test(self):
+        result_path = factory.make_string()
+        readwrite = factory.make_string()
+        cmd = deepcopy(fio.CMD)
+        cmd.append('--readwrite=%s' % readwrite)
+        returncode = 0
+        mock_run_cmd = self.patch(fio, "run_cmd")
+        mock_run_cmd.return_value = (
+            FIO_READ_OUTPUT.encode('utf-8'), returncode)
+        mock_re_search = self.patch(re, "search")
+        fio.run_fio_test(readwrite, result_path)
+
+        self.assertThat(mock_run_cmd, MockCalledOnceWith(cmd))
+        self.assertThat(mock_re_search, MockCalledOnceWith(
+            fio.REGEX, FIO_READ_OUTPUT.encode('utf-8')))
+
+    def test_run_fio_test_exits_if_no_match_found(self):
+        result_path = factory.make_string()
+        readwrite = factory.make_string()
+        cmd = deepcopy(fio.CMD)
+        cmd.append('--readwrite=%s' % readwrite)
+        returncode = 0
+        mock_run_cmd = self.patch(fio, "run_cmd")
+        mock_run_cmd.return_value = (
+            FIO_WRITE_OUTPUT.encode('utf-8'), returncode)
+        mock_re_search = self.patch(re, "search")
+        mock_re_search.return_value = None
+        fio.run_fio_test(readwrite, result_path)
+
+        self.assertThat(mock_run_cmd, MockCalledOnceWith(cmd))
+        self.assertThat(mock_re_search, MockCalledOnceWith(
+            fio.REGEX, FIO_WRITE_OUTPUT.encode('utf-8')))
+
+    def test_run_fio_writes_yaml_file(self):
+        self.patch(os, 'environ', {
+            "RESULT_PATH": factory.make_name()
+        })
+        disk = factory.make_name('disk')
+        read_match = re.search(fio.REGEX, FIO_READ_OUTPUT.encode('utf-8'))
+        write_match = re.search(fio.REGEX, FIO_WRITE_OUTPUT.encode('utf-8'))
+        mock_run_fio_test = self.patch(fio, "run_fio_test")
+        mock_run_fio_test.side_effect = [
+            read_match, read_match, write_match, write_match]
+        mock_open = self.patch(fio, "open")
+        mock_open.return_value = io.StringIO()
+        mock_yaml_safe_dump = self.patch(yaml, "safe_dump")
+        # For the test, we will just use the same fio result for both
+        # random and sequential.
+        results = {
+            'status': "passed",
+            'results': {
+                'random_read': read_match.group(1).decode(),
+                'random_read_iops': read_match.group(2).decode(),
+                'sequential_read': read_match.group(1).decode(),
+                'sequential_read_iops': read_match.group(2).decode(),
+                'random_write': write_match.group(1).decode(),
+                'random_write_iops': write_match.group(2).decode(),
+                'sequential_write': write_match.group(1).decode(),
+                'sequential_write_iops': write_match.group(2).decode(),
+            }
+        }
+        fio.run_fio(disk)
+
+        self.assertThat(mock_open, MockCalledOnceWith(ANY, "w"))
+        self.assertThat(mock_yaml_safe_dump, MockCalledOnceWith(
+            results, mock_open.return_value))