Merge lp:~bladernr/checkbox/network-fail-value-opton into lp:checkbox

[Jeff Lane ]

This addresses the issue of bumping the fail rate to 80% for crappy client systems that can't seem to do better than 40% ;-)

Make fail threshold an option and sets the default to 40% and changes the server multi-nic tests to fail at 80% instead (well, anything that runs ethernet/multi-nic will fail at 80%)

This is a follow on to a previous checkbox merge that changed fail rate without making it optional and the optional route is at request of roadmr.

[lp:~bladernr/checkbox/network-fail-value-opton]

2614. By Zygmunt Krynicki on 2014-01-27

"automatic merge by tarmac [r=roadmr][bug=][author=zkrynicki]"

[Daniel Manrique (roadmr)]

Jeff is aware of a couple of needed fixes, just marking this as needs fixing to avoid review duplication, details:

11:56 < roadmr> bladernr: :( changelog conflict :(
11:57 < roadmr> bladernr: would you be opposed to changing fail_speed to
                fail_threshold? fail_speed suggests, well, a speed value, so
                it's a bit confusing (what are the units? why is the default 40
                of those units? and so on)

review: Needs Fixing

[Zygmunt Krynicki (zyga)]

Conflict, please fix

On Mon, Jan 27, 2014 at 5:55 PM, Jeff Lane <email address hidden>wrote:

> Jeff Lane has proposed merging
> lp:~bladernr/checkbox/network-fail-value-opton into lp:checkbox.
>
> Requested reviews:
> Checkbox Developers (checkbox-dev)
> Related bugs:
> Bug #1267918 in checkbox: "multi-nic iperf test should fail at <80%"
> https://bugs.launchpad.net/checkbox/+bug/1267918
>
> For more details, see:
>
> https://code.launchpad.net/~bladernr/checkbox/network-fail-value-opton/+merge/203372
>
> This addresses the issue of bumping the fail rate to 80% for crappy client
> systems that can't seem to do better than 40% ;-)
>
> Make fail threshold an option and sets the default to 40% and changes the
> server multi-nic tests to fail at 80% instead (well, anything that runs
> ethernet/multi-nic will fail at 80%)
>
> This is a follow on to a previous checkbox merge that changed fail rate
> without making it optional and the optional route is at request of roadmr.
> --
>
> https://code.launchpad.net/~bladernr/checkbox/network-fail-value-opton/+merge/203372
> Your team Checkbox Developers is requested to review the proposed merge of
> lp:~bladernr/checkbox/network-fail-value-opton into lp:checkbox.
>
> === modified file 'checkbox-old/debian/changelog'
> --- checkbox-old/debian/changelog 2014-01-21 22:17:57 +0000
> +++ checkbox-old/debian/changelog 2014-01-27 16:55:02 +0000
> @@ -11,12 +11,20 @@
> [Daniel Manrique]
> * jobs/graphics.txt.in: Use grep to work around bad exit codes from
> unity_support_test (LP: #1212618)
> +<<<<<<< TREE
> * Added tests for 802.11ac networking, including environment variables
> for
> their settings, and updated whitelists accordingly.
> * scripts/sleep_test_log_check: Rewrote the parser logic to avoid
> missing
> failures due to file format changes (LP: #1201667)
> * parsers/udevadm: Added additional heuristics for video devices with
> PCI subclass "2", like newer NVidia GPUs (LP: #1270139)
> +=======
> +
> + [Jeff Lane]
> + * scripts/network: fail threshold for iperf is now an option and
> defaults to
> + 40%
> + * jobs/ethernet.txt.in: modified multinic to fail on 80% threshold
> +>>>>>>> MERGE-SOURCE
>
> -- Brendan Donegan <email address hidden> Fri, 10 Jan 2014
> 11:02:47 +0000
>
>
> === modified file 'checkbox-old/jobs/ethernet.txt.in'
> --- checkbox-old/jobs/ethernet.txt.in 2014-01-24 18:13:09 +0000
> +++ checkbox-old/jobs/ethernet.txt.in 2014-01-27 16:55:02 +0000
> @@ -45,7 +45,7 @@
> package.name == 'nmap'
> device.path == "$1"
> user: root
> - command: network test -i $2 -t iperf
> + command: network test -i $2 -t iperf --fail-speed 80
> estimated_duration: 330.0
> description:
> Testing for NIC $2
>
> === modified file 'checkbox-old/scripts/network'
> --- checkbox-old/scripts/network 2014-01-15 22:21:57 +0000
> +++ checkbox-old/scripts/network 2014-01-27 16:55:02 +0000
> @@ -53,6 +53,7 @@
> self,
> interface,
> target,
> + fail_speed,
> protocol="tcp",
> mbytes="1024M"):
>
> @@ -112,7 +113,7 @@
> print(...

[lp:~bladernr/checkbox/network-fail-value-opton]

2615. By Zygmunt Krynicki on 2014-01-27

"automatic merge by tarmac [r=roadmr][bug=1236377,1236505,1243610,1254189,1254191,1255066,1255085,1265748,1267794,1273100][author=zkrynicki]"

2616. By Zygmunt Krynicki on 2014-01-27

"automatic merge by tarmac [r=roadmr][bug=][author=zkrynicki]"

2617. By Sylvain Pineau on 2014-01-27

"automatic merge by tarmac [r=sylvain-pineau,roadmr][bug=][author=sylvain-pineau]"

2618. By Zygmunt Krynicki on 2014-01-27

"automatic merge by tarmac [r=roadmr][bug=][author=zkrynicki]"

2619. By Zygmunt Krynicki on 2014-01-28

"automatic merge by tarmac [r=zkrynicki][bug=][author=zkrynicki]"

2620. By Daniel Manrique on 2014-01-28

"checkbox/scripts/audio_settings: Place/read active_profiles in an absolute, known-writable location. [r=zkrynicki][bug=1271707][author=roadmr]"

2621. By Zygmunt Krynicki on 2014-01-28

Incremented changelog

2622. By Sylvain Pineau on 2014-01-28

"automatic merge by tarmac [r=roadmr][bug=1273497][author=sylvain-pineau]"

2623. By Brendan Donegan on 2014-01-29

"automatic merge by tarmac [r=zkrynicki][bug=][author=brendan-donegan]"

2624. By Sylvain Pineau on 2014-01-29

"automatic merge by tarmac [r=sylvain-pineau][bug=1274152][author=sylvain-pineau]"

2625. By Zygmunt Krynicki on 2014-01-29

"automatic merge by tarmac [r=brendan-donegan][bug=][author=zkrynicki]"

2626. By Sylvain Pineau on 2014-01-30

"automatic merge by tarmac [r=sylvain-pineau][bug=][author=brendan-donegan]"

2627. By Zygmunt Krynicki on 2014-01-30

"automatic merge by tarmac [r=zkrynicki][bug=1236377,1236505,1243610,1254189,1254191,1255066,1255085,1265748,1267794,1273100][author=zkrynicki]"

2628. By Zygmunt Krynicki on 2014-01-30

"automatic merge by tarmac [r=zkrynicki][bug=][author=zkrynicki]"

2629. By Zygmunt Krynicki on 2014-01-30

"automatic merge by tarmac [r=brendan-donegan][bug=][author=zkrynicki]"

2630. By Zygmunt Krynicki on 2014-01-30

"automatic merge by tarmac [r=brendan-donegan][bug=][author=zkrynicki]"

2631. By Daniel Manrique on 2014-01-30

"scripts/network and scripts/virtualization can now be configured via environment variables, in addition to the existing config file mechanism and command-line parameters.

This is most useful with canonical-certification-server or other plainbox-based testing tools (i.e. in the absence of checkbox proper). In this case all configuration is centralized in the plainbox config file (/etc/xdg/plainbox.conf). These variables can be defined in the [environment] section with the familiar config file syntax.

For virtualization, the KVM_IMAGE and KVM_TIMEOUT variables are picked up.

For network, it's TEST_TARGET_FTP, TEST_TARGET_IPERF, TEST_USER and TEST_PASS. Separate variables are used for iperf and ftp to mimic the existing config file behavior (but note that a single --target command-line parameter is used in all cases).

Across the board, the preference order is command line -> environment -> config file. [r=zkrynicki][bug=1260507][author=roadmr]"

2632. By Zygmunt Krynicki on 2014-01-30

"automatic merge by tarmac [r=roadmr][bug=1236377,1236505,1243610,1254189,1254191,1255066,1255085,1265748,1267794,1273100][author=zkrynicki]"

2633. By Zygmunt Krynicki on 2014-01-30

"automatic merge by tarmac [r=roadmr][bug=1213893][author=zkrynicki]"

2634. By Sylvain Pineau on 2014-01-30

"automatic merge by tarmac [r=zkrynicki][bug=][author=sylvain-pineau]"

2635. By Launchpad Translations on behalf of checkbox-dev on 2014-02-01

Launchpad automatic translations update.

2636. By Zygmunt Krynicki on 2014-02-03

"automatic merge by tarmac [r=sylvain-pineau,apulido][bug=][author=zkrynicki]"

2637. By Zygmunt Krynicki on 2014-02-03

"automatic merge by tarmac [r=roadmr][bug=][author=zkrynicki]"

2638. By Zygmunt Krynicki on 2014-02-03

"automatic merge by tarmac [r=roadmr][bug=][author=zkrynicki]"

2639. By Brendan Donegan on 2014-02-03

"automatic merge by tarmac [r=][bug=][author=brendan-donegan]"

2640. By Zygmunt Krynicki on 2014-02-04

"automatic merge by tarmac [r=zkrynicki][bug=][author=zkrynicki]"

2641. By Daniel Manrique on 2014-02-04

"Removes the packaging directory from checkbox-old/debian. Changelog is relocated to checkbox-old/CHANGELOG for historical and procedural reasons.

 [r=zkrynicki][bug=][author=roadmr]"

2642. By Daniel Manrique on 2014-02-04

"Relocate the man page from checkbox.1 to checkbox-invocation.1. [r=zkrynicki][bug=][author=roadmr]"

2643. By Sylvain Pineau on 2014-02-04

"automatic merge by tarmac [r=zkrynicki][bug=][author=sylvain-pineau]"

2644. By Zygmunt Krynicki on 2014-02-04

"automatic merge by tarmac [r=roadmr][bug=][author=zkrynicki]"

2645. By Brendan Donegan on 2014-02-04

"automatic merge by tarmac [r=roadmr,zkrynicki][bug=][author=brendan-donegan]"

2646. By Daniel Manrique on 2014-02-04

"Fixes incorrect determination of whether to download a cloud image.

If no image is specified, the script downloads one. With recent changes, the "no image specified" value changed from None to maybe an empty string. Just using "not self.image" instead of the more strict "self.image is None" works. [r=zkrynicki][bug=][author=roadmr]"

2647. By Zygmunt Krynicki on 2014-02-05

"automatic merge by tarmac [r=sylvain-pineau][bug=1236377,1236505,1243610,1254189,1254191,1255066,1255085,1265748,1267794,1273100][author=zkrynicki]"

2648. By Sylvain Pineau on 2014-02-06

"automatic merge by tarmac [r=sylvain-pineau][bug=][author=zkrynicki]"

2649. By Daniel Manrique on 2014-02-06

"jobs/disk.txt.in: Remove max_diskspace* tests [r=zkrynicki][bug=][author=roadmr]"

2650. By Brendan Donegan on 2014-02-06

"automatic merge by tarmac [r=sylvain-pineau][bug=][author=brendan-donegan]"

2651. By Brendan Donegan on 2014-02-06

"automatic merge by tarmac [r=zkrynicki][bug=1270286][author=brendan-donegan]"

2652. By Brendan Donegan on 2014-02-06

"automatic merge by tarmac [r=zkrynicki][bug=][author=brendan-donegan]"

2653. By Ara Pulido on 2014-02-06

"automatic merge by tarmac [r=roadmr][bug=][author=apulido]"

2654. By Zygmunt Krynicki on 2014-02-06

"automatic merge by tarmac [r=roadmr][bug=][author=zkrynicki]"

2655. By Brendan Donegan on 2014-02-07

"automatic merge by tarmac [r=zkrynicki][bug=][author=brendan-donegan]"

2656. By Sylvain Pineau on 2014-02-07

"automatic merge by tarmac [r=zkrynicki][bug=][author=sylvain-pineau]"

2657. By Sylvain Pineau on 2014-02-07

"automatic merge by tarmac [r=zkrynicki][bug=][author=sylvain-pineau]"

2658. By Sylvain Pineau on 2014-02-07

"automatic merge by tarmac [r=zkrynicki][bug=][author=sylvain-pineau]"

2659. By Zygmunt Krynicki on 2014-02-07

"automatic merge by tarmac [r=zkrynicki][bug=][author=checkbox-dev]"

2660. By Zygmunt Krynicki on 2014-02-07

"automatic merge by tarmac [r=roadmr][bug=][author=zkrynicki]"

2661. By Launchpad Translations on behalf of checkbox-dev on 2014-02-11

Launchpad automatic translations update.

2662. By Zygmunt Krynicki on 2014-02-11

"automatic merge by tarmac [r=brendan-donegan][bug=][author=zkrynicki]"

2663. By Jeff Lane  on 2014-02-11

"automatic merge by tarmac [r=roadmr][bug=][author=bladernr]"

2664. By Zygmunt Krynicki on 2014-02-12

"automatic merge by tarmac [r=roadmr][bug=][author=zkrynicki]"

2665. By Po-Hsu Lin on 2014-02-12

"automatic merge by tarmac [r=zkrynicki][bug=1279222][author=cypressyew]"

2666. By Brendan Donegan on 2014-02-12

"automatic merge by tarmac [r=zkrynicki][bug=][author=brendan-donegan]"

2667. By Brendan Donegan on 2014-02-12

"automatic merge by tarmac [r=roadmr][bug=][author=brendan-donegan]"

2668. By Sylvain Pineau on 2014-02-13

"automatic merge by tarmac [r=zkrynicki][bug=][author=sylvain-pineau]"

2669. By Jeff Lane  on 2014-02-13

fixed changelog conflict and also changed fail_speed to fail_threshold to be more clear about the intention

2670. By Jeff Lane  on 2014-02-13

missed the fail-speed reference in the job description for multi-nic

[Jeff Lane  (bladernr)]

Fixed the changelog conflict (and it'll probably happen yet again depending on when this lands). Also per my discussion with Daniel, changed fail_speed to fail_threshold which I agree is more clear about the intent of the opton.

Also just pushed a second update beause I missed a reference to fail-speed in the multi-nic job description originally.

review: Needs Resubmitting

[Daniel Manrique (roadmr)]

Thanks! Looks OK, let's merge this.

review: Approve

=== modified file 'checkbox-old/CHANGELOG'
--- checkbox-old/CHANGELOG	2014-02-12 10:38:09 +0000
+++ checkbox-old/CHANGELOG	2014-02-13 18:51:18 +0000
@@ -5,6 +5,9 @@
     12.04.4 (LP: #1279222) 
 
   [ Jeff Lane ]
+  * scripts/network: fail threshold for iperf is now an option and defaults to
+    40%
+  * jobs/ethernet.txt.in: modified multinic to fail on 80% threshold
   * Removed server-cert.whitelist.  Forked server-selftest.whitelist to have a
     new 14.04 only version for Trusty testing. This is the foundation for 14.04
     server cert changes.

=== modified file 'checkbox-old/jobs/ethernet.txt.in'
--- checkbox-old/jobs/ethernet.txt.in	2014-01-24 18:13:09 +0000
+++ checkbox-old/jobs/ethernet.txt.in	2014-02-13 18:51:18 +0000
@@ -45,7 +45,7 @@
   package.name == 'nmap'
   device.path == "$1"
  user: root
- command: network test -i $2 -t iperf
+ command: network test -i $2 -t iperf --fail-threshold 80
  estimated_duration: 330.0
  description:
   Testing for NIC $2

=== modified file 'checkbox-old/scripts/network'
--- checkbox-old/scripts/network	2014-01-30 14:27:28 +0000
+++ checkbox-old/scripts/network	2014-02-13 18:51:18 +0000
@@ -54,6 +54,7 @@
             self,
             interface,
             target,
+            fail_threshold,
             protocol="tcp",
             mbytes="1024M"):
 
@@ -113,7 +114,7 @@
             print("%3.2f%% of " % percent, end="")
             print("theoretical max %sMb/s\n" % int(self.iface.max_speed))
 
-            if percent < 80:
+            if percent < self.fail_threshold:
                 logging.warn("Poor network performance detected")
                 return 30
 
@@ -482,7 +483,8 @@
             result = ftp_benchmark.run()
 
         elif args.test_type.lower() == "iperf":
-            iperf_benchmark = IPerfPerformanceTest(args.interface, test_target)
+            iperf_benchmark = IPerfPerformanceTest(args.interface, test_target, 
+                                                   args.fail_threshold)
             result = iperf_benchmark.run()
 
         elif args.test_type.lower() == "stress":
@@ -598,6 +600,12 @@
         '--config', type=str,
         default="/etc/checkbox.d/network.cfg",
         help="Supply config file for target/host network parameters")
+    test_parser.add_argument(
+        '--fail-threshold', type=int,
+        default=40,
+        help=("IPERF Test ONLY. Set the failure threshold (Percent of maximum "
+              "theoretical bandwidth) as a number like 80.  (Default is "
+              "%(default)s)"))
 
     # Sub info options
     info_parser.add_argument(