Merge ~gavin.lin/plainbox-provider-snappy:wifi-client-np into plainbox-provider-snappy:master

[Gavin Lin]

Add automated wifi client test using netplan, this is the updated version.

[Maciej Kisielewski (kissiel)]

One big problem is a potential loss of config due to imperative moves on /etc/netplan.

I recommend using ContextManagers to guarantee proper rollback.

As for things that are not dangerous, but could be improved:
ping and connection tests should be functions returning a boolean indicating success, this way nested if/elses would disappear, as only return test_ping() would suffice.

Instead of using shell utilities I recommend using builtin shutil module. It's cleaner, less error prone and more informative when something goes wrong.

Also see inlines.

review: Needs Fixing

[Gavin Lin (gavin.lin)]

Updated according to Maciej's comments, thanks a lot.

Separated the copy part when backing up configuration file, so if anything goes wrong and unrecoverable, at least the backup is still there.

Also replaced all file manipulation with shutil module, and try to recover when things not going as expected.

Configuration file template and ping test are also updated.

[Maciej Kisielewski (kissiel)]

Personally I would find it much easier to work with if the wifi_client_test_netplan program had some structure. But this, I guess, can wait for the next rewrite.

Minor typo inline + PEP8 issues. The top-level comment in the script is misfit.
If you've got a minute run flake8 on the script to make it clean. Provisory +1

review: Approve

[Gavin Lin (gavin.lin)]

I'll update the script base on comments later, thanks a lot for pointing them out!

Unmerged commits

4cc9630... by Gavin Lin on 2017-06-07

Add automated wifi client test using netplan

diff --git a/bin/wifi_client_test_netplan b/bin/wifi_client_test_netplan
new file mode 100755
index 0000000..86b6f01
--- /dev/null
+++ b/bin/wifi_client_test_netplan
@@ -0,0 +1,172 @@
+#!/usr/bin/env python3
+# Copyright 2018 Canonical Ltd.
+# All rights reserved.
+#
+# Written by:
+#    Gavin Lin <[e-mail]>
+
+"""
+This script will test Wi-Fi device with netplan automatically.
+
+To see how to use, please run "./wifi_client_test_netplan --help"
+"""
+
+import argparse
+import os
+import subprocess
+import sys
+import time
+import shutil
+
+
+# Configuration file path
+if "SNAP_USER_DATA" in os.environ:
+    TMP_PATH = os.path.join(os.environ["SNAP_USER_DATA"],
+                            "WIFI-TEST-NETPLAN-CREATED-BY-CHECKBOX")
+else:
+    TMP_PATH = os.path.join("/tmp", "WIFI-TEST-NETPLAN-CREATED-BY-CHECKBOX")
+CFG_PATH = "/etc/netplan"
+CFG_FILE = "/etc/netplan/99-CREATED-BY-CHECKBOX.yaml"
+
+# Read arguments
+parser = argparse.ArgumentParser(
+    description=("This script will test wireless network with netplan"
+                 " in client mode."))
+parser.add_argument(
+    "-i", "--interface", type=str, help=(
+        "The interface which will be tested, default is wlan0"),
+    default="wlan0")
+parser.add_argument(
+    "-s", "--ssid", type=str, help=(
+        "SSID of target network, this is required argument"),
+    required=True)
+parser.add_argument(
+    "-k", "--psk", type=str, help=(
+        "Pre-shared key of target network, this is optional argument,"
+        " only for PSK protected network"))
+parser.add_argument(
+    "-d", "--dhcp", action='store_true', help=(
+        "Enable DHCP for IPv4"),
+    default=False)
+parser.add_argument(
+    "-a", "--address", type=str, help=(
+        "Set ip address for the test interface, optional argument,"
+        " example: 192.168.1.1/24"),
+    default="")
+parser.add_argument(
+    "-p", "--ping", action='store_true', help=(
+        "Enable ping test, will only check link status if not enabled"),
+    default=False)
+parser.add_argument(
+    "-v", "--server", type=str, help=(
+        "Set ip of server which respond to ping, default is default gateway,"
+        " example: 192.168.1.2"),
+    default="")
+args = parser.parse_args()
+
+# Check if executed with root
+if os.geteuid() != 0:
+    raise SystemExit("Error: please run this command as root")
+
+# Backup current network configuration file
+shutil.rmtree(TMP_PATH, ignore_errors=True)
+try:
+    shutil.copytree(CFG_PATH, TMP_PATH, symlinks=True)
+except:
+    shutil.rmtree(TMP_PATH, ignore_errors=True)
+    raise SystemExit("Failed to backup network config files, exiting...")
+
+# Remove configuration file, since backup created successfully,
+# ignore exception while deleting
+for item in os.listdir(CFG_PATH):
+    target_path = os.path.join(CFG_PATH, item)
+    if os.path.isfile(target_path):
+        try:
+            os.remove(target_path)
+        except:
+            pass
+    elif os.path.isdir(target_path):
+        shutil.rmtree(target_path, ignore_errors=True)
+
+# If there's any file left in configuration folder,
+# then there's something not excepted, stop the test
+if os.listdir(CFG_PATH):
+    print("Failed to manipulate network config file")
+    print("Restoring network config file from backup...")
+    try:
+        for item in os.listdir(TMP_PATH):
+            target_path = os.path.join(TMP_PATH, item)
+            shutil.move(target_path, CFG_PATH)
+    except:
+        raise SystemExit("Failed to restore network config file from " +
+                         TMP_PATH + ", please check it manually")
+    shutil.rmtree(TMP_PATH, ignore_errors=True)
+    raise SystemExit("Configutation file restored, exiting...")
+
+# Create wireless network test configuration file
+if args.psk:
+    pass_field = "password: " + args.psk
+else:
+    pass_field = ""
+
+# Template
+np_cfg = """# This is the network config written by checkbox
+network:
+  version: 2
+  wifis:
+    {interface}:
+      access-points:
+        {ssid}: {{{password}}}
+      addresses: [{address}]
+      dhcp4: {dhcp}
+      nameservers: {{}}"""
+
+# Fill data and write to file
+cfg_data = {"interface": args.interface, "ssid": args.ssid,
+            "password": pass_field, "address": args.address,
+            "dhcp": str(args.dhcp)}
+with open(CFG_FILE, "w", encoding="utf-8") as wifi_cfg_file:
+    wifi_cfg_file.write(np_cfg.format(**cfg_data))
+
+# Bring up the interface
+subprocess.call(["netplan", "apply"])
+time.sleep(10)
+
+# Check connection by ping or link status
+server_ip = ""
+if args.ping:
+    # Get gateway ip for ping test
+    if args.server == "":
+        route_table = subprocess.check_output(["route", "-n"]).decode()
+        for line in route_table.split("\n"):
+            if args.interface in line and "UG" in line:
+                server_ip = line.split()[1]
+    else:
+        server_ip = args.server
+    # Test connection by ping
+    exit_code = subprocess.call([
+        "ping", "-c", "5", "-I", args.interface, server_ip])
+else:
+    # Test connection by checking if link established
+    test_output = subprocess.check_output([
+        "iw", args.interface, "link"]).decode()
+    if test_output.find("SSID") != -1:
+        print("Connection test passed")
+        exit_code = 0
+    else:
+        print("Connection test failed")
+        exit_code = 1
+
+# Restore configuration file and apply
+os.remove(CFG_FILE)
+try:
+    for item in os.listdir(TMP_PATH):
+        target_path = os.path.join(TMP_PATH, item)
+        shutil.move(target_path, CFG_PATH)
+except:
+    raise SystemExit("Failed to restore network config file from " +
+                     TMP_PATH + ", please check it manually")
+shutil.rmtree(TMP_PATH, ignore_errors=True)
+subprocess.call(["netplan", "apply"])
+time.sleep(10)
+sys.exit(exit_code)
diff --git a/units/wireless/jobs.pxu b/units/wireless/jobs.pxu
index dded027..ac8b4da 100644
--- a/units/wireless/jobs.pxu
+++ b/units/wireless/jobs.pxu
@@ -130,6 +130,120 @@ flags: preserve-locale also-after-suspend
 
 unit: template
 template-resource: device
+template-filter: device.category == 'WIRELESS'
+template-engine: jinja2
+template-unit: job
+id: wireless/wireless_connection_open_ac_np_{{ interface }}
+_summary:
+ Connect to unencrypted 802.11ac Wi-Fi network on {{ interface }} - netplan
+_purpose:
+ Check system can connect to insecure 802.11ac AP using netplan
+plugin: shell
+command:
+    wifi_client_test_netplan -i {{ interface }} -s $OPEN_AC_SSID -dp
+user: root
+environ: LD_LIBRARY_PATH OPEN_AC_SSID
+category_id: wifi
+estimated_duration: 15
+flags: preserve-locale also-after-suspend
+
+unit: template
+template-resource: device
+template-filter: device.category == 'WIRELESS'
+template-engine: jinja2
+template-unit: job
+id: wireless/wireless_connection_open_bg_np_{{ interface }}
+_summary:
+ Connect to unencrypted 802.11b/g Wi-Fi network on {{ interface }} - netplan
+_purpose:
+ Check system can connect to insecure 802.11b/g AP using netplan
+plugin: shell
+command:
+    wifi_client_test_netplan -i {{ interface }} -s $OPEN_BG_SSID -dp
+user: root
+environ: LD_LIBRARY_PATH OPEN_BG_SSID
+category_id: wifi
+estimated_duration: 15
+flags: preserve-locale also-after-suspend
+
+unit: template
+template-resource: device
+template-filter: device.category == 'WIRELESS'
+template-engine: jinja2
+template-unit: job
+id: wireless/wireless_connection_open_n_np_{{ interface }}
+_summary:
+ Connect to unencrypted 802.11n Wi-Fi network on {{ interface }} - netplan
+_purpose:
+ Check system can connect to insecure 802.11n AP using netplan
+plugin: shell
+command:
+    wifi_client_test_netplan -i {{ interface }} -s $OPEN_N_SSID -dp
+user: root
+environ: LD_LIBRARY_PATH OPEN_N_SSID
+category_id: wifi
+estimated_duration: 15
+flags: preserve-locale also-after-suspend
+
+unit: template
+template-resource: device
+template-filter: device.category == 'WIRELESS'
+template-engine: jinja2
+template-unit: job
+id: wireless/wireless_connection_wpa_ac_np_{{ interface }}
+_summary:
+ Connect to WPA-encrypted 802.11ac Wi-Fi network on {{ interface }} - netplan
+_purpose:
+ Check system can connect to 802.11ac AP with wpa security using netplan
+plugin: shell
+command:
+    wifi_client_test_netplan -i {{ interface }} -s $WPA_AC_SSID -k $WPA_AC_PSK -dp
+user: root
+environ: LD_LIBRARY_PATH WPA_AC_SSID WPA_AC_PSK
+category_id: wifi
+estimated_duration: 15
+flags: preserve-locale also-after-suspend
+
+unit: template
+template-resource: device
+template-filter: device.category == 'WIRELESS'
+template-engine: jinja2
+template-unit: job
+id: wireless/wireless_connection_wpa_bg_np_{{ interface }}
+_summary:
+ Connect to WPA-encrypted 802.11b/g Wi-Fi network on {{ interface }} - netplan
+_purpose:
+ Check system can connect to 802.11b/g AP with wpa security using netplan
+plugin: shell
+command:
+    wifi_client_test_netplan -i {{ interface }} -s $WPA_BG_SSID -k $WPA_BG_PSK -dp
+user: root
+environ: LD_LIBRARY_PATH WPA_BG_SSID WPA_BG_PSK
+category_id: wifi
+estimated_duration: 15
+flags: preserve-locale also-after-suspend
+
+unit: template
+template-resource: device
+template-filter: device.category == 'WIRELESS'
+template-engine: jinja2
+template-unit: job
+id: wireless/wireless_connection_wpa_n_np_{{ interface }}
+_summary:
+ Connect to WPA-encrypted 802.11n Wi-Fi network on {{ interface }} - netplan
+_purpose:
+ Check system can connect to 802.11n AP with wpa security using netplan
+plugin: shell
+command:
+    wifi_client_test_netplan -i {{ interface }} -s $WPA_N_SSID -k $WPA_N_PSK -dp
+user: root
+environ: LD_LIBRARY_PATH WPA_N_SSID WPA_N_PSK
+category_id: wifi
+estimated_duration: 15
+flags: preserve-locale also-after-suspend
+
+unit: template
+template-resource: device
 template-filter: device.category == 'WIRELESS' and device.mac != 'UNKNOWN'
 id: wireless/wowlan_S5_{interface}_wakeonlan
 _summary: Wake on Wireless LAN (WoWLAN) test from S5 - {interface} - wakeonlan
diff --git a/units/wireless/test-plan.pxu b/units/wireless/test-plan.pxu
index 0650f97..f7817ce 100644
--- a/units/wireless/test-plan.pxu
+++ b/units/wireless/test-plan.pxu
@@ -40,6 +40,22 @@ include:
 bootstrap_include:
     device
 
+id: wireless-netplan-automated
+unit: test plan
+_name: Automated tests for wireless using netplan
+_description:
+ Automated connection tests for unencrypted or WPA-encrypted 802.11 bg, n, ac
+ networks using netplan.
+include:
+    wireless/wireless_connection_open_ac_np_.*
+    wireless/wireless_connection_open_bg_np_.*
+    wireless/wireless_connection_open_n_np_.*
+    wireless/wireless_connection_wpa_ac_np_.*
+    wireless/wireless_connection_wpa_bg_np_.*
+    wireless/wireless_connection_wpa_n_np_.*
+bootstrap_include:
+    device
+
 id: wireless-wifi-master-mode
 unit: test plan
 _name: QA tests for wifi master mode
@@ -124,6 +140,22 @@ include:
 bootstrap_include:
     device
 
+id: after-suspnend-wireless-netplan-automated
+unit: test plan
+_name: Automated tests for wireless using netplan (after suspend)
+_description:
+ Automated connection tests for unencrypted or WPA-encrypted 802.11 bg, n, ac
+ networks using netplan.
+include:
+    after-suspend-wireless/wireless_connection_open_ac_np_.*
+    after-suspend-wireless/wireless_connection_open_bg_np_.*
+    after-suspend-wireless/wireless_connection_open_n_np_.*
+    after-suspend-wireless/wireless_connection_wpa_ac_np_.*
+    after-suspend-wireless/wireless_connection_wpa_bg_np_.*
+    after-suspend-wireless/wireless_connection_wpa_n_np_.*
+bootstrap_include:
+    device
+
 id: after-suspend-wireless-wifi-master-mode
 unit: test plan
 _name: QA tests for wifi master mode (after suspend)