Merge ~djacobs98/checkbox-iiotg/+git/checkbox-provider-intliotg-3:tsn-refactor into ~checkbox-dev/checkbox-iiotg/+git/checkbox-provider-intliotg:master

[Doug Jacobs]

Change. Refactor the TSN tests to refer to the test document for the commands.
https://docs.google.com/document/d/14fNcjUzxh6H-mWK8tXV8HoUxrgFirOhDksUZfd44OTY/edit#heading=h.v0ljmsfxoxqo

The old method just created tests with too many instructions making it hard to read. This should be easier for the tester. Each test links to the section of the document for that test case.

[Doug Jacobs (djacobs98)]

Pastebin showing output from checkbox running refactored test cases:
https://pastebin.canonical.com/p/dWfVb9nWCq/

[Vic Liu (zongminl)]

There was a "TSN setup for Checkbox" document mentioned in steps field from the previous version but now is missing, is this no longer required?

review: Needs Information

[Vic Liu (zongminl)]

There was a "TSN setup for Checkbox" document mentioned in steps field from the previous version but now is missing, is this no longer required?

review: Needs Information

[Doug Jacobs (djacobs98)]

> There was a "TSN setup for Checkbox" document mentioned in steps field from
> the previous version but now is missing, is this no longer required?

The setup document pointed to the setup chapter of the Test Plan.

I've incorporated (copied) this information into the Test Instructions document for simplicity.

[Vic Liu (zongminl)]

Then it's good enough, thank you

review: Approve

diff --git a/units/tsn/jobs.pxu b/units/tsn/jobs.pxu
index f454d6f..701398b 100644
--- a/units/tsn/jobs.pxu
+++ b/units/tsn/jobs.pxu
@@ -77,24 +77,8 @@ _purpose:
 depends: tsn/enabled
 plugin: manual
 _steps:
-    Before performing the test steps, please follow the instructions in the
-    configuration document at:
-    https://docs.google.com/document/d/1cN34NR4mZm43-43lj9J93etmL7mjn2dIoWyuHiYqFlQ/edit?usp=share_link
-    Additional SW required: ptp4l (provided by linuxptp)
-    1. Connect an Ethernet cable from the Secondary System's TSN port directly
-       to the {eth-interface} Ethernet interface of the SUT.
-    2. Copy the gPTP.cfg file to user u's home directory on the SUT and Secondary.
-    3. On the secondary, execute the command:
-       sudo ptp4l -P2Hi <secondary-inerface> -f ~/gPTP.cfg --step_threshold=1 -m
-    3. Wait for the system to announce it is the Grand Master.
-    4. run the same command on the SUT.
-    5. The SUT should announce it has selected a foreign master (the Secondary
-       system)
-    6. Clock readings will appear on both the Secondary and SUT.  When the rms
-       value remains below 100, the SUT has successfully synchronized to the clock
-       on the Secondary system.
-    7. Leave the clock windows running and open - they will be required in futher
-       test cases.
+    Follow the instructions as described here:
+    https://docs.google.com/document/d/14fNcjUzxh6H-mWK8tXV8HoUxrgFirOhDksUZfd44OTY/edit#heading=h.pzr6mq4dyb0u
 
 unit: template
 template-resource: tsn/tsn-devices
@@ -108,35 +92,8 @@ _purpose:
 depends: tsn/enabled
 plugin: manual
 _steps:
-    Before performing the test steps, please follow the instructions in the
-    configuration document at:
-    https://docs.google.com/document/d/1cN34NR4mZm43-43lj9J93etmL7mjn2dIoWyuHiYqFlQ/edit?usp=share_link
-    Leave the gPTP clock windows open and execute additional commands in a
-    separate window.
-    1. Execute the following command on the Secondary, then the SUT:
-       $ sudo pmc -u -b 0 -t 1 "SET GRANDMASTER_SETTINGS_NP clockClass 248\
-       clockAccuracy 0xfe offsetScaledLogVariance 0xffff currentUtcOffset 37\
-       leap61 0 leap59 0 currentUtcOffsetValid 1 ptpTimescale 1 timeTraceable 1\
-       frequencyTraceable 0 timeSource 0xa0"
-    2. Both the Secondary and SUT should display the settings for its
-       GRANDMASTER_SETTINGS_NP
-    3. Execute the following command on the Secondary, then the SUT:
-        $ phc2sys -s <interface> -O 0 -c CLOCK_REALTIME --step_threshold=1\
-          kk--transportSpecific=1 -w -m --first_step_threshold=0.0
-    4. Both systems will display CLOCK_REALTIME data in 4 columns:
-       phc offset : when this is +/- 100ns the offset between the PHC and the local
-       system clock are in sync.
-       \
-       State : This should transition from S0 to S2.  S2 means the system is
-       tracking the Primary Clock.
-       \
-       Frequency Offset : (in parts per billion) This value does not affect this
-       test.
-       \
-       Path Delay : This should be 0, indicating the device is using hardware
-       cross-timestamping.
-       \
-       When all of these conditions are met, this test has passed.
+    Follow the instructions as described here:
+    https://docs.google.com/document/d/14fNcjUzxh6H-mWK8tXV8HoUxrgFirOhDksUZfd44OTY/edit#heading=h.dxubm4m7zzag
 
 unit: template
 template-resource: tsn/tsn-devices
@@ -150,66 +107,8 @@ _purpose:
 depends: tsn/enabled
 plugin: manual
 _steps:
-    Before performing the test steps, please follow the instructions in the
-    configuration document at:
-    https://docs.google.com/document/d/1cN34NR4mZm43-43lj9J93etmL7mjn2dIoWyuHiYqFlQ/edit?usp=share_link
-    Leave the gPTP clock windows open and execute additional commands in a
-    separate window.
-    Additional SW required:
-       ptp4l (provided by linuxptp)
-    1. Execute this command on both the SUT and Secondary system to clear any
-       queue configuration:
-       $ sudo tc qdisc del dev <interface> root
-    2. Execute this command on both the SUT and Secondary system to configure a
-       TAPRIO queue discipline:
-        $ BASE=$(date +%s%N)
-        $ echo "$BASE"
-        $ sudo tc qdisc add dev <interface> parent root handle 100 taprio num_tc 4\
-          queues 1@0 1@1 1@2 1@3 base-time $BASE sched-entry S 01 5000000\
-          S 02 5000000 S 04 5000000 S 08 5000000 flags 0x2 txtime-delay 0
-    3. Configure an IP address for the TSN Ethernet interfaces on both the SUT
-       and Secondary systems:
-       SUT (Talker): 
-       $ sudo ip addr add 192.168.1.11/24 dev <interface>
-       $ sudo ip link set dev <interface> up
-       Secondary (Listener): 
-       $ sudo ip addr add 192.168.1.22/24 dev <interface>
-       $ sudo ip link set dev <interface> up
-    4. On the Secondary system, start iperf3 to listen for traffic:
-       $ iperf3 -s -p 5100 &
-       $ iperf3 -s -p 5101 &
-       $ iperf3 -s -p 5102 &
-       $ iperf3 -s -p 5103 &
-    5. The output for each command should indicate iperf3 is listening on the 
-       indicated port.
-    6. Prepare tcpdump to capture test traffic on the Secondary:
-       $ tcpdump -Q in -nvvvXXi <interface> –time-stamp-percision=nano\
-       -j adapter_unsynched -B 524288 -s 54 -w filename.pcap
-    7. On the Secondary, configure cgroups for transmit queues:
-       $ sudo mkdir -p /sys/fs/cgroup/net_prio/grp1
-       $ sudo echo "<interface> 1" > /sys/fs/cgroup/net_prio/grp1/net_prio.ifpriomap
-       \
-       $ sudo mkdir -p /sys/fs/cgroup/net_prio/grp2
-       $ sudo echo "<interface> 2" > /sys/fs/cgroup/net_prio/grp2/net_prio.ifpriomap 
-       \
-       $ sudo mkdir -p /sys/fs/cgroup/net_prio/grp3
-       $ sudo echo "<interface> 2" > /sys/fs/cgroup/net_prio/grp3/net_prio.ifpriomap
-    8. Start transmitting test data from the SUT:
-       $ export TARGET_IP=192.168.1.22
-       \
-       $ iperf3 -u -11000 -b0 -p 5100 -c $TARGET_IP -t 150 >> /dev/null &
-       $ echo $! > /sys/fs/cgroup/net_prio/grp1/cgroup.procs
-       \
-       $ iperf3 -u -11000 -b0 -p 5101 -c $TARGET_IP -t 150 >> /dev/null &
-       $ echo $! > /sys/fs/cgroup/net_prio/grp2/cgroup.procs
-       \
-       $ iperf3 -u -11000 -b0 -p 5102 -c $TARGET_IP -t 150 >> /dev/null &
-       $ echo $! > /sys/fs/cgroup/net_prio/grp3/cgroup.procs
-    9. On the SUT, watch the queues for transmitted data:
-       $ sudo watch -d -n 0.5 "tc -s qdisc show dev <interface>"
-       \
-       The byte and packet count for 100:1 - 100:4 each increase for half a second
-       before the next queue increases.
+    Follow the instructions as described here:
+    https://docs.google.com/document/d/14fNcjUzxh6H-mWK8tXV8HoUxrgFirOhDksUZfd44OTY/edit#heading=h.2mpce0amhrf9
 
 unit: template
 template-resource: tsn/tsn-devices
@@ -223,32 +122,8 @@ _purpose:
 depends: tsn/enabled
 plugin: manual
 _steps:
-    Before performing the test steps, please follow the instructions in the
-    configuration document at:
-    https://docs.google.com/document/d/1cN34NR4mZm43-43lj9J93etmL7mjn2dIoWyuHiYqFlQ/edit?usp=share_link
-    Leave the gPTP clock windows open and execute additional commands in a
-    separate window.
-    Additional software required: udp_tai (part of sw_tsn_tools)
-    1. Clear any queue discipline (qdisc) configuration on both the SUT and 
-       Secondary's TSN interface by running this command on both systems:
-       $ sudo qdisc del dev <interface> root
-    2. Configure Earliest Transmit-Time First (ETF) Queue Discipline on both 
-       systems:
-       $ sudo tc qdisc add dev <interface> handle 8001: parent root mqprio num_tc 4\
-       map 0 1 2 3 3 3 3 3 3 3 3 3 3 3 3 3 queues 1@0 1@1 1@2 1@3 hw 0
-       $ sudo tc replace dev <interface> parent 8001:4 etf offload clockid CLOCK_TAI delta 500000
-    3. Verify the qdisc is configured properly on the SUT and Secondary:
-       $ sudo tc qdisc show dev <interface> 
-       Should see that ETF qdisc is configured on 8001:4 with a delta of 500000 on both systems
-    4. Start transmitting data from the SUT every 1ms (1000000ns) starting 6ms in the future.
-       Capture the packets on the Secondary system.
-       (SUT)
-       $ sudo ./udp_tai -c 3 -i <interface> -P 1000000 -p 90 -d 600000
-       (Secondary)
-       $ sudo tcpdump -Q in -ttt -ni <interface> --time-stamp-precision=nano -j adapter_unsynched port 7788 -c 20000
-    5. Verification:
-       SUT: You should see packets being transmitted.
-       Secondary: >95% of packets should arrive every 1000000ns (+/-500ns) 
+    Follow the instructions as described here:
+    https://docs.google.com/document/d/14fNcjUzxh6H-mWK8tXV8HoUxrgFirOhDksUZfd44OTY/edit#heading=h.2m1zswmirdvb
 
 unit: template
 template-resource: tsn/tsn-devices
@@ -261,19 +136,8 @@ _purpose:
 depends: tsn/enabled
 plugin: manual
 _steps:
-    Before performing the test steps, please follow the instructions in the
-    configuration document at:
-    https://docs.google.com/document/d/1cN34NR4mZm43-43lj9J93etmL7mjn2dIoWyuHiYqFlQ/edit?usp=share_link
-    This test does not require clock synchronization.
-    NOTE: ethtool 5.16 does not support the --set-frame-preemption feature. 
-          You must download a later version from: 
-          https://git.kernel.org/pub/scm/network/ethtool/ethtool.git
-          and compile it yourself.
-    1. Enable FPE on the DUT:
-       $ sudo ethtool --set-frame-preemption <interface> fp on
-    2. Verify FPE is enabled:
-       $ sudo ethtool --show-frame-preemption <interface>
-       The command should show that frame-preemtpion is enabled.
+    Follow the instructions as described here:
+    https://docs.google.com/document/d/14fNcjUzxh6H-mWK8tXV8HoUxrgFirOhDksUZfd44OTY/edit#heading=h.ge4dauxlinm3
 
 unit: template
 template-resource: tsn/tsn-devices
@@ -287,23 +151,8 @@ _purpose:
 depends: tsn/enabled
 plugin: manual
 _steps:
-    Before performing the test steps, please follow the instructions in the
-    configuration document at:
-    https://docs.google.com/document/d/1cN34NR4mZm43-43lj9J93etmL7mjn2dIoWyuHiYqFlQ/edit?usp=share_link
-    Additional SW required: ptp4l (provided by linuxptp)
-    1. Connect an Ethernet cable from the Secondary System's TSN port directly
-       to the {eth-interface} Ethernet interface of the SUT.
-    2. Copy the gPTP.cfg file to user u's home directory on the SUT and Secondary.
-    3. On the SUT, execute the command:
-       sudo ptp4l -P2Hi <secondary-inerface> -f ~/gPTP.cfg --step_threshold=1 -m
-    3. Wait for the SUT to announce it is the Grand Master.
-    4. run the same command on the Secondary.
-    5. The Secondary should announce it has selected a foreign master (the SUT)
-    6. Clock readings will appear on both the Secondary and SUT.  When the rms
-       value remains below 100, the Secondary has successfully synchronized to the clock
-       on the SUT system.
-    7. Leave the clock windows running and open - they will be required in futher
-       test cases.
+    Follow the instructions as described here:
+    https://docs.google.com/document/d/14fNcjUzxh6H-mWK8tXV8HoUxrgFirOhDksUZfd44OTY/edit#heading=h.15f1v79u3km0
 
 unit: template
 template-resource: tsn/tsn-devices
@@ -316,35 +165,8 @@ _purpose:
 depends: tsn/enabled
 plugin: manual
 _steps:
-    Before performing the test steps, please follow the instructions in the
-    configuration document at:
-    https://docs.google.com/document/d/1cN34NR4mZm43-43lj9J93etmL7mjn2dIoWyuHiYqFlQ/edit?usp=share_link
-    Leave the gPTP clock windows open and execute additional commands in a
-    separate window.
-    1. Execute the following command on the SUT, then the Secondary:
-       $ sudo pmc -u -b 0 -t 1 "SET GRANDMASTER_SETTINGS_NP clockClass 248\
-       clockAccuracy 0xfe offsetScaledLogVariance 0xffff currentUtcOffset 37\
-       leap61 0 leap59 0 currentUtcOffsetValid 1 ptpTimescale 1 timeTraceable 1\
-       frequencyTraceable 0 timeSource 0xa0"
-    2. Both the Secondary and SUT should display the settings for its
-       GRANDMASTER_SETTINGS_NP
-    3. Execute the following command on the SUT, then the Secondary:
-        $ phc2sys -s <interface> -O 0 -c CLOCK_REALTIME --step_threshold=1\
-          kk--transportSpecific=1 -w -m --first_step_threshold=0.0
-    4. Both systems will display CLOCK_REALTIME data in 4 columns:
-       phc offset : when this is +/- 100ns the offset between the PHC and the local
-       system clock are in sync.
-       \
-       State : This should transition from S0 to S2.  S2 means the system is
-       tracking the Primary Clock.
-       \
-       Frequency Offset : (in parts per billion) This value does not affect this
-       test.
-       \
-       Path Delay : This should be 0, indicating the device is using hardware
-       cross-timestamping.
-       \
-       When all of these conditions are met, this test has passed.
+    Follow the instructions as described here:
+    https://docs.google.com/document/d/14fNcjUzxh6H-mWK8tXV8HoUxrgFirOhDksUZfd44OTY/edit#heading=h.rb5hx7uq4hcr
 
 unit: template
 template-resource: tsn/tsn-devices
@@ -358,66 +180,8 @@ _purpose:
 depends: tsn/enabled
 plugin: manual
 _steps:
-    Before performing the test steps, please follow the instructions in the
-    configuration document at:
-    https://docs.google.com/document/d/1cN34NR4mZm43-43lj9J93etmL7mjn2dIoWyuHiYqFlQ/edit?usp=share_link
-    Leave the gPTP clock windows open and execute additional commands in a
-    separate window.
-    Additional SW required:
-       ptp4l (provided by linuxptp)
-    1. Execute this command on both the SUT and Secondary system to clear any
-       queue configuration:
-       $ sudo tc qdisc del dev <interface> root
-    2. Execute this command on both the SUT and Secondary system to configure a
-       TAPRIO queue discipline:
-        $ BASE=$(date +%s%N)
-        $ echo "$BASE"
-        $ sudo tc qdisc add dev <interface> parent root handle 100 taprio num_tc 4\
-          queues 1@0 1@1 1@2 1@3 base-time $BASE sched-entry S 01 5000000\
-          S 02 5000000 S 04 5000000 S 08 5000000 flags 0x2 txtime-delay 0
-    3. Configure an IP address for the TSN Ethernet interfaces on both the SUT
-       and Secondary systems:
-       SUT (Listener):
-       $ sudo ip addr add 192.168.1.11/24 dev <interface>
-       $ sudo ip link set dev <interface> up
-       Secondary (Talker):
-       $ sudo ip addr add 192.168.1.22/24 dev <interface>
-       $ sudo ip link set dev <interface> up
-    4. On the SUT, start iperf3 to listen for traffic:
-       $ iperf3 -s -p 5100 &
-       $ iperf3 -s -p 5101 &
-       $ iperf3 -s -p 5102 &
-       $ iperf3 -s -p 5103 &
-    5. The output for each command should indicate iperf3 is listening on the
-       indicated port.
-    6. Prepare tcpdump to capture test traffic on the SUT:
-       $ tcpdump -Q in -nvvvXXi <interface> –time-stamp-percision=nano\
-       -j adapter_unsynched -B 524288 -s 54 -w filename.pcap
-    7. On the SUT, configure cgroups for transmit queues:
-       $ sudo mkdir -p /sys/fs/cgroup/net_prio/grp1
-       $ sudo echo "<interface> 1" > /sys/fs/cgroup/net_prio/grp1/net_prio.ifpriomap
-       \
-       $ sudo mkdir -p /sys/fs/cgroup/net_prio/grp2
-       $ sudo echo "<interface> 2" > /sys/fs/cgroup/net_prio/grp2/net_prio.ifpriomap
-       \
-       $ sudo mkdir -p /sys/fs/cgroup/net_prio/grp3
-       $ sudo echo "<interface> 2" > /sys/fs/cgroup/net_prio/grp3/net_prio.ifpriomap
-    8. Start transmitting test data from the Secondary:
-       $ export TARGET_IP=192.168.1.11
-       \
-       $ iperf3 -u -11000 -b0 -p 5100 -c $TARGET_IP -t 150 >> /dev/null &
-       $ echo $! > /sys/fs/cgroup/net_prio/grp1/cgroup.procs
-       \
-       $ iperf3 -u -11000 -b0 -p 5101 -c $TARGET_IP -t 150 >> /dev/null &
-       $ echo $! > /sys/fs/cgroup/net_prio/grp2/cgroup.procs
-       \
-       $ iperf3 -u -11000 -b0 -p 5102 -c $TARGET_IP -t 150 >> /dev/null &
-       $ echo $! > /sys/fs/cgroup/net_prio/grp3/cgroup.procs
-    9. On the Secondary, watch the queues for transmitted data:
-       $ sudo watch -d -n 0.5 "tc -s qdisc show dev <interface>"
-       \
-       The byte and packet count for 100:1 - 100:4 each increase for half a second
-       before the next queue increases.
+    Follow the instructions as described here:
+    https://docs.google.com/document/d/14fNcjUzxh6H-mWK8tXV8HoUxrgFirOhDksUZfd44OTY/edit#heading=h.9x7mdm368f70
 
 unit: template
 template-resource: tsn/tsn-devices
@@ -431,29 +195,5 @@ _purpose:
 depends: tsn/enabled
 plugin: manual
 _steps:
-    Before performing the test steps, please follow the instructions in the
-    configuration document at:
-    https://docs.google.com/document/d/1cN34NR4mZm43-43lj9J93etmL7mjn2dIoWyuHiYqFlQ/edit?usp=share_link
-    Leave the gPTP clock windows open and execute additional commands in a
-    separate window.
-    Additional software required: udp_tai (part of sw_tsn_tools)
-    1. Clear any queue discipline (qdisc) configuration on both the SUT and
-       Secondary's TSN interface by running this command on both systems:
-       $ sudo qdisc del dev <interface> root
-    2. Configure Earliest Transmit-Time First (ETF) Queue Discipline on both
-       systems:
-       $ sudo tc qdisc add dev <interface> handle 8001: parent root mqprio num_tc 4\
-       map 0 1 2 3 3 3 3 3 3 3 3 3 3 3 3 3 queues 1@0 1@1 1@2 1@3 hw 0
-       $ sudo tc replace dev <interface> parent 8001:4 etf offload clockid CLOCK_TAI delta 500000
-    3. Verify the qdisc is configured properly on the SUT and Secondary:
-       $ sudo tc qdisc show dev <interface>
-       Should see that ETF qdisc is configured on 8001:4 with a delta of 500000 on both systems
-    4. Start transmitting data from the Secondary system every 1ms (1000000ns) starting 6ms in the future.
-       Capture the packets on the SUT.
-       (Secondary)
-       $ sudo ./udp_tai -c 3 -i <interface> -P 1000000 -p 90 -d 600000
-       (SUT)
-       $ sudo tcpdump -Q in -ttt -ni <interface> --time-stamp-precision=nano -j adapter_unsynched port 7788 -c 20000
-    5. Verification:
-       Secondary: You should see packets being transmitted.
-       SUT: >95% of packets should arrive every 1000000ns (+/-500ns)
+    Follow the instructions as described here:
+    https://docs.google.com/document/d/14fNcjUzxh6H-mWK8tXV8HoUxrgFirOhDksUZfd44OTY/edit#heading=h.ozz3peqlqp6g