Checkbox IIoTG

Merge ~liaou3/checkbox-iiotg/+git/checkbox-provider-intliotg:add_tsn_cbd_testplan into ~checkbox-dev/checkbox-iiotg/+git/checkbox-provider-intliotg:master

  1. Git
  2. lp:~liaou3/checkbox-iiotg/+git/checkbox-provider-intliotg
  3. add_tsn_cbd_testplan
  4. Merge into master
Proposed by Vincent Liao
Status: Merged
Approved by: Vincent Liao
Approved revision: e36d83cfdea129b15f34e2d9f7b31c0629402ed9
Merged at revision: 1c34d6ab952cd00a0fd9b89152f6b027e35dcf91
Proposed branch: ~liaou3/checkbox-iiotg/+git/checkbox-provider-intliotg:add_tsn_cbd_testplan
Merge into: ~checkbox-dev/checkbox-iiotg/+git/checkbox-provider-intliotg:master
Diff against target: 325 lines (+108/-37)
2 files modified
units/tsn/jobs.pxu (+95/-37)
units/tsn/test-plan.pxu (+13/-0)
Reviewer Review Type Date Requested Status
StanleyHuang Approve
PeiYao Chang Approve
Rick Wu Pending
Patrick Liu Pending
Vic Liu Pending
Review via email: mp+452898@code.launchpad.net

Commit message

Add ieee802.1qav tsn test plan and cases

Description of the change

We are missing the test cases related to ieee802.1qav credit-based shaper, so I now add those things into the provider.
This test case is the implementation of the TSN document provided by intel [1] on page 33 to 38.

submission: https://certification.canonical.com/hardware/202305-31589/submission/335763/test/203414/result/36610931/

[1]https://drive.google.com/file/d/1McvVL5Xo5CO2fHsqkEV6L6KUTVPcGss9/view

To post a comment you must log in.
Revision history for this message
PeiYao Chang (baconyao) wrote :

LGTM

review: Approve
Revision history for this message
StanleyHuang (stanley31) wrote :

I would suggest we modify this job as a automated tests, please see my inline comments

review: Needs Fixing
Revision history for this message
Vincent Liao (liaou3) wrote :

> I would suggest we modify this job as a automated tests, please see my inline
> comments
@Stanley
I've made test cases automatically, here's the submissions.

Automated: https://certification.canonical.com/hardware/202305-31589/submission/335963/
Manual: https://certification.canonical.com/hardware/202305-31589/submission/335965/

Revision history for this message
Vincent Liao (liaou3) wrote :

submission: https://certification.canonical.com/hardware/202305-31589/submission/335975/

Revision history for this message
StanleyHuang (stanley31) wrote :

LGTM

review: Approve

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1diff --git a/units/tsn/jobs.pxu b/units/tsn/jobs.pxu
2index c8db1dc..96f4ab3 100644
3--- a/units/tsn/jobs.pxu
4+++ b/units/tsn/jobs.pxu
5@@ -49,7 +49,7 @@ command:
6 mystr=$(lspci -nnvv -s "$DEVICE" |grep -i 'precision time measurement')
7 if [ -n "$mystr" ]; then
8 INTERFACE=$(ls -l /sys/class/net |grep "$DEVICE" |cut -d'/' -f8)
9- echo "eth-interface: $INTERFACE"
10+ echo "interface: $INTERFACE"
11 echo ""
12 fi
13 done
14@@ -58,11 +58,11 @@ flags: preserve-locale
15 unit: template
16 template-resource: tsn/tsn-devices
17 template-unit: job
18-id: tsn/verify-PTP-for-{eth-interface}
19+id: tsn/verify-PTP-for-{interface}
20 category_id: tsn
21-_summary: Verify Ethernet interface {eth-interface} supports PTP HW Clock
22+_summary: Verify Ethernet interface {interface} supports PTP HW Clock
23 _description:
24- Check that the Ethernet interface {eth-interface} has a PTP Hardware Clock\
25+ Check that the Ethernet interface {interface} has a PTP Hardware Clock\
26 If it does, it will have a number identifying which clock ID it is using,\
27 and a corresponding entry in /dev/ptp#.\
28 If it does not support PTP, it will list "none", and the test will fail.
29@@ -70,9 +70,9 @@ depends: tsn/tsn-detected
30 plugin: shell
31 user: root
32 command:
33- PTP_ID=$(ethtool -T {eth-interface} |grep "PTP Hardware Clock:" |cut -d' ' -f4)
34+ PTP_ID=$(ethtool -T {interface} |grep "PTP Hardware Clock:" |cut -d' ' -f4)
35 if [ "$PTP_ID" = "none" ]; then
36- echo "Device {eth-interface} does not support PTP."
37+ echo "Device {interface} does not support PTP."
38 exit 1
39 else
40 if [ -e "/dev/ptp$PTP_ID" ]; then
41@@ -87,9 +87,9 @@ command:
42 unit: template
43 template-resource: tsn/tsn-devices
44 template-unit: job
45-id: tsn/test-env-setup-for-{eth-interface}
46+id: tsn/test-env-setup-for-{interface}
47 category_id: tsn
48-_summary: Setup TSN testing environment for Ethernet interface {eth-interface}
49+_summary: Setup TSN testing environment for Ethernet interface {interface}
50 _description:
51 Setup Testing environment for TSN
52 depends: tsn/tsn-detected
53@@ -98,7 +98,7 @@ user: root
54 environ:
55 GPTP_CONFIG_PATH
56 command:
57- tsn_test_env_setup.sh {eth-interface} "${{GPTP_CONFIG_PATH:-/usr/share/gPTP.cfg}}"
58+ tsn_test_env_setup.sh {interface} "${{GPTP_CONFIG_PATH:-/usr/share/gPTP.cfg}}"
59
60 unit: job
61 category_id: tsn
62@@ -119,12 +119,12 @@ command:
63 unit: template
64 template-resource: tsn/tsn-devices
65 template-unit: job
66-id: tsn/ieee802.1as-2011/time-sync-between-two-system-as-client-{eth-interface}
67+id: tsn/ieee802.1as-2011/time-sync-between-two-system-as-client-{interface}
68 category_id: tsn
69 _summary:
70- Check establish time-sync with another machine as client via {eth-interface}
71+ Check establish time-sync with another machine as client via {interface}
72 _purpose:
73- Use {eth-interface} on the SUT to achieve time sync with the second machine as client
74+ Use {interface} on the SUT to achieve time sync with the second machine as client
75 depends:
76 tsn/tsn-detected
77 plugin: user-interact-verify
78@@ -136,19 +136,19 @@ environ:
79 TSN_SERVER_INTERFACE
80 GPTP_CONFIG_PATH
81 command:
82- tsn_time_sync.sh 60 client "${{GPTP_CONFIG_PATH:-/usr/share/gPTP.cfg}}" {eth-interface} "$TSN_SERVER_USER" "$TSN_SERVER_IP" "$TSN_SERVER_PWD" "$TSN_SERVER_INTERFACE"
83+ tsn_time_sync.sh 60 client "${{GPTP_CONFIG_PATH:-/usr/share/gPTP.cfg}}" {interface} "$TSN_SERVER_USER" "$TSN_SERVER_IP" "$TSN_SERVER_PWD" "$TSN_SERVER_INTERFACE"
84 _verification:
85 Is the `rms` value within 100ns?
86
87 unit: template
88 template-resource: tsn/tsn-devices
89 template-unit: job
90-id: tsn/ieee802.1as-2011/time-sync-between-two-system-as-master-{eth-interface}
91+id: tsn/ieee802.1as-2011/time-sync-between-two-system-as-master-{interface}
92 category_id: tsn
93 _summary:
94- Check establish time-sync with another machine as master via {eth-interface}
95+ Check establish time-sync with another machine as master via {interface}
96 _purpose:
97- Use {eth-interface} on the SUT to achieve time sync with the second machine as master
98+ Use {interface} on the SUT to achieve time sync with the second machine as master
99 depends:
100 tsn/tsn-detected
101 plugin: user-interact-verify
102@@ -160,20 +160,20 @@ environ:
103 TSN_SERVER_INTERFACE
104 GPTP_CONFIG_PATH
105 command:
106- tsn_time_sync.sh 60 master "${{GPTP_CONFIG_PATH:-/usr/share/gPTP.cfg}}" {eth-interface} "$TSN_SERVER_USER" "$TSN_SERVER_IP" "$TSN_SERVER_PWD" "$TSN_SERVER_INTERFACE"
107+ tsn_time_sync.sh 60 master "${{GPTP_CONFIG_PATH:-/usr/share/gPTP.cfg}}" {interface} "$TSN_SERVER_USER" "$TSN_SERVER_IP" "$TSN_SERVER_PWD" "$TSN_SERVER_INTERFACE"
108 _verification:
109 Is the `rms` value within 100ns?
110
111 unit: template
112 template-resource: tsn/tsn-devices
113 template-unit: job
114-id: tsn/ieee802.1as-2011/time-sync-system-clock-with-PHC-as-client-{eth-interface}
115+id: tsn/ieee802.1as-2011/time-sync-system-clock-with-PHC-as-client-{interface}
116 category_id: tsn
117 _summary:
118 Check system time sync with PTP hardware clock as client
119 depends:
120 tsn/tsn-detected
121- tsn/ieee802.1as-2011/time-sync-between-two-system-as-client-{eth-interface}
122+ tsn/ieee802.1as-2011/time-sync-between-two-system-as-client-{interface}
123 plugin: user-interact-verify
124 user: root
125 _purpose:
126@@ -185,12 +185,12 @@ environ:
127 TSN_SERVER_INTERFACE
128 GPTP_CONFIG_PATH
129 command:
130- tsn_time_sync.sh 60 client "${{GPTP_CONFIG_PATH:-/usr/share/gPTP.cfg}}" {eth-interface} "$TSN_SERVER_USER" "$TSN_SERVER_IP" "$TSN_SERVER_PWD" "$TSN_SERVER_INTERFACE" > /dev/null &
131+ tsn_time_sync.sh 60 client "${{GPTP_CONFIG_PATH:-/usr/share/gPTP.cfg}}" {interface} "$TSN_SERVER_USER" "$TSN_SERVER_IP" "$TSN_SERVER_PWD" "$TSN_SERVER_INTERFACE" > /dev/null &
132 pmc -u -b 0 -t 1 "SET GRANDMASTER_SETTINGS_NP clockClass 248\
133 clockAccuracy 0xfe offsetScaledLogVariance 0xffff currentUtcOffset 37\
134 leap61 0 leap59 0 currentUtcOffsetValid 1 ptpTimescale 1 timeTraceable 1\
135 frequencyTraceable 0 timeSource 0xa0"
136- timeout 60 phc2sys -s {eth-interface} -O 0 -c CLOCK_REALTIME --step_threshold=1 \
137+ timeout 60 phc2sys -s {interface} -O 0 -c CLOCK_REALTIME --step_threshold=1 \
138 --transportSpecific=1 -w -m --first_step_threshold=0.0
139 _verification:
140 Is the `phc offset` value within `100ns`?
141@@ -200,13 +200,13 @@ _verification:
142 unit: template
143 template-resource: tsn/tsn-devices
144 template-unit: job
145-id: tsn/ieee802.1as-2011/time-sync-system-clock-with-PHC-as-master-{eth-interface}
146+id: tsn/ieee802.1as-2011/time-sync-system-clock-with-PHC-as-master-{interface}
147 category_id: tsn
148 _summary:
149 Check system time sync with PTP hardware clock as client
150 depends:
151 tsn/tsn-detected
152- tsn/ieee802.1as-2011/time-sync-between-two-system-as-master-{eth-interface}
153+ tsn/ieee802.1as-2011/time-sync-between-two-system-as-master-{interface}
154 plugin: user-interact-verify
155 user: root
156 _purpose:
157@@ -218,12 +218,12 @@ environ:
158 TSN_SERVER_INTERFACE
159 GPTP_CONFIG_PATH
160 command:
161- tsn_time_sync.sh 60 master "${{GPTP_CONFIG_PATH:-/usr/share/gPTP.cfg}}" {eth-interface} "$TSN_SERVER_USER" "$TSN_SERVER_IP" "$TSN_SERVER_PWD" "$TSN_SERVER_INTERFACE" > /dev/null &
162+ tsn_time_sync.sh 60 master "${{GPTP_CONFIG_PATH:-/usr/share/gPTP.cfg}}" {interface} "$TSN_SERVER_USER" "$TSN_SERVER_IP" "$TSN_SERVER_PWD" "$TSN_SERVER_INTERFACE" > /dev/null &
163 pmc -u -b 0 -t 1 "SET GRANDMASTER_SETTINGS_NP clockClass 248\
164 clockAccuracy 0xfe offsetScaledLogVariance 0xffff currentUtcOffset 37\
165 leap61 0 leap59 0 currentUtcOffsetValid 1 ptpTimescale 1 timeTraceable 1\
166 frequencyTraceable 0 timeSource 0xa0"
167- timeout 60 phc2sys -s {eth-interface} -O 0 -c CLOCK_REALTIME --step_threshold=1 \
168+ timeout 60 phc2sys -s {interface} -O 0 -c CLOCK_REALTIME --step_threshold=1 \
169 --transportSpecific=1 -w -m --first_step_threshold=0.0
170 _verification:
171 Is the `phc offset` value within `100ns`?
172@@ -233,7 +233,7 @@ _verification:
173 unit: template
174 template-resource: tsn/tsn-devices
175 template-unit: job
176-id: tsn/time-based-shaping/udp-packet-time-check-{eth-interface}
177+id: tsn/time-based-shaping/udp-packet-time-check-{interface}
178 category_id: tsn
179 _summary: Check the packet time intervals
180 depends:
181@@ -248,11 +248,11 @@ environ:
182 TSN_SERVER_PWD
183 TSN_SERVER_INTERFACE
184 command:
185- tc qdisc del dev {eth-interface} root
186- tc qdisc add dev {eth-interface} handle 8001: parent root mqprio num_tc 4 map 0 1 2 3 3 3 3 3 3 3 3 3 3 3 3 3 queues 1@0 1@1 1@2 1@3 hw 0
187- tc qdisc replace dev {eth-interface} parent 8001:4 etf offload clockid CLOCK_TAI delta 500000
188- tc qdisc show dev {eth-interface}
189- timeout 30 udp_tai -c 3 -i {eth-interface} -P 1000000 -p 90 -d 600000 > /dev/null &
190+ tc qdisc del dev {interface} root
191+ tc qdisc add dev {interface} handle 8001: parent root mqprio num_tc 4 map 0 1 2 3 3 3 3 3 3 3 3 3 3 3 3 3 queues 1@0 1@1 1@2 1@3 hw 0
192+ tc qdisc replace dev {interface} parent 8001:4 etf offload clockid CLOCK_TAI delta 500000
193+ tc qdisc show dev {interface}
194+ timeout 30 udp_tai -c 3 -i {interface} -P 1000000 -p 90 -d 600000 > /dev/null &
195 timeout 30 sshpass -p "u" ssh -o StrictHostKeyChecking=no "$TSN_SERVER_USER"@"$TSN_SERVER_IP" "echo "$TSN_SERVER_PWD" | sudo -S tcpdump -Q in -ttt -ni "$TSN_SERVER_INTERFACE" --time-stamp-precision=nano -j adapter_unsynced port 7788 -c 20000"
196 _verification:
197 Is the packets are being transmitted every 1000000ns (+/-500ns)?
198@@ -261,7 +261,65 @@ _verification:
199 unit: template
200 template-resource: tsn/tsn-devices
201 template-unit: job
202-id: tsn/ieee802.1qbv/udp-traffic-check-{eth-interface}
203+template-engine: jinja2
204+id: tsn/ieee802.1qav/check-credit-based-shaper-{{ interface }}
205+category_id: tsn
206+_summary: Credit-Based Shaper algorithm check
207+depends:
208+ tsn/tsn-detected
209+plugin: shell
210+user: root
211+_purpose:
212+ To check if the system can send packets in the certain rate
213+environ:
214+ TSN_SERVER_IP
215+ TSN_SERVER_USER
216+ TSN_SERVER_PWD
217+ TSN_SERVER_INTERFACE
218+ GPTP_CONFIG_PATH
219+command:
220+ echo "## Clearing previous queue discipline configuration..."
221+ echo "## You will get error if there's no previous configuration, but you could ignore it."
222+ echo ""
223+ sshpass -p "$TSN_SERVER_PWD" ssh -o StrictHostKeyChecking=no "$TSN_SERVER_USER"@"$TSN_SERVER_IP" "echo $TSN_SERVER_PWD | sudo -S tc qdisc del dev $TSN_SERVER_INTERFACE root"
224+ tc qdisc del dev {{ interface }} root
225+ echo ""
226+ echo "## Setting 4 traffic classes mapping..."
227+ tc qdisc replace dev {{ interface }} handle 100: parent root mqprio num_tc 4 map 0 1 2 3 3 3 3 3 3 3 3 3 3 3 3 3 queues 1@0 1@1 1@2 1@3 hw 0
228+ tc -g class show dev {{ interface }}
229+ echo ""
230+ echo "## Setting Traffic Class 1 (Queue 1) to 10% bandwidth of 1 Gbps..."
231+ tc qdisc replace dev {{ interface }} parent 100:1 cbs locredit -1350 hicredit 150 sendslope -900000 idleslope 100000 offload 1
232+ tc qdisc show dev {{ interface }}
233+ echo ""
234+ echo "## Start sending UDP packets of 1500 bytes using socket priority 1 with tsn-talker..."
235+ stdbuf -oL timeout 60 tsn-talker -d 01:AA:AA:AA:AA:AA -i {{ interface }} -p 0 -s 1500 &
236+ sleep 5
237+ echo ""
238+ echo "## Start listening on TSN server..."
239+ output=$(mktemp)
240+ (sshpass -p "$TSN_SERVER_PWD" ssh -o StrictHostKeyChecking=no "$TSN_SERVER_USER"@"$TSN_SERVER_IP" "echo $TSN_SERVER_PWD | sudo -S stdbuf -oL timeout 55 ./tsn-listener -d 01:AA:AA:AA:AA:AA -i $TSN_SERVER_INTERFACE -s 1500") | tee $output
241+ lines=$(cat $output | tail -n 50)
242+ while read -r line
243+ do
244+ speed=$(echo "$line" | awk 'match($0, /: [0-9]* kbps/) {print substr($0, RSTART+2, RLENGTH-7)}')
245+ if [ $speed -gt 100000 ]; then
246+ echo "FAIL: Data rate over 1000000 kbps!"
247+ exit 1
248+ fi
249+ if [ $speed -lt 90000 ]; then
250+ echo "FAIL: Data rate cannot reach 90% of 1000000 kbps!"
251+ exit 1
252+ fi
253+ done <<< $lines
254+ echo ""
255+ echo "Pass: Data rate within -10% of 1000000kbps"
256+ exit 0
257+
258+unit: template
259+template-resource: tsn/tsn-devices
260+template-unit: job
261+id: tsn/ieee802.1qbv/udp-traffic-check-{interface}
262 category_id: tsn
263 _summary: Check if the streams packet are handled properly
264 depends:
265@@ -278,7 +336,7 @@ environ:
266 TSN_SERVER_INTERFACE
267 GPTP_CONFIG_PATH
268 command:
269- udp_traffic_test.sh {eth-interface} 30 "${{GPTP_CONFIG_PATH:-/usr/share/gPTP.cfg}}" "$TSN_SERVER_USER" "$TSN_SERVER_IP" "$TSN_SERVER_PWD" "$TSN_SERVER_INTERFACE"
270+ udp_traffic_test.sh {interface} 30 "${{GPTP_CONFIG_PATH:-/usr/share/gPTP.cfg}}" "$TSN_SERVER_USER" "$TSN_SERVER_IP" "$TSN_SERVER_PWD" "$TSN_SERVER_INTERFACE"
271 _verification:
272 Is the send bytes and pkt are increasing for 100:1 - 100:4?
273
274@@ -286,7 +344,7 @@ _verification:
275 unit: template
276 template-resource: tsn/tsn-devices
277 template-unit: job
278-id: tsn/ieee802.1qbu/check-frame-preemption-{eth-interface}
279+id: tsn/ieee802.1qbu/check-frame-preemption-{interface}
280 category_id: tsn
281 _summary: Check if could set frame preemption
282 depends:
283@@ -303,8 +361,8 @@ command:
284 exit 1
285 fi
286 echo "## Setting frame preemption on..."
287- ethtool-rt.ethtool --set-frame-preemption {eth-interface} fp on
288+ ethtool-rt.ethtool --set-frame-preemption {interface} fp on
289 echo "## Showing frame preemption settings..."
290- ethtool-rt.ethtool --show-frame-preemption {eth-interface}
291+ ethtool-rt.ethtool --show-frame-preemption {interface}
292 _verification:
293- Is the frame preemption enabled for {eth-interface}?
294+ Is the frame preemption enabled for {interface}?
295diff --git a/units/tsn/test-plan.pxu b/units/tsn/test-plan.pxu
296index 1b6d405..696e61b 100644
297--- a/units/tsn/test-plan.pxu
298+++ b/units/tsn/test-plan.pxu
299@@ -29,6 +29,9 @@ bootstrap_include:
300 tsn/tsn-devices
301 include:
302 tsn/verify-PTP-for-.*
303+ tsn/test-env-setup-for-.*
304+nested_part:
305+ tsn-ieee802.1qav
306
307 id: tsn-ieee802.1as-2011
308 _name: Time Sensitive Networking Test Plan for IEEE 802.1AS-2011
309@@ -58,6 +61,16 @@ bootstrap_include:
310 include:
311 tsn/time-based-shaping/udp-packet-time-check-.*
312
313+id: tsn-ieee802.1qav
314+_name: Time Sensitive Networking Test Plan for IEEE 802.1Qav
315+_description:
316+ This testplan is to check if it could follow IEEE 802.1Qav Credit-Based Shaper.
317+unit: test plan
318+bootstrap_include:
319+ tsn/tsn-devices
320+include:
321+ tsn/ieee802.1qav/check-credit-based-shaper-.*
322+
323 id: tsn-ieee802.1qbv
324 _name: Time Sensitive Networking Test Plan for IEEE 802.1Qbv
325 _description:

Subscribers

People subscribed via source and target branches