Merge lp:~bladernr/checkbox/story759-checkbox-add-suspend-log-check into lp:checkbox
- story759-checkbox-add-suspend-log-check
- Merge into trunk
Proposed by
Jeff Lane
Status: | Merged | ||||
---|---|---|---|---|---|
Merged at revision: | 2080 | ||||
Proposed branch: | lp:~bladernr/checkbox/story759-checkbox-add-suspend-log-check | ||||
Merge into: | lp:checkbox | ||||
Diff against target: |
285 lines (+137/-29) 7 files modified
data/whitelists/sru.whitelist (+1/-0) debian/changelog (+8/-0) jobs/hibernate.txt.in (+8/-2) jobs/stress.txt.in (+14/-2) jobs/suspend.txt.in (+8/-2) scripts/fwts_test (+26/-23) scripts/sleep_test_log_check (+72/-0) |
||||
To merge this branch: | bzr merge lp:~bladernr/checkbox/story759-checkbox-add-suspend-log-check | ||||
Related bugs: |
|
Reviewer | Review Type | Date Requested | Status |
---|---|---|---|
Daniel Manrique (community) | Approve | ||
Review via email:
|
Commit message
Description of the change
Wrote the parser script to look at the FWTS log and give us the error count and/or report of errors encountered in testing
modified fwts_test to include a new fail level "none" to tell fwts_test to NOT fail on any error.
modified sleep test jobs in hibernate.txt.in, suspend.txt.in and stress.txt.in to use the new fail level.
Also added log check jobs to each of those files to use the new parser script.
Also modified SRU whitelist to make sure that the log check runs after suspend as well.
To post a comment you must log in.
Revision history for this message
![](/+icing/build/overlay/assets/skins/sam/images/close.gif)
Jeff Lane (bladernr) wrote : | # |
Revision history for this message
![](/+icing/build/overlay/assets/skins/sam/images/close.gif)
Daniel Manrique (roadmr) wrote : | # |
Looks good and I even tested the script locally.
Thanks!
review:
Approve
Preview Diff
[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1 | === modified file 'data/whitelists/sru.whitelist' | |||
2 | --- data/whitelists/sru.whitelist 2013-04-22 23:48:18 +0000 | |||
3 | +++ data/whitelists/sru.whitelist 2013-04-24 14:32:29 +0000 | |||
4 | @@ -95,6 +95,7 @@ | |||
5 | 95 | suspend/network_before_suspend | 95 | suspend/network_before_suspend |
6 | 96 | suspend/memory_before_suspend | 96 | suspend/memory_before_suspend |
7 | 97 | suspend/suspend_advanced_auto | 97 | suspend/suspend_advanced_auto |
8 | 98 | suspend/suspend-single-log-check | ||
9 | 98 | suspend/audio_after_suspend_auto | 99 | suspend/audio_after_suspend_auto |
10 | 99 | suspend/network_resume_time | 100 | suspend/network_resume_time |
11 | 100 | suspend/wifi_resume_time | 101 | suspend/wifi_resume_time |
12 | 101 | 102 | ||
13 | === modified file 'debian/changelog' | |||
14 | --- debian/changelog 2013-04-24 13:33:15 +0000 | |||
15 | +++ debian/changelog 2013-04-24 14:32:29 +0000 | |||
16 | @@ -18,6 +18,14 @@ | |||
17 | 18 | plugins/persist_prompt.py: promoted save to run before lock release. | 18 | plugins/persist_prompt.py: promoted save to run before lock release. |
18 | 19 | checkbox/lib/fifo.py: trap OSError exception at close when the input/output | 19 | checkbox/lib/fifo.py: trap OSError exception at close when the input/output |
19 | 20 | fifo fds disappear before fifo.close() can get to them. (LP: #115561) | 20 | fifo fds disappear before fifo.close() can get to them. (LP: #115561) |
20 | 21 | * scripts/sleep_test_log_check: added new script to parse fwts logs for | ||
21 | 22 | errors | ||
22 | 23 | jobs/hibernate.txt.in, jobs/stress.txt.in, jobs/suspend.txt.in: added jobs | ||
23 | 24 | to use the sleep_test_log_check script after s3/s4 tests. Modified current | ||
24 | 25 | s3/s4 tests to use the new None fail level for fwts_test | ||
25 | 26 | scripts/fwts_test: no longer fails on fwts errors if "-f none" is chosen. | ||
26 | 27 | Now it should only fail by manual indication if the actual sleep action | ||
27 | 28 | fails. (LP: #1169922) | ||
28 | 21 | 29 | ||
29 | 22 | [ Brendan Donegan ] | 30 | [ Brendan Donegan ] |
30 | 23 | * scripts/sources_test - modified script so that it takes sources list | 31 | * scripts/sources_test - modified script so that it takes sources list |
31 | 24 | 32 | ||
32 | === modified file 'jobs/hibernate.txt.in' | |||
33 | --- jobs/hibernate.txt.in 2013-01-03 14:58:32 +0000 | |||
34 | +++ jobs/hibernate.txt.in 2013-04-24 14:32:29 +0000 | |||
35 | @@ -6,7 +6,7 @@ | |||
36 | 6 | command: | 6 | command: |
37 | 7 | if type -P fwts >/dev/null; then | 7 | if type -P fwts >/dev/null; then |
38 | 8 | echo "Calling fwts" | 8 | echo "Calling fwts" |
40 | 9 | fwts_test -l $CHECKBOX_DATA/hibernate-single -s s4 --s4-device-check --s4-device-check-delay=45 --s4-sleep-delay=120 | 9 | fwts_test -l $CHECKBOX_DATA/hibernate-single -f none -s s4 --s4-device-check --s4-device-check-delay=45 --s4-sleep-delay=120 |
41 | 10 | else | 10 | else |
42 | 11 | echo "Calling sleep_test" | 11 | echo "Calling sleep_test" |
43 | 12 | sleep_test -s disk -w 120 | 12 | sleep_test -s disk -w 120 |
44 | @@ -22,7 +22,13 @@ | |||
45 | 22 | VERIFICATION: | 22 | VERIFICATION: |
46 | 23 | Did the system successfully hibernate and did it work properly after waking up? | 23 | Did the system successfully hibernate and did it work properly after waking up? |
47 | 24 | 24 | ||
48 | 25 | plugin: shell | ||
49 | 26 | name: power-management/hibernate-single-log-check | ||
50 | 27 | command: [ -e $CHECKBOX_DATA/hibernate-single.log ] && sleep_test_log_check -v s4 $CHECKBOX_DATA/hibernate-single.log | ||
51 | 28 | _description: | ||
52 | 29 | Automated check of the hibernate log for errors discovered by fwts | ||
53 | 30 | |||
54 | 25 | plugin: attachment | 31 | plugin: attachment |
55 | 26 | name: power-management/hibernate-single-log-attach | 32 | name: power-management/hibernate-single-log-attach |
57 | 27 | command: [ -e $CHECKBOX_DATA/hibernate-single.log ] && cat $CHECBOX_DATA/hibernate-single.log | 33 | command: [ -e $CHECKBOX_DATA/hibernate-single.log ] && cat $CHECKBOX_DATA/hibernate-single.log |
58 | 28 | description: attaches log from single hibernate/resume test to results | 34 | description: attaches log from single hibernate/resume test to results |
59 | 29 | 35 | ||
60 | === modified file 'jobs/stress.txt.in' | |||
61 | --- jobs/stress.txt.in 2013-04-04 16:06:42 +0000 | |||
62 | +++ jobs/stress.txt.in 2013-04-24 14:32:29 +0000 | |||
63 | @@ -16,7 +16,7 @@ | |||
64 | 16 | command: | 16 | command: |
65 | 17 | if type -P fwts >/dev/null; then | 17 | if type -P fwts >/dev/null; then |
66 | 18 | echo "Calling fwts" | 18 | echo "Calling fwts" |
68 | 19 | fwts_test -l $CHECKBOX_DATA/hibernate_30_cycles -s s4 --s4-device-check --s4-device-check-delay=45 --s4-sleep-delay=120 --s4-multiple=30 | 19 | fwts_test -l $CHECKBOX_DATA/hibernate_30_cycles -f none -s s4 --s4-device-check --s4-device-check-delay=45 --s4-sleep-delay=120 --s4-multiple=30 |
69 | 20 | else | 20 | else |
70 | 21 | echo "Calling sleep_test" | 21 | echo "Calling sleep_test" |
71 | 22 | sleep_test -s disk -i 30 -w 120 | 22 | sleep_test -s disk -i 30 -w 120 |
72 | @@ -25,6 +25,12 @@ | |||
73 | 25 | PURPOSE: | 25 | PURPOSE: |
74 | 26 | This is an automated stress test that will force the system to hibernate/resume for 30 cycles | 26 | This is an automated stress test that will force the system to hibernate/resume for 30 cycles |
75 | 27 | 27 | ||
76 | 28 | plugin: shell | ||
77 | 29 | name: power-management/hibernate-30-cycles-log-check | ||
78 | 30 | command: [ -e $CHECKBOX_DATA/hibernate_30_cycles.log ] && sleep_test_log_check -v s4 $CHECKBOX_DATA/hibernate_30_cycles.log | ||
79 | 31 | _description: | ||
80 | 32 | Automated check of the 30 cycle hibernate log for errors detected by fwts. | ||
81 | 33 | |||
82 | 28 | plugin: attachment | 34 | plugin: attachment |
83 | 29 | name: power-management/hibernate-30-cycle-log-attach | 35 | name: power-management/hibernate-30-cycle-log-attach |
84 | 30 | command: [ -e $CHECKBOX_DATA/hibernate_30_cycles.log ] && cat $CHECKBOX_DATA/hibernate_30_cycles.log | 36 | command: [ -e $CHECKBOX_DATA/hibernate_30_cycles.log ] && cat $CHECKBOX_DATA/hibernate_30_cycles.log |
85 | @@ -39,7 +45,7 @@ | |||
86 | 39 | command: | 45 | command: |
87 | 40 | if type -P fwts >/dev/null; then | 46 | if type -P fwts >/dev/null; then |
88 | 41 | echo "Calling fwts" | 47 | echo "Calling fwts" |
90 | 42 | set -o pipefail; fwts_test -l $CHECKBOX_DATA/suspend_30_cycles -s s3 --s3-device-check --s3-device-check-delay=45 --s3-sleep-delay=30 --s3-multiple=30 | tee $CHECKBOX_DATA/suspend_30_cycles_times.log | 48 | set -o pipefail; fwts_test -l $CHECKBOX_DATA/suspend_30_cycles -f none -s s3 --s3-device-check --s3-device-check-delay=45 --s3-sleep-delay=30 --s3-multiple=30 | tee $CHECKBOX_DATA/suspend_30_cycles_times.log |
91 | 43 | else | 49 | else |
92 | 44 | echo "Calling sleep_test" | 50 | echo "Calling sleep_test" |
93 | 45 | set -o pipefail; sleep_test -p s mem -i 30 | tee $CHECKBOX_DATA/suspend_30_cycles_times.log | 51 | set -o pipefail; sleep_test -p s mem -i 30 | tee $CHECKBOX_DATA/suspend_30_cycles_times.log |
94 | @@ -48,6 +54,12 @@ | |||
95 | 48 | PURPOSE: | 54 | PURPOSE: |
96 | 49 | This is an automated stress test that will force the system to suspend/resume for 30 cycles. | 55 | This is an automated stress test that will force the system to suspend/resume for 30 cycles. |
97 | 50 | 56 | ||
98 | 57 | plugin: shell | ||
99 | 58 | name: power-management/suspend-30-cycles-log-check | ||
100 | 59 | command: [ -e $CHECKBOX_DATA/suspend_30_cycles.log ] && sleep_test_log_check -v s3 $CHECKBOX_DATA/suspend_30_cycles.log | ||
101 | 60 | _description: | ||
102 | 61 | Automated check of the 30 cycle hibernate log for errors detected by fwts. | ||
103 | 62 | |||
104 | 51 | plugin: attachment | 63 | plugin: attachment |
105 | 52 | name: power-management/suspend-30-cycle-log-attach | 64 | name: power-management/suspend-30-cycle-log-attach |
106 | 53 | command: [ -e $CHECKBOX_DATA/suspend_30_cycles.log ] && cat $CHECKBOX_DATA/suspend_30_cycles.log | 65 | command: [ -e $CHECKBOX_DATA/suspend_30_cycles.log ] && cat $CHECKBOX_DATA/suspend_30_cycles.log |
107 | 54 | 66 | ||
108 | === modified file 'jobs/suspend.txt.in' | |||
109 | --- jobs/suspend.txt.in 2013-04-24 03:40:17 +0000 | |||
110 | +++ jobs/suspend.txt.in 2013-04-24 14:32:29 +0000 | |||
111 | @@ -168,7 +168,7 @@ | |||
112 | 168 | command: | 168 | command: |
113 | 169 | if type -P fwts >/dev/null; then | 169 | if type -P fwts >/dev/null; then |
114 | 170 | echo "Calling fwts" | 170 | echo "Calling fwts" |
116 | 171 | set -o pipefail; fwts_test -f critical -l $CHECKBOX_DATA/suspend_single -s s3 --s3-sleep-delay=30 --s3-device-check --s3-device-check-delay=45 | tee $CHECKBOX_DATA/suspend_single_times.log | 171 | set -o pipefail; fwts_test -f none -l $CHECKBOX_DATA/suspend_single -s s3 --s3-sleep-delay=30 --s3-device-check --s3-device-check-delay=45 | tee $CHECKBOX_DATA/suspend_single_times.log |
117 | 172 | else | 172 | else |
118 | 173 | echo "Calling sleep_test" | 173 | echo "Calling sleep_test" |
119 | 174 | set -o pipefail; sleep_test -p | tee $CHECKBOX_DATA/suspend_single_times.log | 174 | set -o pipefail; sleep_test -p | tee $CHECKBOX_DATA/suspend_single_times.log |
120 | @@ -194,7 +194,13 @@ | |||
121 | 194 | This is the automated version of suspend/suspend_advanced. | 194 | This is the automated version of suspend/suspend_advanced. |
122 | 195 | user: root | 195 | user: root |
123 | 196 | environ: CHECKBOX_DATA | 196 | environ: CHECKBOX_DATA |
125 | 197 | command: set -o pipefail; fwts_test -f critical -l $CHECKBOX_DATA/suspend_single -s s3 --s3-sleep-delay=30 --s3-device-check --s3-device-check-delay=45 | tee $CHECKBOX_DATA/suspend_single_times.log | 197 | command: set -o pipefail; fwts_test -f none -l $CHECKBOX_DATA/suspend_single -s s3 --s3-sleep-delay=30 --s3-device-check --s3-device-check-delay=45 | tee $CHECKBOX_DATA/suspend_single_times.log |
126 | 198 | |||
127 | 199 | plugin: shell | ||
128 | 200 | name: suspend/suspend-single-log-check | ||
129 | 201 | command: [ -e $CHECKBOX_DATA/suspend_single.log ] && sleep_test_log_check -v s3 $CHECKBOX_DATA/suspend_single.log | ||
130 | 202 | _description: | ||
131 | 203 | Automated check of the suspend log to look for errors reported by fwts | ||
132 | 198 | 204 | ||
133 | 199 | plugin: attachment | 205 | plugin: attachment |
134 | 200 | name: suspend/suspend-single-log-attach | 206 | name: suspend/suspend-single-log-attach |
135 | 201 | 207 | ||
136 | === modified file 'scripts/fwts_test' | |||
137 | --- scripts/fwts_test 2013-01-15 23:03:42 +0000 | |||
138 | +++ scripts/fwts_test 2013-04-24 14:32:29 +0000 | |||
139 | @@ -125,7 +125,7 @@ | |||
140 | 125 | '[Default: %(default)s]')) | 125 | '[Default: %(default)s]')) |
141 | 126 | parser.add_argument('-f', '--fail-level', | 126 | parser.add_argument('-f', '--fail-level', |
142 | 127 | default='high', | 127 | default='high', |
144 | 128 | choices=['critical', 'high', 'medium', 'low'], | 128 | choices=['critical', 'high', 'medium', 'low', 'none'], |
145 | 129 | help=('Specify the FWTS failure level that will trigger ' | 129 | help=('Specify the FWTS failure level that will trigger ' |
146 | 130 | 'this script to return a failing exit code. For ' | 130 | 'this script to return a failing exit code. For ' |
147 | 131 | 'example, if you chose "critical" as the ' | 131 | 'example, if you chose "critical" as the ' |
148 | @@ -188,15 +188,17 @@ | |||
149 | 188 | passed = [] | 188 | passed = [] |
150 | 189 | 189 | ||
151 | 190 | # Set correct fail level | 190 | # Set correct fail level |
161 | 191 | args.fail_level = 'FAILED_%s' % args.fail_level.upper() | 191 | if args.fail_level is not 'none': |
162 | 192 | 192 | args.fail_level = 'FAILED_%s' % args.fail_level.upper() | |
163 | 193 | # Get our failure priority and create the priority values | 193 | |
164 | 194 | fail_levels = {'FAILED_CRITICAL':4, | 194 | # Get our failure priority and create the priority values |
165 | 195 | 'FAILED_HIGH':3, | 195 | fail_levels = {'FAILED_CRITICAL':4, |
166 | 196 | 'FAILED_MEDIUM':2, | 196 | 'FAILED_HIGH':3, |
167 | 197 | 'FAILED_LOW':1} | 197 | 'FAILED_MEDIUM':2, |
168 | 198 | fail_priority = fail_levels[args.fail_level] | 198 | 'FAILED_LOW':1, |
169 | 199 | 199 | 'FAILED_NONE':0} | |
170 | 200 | fail_priority = fail_levels[args.fail_level] | ||
171 | 201 | |||
172 | 200 | # Enforce only using sleep opts with --sleep | 202 | # Enforce only using sleep opts with --sleep |
173 | 201 | if args.sleep_time or args.resume_time and not args.sleep: | 203 | if args.sleep_time or args.resume_time and not args.sleep: |
174 | 202 | parser.error('--sleep-time and --resume-time only apply to the ' | 204 | parser.error('--sleep-time and --resume-time only apply to the ' |
175 | @@ -327,19 +329,20 @@ | |||
176 | 327 | print("Passed: %d" % len(passed)) | 329 | print("Passed: %d" % len(passed)) |
177 | 328 | for test in passed: | 330 | for test in passed: |
178 | 329 | print(" - " + test) | 331 | print(" - " + test) |
192 | 330 | 332 | ||
193 | 331 | if fail_priority == fail_levels['FAILED_CRITICAL']: | 333 | if args.fail_level is not 'none': |
194 | 332 | if critical_fails: | 334 | if fail_priority == fail_levels['FAILED_CRITICAL']: |
195 | 333 | return 1 | 335 | if critical_fails: |
196 | 334 | if fail_priority == fail_levels['FAILED_HIGH']: | 336 | return 1 |
197 | 335 | if critical_fails or high_fails: | 337 | if fail_priority == fail_levels['FAILED_HIGH']: |
198 | 336 | return 1 | 338 | if critical_fails or high_fails: |
199 | 337 | if fail_priority == fail_levels['FAILED_MEDIUM']: | 339 | return 1 |
200 | 338 | if critical_fails or high_fails or medium_fails: | 340 | if fail_priority == fail_levels['FAILED_MEDIUM']: |
201 | 339 | return 1 | 341 | if critical_fails or high_fails or medium_fails: |
202 | 340 | if fail_priority == fail_levels['FAILED_LOW']: | 342 | return 1 |
203 | 341 | if critical_fails or high_fails or medium_fails or low_fails: | 343 | if fail_priority == fail_levels['FAILED_LOW']: |
204 | 342 | return 1 | 344 | if critical_fails or high_fails or medium_fails or low_fails: |
205 | 345 | return 1 | ||
206 | 343 | 346 | ||
207 | 344 | return 0 | 347 | return 0 |
208 | 345 | 348 | ||
209 | 346 | 349 | ||
210 | === added file 'scripts/sleep_test_log_check' | |||
211 | --- scripts/sleep_test_log_check 1970-01-01 00:00:00 +0000 | |||
212 | +++ scripts/sleep_test_log_check 2013-04-24 14:32:29 +0000 | |||
213 | @@ -0,0 +1,72 @@ | |||
214 | 1 | #!/usr/bin/env python3 | ||
215 | 2 | ''' | ||
216 | 3 | This script is used to parse the log generated by fwts and check it for certain | ||
217 | 4 | errors detected during testing. It expects that this is a log file created by | ||
218 | 5 | fwts at runtime using the -l <log name> option. | ||
219 | 6 | |||
220 | 7 | It's written now specifically for checking ater the fwts s3 and s4 tests but | ||
221 | 8 | can be adapted to look for other tests, or all tests. | ||
222 | 9 | ''' | ||
223 | 10 | |||
224 | 11 | import sys | ||
225 | 12 | |||
226 | 13 | from argparse import ArgumentParser, RawTextHelpFormatter | ||
227 | 14 | |||
228 | 15 | |||
229 | 16 | def main(): | ||
230 | 17 | parser = ArgumentParser() | ||
231 | 18 | parser.add_argument('-v', '--verbose', | ||
232 | 19 | action='store_true', | ||
233 | 20 | default=False, | ||
234 | 21 | help="Display each error discovered. May provide \ | ||
235 | 22 | very long output. Also, this option will only \ | ||
236 | 23 | provide a list of UNIQUE errors encountered in \ | ||
237 | 24 | the log file. It will not display duplicates. \ | ||
238 | 25 | Default is [%(default)s]") | ||
239 | 26 | parser.add_argument('test', | ||
240 | 27 | action='store', | ||
241 | 28 | help='The test to check (s3 or s4)') | ||
242 | 29 | parser.add_argument('logfile', | ||
243 | 30 | action='store', | ||
244 | 31 | help='The log file to parse') | ||
245 | 32 | |||
246 | 33 | args = parser.parse_args() | ||
247 | 34 | |||
248 | 35 | return_code = 0 | ||
249 | 36 | |||
250 | 37 | #Create a generator and get our lines | ||
251 | 38 | log = (line.rstrip() for line in open(args.logfile,'r')) | ||
252 | 39 | |||
253 | 40 | #Now parse and find out error lines. We're looking for specific keys that | ||
254 | 41 | #occur in the summary section for each test run. | ||
255 | 42 | #Error levels we care about: | ||
256 | 43 | levels = ('critical', 'high', 'medium') | ||
257 | 44 | errors = {} | ||
258 | 45 | for level in levels: | ||
259 | 46 | errors[level] = [] | ||
260 | 47 | for logline in log: | ||
261 | 48 | for level in levels: | ||
262 | 49 | if "%s: %s" % (args.test, level.upper()) in logline: | ||
263 | 50 | line = logline.split(']',1)[1].strip() | ||
264 | 51 | #Only include unique lines to avoid hundreds of repititions | ||
265 | 52 | if line not in errors[level]: | ||
266 | 53 | errors[level].append(line) | ||
267 | 54 | |||
268 | 55 | for level in levels: | ||
269 | 56 | if errors[level]: | ||
270 | 57 | return_code = 1 | ||
271 | 58 | print("%s errors: %s" % (level.upper(), len(errors[level]))) | ||
272 | 59 | if args.verbose: | ||
273 | 60 | #Print the actual errors) | ||
274 | 61 | print('='*40) | ||
275 | 62 | for line in errors[level]: | ||
276 | 63 | print(line) | ||
277 | 64 | print('\n') | ||
278 | 65 | |||
279 | 66 | if return_code == 0: | ||
280 | 67 | print("No errors detected") | ||
281 | 68 | |||
282 | 69 | return return_code | ||
283 | 70 | |||
284 | 71 | if __name__ == '__main__': | ||
285 | 72 | sys.exit(main()) |
co-depends on this MR: /code.launchpad .net/~bladernr/ checkbox- certification/ story759- cert-add- log-check- jobs/+merge/ 160666
https:/