Merge lp:~joetalbott/utah/fix_result_counts into lp:utah
- fix_result_counts
- Merge into dev
Status: | Merged |
---|---|
Approved by: | Javier Collado |
Approved revision: | 770 |
Merged at revision: | 768 |
Proposed branch: | lp:~joetalbott/utah/fix_result_counts |
Merge into: | lp:utah |
Diff against target: |
678 lines (+432/-76) 9 files modified
tests/test_parser.py (+5/-2) utah/client/examples/test_state_partial_all_failed.yaml (+90/-0) utah/client/examples/test_state_partial_inprogress.yaml (+90/-0) utah/client/examples/test_state_partial_run_all.yaml (+90/-0) utah/client/runner.py (+12/-11) utah/client/testcase.py (+10/-5) utah/client/tests/common.py (+27/-0) utah/client/tests/test_state_agent.py (+102/-55) utah/client/testsuite.py (+6/-3) |
To merge this branch: | bzr merge lp:~joetalbott/utah/fix_result_counts |
Related bugs: |
Reviewer | Review Type | Date Requested | Status |
---|---|---|---|
Joe Talbott (community) | Needs Resubmitting | ||
Javier Collado (community) | Approve | ||
Review via email:
|
Commit message
Description of the change
This branch fixes the result counts for runs that involve a reboot.
- 767. By Joe Talbott
-
test_state_agent - Add a few more tests for resumes.
* make sure different scenarios return the expected results.
![](/+icing/build/overlay/assets/skins/sam/images/close.gif)
Javier Collado (javier.collado) wrote : | # |
- 768. By Joe Talbott
-
testcase - Remove leftover commented out code.
- 769. By Joe Talbott
-
tests - Move YAML content into separtate files.
![](/+icing/build/overlay/assets/skins/sam/images/close.gif)
Joe Talbott (joetalbott) wrote : | # |
I've removed those commented out line. I meant to do that before pushing but forgot.
I also moved the YAML content into separate files.
![](/+icing/build/overlay/assets/skins/sam/images/close.gif)
Javier Collado (javier.collado) wrote : | # |
Looks nice now. Thanks!
![](/+icing/build/overlay/assets/skins/sam/images/close.gif)
Javier Collado (javier.collado) wrote : | # |
By the way, when running the test cases, I get three errors like the one below:
=======
ERROR: Failure: TypeError (__init__() takes at least 4 arguments (1 given))
-------
Traceback (most recent call last):
File "/usr/lib/
return self._makeTest(obj, parent)
File "/usr/lib/
return MethodTestCase(obj)
File "/usr/lib/
self.inst = self.cls()
TypeError: __init__() takes at least 4 arguments (1 given)
-------
I get them for the dev branch as well, so I don't think it's because of any
change here. Anyway, I'm not merging this right now just in case you want to
take a look a this issue.
- 770. By Joe Talbott
-
parser - Fix test for malformed yaml file.
![](/+icing/build/overlay/assets/skins/sam/images/close.gif)
Joe Talbott (joetalbott) wrote : | # |
It was an unrelated issue, but I went ahead and fixed it here.
![](/+icing/build/overlay/assets/skins/sam/images/close.gif)
Javier Collado (javier.collado) wrote : | # |
Actually, I've found that the problem was on my side when running the test
cases.
Instead of passing `utah/client/
side effect of this was that `nosetests` believed that there were some test
cases in `utah/client/
backtrace in my previous comment.
Anyway, merging your changes.
Preview Diff
1 | === modified file 'tests/test_parser.py' | |||
2 | --- tests/test_parser.py 2012-11-14 22:10:40 +0000 | |||
3 | +++ tests/test_parser.py 2012-12-05 15:31:21 +0000 | |||
4 | @@ -3,7 +3,7 @@ | |||
5 | 3 | import tempfile | 3 | import tempfile |
6 | 4 | import unittest | 4 | import unittest |
7 | 5 | 5 | ||
9 | 6 | from utah.parser import UTAHParser | 6 | from utah.parser import UTAHParser, ParserError |
10 | 7 | 7 | ||
11 | 8 | MALFORMED_CONTENT = """ | 8 | MALFORMED_CONTENT = """ |
12 | 9 | --- | 9 | --- |
13 | @@ -22,6 +22,9 @@ | |||
14 | 22 | 22 | ||
15 | 23 | self.tmpfile.write(MALFORMED_CONTENT) | 23 | self.tmpfile.write(MALFORMED_CONTENT) |
16 | 24 | 24 | ||
17 | 25 | # reset the stream for later reading. | ||
18 | 26 | self.tmpfile.seek(0) | ||
19 | 27 | |||
20 | 25 | def tearDown(self): | 28 | def tearDown(self): |
21 | 26 | self.tmpfile.close() | 29 | self.tmpfile.close() |
22 | 27 | 30 | ||
23 | @@ -30,7 +33,7 @@ | |||
24 | 30 | data = self.parser.parse(self.fakelog) | 33 | data = self.parser.parse(self.fakelog) |
25 | 31 | 34 | ||
26 | 32 | def test_malformed_file(self): | 35 | def test_malformed_file(self): |
28 | 33 | with self.assertRaises(jsonschema.ValidationError): | 36 | with self.assertRaises(ParserError): |
29 | 34 | data = self.parser.parse(self.tmpfile) | 37 | data = self.parser.parse(self.tmpfile) |
30 | 35 | 38 | ||
31 | 36 | def test_parser(self): | 39 | def test_parser(self): |
32 | 37 | 40 | ||
33 | === added file 'utah/client/examples/test_state_partial_all_failed.yaml' | |||
34 | --- utah/client/examples/test_state_partial_all_failed.yaml 1970-01-01 00:00:00 +0000 | |||
35 | +++ utah/client/examples/test_state_partial_all_failed.yaml 2012-12-05 15:31:21 +0000 | |||
36 | @@ -0,0 +1,90 @@ | |||
37 | 1 | master_runlist: /var/lib/utah/master.run | ||
38 | 2 | status: RUN | ||
39 | 3 | result_class: !!python/name:utah.client.result.ResultYAML '' | ||
40 | 4 | fetched_suites: | ||
41 | 5 | - examples | ||
42 | 6 | - utah_tests | ||
43 | 7 | - utah_tests_sample | ||
44 | 8 | suites: | ||
45 | 9 | - build_cmd: null | ||
46 | 10 | name: examples | ||
47 | 11 | passes: 0 | ||
48 | 12 | errors: 0 | ||
49 | 13 | failures: 2 | ||
50 | 14 | status: DONE | ||
51 | 15 | tests: | ||
52 | 16 | - build_cmd: echo "building for test_one" | ||
53 | 17 | command: python test_one.py | ||
54 | 18 | description: A first sample test case. | ||
55 | 19 | name: test_one | ||
56 | 20 | path: examples/test_one | ||
57 | 21 | status: DONE | ||
58 | 22 | summary: FAIL | ||
59 | 23 | tc_cleanup: echo "cleanup for test_one" | ||
60 | 24 | tc_setup: echo "setup for test_one" | ||
61 | 25 | timeout: 3 | ||
62 | 26 | type: userland | ||
63 | 27 | - build_cmd: echo "building for test_two" | ||
64 | 28 | command: python test_two.py | ||
65 | 29 | description: A second sample test case. | ||
66 | 30 | name: test_two | ||
67 | 31 | path: examples/test_two | ||
68 | 32 | status: DONE | ||
69 | 33 | summary: FAIL | ||
70 | 34 | tc_cleanup: echo "cleanup for test_two" | ||
71 | 35 | tc_setup: echo "setup for test_two" | ||
72 | 36 | timeout: 300 | ||
73 | 37 | type: userland | ||
74 | 38 | - build_cmd: null | ||
75 | 39 | name: utah_tests | ||
76 | 40 | passes: 0 | ||
77 | 41 | errors: 0 | ||
78 | 42 | failures: 2 | ||
79 | 43 | status: DONE | ||
80 | 44 | tests: | ||
81 | 45 | - build_cmd: echo "building for test_one" | ||
82 | 46 | command: python test_one.py | ||
83 | 47 | description: A first sample test case. | ||
84 | 48 | name: test_one | ||
85 | 49 | path: utah_tests/test_one | ||
86 | 50 | status: DONE | ||
87 | 51 | summary: FAIL | ||
88 | 52 | tc_cleanup: echo "cleanup for test_one" | ||
89 | 53 | tc_setup: echo "setup for test_one" | ||
90 | 54 | timeout: 3 | ||
91 | 55 | type: userland | ||
92 | 56 | - build_cmd: echo "building for test_two" | ||
93 | 57 | command: python test_two.py | ||
94 | 58 | description: A second sample test case. | ||
95 | 59 | name: test_two | ||
96 | 60 | path: utah_tests/test_two | ||
97 | 61 | status: DONE | ||
98 | 62 | summary: FAIL | ||
99 | 63 | tc_cleanup: echo "cleanup for test_two" | ||
100 | 64 | tc_setup: echo "setup for test_two" | ||
101 | 65 | timeout: 300 | ||
102 | 66 | type: userland | ||
103 | 67 | timeout: 0 | ||
104 | 68 | ts_cleanup: null | ||
105 | 69 | ts_setup: null | ||
106 | 70 | - build_cmd: null | ||
107 | 71 | name: utah_tests_sample | ||
108 | 72 | passes: 0 | ||
109 | 73 | errors: 0 | ||
110 | 74 | failures: 1 | ||
111 | 75 | status: DONE | ||
112 | 76 | tests: | ||
113 | 77 | - build_cmd: echo "building for sample_one" | ||
114 | 78 | command: python sample.py | ||
115 | 79 | description: A sample test case. | ||
116 | 80 | name: sample_one | ||
117 | 81 | path: utah_tests_sample/sample_one | ||
118 | 82 | status: DONE | ||
119 | 83 | summary: FAIL | ||
120 | 84 | tc_cleanup: echo "cleanup for sample_one" | ||
121 | 85 | tc_setup: echo "setup for sample_one" | ||
122 | 86 | timeout: 200 | ||
123 | 87 | type: userland | ||
124 | 88 | timeout: 0 | ||
125 | 89 | ts_cleanup: null | ||
126 | 90 | ts_setup: null | ||
127 | 0 | 91 | ||
128 | === added file 'utah/client/examples/test_state_partial_inprogress.yaml' | |||
129 | --- utah/client/examples/test_state_partial_inprogress.yaml 1970-01-01 00:00:00 +0000 | |||
130 | +++ utah/client/examples/test_state_partial_inprogress.yaml 2012-12-05 15:31:21 +0000 | |||
131 | @@ -0,0 +1,90 @@ | |||
132 | 1 | master_runlist: /var/lib/utah/master.run | ||
133 | 2 | status: RUN | ||
134 | 3 | result_class: !!python/name:utah.client.result.ResultYAML '' | ||
135 | 4 | fetched_suites: | ||
136 | 5 | - examples | ||
137 | 6 | - utah_tests | ||
138 | 7 | - utah_tests_sample | ||
139 | 8 | suites: | ||
140 | 9 | - build_cmd: null | ||
141 | 10 | name: examples | ||
142 | 11 | passes: 2 | ||
143 | 12 | errors: 0 | ||
144 | 13 | failures: 0 | ||
145 | 14 | status: DONE | ||
146 | 15 | tests: | ||
147 | 16 | - build_cmd: echo "building for test_one" | ||
148 | 17 | command: python test_one.py | ||
149 | 18 | description: A first sample test case. | ||
150 | 19 | name: test_one | ||
151 | 20 | path: examples/test_one | ||
152 | 21 | status: DONE | ||
153 | 22 | summary: PASS | ||
154 | 23 | tc_cleanup: echo "cleanup for test_one" | ||
155 | 24 | tc_setup: echo "setup for test_one" | ||
156 | 25 | timeout: 3 | ||
157 | 26 | type: userland | ||
158 | 27 | - build_cmd: echo "building for test_two" | ||
159 | 28 | command: python test_two.py | ||
160 | 29 | description: A second sample test case. | ||
161 | 30 | name: test_two | ||
162 | 31 | path: examples/test_two | ||
163 | 32 | status: DONE | ||
164 | 33 | summary: PASS | ||
165 | 34 | tc_cleanup: echo "cleanup for test_two" | ||
166 | 35 | tc_setup: echo "setup for test_two" | ||
167 | 36 | timeout: 300 | ||
168 | 37 | type: userland | ||
169 | 38 | - build_cmd: null | ||
170 | 39 | name: utah_tests | ||
171 | 40 | passes: 1 | ||
172 | 41 | errors: 0 | ||
173 | 42 | failures: 0 | ||
174 | 43 | status: INPROGRESS | ||
175 | 44 | tests: | ||
176 | 45 | - build_cmd: echo "building for test_one" | ||
177 | 46 | command: python test_one.py | ||
178 | 47 | description: A first sample test case. | ||
179 | 48 | name: test_one | ||
180 | 49 | path: utah_tests/test_one | ||
181 | 50 | status: DONE | ||
182 | 51 | summary: PASS | ||
183 | 52 | tc_cleanup: echo "cleanup for test_one" | ||
184 | 53 | tc_setup: echo "setup for test_one" | ||
185 | 54 | timeout: 3 | ||
186 | 55 | type: userland | ||
187 | 56 | - build_cmd: echo "building for test_two" | ||
188 | 57 | command: python test_two.py | ||
189 | 58 | description: A second sample test case. | ||
190 | 59 | name: test_two | ||
191 | 60 | path: utah_tests/test_two | ||
192 | 61 | status: SETUP | ||
193 | 62 | summary: PASS | ||
194 | 63 | tc_cleanup: echo "cleanup for test_two" | ||
195 | 64 | tc_setup: echo "setup for test_two" | ||
196 | 65 | timeout: 300 | ||
197 | 66 | type: userland | ||
198 | 67 | timeout: 0 | ||
199 | 68 | ts_cleanup: null | ||
200 | 69 | ts_setup: null | ||
201 | 70 | - build_cmd: null | ||
202 | 71 | name: utah_tests_sample | ||
203 | 72 | passes: 0 | ||
204 | 73 | errors: 0 | ||
205 | 74 | failures: 1 | ||
206 | 75 | status: DONE | ||
207 | 76 | tests: | ||
208 | 77 | - build_cmd: echo "building for sample_one" | ||
209 | 78 | command: python sample.py | ||
210 | 79 | description: A sample test case. | ||
211 | 80 | name: sample_one | ||
212 | 81 | path: utah_tests_sample/sample_one | ||
213 | 82 | status: DONE | ||
214 | 83 | summary: FAIL | ||
215 | 84 | tc_cleanup: echo "cleanup for sample_one" | ||
216 | 85 | tc_setup: echo "setup for sample_one" | ||
217 | 86 | timeout: 200 | ||
218 | 87 | type: userland | ||
219 | 88 | timeout: 0 | ||
220 | 89 | ts_cleanup: null | ||
221 | 90 | ts_setup: null | ||
222 | 0 | 91 | ||
223 | === added file 'utah/client/examples/test_state_partial_run_all.yaml' | |||
224 | --- utah/client/examples/test_state_partial_run_all.yaml 1970-01-01 00:00:00 +0000 | |||
225 | +++ utah/client/examples/test_state_partial_run_all.yaml 2012-12-05 15:31:21 +0000 | |||
226 | @@ -0,0 +1,90 @@ | |||
227 | 1 | master_runlist: /var/lib/utah/master.run | ||
228 | 2 | status: RUN | ||
229 | 3 | result_class: !!python/name:utah.client.result.ResultYAML '' | ||
230 | 4 | fetched_suites: | ||
231 | 5 | - examples | ||
232 | 6 | - utah_tests | ||
233 | 7 | - utah_tests_sample | ||
234 | 8 | suites: | ||
235 | 9 | - build_cmd: null | ||
236 | 10 | name: examples | ||
237 | 11 | passes: 0 | ||
238 | 12 | errors: 0 | ||
239 | 13 | failures: 0 | ||
240 | 14 | status: NOTRUN | ||
241 | 15 | tests: | ||
242 | 16 | - build_cmd: echo "building for test_one" | ||
243 | 17 | command: python test_one.py | ||
244 | 18 | description: A first sample test case. | ||
245 | 19 | name: test_one | ||
246 | 20 | path: examples/test_one | ||
247 | 21 | status: NOTRUN | ||
248 | 22 | summary: PASS | ||
249 | 23 | tc_cleanup: echo "cleanup for test_one" | ||
250 | 24 | tc_setup: echo "setup for test_one" | ||
251 | 25 | timeout: 3 | ||
252 | 26 | type: userland | ||
253 | 27 | - build_cmd: echo "building for test_two" | ||
254 | 28 | command: python test_two.py | ||
255 | 29 | description: A second sample test case. | ||
256 | 30 | name: test_two | ||
257 | 31 | path: examples/test_two | ||
258 | 32 | status: NOTRUN | ||
259 | 33 | summary: PASS | ||
260 | 34 | tc_cleanup: echo "cleanup for test_two" | ||
261 | 35 | tc_setup: echo "setup for test_two" | ||
262 | 36 | timeout: 300 | ||
263 | 37 | type: userland | ||
264 | 38 | - build_cmd: null | ||
265 | 39 | name: utah_tests | ||
266 | 40 | passes: 1 | ||
267 | 41 | errors: 0 | ||
268 | 42 | failures: 0 | ||
269 | 43 | status: NOTRUN | ||
270 | 44 | tests: | ||
271 | 45 | - build_cmd: echo "building for test_one" | ||
272 | 46 | command: python test_one.py | ||
273 | 47 | description: A first sample test case. | ||
274 | 48 | name: test_one | ||
275 | 49 | path: utah_tests/test_one | ||
276 | 50 | status: NOTRUN | ||
277 | 51 | summary: PASS | ||
278 | 52 | tc_cleanup: echo "cleanup for test_one" | ||
279 | 53 | tc_setup: echo "setup for test_one" | ||
280 | 54 | timeout: 3 | ||
281 | 55 | type: userland | ||
282 | 56 | - build_cmd: echo "building for test_two" | ||
283 | 57 | command: python test_two.py | ||
284 | 58 | description: A second sample test case. | ||
285 | 59 | name: test_two | ||
286 | 60 | path: utah_tests/test_two | ||
287 | 61 | status: NOTRUN | ||
288 | 62 | summary: PASS | ||
289 | 63 | tc_cleanup: echo "cleanup for test_two" | ||
290 | 64 | tc_setup: echo "setup for test_two" | ||
291 | 65 | timeout: 300 | ||
292 | 66 | type: userland | ||
293 | 67 | timeout: 0 | ||
294 | 68 | ts_cleanup: null | ||
295 | 69 | ts_setup: null | ||
296 | 70 | - build_cmd: null | ||
297 | 71 | name: utah_tests_sample | ||
298 | 72 | passes: 0 | ||
299 | 73 | errors: 0 | ||
300 | 74 | failures: 1 | ||
301 | 75 | status: NOTRUN | ||
302 | 76 | tests: | ||
303 | 77 | - build_cmd: echo "building for sample_one" | ||
304 | 78 | command: python sample.py | ||
305 | 79 | description: A sample test case. | ||
306 | 80 | name: sample_one | ||
307 | 81 | path: utah_tests_sample/sample_one | ||
308 | 82 | status: NOTRUN | ||
309 | 83 | summary: PASS | ||
310 | 84 | tc_cleanup: echo "cleanup for sample_one" | ||
311 | 85 | tc_setup: echo "setup for sample_one" | ||
312 | 86 | timeout: 200 | ||
313 | 87 | type: userland | ||
314 | 88 | timeout: 0 | ||
315 | 89 | ts_cleanup: null | ||
316 | 90 | ts_setup: null | ||
317 | 0 | 91 | ||
318 | === modified file 'utah/client/runner.py' | |||
319 | --- utah/client/runner.py 2012-11-07 18:02:22 +0000 | |||
320 | +++ utah/client/runner.py 2012-12-05 15:31:21 +0000 | |||
321 | @@ -251,9 +251,9 @@ | |||
322 | 251 | if not keep_going: | 251 | if not keep_going: |
323 | 252 | return self.process_results() | 252 | return self.process_results() |
324 | 253 | 253 | ||
328 | 254 | self.errors += suite.errors | 254 | self.errors += suite.errors |
329 | 255 | self.passes += suite.passes | 255 | self.passes += suite.passes |
330 | 256 | self.failures += suite.failures | 256 | self.failures += suite.failures |
331 | 257 | 257 | ||
332 | 258 | self.status = "DONE" | 258 | self.status = "DONE" |
333 | 259 | self.save_state() | 259 | self.save_state() |
334 | @@ -443,14 +443,15 @@ | |||
335 | 443 | 443 | ||
336 | 444 | self.fetched_suites.append(name) | 444 | self.fetched_suites.append(name) |
337 | 445 | 445 | ||
346 | 446 | # Create a TestSuite | 446 | # Create a TestSuite |
347 | 447 | s = TestSuite(name=name, runlist_file=suite_runlist, | 447 | s = TestSuite(name=name, runlist_file=suite_runlist, |
348 | 448 | includes=includes, excludes=excludes, | 448 | includes=includes, excludes=excludes, |
349 | 449 | result=self.result, path=self.testsuitedir, | 449 | result=self.result, path=self.testsuitedir, |
350 | 450 | timeout=self.timeout, | 450 | timeout=self.timeout, |
351 | 451 | _save_state_callback=self.save_state, | 451 | _save_state_callback=self.save_state, |
352 | 452 | _reboot_callback=self.reboot) | 452 | _reboot_callback=self.reboot) |
353 | 453 | self.add_suite(s) | 453 | self.add_suite(s) |
354 | 454 | |||
355 | 454 | 455 | ||
356 | 455 | def get_next_suite(self): | 456 | def get_next_suite(self): |
357 | 456 | """ | 457 | """ |
358 | 457 | 458 | ||
359 | === modified file 'utah/client/testcase.py' | |||
360 | --- utah/client/testcase.py 2012-10-17 10:22:00 +0000 | |||
361 | +++ utah/client/testcase.py 2012-12-05 15:31:21 +0000 | |||
362 | @@ -27,6 +27,7 @@ | |||
363 | 27 | """ | 27 | """ |
364 | 28 | 28 | ||
365 | 29 | status = 'NOTRUN' | 29 | status = 'NOTRUN' |
366 | 30 | summary = 'PASS' # 'PASS', 'FAIL', of 'ERROR' | ||
367 | 30 | build_cmd = None | 31 | build_cmd = None |
368 | 31 | tc_setup = None | 32 | tc_setup = None |
369 | 32 | tc_cleanup = None | 33 | tc_cleanup = None |
370 | @@ -181,8 +182,7 @@ | |||
371 | 181 | """ | 182 | """ |
372 | 182 | 183 | ||
373 | 183 | if self.is_done(): | 184 | if self.is_done(): |
376 | 184 | self.run_status = 'PASS' | 185 | return 'PASS' |
375 | 185 | return self.run_status | ||
377 | 186 | 186 | ||
378 | 187 | # Do not proceed if we are missing data | 187 | # Do not proceed if we are missing data |
379 | 188 | if (self.description is None | 188 | if (self.description is None |
380 | @@ -203,7 +203,7 @@ | |||
381 | 203 | 203 | ||
382 | 204 | # if the build fails don't run the test command | 204 | # if the build fails don't run the test command |
383 | 205 | if result.status != 'PASS': | 205 | if result.status != 'PASS': |
385 | 206 | self.run_status = result.status | 206 | self.summary = 'ERROR' |
386 | 207 | 207 | ||
387 | 208 | # reset status | 208 | # reset status |
388 | 209 | result.status = 'PASS' | 209 | result.status = 'PASS' |
389 | @@ -213,7 +213,7 @@ | |||
390 | 213 | 213 | ||
391 | 214 | # if the setup fails don't run the test command | 214 | # if the setup fails don't run the test command |
392 | 215 | if result.status != 'PASS': | 215 | if result.status != 'PASS': |
394 | 216 | self.run_status = result.status | 216 | self.summary = 'ERROR' |
395 | 217 | 217 | ||
396 | 218 | # reset status | 218 | # reset status |
397 | 219 | result.status = 'PASS' | 219 | result.status = 'PASS' |
398 | @@ -237,6 +237,11 @@ | |||
399 | 237 | ), | 237 | ), |
400 | 238 | extra_info=extra_info) | 238 | extra_info=extra_info) |
401 | 239 | 239 | ||
402 | 240 | # only if we haven't failed or errored so far do | ||
403 | 241 | # we set the summary for the testcase. | ||
404 | 242 | if self.summary == 'PASS': | ||
405 | 243 | self.summary = result.status | ||
406 | 244 | |||
407 | 240 | # Clean up whether 'command' failed or not. | 245 | # Clean up whether 'command' failed or not. |
408 | 241 | self.cleanup(result) | 246 | self.cleanup(result) |
409 | 242 | 247 | ||
410 | @@ -246,7 +251,6 @@ | |||
411 | 246 | and result.status == 'PASS'): | 251 | and result.status == 'PASS'): |
412 | 247 | need_reboot = True | 252 | need_reboot = True |
413 | 248 | 253 | ||
414 | 249 | self.run_status = result.status | ||
415 | 250 | result.status = 'PASS' | 254 | result.status = 'PASS' |
416 | 251 | 255 | ||
417 | 252 | if self.status == 'CLEANUP': | 256 | if self.status == 'CLEANUP': |
418 | @@ -285,6 +289,7 @@ | |||
419 | 285 | 'command': self.command, | 289 | 'command': self.command, |
420 | 286 | 'timeout': self.timeout, | 290 | 'timeout': self.timeout, |
421 | 287 | 'status': self.status, | 291 | 'status': self.status, |
422 | 292 | 'summary': self.summary, | ||
423 | 288 | 'build_cmd': self.build_cmd, | 293 | 'build_cmd': self.build_cmd, |
424 | 289 | 'tc_setup': self.tc_setup, | 294 | 'tc_setup': self.tc_setup, |
425 | 290 | 'tc_cleanup': self.tc_cleanup, | 295 | 'tc_cleanup': self.tc_cleanup, |
426 | 291 | 296 | ||
427 | === modified file 'utah/client/tests/common.py' | |||
428 | --- utah/client/tests/common.py 2012-10-10 14:44:00 +0000 | |||
429 | +++ utah/client/tests/common.py 2012-12-05 15:31:21 +0000 | |||
430 | @@ -89,3 +89,30 @@ | |||
431 | 89 | 89 | ||
432 | 90 | if os.path.exists(examples_dir): | 90 | if os.path.exists(examples_dir): |
433 | 91 | shutil.rmtree(examples_dir) | 91 | shutil.rmtree(examples_dir) |
434 | 92 | |||
435 | 93 | def _get_partial_state_file(filename): | ||
436 | 94 | """ | ||
437 | 95 | Read a partial state file from disk. | ||
438 | 96 | """ | ||
439 | 97 | |||
440 | 98 | state = None | ||
441 | 99 | |||
442 | 100 | with open(filename, 'r') as fp: | ||
443 | 101 | state = fp.read() | ||
444 | 102 | |||
445 | 103 | return state | ||
446 | 104 | |||
447 | 105 | partial_state_file_content = _get_partial_state_file( | ||
448 | 106 | os.path.join(os.path.dirname(__file__), "..", "examples", | ||
449 | 107 | "test_state_partial_inprogress.yaml") | ||
450 | 108 | ) | ||
451 | 109 | |||
452 | 110 | partial_state_file_content_run_all = _get_partial_state_file( | ||
453 | 111 | os.path.join(os.path.dirname(__file__), "..", "examples", | ||
454 | 112 | "test_state_partial_run_all.yaml") | ||
455 | 113 | ) | ||
456 | 114 | |||
457 | 115 | partial_state_file_content_done_all_failed = _get_partial_state_file( | ||
458 | 116 | os.path.join(os.path.dirname(__file__), "..", "examples", | ||
459 | 117 | "test_state_partial_all_failed.yaml") | ||
460 | 118 | ) | ||
461 | 92 | 119 | ||
462 | === modified file 'utah/client/tests/test_state_agent.py' | |||
463 | --- utah/client/tests/test_state_agent.py 2012-10-03 09:07:18 +0000 | |||
464 | +++ utah/client/tests/test_state_agent.py 2012-12-05 15:31:21 +0000 | |||
465 | @@ -12,60 +12,11 @@ | |||
466 | 12 | setUp, # Used by nosetests | 12 | setUp, # Used by nosetests |
467 | 13 | tearDown, # Used by nosetests | 13 | tearDown, # Used by nosetests |
468 | 14 | get_module_path, | 14 | get_module_path, |
469 | 15 | partial_state_file_content, | ||
470 | 16 | partial_state_file_content_run_all, | ||
471 | 17 | partial_state_file_content_done_all_failed, | ||
472 | 15 | ) | 18 | ) |
473 | 16 | 19 | ||
474 | 17 | partial_state_file_content = """master_runlist: /var/lib/utah/master.run | ||
475 | 18 | status: RUN | ||
476 | 19 | result_class: !!python/name:utah.client.result.ResultYAML '' | ||
477 | 20 | fetched_suites: | ||
478 | 21 | - utah_tests | ||
479 | 22 | - utah_tests_sample | ||
480 | 23 | suites: | ||
481 | 24 | - build_cmd: null | ||
482 | 25 | name: utah_tests | ||
483 | 26 | status: INPROGRESS | ||
484 | 27 | tests: | ||
485 | 28 | - build_cmd: echo "building for test_one" | ||
486 | 29 | command: python test_one.py | ||
487 | 30 | description: A first sample test case. | ||
488 | 31 | name: test_one | ||
489 | 32 | path: utah_tests/test_one | ||
490 | 33 | status: DONE | ||
491 | 34 | tc_cleanup: echo "cleanup for test_one" | ||
492 | 35 | tc_setup: echo "setup for test_one" | ||
493 | 36 | timeout: 3 | ||
494 | 37 | type: userland | ||
495 | 38 | - build_cmd: echo "building for test_two" | ||
496 | 39 | command: python test_two.py | ||
497 | 40 | description: A second sample test case. | ||
498 | 41 | name: test_two | ||
499 | 42 | path: utah_tests/test_two | ||
500 | 43 | status: SETUP | ||
501 | 44 | tc_cleanup: echo "cleanup for test_two" | ||
502 | 45 | tc_setup: echo "setup for test_two" | ||
503 | 46 | timeout: 300 | ||
504 | 47 | type: userland | ||
505 | 48 | timeout: 0 | ||
506 | 49 | ts_cleanup: null | ||
507 | 50 | ts_setup: null | ||
508 | 51 | - build_cmd: null | ||
509 | 52 | name: utah_tests_sample | ||
510 | 53 | status: DONE | ||
511 | 54 | tests: | ||
512 | 55 | - build_cmd: echo "building for sample_one" | ||
513 | 56 | command: python sample.py | ||
514 | 57 | description: A sample test case. | ||
515 | 58 | name: sample_one | ||
516 | 59 | path: utah_tests_sample/sample_one | ||
517 | 60 | status: DONE | ||
518 | 61 | tc_cleanup: echo "cleanup for sample_one" | ||
519 | 62 | tc_setup: echo "setup for sample_one" | ||
520 | 63 | timeout: 200 | ||
521 | 64 | type: userland | ||
522 | 65 | timeout: 0 | ||
523 | 66 | ts_cleanup: null | ||
524 | 67 | ts_setup: null | ||
525 | 68 | """ | ||
526 | 69 | 20 | ||
527 | 70 | 21 | ||
528 | 71 | class TestStateAgentYAML(unittest.TestCase): | 22 | class TestStateAgentYAML(unittest.TestCase): |
529 | @@ -83,6 +34,8 @@ | |||
530 | 83 | self.runner = Runner(install_type="desktop", result_class=ResultYAML, | 34 | self.runner = Runner(install_type="desktop", result_class=ResultYAML, |
531 | 84 | state_agent=self.state_agent) | 35 | state_agent=self.state_agent) |
532 | 85 | self.partial_state_file_content = partial_state_file_content | 36 | self.partial_state_file_content = partial_state_file_content |
533 | 37 | self.partial_state_file_content_run_all = partial_state_file_content_run_all | ||
534 | 38 | self.partial_state_file_content_done_all_failed = partial_state_file_content_done_all_failed | ||
535 | 86 | 39 | ||
536 | 87 | def tearDown(self): | 40 | def tearDown(self): |
537 | 88 | """ | 41 | """ |
538 | @@ -159,14 +112,108 @@ | |||
539 | 159 | shutil.copytree(test_src, test_dir) | 112 | shutil.copytree(test_src, test_dir) |
540 | 160 | 113 | ||
541 | 161 | state_agent = StateAgentYAML(state_file=self.state_file) | 114 | state_agent = StateAgentYAML(state_file=self.state_file) |
545 | 162 | runner = Runner(install_type="desktop", state_agent=state_agent, | 115 | runner = Runner(install_type="desktop", |
546 | 163 | resume=True) | 116 | state_agent=state_agent, |
547 | 164 | runner.load_state() | 117 | result_class=ResultYAML, |
548 | 118 | resume=True) | ||
549 | 165 | 119 | ||
550 | 166 | self.assertEqual(runner.status, 'RUN') | 120 | self.assertEqual(runner.status, 'RUN') |
551 | 121 | self.assertEqual(runner.count_suites(), 3) | ||
552 | 122 | self.assertEqual(runner.count_tests(), 5) | ||
553 | 167 | 123 | ||
554 | 168 | next_test = runner.get_next_test() | 124 | next_test = runner.get_next_test() |
555 | 169 | 125 | ||
556 | 170 | self.assertEqual(next_test.name, 'test_two') | 126 | self.assertEqual(next_test.name, 'test_two') |
557 | 171 | 127 | ||
558 | 172 | runner.run() | 128 | runner.run() |
559 | 129 | |||
560 | 130 | self.assertEqual(runner.passes, 4) | ||
561 | 131 | self.assertEqual(runner.failures, 1) | ||
562 | 132 | self.assertEqual(runner.errors, 0) | ||
563 | 133 | self.assertEqual(runner.passes + runner.failures + runner.errors, 5) | ||
564 | 134 | |||
565 | 135 | def test_load_state_partial_run_all(self): | ||
566 | 136 | """ | ||
567 | 137 | Test that a partially run state file can be resumed. | ||
568 | 138 | """ | ||
569 | 139 | fp = open(self.state_file, 'w') | ||
570 | 140 | fp.write(self.partial_state_file_content_run_all) | ||
571 | 141 | fp.close() | ||
572 | 142 | |||
573 | 143 | # Need to set up utah_tests since we're simulating a restart | ||
574 | 144 | test_src = os.path.join(get_module_path(), 'client', | ||
575 | 145 | 'examples', 'utah_tests') | ||
576 | 146 | test_dir = os.path.join(UTAH_DIR, 'testsuites', | ||
577 | 147 | 'utah_tests', 'utah_tests') | ||
578 | 148 | if not os.path.exists(test_dir): | ||
579 | 149 | shutil.copytree(test_src, test_dir) | ||
580 | 150 | |||
581 | 151 | # Need to set up utah_tests_sample since we're simulating a restart | ||
582 | 152 | test_src = os.path.join(get_module_path(), 'client', | ||
583 | 153 | 'examples', 'utah_tests_sample') | ||
584 | 154 | test_dir = os.path.join(UTAH_DIR, 'testsuites', 'utah_tests_sample', | ||
585 | 155 | 'utah_tests_sample') | ||
586 | 156 | if not os.path.exists(test_dir): | ||
587 | 157 | shutil.copytree(test_src, test_dir) | ||
588 | 158 | |||
589 | 159 | state_agent = StateAgentYAML(state_file=self.state_file) | ||
590 | 160 | runner = Runner(install_type="desktop", | ||
591 | 161 | state_agent=state_agent, | ||
592 | 162 | result_class=ResultYAML, | ||
593 | 163 | resume=True) | ||
594 | 164 | |||
595 | 165 | self.assertEqual(runner.status, 'RUN') | ||
596 | 166 | self.assertEqual(runner.count_suites(), 3) | ||
597 | 167 | self.assertEqual(runner.count_tests(), 5) | ||
598 | 168 | |||
599 | 169 | next_test = runner.get_next_test() | ||
600 | 170 | |||
601 | 171 | self.assertEqual(next_test.name, 'test_one') | ||
602 | 172 | |||
603 | 173 | runner.run() | ||
604 | 174 | |||
605 | 175 | self.assertEqual(runner.passes, 3) | ||
606 | 176 | self.assertEqual(runner.failures, 0) | ||
607 | 177 | self.assertEqual(runner.errors, 2) | ||
608 | 178 | self.assertEqual(runner.passes + runner.failures + runner.errors, 5) | ||
609 | 179 | |||
610 | 180 | def test_load_state_partial_done_all_failed(self): | ||
611 | 181 | """ | ||
612 | 182 | Test that a partially run state file can be resumed. | ||
613 | 183 | """ | ||
614 | 184 | fp = open(self.state_file, 'w') | ||
615 | 185 | fp.write(self.partial_state_file_content_done_all_failed) | ||
616 | 186 | fp.close() | ||
617 | 187 | |||
618 | 188 | # Need to set up utah_tests since we're simulating a restart | ||
619 | 189 | test_src = os.path.join(get_module_path(), 'client', | ||
620 | 190 | 'examples', 'utah_tests') | ||
621 | 191 | test_dir = os.path.join(UTAH_DIR, 'testsuites', | ||
622 | 192 | 'utah_tests', 'utah_tests') | ||
623 | 193 | if not os.path.exists(test_dir): | ||
624 | 194 | shutil.copytree(test_src, test_dir) | ||
625 | 195 | |||
626 | 196 | # Need to set up utah_tests_sample since we're simulating a restart | ||
627 | 197 | test_src = os.path.join(get_module_path(), 'client', | ||
628 | 198 | 'examples', 'utah_tests_sample') | ||
629 | 199 | test_dir = os.path.join(UTAH_DIR, 'testsuites', 'utah_tests_sample', | ||
630 | 200 | 'utah_tests_sample') | ||
631 | 201 | if not os.path.exists(test_dir): | ||
632 | 202 | shutil.copytree(test_src, test_dir) | ||
633 | 203 | |||
634 | 204 | state_agent = StateAgentYAML(state_file=self.state_file) | ||
635 | 205 | runner = Runner(install_type="desktop", | ||
636 | 206 | state_agent=state_agent, | ||
637 | 207 | result_class=ResultYAML, | ||
638 | 208 | resume=True) | ||
639 | 209 | |||
640 | 210 | self.assertEqual(runner.status, 'RUN') | ||
641 | 211 | self.assertEqual(runner.count_suites(), 3) | ||
642 | 212 | self.assertEqual(runner.count_tests(), 5) | ||
643 | 213 | |||
644 | 214 | runner.run() | ||
645 | 215 | |||
646 | 216 | self.assertEqual(runner.passes, 0) | ||
647 | 217 | self.assertEqual(runner.failures, 5) | ||
648 | 218 | self.assertEqual(runner.errors, 0) | ||
649 | 219 | self.assertEqual(runner.passes + runner.failures + runner.errors, 5) | ||
650 | 173 | 220 | ||
651 | === modified file 'utah/client/testsuite.py' | |||
652 | --- utah/client/testsuite.py 2012-10-22 08:15:08 +0000 | |||
653 | +++ utah/client/testsuite.py 2012-12-05 15:31:21 +0000 | |||
654 | @@ -204,6 +204,9 @@ | |||
655 | 204 | Run the test cases; including any build, setup, and cleanup commands. | 204 | Run the test cases; including any build, setup, and cleanup commands. |
656 | 205 | """ | 205 | """ |
657 | 206 | 206 | ||
658 | 207 | # we gather this info from the testcases | ||
659 | 208 | self.passes = self.errors = self.failures = 0 | ||
660 | 209 | |||
661 | 207 | # Return value to indicate whether processing of a Runner should | 210 | # Return value to indicate whether processing of a Runner should |
662 | 208 | # continue. This is to avoid a shutdown race on reboot cases. | 211 | # continue. This is to avoid a shutdown race on reboot cases. |
663 | 209 | keep_going = True | 212 | keep_going = True |
664 | @@ -226,11 +229,11 @@ | |||
665 | 226 | for test in self.tests: | 229 | for test in self.tests: |
666 | 227 | keep_going = test.run() | 230 | keep_going = test.run() |
667 | 228 | 231 | ||
669 | 229 | if test.run_status == 'PASS': | 232 | if test.summary == 'PASS': |
670 | 230 | self.passes += 1 | 233 | self.passes += 1 |
672 | 231 | elif test.run_status == 'FAIL': | 234 | elif test.summary == 'FAIL': |
673 | 232 | self.failures += 1 | 235 | self.failures += 1 |
675 | 233 | elif test.run_status == 'ERROR': | 236 | elif test.summary == 'ERROR': |
676 | 234 | self.errors += 1 | 237 | self.errors += 1 |
677 | 235 | 238 | ||
678 | 236 | if not keep_going: | 239 | if not keep_going: |
I've got a couple of questions:
- What does JAT stand for? I'd prefer all those commented lines to be removed.
- Why is there a long yaml file embedded in the code? I think maintenance would be easier if the file was in a separate location on its own.