Merge lp:~kissiel/checkbox/rerun-in-cbt into lp:checkbox
- rerun-in-cbt
- Merge into trunk
Status: | Merged |
---|---|
Approved by: | Sylvain Pineau |
Approved revision: | 3917 |
Merged at revision: | 3914 |
Proposed branch: | lp:~kissiel/checkbox/rerun-in-cbt |
Merge into: | lp:checkbox |
Diff against target: |
323 lines (+168/-33) 6 files modified
checkbox-touch/checkbox-touch.qml (+34/-1) checkbox-touch/components/CheckboxTouchApplication.qml (+6/-0) checkbox-touch/components/SelectionPage.qml (+3/-1) checkbox-touch/py/checkbox_touch.py (+22/-0) checkbox-touch/tests/autopilot/checkbox_touch/__init__.py (+33/-0) checkbox-touch/tests/autopilot/checkbox_touch/test_checkbox_touch.py (+70/-31) |
To merge this branch: | bzr merge lp:~kissiel/checkbox/rerun-in-cbt |
Related bugs: |
Reviewer | Review Type | Date Requested | Status |
---|---|---|---|
Sylvain Pineau (community) | Approve | ||
Review via email: mp+265232@code.launchpad.net |
Commit message
Description of the change
This MR brings re-run feature to checkbox-
6e98004 chekcbox-touch: add function getting rerun candidates to py logic
d873a99 checkbox-touch: invalidate cache and reset index when tests are started
a7feee8 checkbox-touch: add proxy-ing to get rerun candidates in qml logic
214224d checkbox-touch: add emptyAllowed option to selection screen
e58246e checkbox-touch: add rerun screen
2db9319 checkbox-touch: run re-run before results screen
5d71b5b checkbox-
c0e491a checkbox-
7d49391 checkbox-
- 3905. By Zygmunt Krynicki
-
"automatic merge of lp:~zyga/checkbox/fix-1476136/ by tarmac [r=sylvain-pineau][bug=1476136][author=zyga]"
- 3906. By Maciej Kisielewski
-
"automatic merge of lp:~kissiel/checkbox/remove-lxml-from-cbt-reqs/ by tarmac [r=sylvain-
pineau, zyga][bug= ][author= kissiel] "
- 3907. By Maciej Kisielewski
-
checkbox-touch: remember test_plan_id once it's selected
Signed-off-by: Maciej Kisielewski <email address hidden>
- 3908. By Maciej Kisielewski
-
checkbox-touch: make resuming re-initiate test-plan
Signed-off-by: Maciej Kisielewski <email address hidden>
- 3909. By Maciej Kisielewski
-
chekcbox-touch: add function getting rerun candidates to py logic
Signed-off-by: Maciej Kisielewski <email address hidden>
- 3910. By Maciej Kisielewski
-
checkbox-touch: invalidate cache and reset index when tests are started
Signed-off-by: Maciej Kisielewski <email address hidden>
- 3911. By Maciej Kisielewski
-
checkbox-touch: add proxy-ing to get rerun candidates in qml logic
Signed-off-by: Maciej Kisielewski <email address hidden>
- 3912. By Maciej Kisielewski
-
checkbox-touch: add emptyAllowed option to selection screen
This patch adds an option to SelectionPage component that lets user proceed
even if nothing is selected.Signed-off-by: Maciej Kisielewski <email address hidden>
- 3913. By Maciej Kisielewski
-
checkbox-touch: add rerun screen
Signed-off-by: Maciej Kisielewski <email address hidden>
- 3914. By Maciej Kisielewski
-
checkbox-touch: run re-run before results screen
Signed-off-by: Maciej Kisielewski <email address hidden>
- 3915. By Maciej Kisielewski
-
checkbox-
touch:autopilot : move processing clicks helper to top-class This patch moves the helper that executes common scenario of clicking through
pages, to the ClickAppTestCase class, so all our test classes inherit this.Signed-off-by: Maciej Kisielewski <email address hidden>
- 3916. By Maciej Kisielewski
-
checkbox-
touch:autopilot : add generic check_results helper This patch adds a helper function that checks if the result screen is shown
with the expected values.Signed-off-by: Maciej Kisielewski <email address hidden>
- 3917. By Maciej Kisielewski
-
checkbox-
touch:autopilot : add re-run scenarios for AP Signed-off-by: Maciej Kisielewski <email address hidden>
Maciej Kisielewski (kissiel) wrote : | # |
> When we landed the same feature for checkbox-cli, I thought that being unable
> to rerun every test would miss. But actually the only concern was about
> selection of all jobs dependencies (i.e suspend again if the test was
> depending on suspend_auto). So I agree with the decision to only select
> failed/crash test results.
>
> Tests are presented with their respective categories which is great (clearly
> lacking in checkbox-gui).
>
> So conclusion is a +1 for the design choices.
>
> I ran the following sequence which was crashing the app:
>
> - selected both sensor and screen categories from the qml provider
> - failed one in each category
> - rerun one succesfully
> - killed cbt with a swipe move on the screen
> - when prompted to resume the session selected rerun last test
> - failed again the test "ensure haptics are working"
> - at the final screen, select finish
> - cbt controls are greyed out, app is blocked
>
> The upstart log: http://
Rerun + resume now works. Tested with your scenario, and the scenario when resume is done before any job is rerun.
Sylvain Pineau (sylvain-pineau) wrote : | # |
Re tested, it works! Thanks
Preview Diff
1 | === modified file 'checkbox-touch/checkbox-touch.qml' |
2 | --- checkbox-touch/checkbox-touch.qml 2015-07-16 13:58:31 +0000 |
3 | +++ checkbox-touch/checkbox-touch.qml 2015-07-23 08:26:16 +0000 |
4 | @@ -357,6 +357,30 @@ |
5 | } |
6 | } |
7 | |
8 | + SelectionPage { |
9 | + id: rerunSelectionPage |
10 | + objectName: "rerunSelectionPage" |
11 | + title: i18n.tr("Select tests to re-run") |
12 | + continueText: state == "empty selection" ? |
13 | + i18n.tr("Finish") : i18n.tr("Re-run") |
14 | + emptyAllowed: true |
15 | + |
16 | + function setup(rerunCandidates, continuation) { |
17 | + model.clear(); |
18 | + for (var i=0; i<rerunCandidates.length; i++) { |
19 | + model.append(rerunCandidates[i]); |
20 | + } |
21 | + modelUpdated(); |
22 | + pageStack.push(rerunSelectionPage) |
23 | + } |
24 | + onSelectionDone: { |
25 | + app.rememberTestSelection(selected_id_list, function() { |
26 | + processNextTest(); |
27 | + unlatchContinue(); |
28 | + }); |
29 | + } |
30 | + } |
31 | + |
32 | PasswordDialog { |
33 | id: passwordDialog |
34 | } |
35 | @@ -388,7 +412,7 @@ |
36 | app.getNextTest(function(test) { |
37 | pageStack.clear(); |
38 | if (test.plugin === undefined) { |
39 | - return showResultsScreen(); |
40 | + return maybeShowRerunScreen(); |
41 | } |
42 | if (test.user) { |
43 | // running this test will require to be run as a different |
44 | @@ -449,6 +473,15 @@ |
45 | app.runTestActivity(test, continuation); |
46 | |
47 | } |
48 | + function maybeShowRerunScreen() { |
49 | + app.getRerunCandidates(function(result) { |
50 | + if (result == false) { |
51 | + showResultsScreen(); |
52 | + return; |
53 | + } |
54 | + rerunSelectionPage.setup(result); |
55 | + }); |
56 | + } |
57 | |
58 | function showResultsScreen() { |
59 | pageStack.clear(); |
60 | |
61 | === modified file 'checkbox-touch/components/CheckboxTouchApplication.qml' |
62 | --- checkbox-touch/components/CheckboxTouchApplication.qml 2015-06-30 22:20:49 +0000 |
63 | +++ checkbox-touch/components/CheckboxTouchApplication.qml 2015-07-23 08:26:16 +0000 |
64 | @@ -138,6 +138,12 @@ |
65 | }); |
66 | } |
67 | |
68 | + function getRerunCandidates(continuation) { |
69 | + request("get_rerun_candidates", [], continuation, function(error) { |
70 | + console.error("Unable to get rerun candidates"); |
71 | + }); |
72 | + } |
73 | + |
74 | function getResults(continuation) { |
75 | request("get_results", [], continuation, function(error) { |
76 | console.error("Unable to get test results"); |
77 | |
78 | === modified file 'checkbox-touch/components/SelectionPage.qml' |
79 | --- checkbox-touch/components/SelectionPage.qml 2015-07-17 08:36:57 +0000 |
80 | +++ checkbox-touch/components/SelectionPage.qml 2015-07-23 08:26:16 +0000 |
81 | @@ -36,6 +36,7 @@ |
82 | property string continueText: i18n.tr("Continue") |
83 | readonly property alias model: selectionModel |
84 | property bool onlyOneAllowed: false |
85 | + property bool emptyAllowed: false |
86 | |
87 | visible: false |
88 | flickable: null |
89 | @@ -100,7 +101,8 @@ |
90 | states: [ |
91 | State { |
92 | name: "empty selection" |
93 | - PropertyChanges { target: continueButton; enabled: false } |
94 | + PropertyChanges { target: continueButton; |
95 | + enabled: false || emptyAllowed } |
96 | }, |
97 | State { |
98 | name: "nonempty selection" |
99 | |
100 | === modified file 'checkbox-touch/py/checkbox_touch.py' |
101 | --- checkbox-touch/py/checkbox_touch.py 2015-07-16 21:25:01 +0000 |
102 | +++ checkbox-touch/py/checkbox_touch.py 2015-07-23 08:26:16 +0000 |
103 | @@ -248,6 +248,7 @@ |
104 | self.context.provider_list, |
105 | os.path.join(self.manager.storage.location, 'io-logs')) |
106 | self.index = app_blob['index_in_run_list'] |
107 | + self._init_test_plan_id(app_blob['test_plan_id']) |
108 | _logger.error(self.context.state.run_list) |
109 | _logger.error(self.index) |
110 | if not rerun_last_test: |
111 | @@ -399,6 +400,24 @@ |
112 | return { |
113 | 'test_info_list': test_info_list |
114 | } |
115 | + @view |
116 | + def get_rerun_candidates(self): |
117 | + def rerun_predicate(job_state): |
118 | + return job_state.result.outcome in ( |
119 | + IJobResult.OUTCOME_FAIL, IJobResult.OUTCOME_CRASH) |
120 | + id_map = self.context.compute_shared( |
121 | + 'id_map', compute_value_map, self.context, 'id') |
122 | + rerun_candidates = [] |
123 | + for job in self.manager.state.run_list: |
124 | + if rerun_predicate(self.manager.state.job_state_map[job.id]): |
125 | + rerun_candidates.append({ |
126 | + "mod_id": job.id, |
127 | + "mod_name": job.tr_summary(), |
128 | + "mod_group": id_map[job.category_id][0].tr_name(), |
129 | + "mod_selected": False |
130 | + |
131 | + }) |
132 | + return rerun_candidates |
133 | |
134 | @view |
135 | def remember_tests(self, selected_id_list): |
136 | @@ -407,6 +426,8 @@ |
137 | """ |
138 | self.desired_test_ids = frozenset(selected_id_list) |
139 | _logger.info("Selected tests: %s", self.desired_test_ids) |
140 | + self.index = 0 |
141 | + self.context.invalidate_shared('desired_job_list') |
142 | desired_job_list = self.context.compute_shared( |
143 | 'desired_job_list', select_jobs, |
144 | self.context.state.job_list, [ |
145 | @@ -611,6 +632,7 @@ |
146 | if test_plan.Meta.name != 'test plan': |
147 | raise ValueError( |
148 | "unit {!r} is not a test plan".format(test_plan_id)) |
149 | + self.test_plan_id = test_plan_id |
150 | self.test_plan = test_plan |
151 | |
152 | def _init_session_storage_repo(self): |
153 | |
154 | === modified file 'checkbox-touch/tests/autopilot/checkbox_touch/__init__.py' |
155 | --- checkbox-touch/tests/autopilot/checkbox_touch/__init__.py 2015-07-15 15:32:59 +0000 |
156 | +++ checkbox-touch/tests/autopilot/checkbox_touch/__init__.py 2015-07-23 08:26:16 +0000 |
157 | @@ -89,6 +89,39 @@ |
158 | objectName='continueButton') |
159 | self.pointing_device.click_object(continue_btn) |
160 | |
161 | + def process_sequence_of_clicks_on_pages(self, steps): |
162 | + """ |
163 | + Do a sequence of clicks on simple page->component hierarchies. |
164 | + |
165 | + :param steps: |
166 | + sequence of (page-objectName, component-objectName) pairs to go |
167 | + through. |
168 | + |
169 | + Typical run of checkbox-touch requires user to go through a sequence of |
170 | + pages that have pass/fail buttons on them. This function helps go |
171 | + through a sequence like that. |
172 | + """ |
173 | + for parent, component in steps: |
174 | + self.app.wait_select_single( |
175 | + objectName=parent, visible=True) |
176 | + clickable = self.main_view.wait_select_single( |
177 | + objectName=component, visible=True) |
178 | + self.pointing_device.click_object(clickable) |
179 | + |
180 | + def check_results(self, results): |
181 | + results_page = self.app.wait_select_single( |
182 | + objectName='resultsPage', visible=True) |
183 | + lbl_passed = results_page.wait_select_single(objectName='passedLabel') |
184 | + self.assertThat(lbl_passed.text.startswith(results['passed']), |
185 | + Equals(True)) |
186 | + lbl_failed = results_page.wait_select_single(objectName='failedLabel') |
187 | + self.assertThat(lbl_failed.text.startswith(results['failed']), |
188 | + Equals(True)) |
189 | + lbl_skipped = results_page.wait_select_single( |
190 | + objectName='skippedLabel') |
191 | + self.assertThat(lbl_skipped.text.startswith(results['skipped']), |
192 | + Equals(True)) |
193 | + |
194 | def launch_application(self): |
195 | if platform.model() == 'Desktop': |
196 | self._launch_application_from_desktop() |
197 | |
198 | === modified file 'checkbox-touch/tests/autopilot/checkbox_touch/test_checkbox_touch.py' |
199 | --- checkbox-touch/tests/autopilot/checkbox_touch/test_checkbox_touch.py 2015-07-15 15:32:57 +0000 |
200 | +++ checkbox-touch/tests/autopilot/checkbox_touch/test_checkbox_touch.py 2015-07-23 08:26:16 +0000 |
201 | @@ -23,25 +23,6 @@ |
202 | yes_btn = dialog.select_single(objectName='yesButton') |
203 | self.pointing_device.click_object(yes_btn) |
204 | |
205 | - def process_sequence_of_clicks_on_pages(self, steps): |
206 | - """ |
207 | - Do a sequence of clicks on simple page->component hierarchies. |
208 | - |
209 | - :param steps: |
210 | - sequence of (page-objectName, component-objectName) pairs to go |
211 | - through. |
212 | - |
213 | - Typical run of checkbox-touch requires user to go through a sequence of |
214 | - pages that have pass/fail buttons on them. This function helps go |
215 | - through a sequence like that. |
216 | - """ |
217 | - for parent, component in steps: |
218 | - self.app.wait_select_single( |
219 | - objectName=parent, visible=True) |
220 | - clickable = self.main_view.wait_select_single( |
221 | - objectName=component, visible=True) |
222 | - self.pointing_device.click_object(clickable) |
223 | - |
224 | def test_launches(self): |
225 | main_view = self.app.select_single(objectName='mainView') |
226 | self.assertThat(main_view.visible, Eventually(Equals(True))) |
227 | @@ -116,22 +97,12 @@ |
228 | next_steps = [ |
229 | ('qmlNativePage', 'continueButton'), |
230 | ('qmlTestPage', 'passButton'), |
231 | + ('rerunSelectionPage', 'continueButton') |
232 | ] |
233 | self.process_sequence_of_clicks_on_pages(next_steps) |
234 | # we should see results screen now |
235 | - results_page = self.app.wait_select_single( |
236 | - objectName='resultsPage', visible=True) |
237 | results = {'passed': '10', 'failed': '5', 'skipped': '5'} |
238 | - lbl_passed = results_page.wait_select_single(objectName='passedLabel') |
239 | - self.assertThat(lbl_passed.text.startswith(results['passed']), |
240 | - Equals(True)) |
241 | - lbl_failed = results_page.wait_select_single(objectName='failedLabel') |
242 | - self.assertThat(lbl_failed.text.startswith(results['failed']), |
243 | - Equals(True)) |
244 | - lbl_skipped = results_page.wait_select_single( |
245 | - objectName='skippedLabel') |
246 | - self.assertThat(lbl_skipped.text.startswith(results['skipped']), |
247 | - Equals(True)) |
248 | + self.check_results(results) |
249 | |
250 | |
251 | class SessionResumeTests(checkbox_touch.ClickAppTestCase): |
252 | @@ -228,3 +199,71 @@ |
253 | text_area = output_page.wait_select_single( |
254 | objectName='textArea', visible=True) |
255 | self.assertThat(text_area.text, Eventually(Equals("foobar\n"))) |
256 | + |
257 | + |
258 | +class RerunTests(checkbox_touch.ClickAppTestCase): |
259 | + def test_rerun_after_rerun(self): |
260 | + test_id = '2015.com.canonical.certification::autopilot/manual-2' |
261 | + self.start_and_select_tests( |
262 | + '2015.com.canonical.certification::normal', [test_id]) |
263 | + next_steps = [ |
264 | + ('manualIntroPage', 'continueButton'), |
265 | + ('testVerificationPage', 'failButton'), |
266 | + ] |
267 | + self.process_sequence_of_clicks_on_pages(next_steps) |
268 | + # we now should see a re-run screen; let's select the only test |
269 | + rerun_page = self.app.wait_select_single( |
270 | + objectName='rerunSelectionPage', visible=True) |
271 | + list_item = rerun_page.wait_select_single( |
272 | + objectName='listItem', item_mod_id=test_id) |
273 | + self.pointing_device.click_object(list_item) |
274 | + continue_btn = rerun_page.wait_select_single( |
275 | + objectName='continueButton', visible=True) |
276 | + self.pointing_device.click_object(continue_btn) |
277 | + # run the same steps as before |
278 | + self.process_sequence_of_clicks_on_pages(next_steps) |
279 | + # we should see the re-run screen again |
280 | + rerun_page = self.app.wait_select_single( |
281 | + objectName='rerunSelectionPage', visible=True) |
282 | + continue_btn = rerun_page.wait_select_single( |
283 | + objectName='continueButton', visible=True) |
284 | + self.pointing_device.click_object(continue_btn) |
285 | + self.check_results({'passed': '0', 'failed': '1', 'skipped': '0'}) |
286 | + |
287 | + def test_rerun_after_fail(self): |
288 | + test_id = '2015.com.canonical.certification::autopilot/manual-2' |
289 | + self.start_and_select_tests( |
290 | + '2015.com.canonical.certification::normal', [test_id]) |
291 | + next_steps = [ |
292 | + ('manualIntroPage', 'continueButton'), |
293 | + ('testVerificationPage', 'failButton'), |
294 | + ] |
295 | + self.process_sequence_of_clicks_on_pages(next_steps) |
296 | + # we now should see a re-run screen; let's select the only test |
297 | + rerun_page = self.app.wait_select_single( |
298 | + objectName='rerunSelectionPage', visible=True) |
299 | + list_item = rerun_page.wait_select_single( |
300 | + objectName='listItem', item_mod_id=test_id) |
301 | + self.pointing_device.click_object(list_item) |
302 | + continue_btn = rerun_page.wait_select_single( |
303 | + objectName='continueButton', visible=True) |
304 | + self.pointing_device.click_object(continue_btn) |
305 | + next_steps = [ |
306 | + ('manualIntroPage', 'continueButton'), |
307 | + ('testVerificationPage', 'passButton'), |
308 | + ] |
309 | + self.process_sequence_of_clicks_on_pages(next_steps) |
310 | + # now set the outcome to 'pass'; we should be on results screen now |
311 | + self.check_results({'passed': '1', 'failed': '0', 'skipped': '0'}) |
312 | + |
313 | + def test_no_rerun_after_pass(self): |
314 | + test_id = '2015.com.canonical.certification::autopilot/manual-1' |
315 | + self.start_and_select_tests( |
316 | + '2015.com.canonical.certification::normal', [test_id]) |
317 | + next_steps = [ |
318 | + ('manualIntroPage', 'continueButton'), |
319 | + ('testVerificationPage', 'passButton'), |
320 | + ] |
321 | + self.process_sequence_of_clicks_on_pages(next_steps) |
322 | + # there should be no re-run screen, just results |
323 | + self.check_results({'passed': '1', 'failed': '0', 'skipped': '0'}) |
When we landed the same feature for checkbox-cli, I thought that being unable to rerun every test would miss. But actually the only concern was about selection of all jobs dependencies (i.e suspend again if the test was depending on suspend_auto). So I agree with the decision to only select failed/crash test results.
Tests are presented with their respective categories which is great (clearly lacking in checkbox-gui).
So conclusion is a +1 for the design choices.
I ran the following sequence which was crashing the app:
- selected both sensor and screen categories from the qml provider
- failed one in each category
- rerun one succesfully
- killed cbt with a swipe move on the screen
- when prompted to resume the session selected rerun last test
- failed again the test "ensure haptics are working"
- at the final screen, select finish
- cbt controls are greyed out, app is blocked
The upstart log: http:// pastebin. ubuntu. com/11918666/