Merge lp:~kissiel/checkbox/rerun-in-cbt into lp:checkbox

[Maciej Kisielewski]

This MR brings re-run feature to checkbox-{touch|converged}

6e98004 chekcbox-touch: add function getting rerun candidates to py logic
d873a99 checkbox-touch: invalidate cache and reset index when tests are started
a7feee8 checkbox-touch: add proxy-ing to get rerun candidates in qml logic
214224d checkbox-touch: add emptyAllowed option to selection screen
e58246e checkbox-touch: add rerun screen
2db9319 checkbox-touch: run re-run before results screen
5d71b5b checkbox-touch:autopilot: move processing clicks helper to top-class
c0e491a checkbox-touch:autopilot: add generic check_results helper
7d49391 checkbox-touch:autopilot: add re-run scenarios for AP

[lp:~kissiel/checkbox/rerun-in-cbt]

3905. By Zygmunt Krynicki on 2015-07-20

"automatic merge of lp:~zyga/checkbox/fix-1476136/ by tarmac [r=sylvain-pineau][bug=1476136][author=zyga]"

3906. By Maciej Kisielewski on 2015-07-21

"automatic merge of lp:~kissiel/checkbox/remove-lxml-from-cbt-reqs/ by tarmac [r=sylvain-pineau,zyga][bug=][author=kissiel]"

[Sylvain Pineau (sylvain-pineau)]

When we landed the same feature for checkbox-cli, I thought that being unable to rerun every test would miss. But actually the only concern was about selection of all jobs dependencies (i.e suspend again if the test was depending on suspend_auto). So I agree with the decision to only select failed/crash test results.

Tests are presented with their respective categories which is great (clearly lacking in checkbox-gui).

So conclusion is a +1 for the design choices.

I ran the following sequence which was crashing the app:

- selected both sensor and screen categories from the qml provider
- failed one in each category
- rerun one succesfully
- killed cbt with a swipe move on the screen
- when prompted to resume the session selected rerun last test
- failed again the test "ensure haptics are working"
- at the final screen, select finish
- cbt controls are greyed out, app is blocked

The upstart log: http://pastebin.ubuntu.com/11918666/

review: Needs Fixing

[lp:~kissiel/checkbox/rerun-in-cbt]

3907. By Maciej Kisielewski on 2015-07-23

checkbox-touch: remember test_plan_id once it's selected

Signed-off-by: Maciej Kisielewski <email address hidden>

3908. By Maciej Kisielewski on 2015-07-23

checkbox-touch: make resuming re-initiate test-plan

Signed-off-by: Maciej Kisielewski <email address hidden>

3909. By Maciej Kisielewski on 2015-07-23

chekcbox-touch: add function getting rerun candidates to py logic

Signed-off-by: Maciej Kisielewski <email address hidden>

3910. By Maciej Kisielewski on 2015-07-23

checkbox-touch: invalidate cache and reset index when tests are started

Signed-off-by: Maciej Kisielewski <email address hidden>

3911. By Maciej Kisielewski on 2015-07-23

checkbox-touch: add proxy-ing to get rerun candidates in qml logic

Signed-off-by: Maciej Kisielewski <email address hidden>

3912. By Maciej Kisielewski on 2015-07-23

checkbox-touch: add emptyAllowed option to selection screen

This patch adds an option to SelectionPage component that lets user proceed
even if nothing is selected.

Signed-off-by: Maciej Kisielewski <email address hidden>

3913. By Maciej Kisielewski on 2015-07-23

checkbox-touch: add rerun screen

Signed-off-by: Maciej Kisielewski <email address hidden>

3914. By Maciej Kisielewski on 2015-07-23

checkbox-touch: run re-run before results screen

Signed-off-by: Maciej Kisielewski <email address hidden>

3915. By Maciej Kisielewski on 2015-07-23

checkbox-touch:autopilot: move processing clicks helper to top-class

This patch moves the helper that executes common scenario of clicking through
pages, to the ClickAppTestCase class, so all our test classes inherit this.

Signed-off-by: Maciej Kisielewski <email address hidden>

3916. By Maciej Kisielewski on 2015-07-23

checkbox-touch:autopilot: add generic check_results helper

This patch adds a helper function that checks if the result screen is shown
with the expected values.

Signed-off-by: Maciej Kisielewski <email address hidden>

3917. By Maciej Kisielewski on 2015-07-23

checkbox-touch:autopilot: add re-run scenarios for AP

Signed-off-by: Maciej Kisielewski <email address hidden>

[Maciej Kisielewski (kissiel)]

> When we landed the same feature for checkbox-cli, I thought that being unable
> to rerun every test would miss. But actually the only concern was about
> selection of all jobs dependencies (i.e suspend again if the test was
> depending on suspend_auto). So I agree with the decision to only select
> failed/crash test results.
>
> Tests are presented with their respective categories which is great (clearly
> lacking in checkbox-gui).
>
> So conclusion is a +1 for the design choices.
>
> I ran the following sequence which was crashing the app:
>
> - selected both sensor and screen categories from the qml provider
> - failed one in each category
> - rerun one succesfully
> - killed cbt with a swipe move on the screen
> - when prompted to resume the session selected rerun last test
> - failed again the test "ensure haptics are working"
> - at the final screen, select finish
> - cbt controls are greyed out, app is blocked
>
> The upstart log: http://pastebin.ubuntu.com/11918666/

Rerun + resume now works. Tested with your scenario, and the scenario when resume is done before any job is rerun.

[Sylvain Pineau (sylvain-pineau)]

Re tested, it works! Thanks

review: Approve

=== modified file 'checkbox-touch/checkbox-touch.qml'
--- checkbox-touch/checkbox-touch.qml	2015-07-16 13:58:31 +0000
+++ checkbox-touch/checkbox-touch.qml	2015-07-23 08:26:16 +0000
@@ -357,6 +357,30 @@
         }
     }
 
+    SelectionPage {
+        id: rerunSelectionPage
+        objectName: "rerunSelectionPage"
+        title: i18n.tr("Select tests to re-run")
+        continueText: state == "empty selection" ?
+            i18n.tr("Finish") : i18n.tr("Re-run")
+        emptyAllowed: true
+
+        function setup(rerunCandidates, continuation) {
+            model.clear();
+            for (var i=0; i<rerunCandidates.length; i++) {
+                model.append(rerunCandidates[i]);
+            }
+            modelUpdated();
+            pageStack.push(rerunSelectionPage)
+        }
+        onSelectionDone: {
+            app.rememberTestSelection(selected_id_list, function() {
+                processNextTest();
+                unlatchContinue();
+            });
+        }
+    }
+
     PasswordDialog {
         id: passwordDialog
     }
@@ -388,7 +412,7 @@
         app.getNextTest(function(test) {
             pageStack.clear();
             if (test.plugin === undefined) { 
-                return showResultsScreen();
+                return maybeShowRerunScreen();
             }
             if (test.user) {
                 // running this test will require to be run as a different
@@ -449,6 +473,15 @@
         app.runTestActivity(test, continuation);
 
     }
+    function maybeShowRerunScreen() {
+        app.getRerunCandidates(function(result) {
+            if (result == false) {
+                 showResultsScreen();
+                 return;
+            }
+            rerunSelectionPage.setup(result);
+        });
+    }
 
     function showResultsScreen() {
         pageStack.clear();
 
=== modified file 'checkbox-touch/components/CheckboxTouchApplication.qml'
--- checkbox-touch/components/CheckboxTouchApplication.qml	2015-06-30 22:20:49 +0000
+++ checkbox-touch/components/CheckboxTouchApplication.qml	2015-07-23 08:26:16 +0000
@@ -138,6 +138,12 @@
         });
     }
 
+    function getRerunCandidates(continuation) {
+        request("get_rerun_candidates", [], continuation, function(error) {
+            console.error("Unable to get rerun candidates");
+        });
+    }
+
     function getResults(continuation) {
         request("get_results", [], continuation, function(error) {
             console.error("Unable to get test results");
 
=== modified file 'checkbox-touch/components/SelectionPage.qml'
--- checkbox-touch/components/SelectionPage.qml	2015-07-17 08:36:57 +0000
+++ checkbox-touch/components/SelectionPage.qml	2015-07-23 08:26:16 +0000
@@ -36,6 +36,7 @@
     property string continueText: i18n.tr("Continue")
     readonly property alias model: selectionModel
     property bool onlyOneAllowed: false
+    property bool emptyAllowed: false
 
     visible: false
     flickable: null
@@ -100,7 +101,8 @@
     states: [
          State {
             name: "empty selection"
-            PropertyChanges { target: continueButton; enabled: false }
+            PropertyChanges { target: continueButton;
+                              enabled: false || emptyAllowed }
          },
          State {
             name: "nonempty selection"
 
=== modified file 'checkbox-touch/py/checkbox_touch.py'
--- checkbox-touch/py/checkbox_touch.py	2015-07-16 21:25:01 +0000
+++ checkbox-touch/py/checkbox_touch.py	2015-07-23 08:26:16 +0000
@@ -248,6 +248,7 @@
             self.context.provider_list,
             os.path.join(self.manager.storage.location, 'io-logs'))
         self.index = app_blob['index_in_run_list']
+        self._init_test_plan_id(app_blob['test_plan_id'])
         _logger.error(self.context.state.run_list)
         _logger.error(self.index)
         if not rerun_last_test:
@@ -399,6 +400,24 @@
         return {
             'test_info_list': test_info_list
         }
+    @view
+    def get_rerun_candidates(self):
+        def rerun_predicate(job_state):
+            return job_state.result.outcome in (
+                IJobResult.OUTCOME_FAIL, IJobResult.OUTCOME_CRASH)
+        id_map = self.context.compute_shared(
+            'id_map', compute_value_map, self.context, 'id')
+        rerun_candidates = []
+        for job in self.manager.state.run_list:
+            if rerun_predicate(self.manager.state.job_state_map[job.id]):
+                rerun_candidates.append({
+                    "mod_id": job.id,
+                    "mod_name": job.tr_summary(),
+                    "mod_group": id_map[job.category_id][0].tr_name(),
+                    "mod_selected": False
+
+                    })
+        return rerun_candidates
 
     @view
     def remember_tests(self, selected_id_list):
@@ -407,6 +426,8 @@
         """
         self.desired_test_ids = frozenset(selected_id_list)
         _logger.info("Selected tests: %s", self.desired_test_ids)
+        self.index = 0
+        self.context.invalidate_shared('desired_job_list')
         desired_job_list = self.context.compute_shared(
             'desired_job_list', select_jobs,
             self.context.state.job_list, [
@@ -611,6 +632,7 @@
         if test_plan.Meta.name != 'test plan':
             raise ValueError(
                 "unit {!r} is not a test plan".format(test_plan_id))
+        self.test_plan_id = test_plan_id
         self.test_plan = test_plan
 
     def _init_session_storage_repo(self):
 
=== modified file 'checkbox-touch/tests/autopilot/checkbox_touch/__init__.py'
--- checkbox-touch/tests/autopilot/checkbox_touch/__init__.py	2015-07-15 15:32:59 +0000
+++ checkbox-touch/tests/autopilot/checkbox_touch/__init__.py	2015-07-23 08:26:16 +0000
@@ -89,6 +89,39 @@
             objectName='continueButton')
         self.pointing_device.click_object(continue_btn)
 
+    def process_sequence_of_clicks_on_pages(self, steps):
+        """
+        Do a sequence of clicks on simple page->component hierarchies.
+
+        :param steps:
+            sequence of (page-objectName, component-objectName) pairs to go
+            through.
+
+        Typical run of checkbox-touch requires user to go through a sequence of
+        pages that have pass/fail buttons on them. This function helps go
+        through a sequence like that.
+        """
+        for parent, component in steps:
+            self.app.wait_select_single(
+                objectName=parent, visible=True)
+            clickable = self.main_view.wait_select_single(
+                objectName=component, visible=True)
+            self.pointing_device.click_object(clickable)
+
+    def check_results(self, results):
+        results_page = self.app.wait_select_single(
+            objectName='resultsPage', visible=True)
+        lbl_passed = results_page.wait_select_single(objectName='passedLabel')
+        self.assertThat(lbl_passed.text.startswith(results['passed']),
+                        Equals(True))
+        lbl_failed = results_page.wait_select_single(objectName='failedLabel')
+        self.assertThat(lbl_failed.text.startswith(results['failed']),
+                        Equals(True))
+        lbl_skipped = results_page.wait_select_single(
+            objectName='skippedLabel')
+        self.assertThat(lbl_skipped.text.startswith(results['skipped']),
+                        Equals(True))
+
     def launch_application(self):
         if platform.model() == 'Desktop':
             self._launch_application_from_desktop()
 
=== modified file 'checkbox-touch/tests/autopilot/checkbox_touch/test_checkbox_touch.py'
--- checkbox-touch/tests/autopilot/checkbox_touch/test_checkbox_touch.py	2015-07-15 15:32:57 +0000
+++ checkbox-touch/tests/autopilot/checkbox_touch/test_checkbox_touch.py	2015-07-23 08:26:16 +0000
@@ -23,25 +23,6 @@
         yes_btn = dialog.select_single(objectName='yesButton')
         self.pointing_device.click_object(yes_btn)
 
-    def process_sequence_of_clicks_on_pages(self, steps):
-        """
-        Do a sequence of clicks on simple page->component hierarchies.
-
-        :param steps:
-            sequence of (page-objectName, component-objectName) pairs to go
-            through.
-
-        Typical run of checkbox-touch requires user to go through a sequence of
-        pages that have pass/fail buttons on them. This function helps go
-        through a sequence like that.
-        """
-        for parent, component in steps:
-            self.app.wait_select_single(
-                objectName=parent, visible=True)
-            clickable = self.main_view.wait_select_single(
-                objectName=component, visible=True)
-            self.pointing_device.click_object(clickable)
-
     def test_launches(self):
         main_view = self.app.select_single(objectName='mainView')
         self.assertThat(main_view.visible, Eventually(Equals(True)))
@@ -116,22 +97,12 @@
         next_steps = [
             ('qmlNativePage', 'continueButton'),
             ('qmlTestPage', 'passButton'),
+            ('rerunSelectionPage', 'continueButton')
         ]
         self.process_sequence_of_clicks_on_pages(next_steps)
         # we should see results screen now
-        results_page = self.app.wait_select_single(
-            objectName='resultsPage', visible=True)
         results = {'passed': '10', 'failed': '5', 'skipped': '5'}
-        lbl_passed = results_page.wait_select_single(objectName='passedLabel')
-        self.assertThat(lbl_passed.text.startswith(results['passed']),
-                        Equals(True))
-        lbl_failed = results_page.wait_select_single(objectName='failedLabel')
-        self.assertThat(lbl_failed.text.startswith(results['failed']),
-                        Equals(True))
-        lbl_skipped = results_page.wait_select_single(
-            objectName='skippedLabel')
-        self.assertThat(lbl_skipped.text.startswith(results['skipped']),
-                        Equals(True))
+        self.check_results(results)
 
 
 class SessionResumeTests(checkbox_touch.ClickAppTestCase):
@@ -228,3 +199,71 @@
         text_area = output_page.wait_select_single(
             objectName='textArea', visible=True)
         self.assertThat(text_area.text, Eventually(Equals("foobar\n")))
+
+
+class RerunTests(checkbox_touch.ClickAppTestCase):
+    def test_rerun_after_rerun(self):
+        test_id = '2015.com.canonical.certification::autopilot/manual-2'
+        self.start_and_select_tests(
+            '2015.com.canonical.certification::normal', [test_id])
+        next_steps = [
+            ('manualIntroPage', 'continueButton'),
+            ('testVerificationPage', 'failButton'),
+        ]
+        self.process_sequence_of_clicks_on_pages(next_steps)
+        # we now should see a re-run screen; let's select the only test
+        rerun_page = self.app.wait_select_single(
+            objectName='rerunSelectionPage', visible=True)
+        list_item = rerun_page.wait_select_single(
+            objectName='listItem', item_mod_id=test_id)
+        self.pointing_device.click_object(list_item)
+        continue_btn = rerun_page.wait_select_single(
+            objectName='continueButton', visible=True)
+        self.pointing_device.click_object(continue_btn)
+        # run the same steps as before
+        self.process_sequence_of_clicks_on_pages(next_steps)
+        # we should see the re-run screen again
+        rerun_page = self.app.wait_select_single(
+            objectName='rerunSelectionPage', visible=True)
+        continue_btn = rerun_page.wait_select_single(
+            objectName='continueButton', visible=True)
+        self.pointing_device.click_object(continue_btn)
+        self.check_results({'passed': '0', 'failed': '1', 'skipped': '0'})
+
+    def test_rerun_after_fail(self):
+        test_id = '2015.com.canonical.certification::autopilot/manual-2'
+        self.start_and_select_tests(
+            '2015.com.canonical.certification::normal', [test_id])
+        next_steps = [
+            ('manualIntroPage', 'continueButton'),
+            ('testVerificationPage', 'failButton'),
+        ]
+        self.process_sequence_of_clicks_on_pages(next_steps)
+        # we now should see a re-run screen; let's select the only test
+        rerun_page = self.app.wait_select_single(
+            objectName='rerunSelectionPage', visible=True)
+        list_item = rerun_page.wait_select_single(
+            objectName='listItem', item_mod_id=test_id)
+        self.pointing_device.click_object(list_item)
+        continue_btn = rerun_page.wait_select_single(
+            objectName='continueButton', visible=True)
+        self.pointing_device.click_object(continue_btn)
+        next_steps = [
+            ('manualIntroPage', 'continueButton'),
+            ('testVerificationPage', 'passButton'),
+        ]
+        self.process_sequence_of_clicks_on_pages(next_steps)
+        # now set the outcome to 'pass'; we should be on results screen now
+        self.check_results({'passed': '1', 'failed': '0', 'skipped': '0'})
+
+    def test_no_rerun_after_pass(self):
+        test_id = '2015.com.canonical.certification::autopilot/manual-1'
+        self.start_and_select_tests(
+            '2015.com.canonical.certification::normal', [test_id])
+        next_steps = [
+            ('manualIntroPage', 'continueButton'),
+            ('testVerificationPage', 'passButton'),
+        ]
+        self.process_sequence_of_clicks_on_pages(next_steps)
+        # there should be no re-run screen, just results
+        self.check_results({'passed': '1', 'failed': '0', 'skipped': '0'})