Merge ~kissiel/plainbox-provider-checkbox:respawn-after-pmtest-reboots into plainbox-provider-checkbox:master

[Maciej Kisielewski]

make pm_test-related jobs respawn checkbox after the reboot cycles

[Maciej Kisielewski (kissiel)]

To test it make sure you're have latest masters of [plainbox, checkbox-ng, plainbox-provider-checkbox] installed (globally, i.e. system-wide).

To do that, enter respective project's directory, and run:

./setup.py build && sudo ./setup.py install

Virtual env won't cut it, as after the reboot restart strategy will use the same 'binary' to run.
In theory re-establishing virtualenv is possible, but I haven't tried that and it seems not worth the hustle ATM.

I recommend doing that on a 'throwaway' installation (Though VM is not suitable as it's difficult - if not impossible - to run S3/S4 inside them)

[Chris Wayne (cwayne)]

Couple inline comments

[Chris Wayne (cwayne)]

Tried to follow your instructions on a new system and not even seeing stress tests after running checkbox-cli.

I'd installed master of plainbox and checkbox-ng, then cloned this branch and did a manage.py install, was there something else I was missing?

[Maciej Kisielewski (kissiel)]

> Couple inline comments
Haven't tried without it. Should I?

[Maciej Kisielewski (kissiel)]

> Tried to follow your instructions on a new system and not even seeing stress
> tests after running checkbox-cli.
>
> I'd installed master of plainbox and checkbox-ng, then cloned this branch and
> did a manage.py install, was there something else I was missing?

You have to `checkbox-cli run` it, or use something like a oem-provider to have a TP that selects it.

[Sylvain Pineau (sylvain-pineau)]

I'm trying to test only the job id:
com.canonical.certification::power-management/suspend_30_cycles_with_reboots

To be sure to save the respawn command in the session share, I've only added flags: noreturn to this job.

But executing the following makes checkbox-cli to crash:

$ checkbox-cli run .*with_reboots
[...]
--------------[ Running job 5 / 5. Estimated time left: 1:30:00 ]---------------
--------------[ power-management/suspend_30_cycles_with_reboots ]---------------
ID: com.canonical.certification::power-management/suspend_30_cycles_with_reboots
Category: com.canonical.plainbox::stress
Waiting for the system to shut down or reboot...

                        Application Malfunction Detected

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/guacamole/core.py", line 248, in eat
    return self._dispatch()
  File "/usr/lib/python3/dist-packages/guacamole/core.py", line 298, in _dispatch
    result = ingredient.dispatch(self.context)
  File "/usr/lib/python3/dist-packages/guacamole/ingredients/cmdtree.py", line 152, in dispatch
    return self._dispatch(context, 0)
  File "/usr/lib/python3/dist-packages/guacamole/ingredients/cmdtree.py", line 179, in _dispatch
    return self._dispatch_None(context, level, retval, command)
  File "/usr/lib/python3/dist-packages/guacamole/ingredients/cmdtree.py", line 208, in _dispatch_None
    return self._dispatch(context, level + 1)
  File "/usr/lib/python3/dist-packages/guacamole/ingredients/cmdtree.py", line 165, in _dispatch
    retval = command.invoked(context)
  File "/home/sylvain/lp/checkbox-ng/checkbox_ng/launcher/subcommands.py", line 852, in invoked
    self.run_matching_jobs(selection)
  File "/home/sylvain/lp/checkbox-ng/checkbox_ng/launcher/subcommands.py", line 876, in run_matching_jobs
    self._run_jobs(self.sa.get_dynamic_todo_list())
  File "/home/sylvain/lp/checkbox-ng/checkbox_ng/launcher/stages.py", line 219, in _run_jobs
    job, self._get_ui_for_job(job))
  File "/home/sylvain/lp/checkbox-ng/checkbox_ng/launcher/stages.py", line 112, in _run_single_job_with_ui_loop
    result_builder = self.sa.run_job(job.id, ui, False)
  File "/home/sylvain/lp/plainbox/plainbox/impl/decorators.py", line 142, in wrapper
    raise exc
  File "/home/sylvain/lp/plainbox/plainbox/impl/decorators.py", line 136, in wrapper
    return func(*args, **kwargs)
  File "/home/sylvain/lp/plainbox/plainbox/impl/session/assistant.py", line 1295, in run_job
    self.get_session_id()))...

review: Needs Fixing

[Maciej Kisielewski (kissiel)]

> I'm trying to test only the job id:
> com.canonical.certification::power-management/suspend_30_cycles_with_reboots
>
> To be sure to save the respawn command in the session share, I've only added
> flags: noreturn to this job.
>
> But executing the following makes checkbox-cli to crash:
>
> $ checkbox-cli run .*with_reboots
> [...]
> --------------[ Running job 5 / 5. Estimated time left: 1:30:00
> ]---------------
> --------------[ power-management/suspend_30_cycles_with_reboots
> ]---------------
> ID: com.canonical.certification::power-
> management/suspend_30_cycles_with_reboots
> Category: com.canonical.plainbox::stress
> Waiting for the system to shut down or reboot...
>
> Application Malfunction Detected
>
> Traceback (most recent call last):
> File "/usr/lib/python3/dist-packages/guacamole/core.py", line 248, in eat
> return self._dispatch()
> File "/usr/lib/python3/dist-packages/guacamole/core.py", line 298, in
> _dispatch
> result = ingredient.dispatch(self.context)
> File "/usr/lib/python3/dist-packages/guacamole/ingredients/cmdtree.py", line
> 152, in dispatch
> return self._dispatch(context, 0)
> File "/usr/lib/python3/dist-packages/guacamole/ingredients/cmdtree.py", line
> 179, in _dispatch
> return self._dispatch_None(context, level, retval, command)
> File "/usr/lib/python3/dist-packages/guacamole/ingredients/cmdtree.py", line
> 208, in _dispatch_None
> return self._dispatch(context, level + 1)
> File "/usr/lib/python3/dist-packages/guacamole/ingredients/cmdtree.py", line
> 165, in _dispatch
> retval = command.invoked(context)
> File "/home/sylvain/lp/checkbox-ng/checkbox_ng/launcher/subcommands.py",
> line 852, in invoked
> self.run_matching_jobs(selection)
> File "/home/sylvain/lp/checkbox-ng/checkbox_ng/launcher/subcommands.py",
> line 876, in run_matching_jobs
> self._run_jobs(self.sa.get_dynamic_todo_list())
> File "/home/sylvain/lp/checkbox-ng/checkbox_ng/launcher/stages.py", line
> 219, in _run_jobs
> job, self._get_ui_for_job(job))
> File "/home/sylvain/lp/checkbox-ng/checkbox_ng/launcher/stages.py", line
> 112, in _run_single_job_with_ui_loop
> result_builder = self.sa.run_job(job.id, ui, False)
> File "/home/sylvain/lp/plainbox/plainbox/impl/decorators.py", line 142, in
> wrapper
> raise exc
> File "/home/sylvain/lp/plainbox/plainbox/impl/decorators.py", line 136, in
> wrapper
> return func(*args, **kwargs)
> File "/home/sylvain/lp/plainbox/plainbox/impl/session/assistant.py", line
> 1295, in run_job
> self.get_session_id()))
> TypeError: 'NoneType' object is not callable
>
>
> Please report a bug including the information from the paragraph above. To
> report the bug visit https://bugs.launchpad.net/checkbox-ng/+filebug
>
> We are sorry for the inconvenience!

Nice catch!
Fix for this is here: https://code.launchpad.net/~kissiel/plainbox/+git/plainbox/+merge/329301

This branch haven't changed.

review: Needs Resubmitting

[Sylvain Pineau (sylvain-pineau)]

The gnome-terminal being immediately closed could be something related to pm_test being the parent process and once killed all children are closed. I'm wondering if using os.execvp() could solve this.

+ the job definition needs the noreturn flag

review: Needs Fixing

[Maciej Kisielewski (kissiel)]

I added `; exec bash` at the end of gnome-terminal -e part, which makes bash stay open \o/

And also missing noreturn flag in one of the jobs.

review: Needs Resubmitting

[Sylvain Pineau (sylvain-pineau)]

IT WORKS !! thanks and big +1 for all the other related MRs.

review: Approve

diff --git a/bin/pm_test b/bin/pm_test
index 868b92b..42de2d8 100755
--- a/bin/pm_test
+++ b/bin/pm_test
@@ -202,7 +202,7 @@ class PowerManagementOperation(object):
             MessageDialog(title, message).run()
         if self.args.checkbox_respawn_cmd:
             subprocess.run(
-                r'DISPLAY=:0 gnome-terminal -e "bash -c \"source {}\""'.format(
+                r'DISPLAY=:0 gnome-terminal -e "bash -c \"source {}; exec bash\""'.format(
                     self.args.checkbox_respawn_cmd), shell=True)
 
     def teardown(self):
diff --git a/jobs/stress.txt.in b/jobs/stress.txt.in
index 75b497c..5eae797 100644
--- a/jobs/stress.txt.in
+++ b/jobs/stress.txt.in
@@ -77,9 +77,10 @@ estimated_duration: 5400.0
 depends:
  power-management/rtc
  suspend/suspend_advanced
+flags: noreturn autorestart
 user: root
 command:
- pm_test reboot --fwts --log-level=debug --log-dir=$PLAINBOX_SESSION_SHARE --suspends-before-reboot=30 -r 3
+ pm_test reboot --checkbox-respawn-cmd $PLAINBOX_SESSION_SHARE/__respawn_checkbox --fwts --log-level=debug --log-dir=$PLAINBOX_SESSION_SHARE --suspends-before-reboot=30 -r 3 --silent
 _description:
  This is an automated stress test that will run a sequence of '30 suspend/resume cycles and one reboot' 3 times.
 
@@ -227,8 +228,8 @@ estimated_duration: 4500.0
 requires:
  package.name == 'upstart'
  package.name == 'fwts'
-command: pm_test -r 100 --silent --log-level=notset reboot --log-dir=$PLAINBOX_SESSION_SHARE
-flags: noreturn
+command: pm_test --checkbox-respawn-cmd $PLAINBOX_SESSION_SHARE/__respawn_checkbox -r 100 --silent --log-level=notset reboot --log-dir=$PLAINBOX_SESSION_SHARE
+flags: noreturn autorestart
 user: root
 environ: PLAINBOX_SESSION_SHARE
 _description:
@@ -249,8 +250,8 @@ id: stress/reboot_30
 requires:
  package.name == 'upstart'
  package.name == 'fwts'
-command: pm_test -r 30 --silent --log-level=notset reboot --log-dir=$PLAINBOX_SESSION_SHARE
-flags: noreturn
+command: pm_test --checkbox-respawn-cmd $PLAINBOX_SESSION_SHARE/__respawn_checkbox -r 30 --silent --log-level=notset reboot --log-dir=$PLAINBOX_SESSION_SHARE
+flags: noreturn autorestart
 estimated_duration: 2700
 user: root
 environ: PLAINBOX_SESSION_SHARE