Merge lp:~zyga/snapcraft/plainbox-app into lp:~snappy-dev/snapcraft/core

[Zygmunt Krynicki]

Add runtests.py

[lp:~zyga/snapcraft/plainbox-app]

213. By Zygmunt Krynicki on 2015-09-29

Add runtests.py

This patch adds a new tool for running tests, runtests.py, intending to
eventually replace the runtests.sh shell script.

This tool uses Plainbox APIs that are only available on python3-plainbox
0.23 (which pulls in python3-guacamole 0.9). The script is feature
complete as far as integration tests go but doesn't (yet) handle unit
tests. I'm looking at how to support that separately.

This patch makes runtests.py take optional advantage of
python3-argcomplete. When installed and properly activated (see below),
runtests.py will complete the partial identifiers of the tests to run.
This makes it easier to run one or a few tests quickly.

To enable that in your active shell, run:

eval "$(register-python-argcomplete3 runtests.py)"

Signed-off-by: Zygmunt Krynicki <email address hidden>

[Federico Gimenez (fgimenez)]

Looks great, the only thing missing that I see is the ability to run different test plans, now it's fixed to "normal". In the runtests.sh script you can specify the kind of tests to be run (unit or plainbox) and, in the case of plainbox tests, the test plan to be run:

$ ./runtests.sh
$ ./runtests.sh unit
$ ./runtests.sh plainbox examples

Are you planning to add this feature to the script?

Perhaps it would be also nice to get a more verbose output through an option, what do you think?

Thanks,

review: Needs Information

[Zygmunt Krynicki (zyga)]

I'm open to ideas, this MR was exactly to provoke feedback and
conversation. The second upside is that this is so trivial to modify
that you don't have to be a plainbox expert to tweak it. As for your
questions:

I'd like to integrate unit tests but there are two options: as a
plainbox job or natively from within the tool. I'll experiment which
works better.

Test plans is that something that the current script ran, we can
default to normal and switch to --examples with an option (it's
trivial as you see). If we integrate unit tests into test plans then
it would just be a selection of a test plan that matters (and we could
have a .* test plan that runs everything).

What kind of verbose output would you like to have, to show live
output as we run (not only when we fail?)

Thanks
ZK

On Wed, Sep 30, 2015 at 10:29 AM, Federico Gimenez
<email address hidden> wrote:
> Review: Needs Information
>
> Looks great, the only thing missing that I see is the ability to run different test plans, now it's fixed to "normal". In the runtests.sh script you can specify the kind of tests to be run (unit or plainbox) and, in the case of plainbox tests, the test plan to be run:
>
> $ ./runtests.sh
> $ ./runtests.sh unit
> $ ./runtests.sh plainbox examples
>
> Are you planning to add this feature to the script?
>
> Perhaps it would be also nice to get a more verbose output through an option, what do you think?
>
> Thanks,
> --
> https://code.launchpad.net/~zyga/snapcraft/plainbox-app/+merge/272720
> You are the owner of lp:~zyga/snapcraft/plainbox-app.

[Federico Gimenez (fgimenez)]

> Test plans is that something that the current script ran, we can
> default to normal and switch to --examples with an option (it's
> trivial as you see).

There's another thing here, for running the examples test plan we had to make some adjustments in the runtests.sh script [1], because plainbox was not able to reach the files out of $PLAINBOX_PROVIDER_DATA. Are you aware of that, will this patch take that into account?

[1]http://bazaar.launchpad.net/~snappy-dev/snapcraft/core/view/head:/integration-tests/runtests.sh#L37

> If we integrate unit tests into test plans then
> it would just be a selection of a test plan that matters (and we could
> have a .* test plan that runs everything).
>

Yes, that would be great

> What kind of verbose output would you like to have, to show live
> output as we run (not only when we fail?)
>

I think that it would be useful for debugging to have the same output as we do now with the plainbox run call used in runtests.sh, which shows all the commands executed.

Also there are some changes required for autopkgtest to work, the new dependencies should be added in [2] (including required version numbers if needed) and the calls at [3] and [4] should be adapted to the new script.

[2] http://bazaar.launchpad.net/~snappy-dev/snapcraft/core/view/head:/debian/tests/control
[3] http://bazaar.launchpad.net/~snappy-dev/snapcraft/core/view/head:/debian/tests/runexamples
[4] http://bazaar.launchpad.net/~snappy-dev/snapcraft/core/view/head:/debian/tests/runtests

Thanks!

[Zygmunt Krynicki (zyga)]

I'm not aware of any issues with $PLAINBOX_PROVIDER_DATA. I'll rebase
this (there's a 2nd patch I didn't push that's a pain to touch as I
changed the IDs of each job) and see if I can reproduce it.

I get your point for other comments. I'll iterate on this. Thanks for
the feedback. :-)

On Wed, Sep 30, 2015 at 3:55 PM, Federico Gimenez
<email address hidden> wrote:
>> Test plans is that something that the current script ran, we can
>> default to normal and switch to --examples with an option (it's
>> trivial as you see).
>
> There's another thing here, for running the examples test plan we had to make some adjustments in the runtests.sh script [1], because plainbox was not able to reach the files out of $PLAINBOX_PROVIDER_DATA. Are you aware of that, will this patch take that into account?
>
> [1]http://bazaar.launchpad.net/~snappy-dev/snapcraft/core/view/head:/integration-tests/runtests.sh#L37
>
>> If we integrate unit tests into test plans then
>> it would just be a selection of a test plan that matters (and we could
>> have a .* test plan that runs everything).
>>
>
> Yes, that would be great
>
>> What kind of verbose output would you like to have, to show live
>> output as we run (not only when we fail?)
>>
>
> I think that it would be useful for debugging to have the same output as we do now with the plainbox run call used in runtests.sh, which shows all the commands executed.
>
> Also there are some changes required for autopkgtest to work, the new dependencies should be added in [2] (including required version numbers if needed) and the calls at [3] and [4] should be adapted to the new script.
>
> [2] http://bazaar.launchpad.net/~snappy-dev/snapcraft/core/view/head:/debian/tests/control
> [3] http://bazaar.launchpad.net/~snappy-dev/snapcraft/core/view/head:/debian/tests/runexamples
> [4] http://bazaar.launchpad.net/~snappy-dev/snapcraft/core/view/head:/debian/tests/runtests
>
> Thanks!
>
> --
> https://code.launchpad.net/~zyga/snapcraft/plainbox-app/+merge/272720
> You are the owner of lp:~zyga/snapcraft/plainbox-app.

Unmerged revisions

213. By Zygmunt Krynicki on 2015-09-29

Add runtests.py

This patch adds a new tool for running tests, runtests.py, intending to
eventually replace the runtests.sh shell script.

This tool uses Plainbox APIs that are only available on python3-plainbox
0.23 (which pulls in python3-guacamole 0.9). The script is feature
complete as far as integration tests go but doesn't (yet) handle unit
tests. I'm looking at how to support that separately.

This patch makes runtests.py take optional advantage of
python3-argcomplete. When installed and properly activated (see below),
runtests.py will complete the partial identifiers of the tests to run.
This makes it easier to run one or a few tests quickly.

To enable that in your active shell, run:

eval "$(register-python-argcomplete3 runtests.py)"

Signed-off-by: Zygmunt Krynicki <email address hidden>

=== added file 'runtests.py'
--- runtests.py	1970-01-01 00:00:00 +0000
+++ runtests.py	2015-09-29 11:47:32 +0000
@@ -0,0 +1,215 @@
+#!/usr/bin/env python3
+# PYTHON_ARGCOMPLETE_OK
+# -*- Mode:Python; indent-tabs-mode:nil; tab-width:4 -*-
+#
+# Copyright (C) 2015 Canonical Ltd
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 3 as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+
+"""Snapcraft Test Runner"""
+
+import os
+
+from guacamole import Command
+from plainbox.abc import IJobRunnerUI
+from plainbox.impl.secure.providers.v1 import Provider1
+from plainbox.impl.secure.providers.v1 import Provider1Definition
+from plainbox.impl.secure.qualifiers import select_jobs
+from plainbox.impl.session.assistant import SessionAssistant
+
+
+class runtests(Command):
+
+    """
+    Tool for running snapcraft tests
+
+    Run snapcraft unit and integration tests
+
+    @EPILOG@
+
+    Without any arguments all tests are run in sequence. Specifying any job
+    identifier runs only the jobs listed.
+    """
+
+    #: Identifier of the application, used by guacamole
+    app_id = 'snapcraft-runtests'
+
+    #: Plainbox namespace used by the snapcraft provider
+    ns = '2015.com.canonical.snapcraft'
+
+    # Plainbox test plan that is started
+    test_plan_id = '{}::normal'.format(ns)
+
+    def register_arguments(self, parser):
+        parser.add_argument(
+            'job_ids', metavar='JOB', nargs="*",
+            help="ID of the job to run"
+        ).completer = JobsInTestPlanCompleter(
+            self.test_plan_id, self._get_snapcraft_provider_def()
+        )
+
+    def invoked(self, ctx):
+        """Method called when the command is invoked."""
+        self._setup_environment()
+        summary = self._run_integration_tests(ctx)
+        # Come up with an useful exit status.
+        #
+        # Ignoring the passing tests. All other outcomes are something we want
+        # to flag on (crash, fail, etc).  Perhaps we want to ignore 'skip'
+        # later on but now nothing should trigger that.
+        if 'pass' in summary:
+            del summary['pass']
+        return 1 if summary else 0
+
+    def _run_integration_tests(self, ctx):
+        sa = SessionAssistant(self.app_id)
+        sa.select_providers(
+            "{}:tests".format(self.ns), additional_providers=[
+                self._get_snapcraft_provider()
+            ])
+        sa.start_new_session("Snapcraft Integration Tests")
+        sa.select_test_plan("{}::normal".format(self.ns))
+        sa.bootstrap()
+        if ctx.args.job_ids:
+            sa.use_alternate_selection([
+                '{}::{}'.format(self.ns, job_id)
+                for job_id in ctx.args.job_ids])
+        ui = _SnapcraftUI(ctx)
+        for job_id in sa.get_static_todo_list():
+            builder = sa.run_job(job_id, ui, native=False)
+            result = builder.get_result()
+            sa.use_job_result(job_id, result)
+        sa._metadata.flags.add('submitted')  # LP: https://pad.lv/1500799
+        sa.finalize_session()
+        return sa.get_summary()
+
+    def _setup_environment(self):
+        cwd = os.getcwd()
+        os.environ['PATH'] = os.path.expandvars("{}/bin:$PATH".format(cwd))
+        os.environ['PYTHONPATH'] = os.path.expandvars(
+            "{}:$PYTHONPATH".format(cwd))
+        os.environ['SNAPCRAFT'] = 'snapcraft'
+
+    def _get_snapcraft_provider_def(self):
+        definition = Provider1Definition()
+        definition.name = "{}:tests".format(self.ns)
+        definition.description = "Plainbox test provider for Snapcraft"
+        definition.relocatable = True
+        definition.version = "1.0"
+        definition.location = os.path.abspath('integration-tests')
+        return definition
+
+    def _get_snapcraft_provider(self):
+        return Provider1.from_definition(
+            self._get_snapcraft_provider_def(), secure=False)
+
+
+class _SnapcraftUI(IJobRunnerUI):
+
+    def __init__(self, ctx):
+        self.ctx = ctx
+
+    def considering_job(self, job, job_state):
+        pass
+
+    def about_to_start_running(self, job, job_state):
+        self.ctx.aprint(
+            "{} ...".format(job.partial_id), end='', flush=True, bold=True)
+
+    def wait_for_interaction_prompt(self, job):
+        pass
+
+    def started_running(self, job, job_state):
+        pass
+
+    def about_to_execute_program(self, args, kwargs):
+        pass
+
+    def finished_executing_program(self, returncode):
+        pass
+
+    def got_program_output(self, stream_name, line):
+        pass
+
+    def finished_running(self, job, job_state, job_result):
+        if job_result.outcome == 'fail':
+            self.ctx.aprint("\r{:50} {}".format(
+                job.partial_id, job_result.outcome_meta().tr_label),
+                flush=True, fg='red', bold=True)
+            for io_log_record in job_result.get_io_log():
+                self.ctx.aprint(
+                    io_log_record.data.decode("UTF-8", 'replace'),
+                    end='', fg='red')
+        elif job_result.outcome == 'pass':
+            self.ctx.aprint("\r{:50} {}".format(
+                job.partial_id, job_result.outcome_meta().tr_label),
+                flush=True, fg='green')
+        else:
+            self.ctx.aprint("\r{:50} {}".format(
+                job.partial_id, job_result.outcome_meta().tr_label),
+                flush=True, fg='yellow')
+
+    def notify_about_description(self, job):
+        pass
+
+    def notify_about_purpose(self, job):
+        pass
+
+    def notify_about_steps(self, job):
+        pass
+
+    def notify_about_verification(self, job):
+        pass
+
+    def job_cannot_start(self, job, job_state, job_result):
+        pass
+
+    def finished(self, job, job_state, job_result):
+        pass
+
+    def pick_action_cmd(self, action_list, prompt=None):
+        pass
+
+    def noreturn_job(self):
+        pass
+
+
+class JobsInTestPlanCompleter:
+
+    def __init__(self, test_plan_id, provider_def):
+        self._test_plan_id = test_plan_id
+        self._provider_def = provider_def
+        self._choices = None
+
+    def __call__(self, prefix, **kwargs):
+        if self. _choices is None:
+            self._choices = self._compute_choices()
+        return (c for c in self._choices if c.startswith(prefix))
+
+    def _compute_choices(self):
+        try:
+            provider = Provider1.from_definition(
+                self._provider_def, secure=False, check=False)
+            test_plan = provider.id_map[self._test_plan_id][0]
+            qualifier_list = [test_plan.get_qualifier()]
+            return [job.partial_id for job in select_jobs(
+                provider.job_list, qualifier_list)]
+        except Exception:
+            # If stuff blows up (maybe the provider is invalid), just don't
+            # suggest anything.
+            return []
+
+
+if __name__ == '__main__':
+    runtests().main()