Merge lp:~kissiel/checkbox/qml-native into lp:checkbox

[Maciej Kisielewski]

This MR includes same changes as https://code.launchpad.net/~kissiel/checkbox/qml-native-test/+merge/247759, but has completely rewrittenhistory for more clarity.

[Zygmunt Krynicki (zyga)]

16:35 <@zyga> kissiel: love the man page
16:35 <@zyga> kissiel: let's merge it
16:35 < kissiel> zyga, \o/
16:36 <@zyga> kissiel: thanks for doing all the work and for iterating :)

review: Approve

[Daniel Manrique (roadmr)]

The attempt to merge lp:~kissiel/checkbox/qml-native into lp:checkbox failed. Below is the output from the failed tests.

[precise] starting container
[precise] (timing) 0.06user 0.01system 0:04.18elapsed 1%CPU (0avgtext+0avgdata 7864maxresident)k
[precise] (timing) 0inputs+32outputs (0major+5788minor)pagefaults 0swaps
[precise] provisioning container
[precise] (timing) 36.91user 10.70system 1:19.85elapsed 59%CPU (0avgtext+0avgdata 51680maxresident)k
[precise] (timing) 0inputs+16408outputs (0major+4472567minor)pagefaults 0swaps
[precise-testing] Starting tests...
Found a test script: ./checkbox-gui/requirements/container-tests-checkbox-gui-build
[precise-testing] container-tests-checkbox-gui-build: PASS
[precise-testing] (timing) 33.14user 2.50system 0:36.09elapsed 98%CPU (0avgtext+0avgdata 116468maxresident)k
[precise-testing] (timing) 0inputs+4216outputs (0major+477769minor)pagefaults 0swaps
Found a test script: ./checkbox-ng/requirements/container-tests-checkbox-ng-unit
[precise-testing] container-tests-checkbox-ng-unit: PASS
[precise-testing] (timing) 0.56user 0.11system 0:01.43elapsed 46%CPU (0avgtext+0avgdata 39924maxresident)k
[precise-testing] (timing) 0inputs+3480outputs (0major+20653minor)pagefaults 0swaps
Found a test script: ./checkbox-support/requirements/container-tests-checkbox-support
[precise-testing] container-tests-checkbox-support: PASS
[precise-testing] (timing) 16.92user 0.16system 0:17.20elapsed 99%CPU (0avgtext+0avgdata 83492maxresident)k
[precise-testing] (timing) 0inputs+1032outputs (0major+28807minor)pagefaults 0swaps
Found a test script: ./checkbox-touch/requirements/container-tests-touch-unit-tests
[precise-testing] container-tests-touch-unit-tests: PASS
[precise-testing] (timing) 0.00user 0.00system 0:00.01elapsed 42%CPU (0avgtext+0avgdata 2020maxresident)k
[precise-testing] (timing) 0inputs+8outputs (0major+2346minor)pagefaults 0swaps
Found a test script: ./plainbox/plainbox/impl/providers/categories/requirements/container-tests-provider-categories
[precise-testing] container-tests-provider-categories: PASS
[precise-testing] (timing) 0.14user 0.05system 0:00.20elapsed 92%CPU (0avgtext+0avgdata 13520maxresident)k
[precise-testing] (timing) 0inputs+176outputs (0major+5874minor)pagefaults 0swaps
Found a test script: ./plainbox/requirements/001-container-tests-plainbox-egg-info
[precise-testing] 001-container-tests-plainbox-egg-info: PASS
[precise-testing] (timing) 0.15user 0.03system 0:00.19elapsed 93%CPU (0avgtext+0avgdata 10520maxresident)k
[precise-testing] (timing) 0inputs+88outputs (0major+4989minor)pagefaults 0swaps
Found a test script: ./plainbox/requirements/container-tests-plainbox
[precise-testing] container-tests-plainbox: PASS
[precise-testing] (timing) 10.64user 0.72system 0:11.57elapsed 98%CPU (0avgtext+0avgdata 66500maxresident)k
[precise-testing] (timing) 0inputs+2776outputs (0major+103963minor)pagefaults 0swaps
Found a test script: ./plainbox/requirements/container-tests-plainbox-documentation
[precise-testing] container-tests-plainbox-documentation: FAIL
[precise-testing] stdout: http://...

[lp:~kissiel/checkbox/qml-native]

3553. By Maciej Kisielewski on 2015-01-29

plainbox:qml_shell: add standalone qml-native shell

This patch adds standalone qml-shell that can be run without a need to run
'plainbox run (...)', which would require provider to be created and loaded.

Signed-off-by: Maciej Kisielewski <email address hidden>

3554. By Maciej Kisielewski on 2015-01-29

plainbox:doc: add manpage for plainbox-qml-shell

Signed-off-by: Maciej Kisielewski <email address hidden>

3555. By Maciej Kisielewski on 2015-01-29

plainbox: add plainbox-qml-shell to setup.py

This patch adds plainbox-qml-shell to console_scripts in setup.py

Signed-off-by: Maciej Kisielewski <email address hidden>

3556. By Maciej Kisielewski on 2015-01-29

providers: add qml-tests provider

This patch adds provider with qml-native jobs. At start there is only one test
- camera test that shows feed from camera attached to the system. The provider
  is added to venv.

Signed-off-by: Maciej Kisielewski <email address hidden>

[Zygmunt Krynicki (zyga)]

FYI:

23:25 < kissiel> zyga, omg
23:25 < kissiel> zyga, i found the problem
23:25 < kissiel> zyga, uncommitted __init__
23:25 < kissiel> in qml_shell module...
23:25 < kissiel> which isn't a module :D
23:26 <@zyga> kissiel: ah
23:26 <@zyga> kissiel: well, python3.4 doesn't need init
23:26 <@zyga> kissiel: so it works there
23:27 < kissiel> zyga, right :)
23:27 <@zyga> kissiel: in 3.2 you still need (even empty) __init__.py

review: Approve

=== added file 'cep/CEP-5.txt'
--- cep/CEP-5.txt	1970-01-01 00:00:00 +0000
+++ cep/CEP-5.txt	2015-01-29 23:33:32 +0000
@@ -0,0 +1,87 @@
+Summary
+=======
+
+Checkbox stack lacks facilities for running GUI driven tests natively.
+This CEP describes a new type of checkbox job, the 'QML job'.
+
+Rationale
+=========
+
+Some of the tests require some form of graphical user interface. Current way of
+handling that is to run some binary (like qmlscene) driving the GUI. This
+approach is acceptable on window-driven environments like Ubuntu Desktop, but
+unfeasible on Ubuntu Touch. Moreover, Checkbox front-end for Ubuntu Touch is
+already driven by qmlscene, so GUI jobs might be as well defined as a
+component resembling native parts of Checkbox Touch.
+
+What qml native test should be able to do (aka Requirements)
+============================================================
+
+1. Determine its own outcome and propagate it to the shell
+2. Run full screen (without system bar, headers, etc.)
+3. Run its own internal page flow mechanism (like PageStack, tabs)
+4. Use all QML APIs that a normal app would use
+5. Get job meta-info (i.e. job id, job name, etc.)
+6. Be fully translatable
+7. Print out ``console.log()``
+
+Constraints
+===========
+
+All that consitutes the job (qml files, translations, images, media files,
+etc.) must be contained within ``data`` directory of the provider.
+
+The API
+=======
+
+Test definition
+---------------
+From testing shell perspective, the test is a root ``Item { }`` of the file
+located in path specified by ``qml_file`` field of the job definition.
+
+When creating the test component, testing environment will initiate
+``testingShell`` property with a hook to the testing shell object (satisfying
+requirement no 5.), and connect to ``testDone(test)`` signal. ``test`` object
+has to have ``outcome`` field, in order to satisfy requirement no 1.
+
+Test boilerplate
+----------------
+
+.. code-block:: guess
+
+    // boilerplate.qml
+    import QtQuick 2.0
+    Item {
+        property var testingShell;
+        signal testDone(test);
+    }
+
+
+Testing Shell object API:
+=========================
+
+``function getTest()`` - get test meta-information object
+
+``string name`` - name of the testing shell
+
+``var pageStack`` - pageStack object to use when implementing internal test
+navigation. This page stack is independent from the page stack used by outside
+application (the one containing testing shell).
+
+Full screen and page stacks
+===========================
+
+Test shell is responsible for clearing current screen, i.e. hiding toolbars,
+headers, etc., and for providing clear page stack to work on.
+This internal page stack is implicitly cleared and destroyed when testDone is
+signalled.
+
+
+Impact
+======
+Introduction of new job type requires changes in plainbox backend (new value of
+plugin field in job definition) and all checkbox front-ends. For front-ends to
+display qml job, shell that runs the job has to be present. For Checkbox Touch
+this shell will be native to the application, for other front-ends auxiliary
+qmlscene has to be run.  All changes required by this CEP should not impact
+exisiting tests.

=== added file 'checkbox-touch/components/QmlNativePage.qml'
--- checkbox-touch/components/QmlNativePage.qml	1970-01-01 00:00:00 +0000
+++ checkbox-touch/components/QmlNativePage.qml	2015-01-29 23:33:32 +0000
@@ -0,0 +1,133 @@
+/*
+ * This file is part of Checkbox
+ *
+ * Copyright 2014 Canonical Ltd.
+ *
+ * Authors:
+ * - Maciej Kisielewski <[e-mail]>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; version 3.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ */
+
+/*! \brief Page for Qml native test
+*/
+
+import QtQuick 2.0
+import Ubuntu.Components 1.1
+import QtQuick.Layouts 1.1
+import "ConfirmationLogic.js" as ConfirmationLogic
+
+Page {
+    id: qmlNativePage
+    property var test: { "name": "", "description": "", "test_number": 0, "tests_count": 0}
+
+    signal testDone(var test);
+
+    objectName: "qmlNativePage"
+    title: i18n.tr("Test Description")
+
+    Object {
+        id: testingShell
+        property string name: "Checkbox-touch qml shell"
+        property alias pageStack: qmlNativePage.pageStack
+        function getTest() {
+            return test;
+        }
+    }
+
+    head {
+        actions: [
+            Action {
+                id: skipAction
+                iconName: "media-seek-forward"
+                text: i18n.tr("Skip")
+                onTriggered: {
+                    var confirmationOptions = {
+                        question : i18n.tr("Do you really want to skip this test?"),
+                        remember : true,
+                    }
+                    ConfirmationLogic.confirmRequest(qmlNativePage,
+                        confirmationOptions, function(res) {
+                            if (res) {
+                                test["outcome"] = "skip";
+                                testDone(test);
+                            }
+                    });
+                }
+            }
+        ]
+    }
+
+    ColumnLayout {
+        id: body
+        spacing: units.gu(3)
+        anchors.fill: parent
+        anchors.margins: units.gu(3)
+
+        Label {
+            fontSize: "large"
+            Layout.fillWidth: true
+            wrapMode: Text.WrapAtWordBoundaryOrAnywhere
+            text: test["name"]
+            font.bold: true
+        }
+
+        Flickable {
+            Layout.fillWidth: true
+            Layout.fillHeight: true
+            contentHeight: childrenRect.height
+            flickableDirection: Flickable.VerticalFlick
+            clip: true
+            Label {
+                fontSize: "medium"
+                anchors.fill: parent
+                wrapMode: Text.WrapAtWordBoundaryOrAnywhere
+                text: test["description"]
+            }
+        }
+
+        LatchButton {
+            objectName: "continueButton"
+            color: UbuntuColors.green
+            Layout.fillWidth: true
+            text: i18n.tr("Continue")
+            onClicked: {
+                pageStack.pop(); // pop the description page
+                // altough there shouldn't be anything on the page stack
+                // dump it to savedStack and clear page stack,
+                // to let qml job use fresh playground
+                var savedStack = [];
+                while(pageStack.depth) {
+                    savedStack.push(pageStack.currentPage);
+                    pageStack.pop();
+                }
+                // prepare page with the test
+                var testItemComponent = Qt.createComponent(Qt.resolvedUrl(test['qml_file']));
+                if (testItemComponent.status == Component.Error) {
+                    console.log("Error creating testPageComponent:", testPageComponent.errorString());
+                }
+
+                var testItem = testItemComponent.createObject(null, {"testingShell": testingShell});
+                testItem.testDone.connect(function(testResult) {
+                    test['outcome'] = testResult['outcome'];
+                    test['result'] = testResult;
+                    pageStack.clear(); // clean test's left-overs from the stack
+                    while(savedStack.length) {
+                        pageStack.push(savedStack.pop());
+                    }
+                    testDone(test);
+                });
+            }
+        }
+    }
+}

=== modified file 'checkbox-touch/main.qml'
--- checkbox-touch/main.qml	2015-01-19 14:56:12 +0000
+++ checkbox-touch/main.qml	2015-01-29 23:33:32 +0000
@@ -327,6 +327,9 @@
                 case 'user-interact':
                     performUserInteractTest(test);
                     break;
+                case 'qml':
+                    performQmlTest(test);
+                    break;
                 default:
                     test.outcome = "skip";
                     completeTest(test);
@@ -430,6 +433,17 @@
         pageStack.push(InteractIntroPage);
     }
 
+    function performQmlTest(test) {
+        var comp = Qt.createComponent(Qt.resolvedUrl("components/QmlNativePage.qml"))
+        console.log(comp.errorString());
+        var qmlNativePage = comp.createObject();
+        qmlNativePage.test = test
+        qmlNativePage.testDone.connect(completeTest);
+        qmlNativePage.__customHeaderContents = progressHeader;
+        progressHeader.update(test);
+        pageStack.push(qmlNativePage);
+    }
+
     function showVerificationScreen(test) {
         var verificationPage = Qt.createComponent(Qt.resolvedUrl("components/TestVerificationPage.qml")).createObject();
         verificationPage.test = test

=== modified file 'checkbox-touch/py/checkbox_touch.py'
--- checkbox-touch/py/checkbox_touch.py	2015-01-07 14:52:42 +0000
+++ checkbox-touch/py/checkbox_touch.py	2015-01-29 23:33:32 +0000
@@ -602,6 +602,7 @@
                 job.tr_verification() is not None else description,
                 "plugin": job.plugin,
                 "id": job.id,
+                "qml_file": job.qml_file,
                 "start_time": time.time(),
                 "test_number": self.index,
                 "tests_count": len(self.context.state.run_list)

=== modified file 'plainbox/docs/conf.py'
--- plainbox/docs/conf.py	2015-01-27 14:35:15 +0000
+++ plainbox/docs/conf.py	2015-01-29 23:33:32 +0000
@@ -298,6 +298,8 @@
      'list and describe various objects', _authors, 1),
     ('manpages/plainbox-device', 'plainbox-device',
      'device management commands', _authors, 1),
+    ('manpages/plainbox-qml-shell', 'plainbox-qml-shell',
+     'standalone qml-native shell', _authors, 1),
     # Section 5
     ('manpages/plainbox.conf', 'plainbox.conf',
      'plainbox configuration file', _authors, 5),

=== modified file 'plainbox/docs/manpages/plainbox-job-units.rst'
--- plainbox/docs/manpages/plainbox-job-units.rst	2014-11-25 11:33:53 +0000
+++ plainbox/docs/manpages/plainbox-job-units.rst	2015-01-29 23:33:32 +0000
@@ -62,6 +62,9 @@
      :resource: A job whose command output results in a set of rfc822
           records, containing key/value pairs, and that can be used in other
           jobs' ``requires`` expressions.
+     :qml: jobs that run a custom QML payload within a test shell (QML
+        application or a generic, minimalistic QML test shell) using test API
+        described in CEP-5
 
 .. warning::
      The following plugin names are deprecated:

=== added file 'plainbox/docs/manpages/plainbox-qml-shell.rst'
--- plainbox/docs/manpages/plainbox-qml-shell.rst	1970-01-01 00:00:00 +0000
+++ plainbox/docs/manpages/plainbox-qml-shell.rst	2015-01-29 23:33:32 +0000
@@ -0,0 +1,18 @@
+======================
+plainbox-qml-shell (1)
+======================
+
+.. argparse::
+    :ref: plainbox.qml_shell.qml_shell.get_parser_for_sphinx
+    :prog: plainbox-qml-shell
+    :manpage:
+    :nodefault:
+
+    This command runs qml job provided by specified file.
+
+
+
+See Also
+========
+
+:doc:`plainbox-run`

=== added directory 'plainbox/plainbox/data/qml-shell'
=== added file 'plainbox/plainbox/data/qml-shell/pipe_handler.py'
--- plainbox/plainbox/data/qml-shell/pipe_handler.py	1970-01-01 00:00:00 +0000
+++ plainbox/plainbox/data/qml-shell/pipe_handler.py	2015-01-29 23:33:32 +0000
@@ -0,0 +1,42 @@
+# This file is part of Checkbox.
+#
+# Copyright 2012-2014 Canonical Ltd.
+# Written by:
+#   Maciej Kisielewski <[e-mail]>
+#
+# Checkbox is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 3,
+# as published by the Free Software Foundation.
+#
+# Checkbox is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with Checkbox.  If not, see <http://www.gnu.org/licenses/>.
+
+import os
+
+
+def write_and_close(s, fd: str) -> None:
+    """
+    Write ``s`` to file descriptor ``fd`` and close ``fd`` afterwards.
+
+    fd is of type str because calling code is written in javascript that
+    doesn't support notion of ints.
+    """
+    with os.fdopen(int(fd), 'wt', encoding='utf-8') as stream:
+        stream.write(s)
+
+
+def read_and_close(fd: str) -> str:
+    """
+    Read from ``fd`` file descriptor and close ``fd`` afterwards.
+    returns read string
+
+    fd is of type str because calling code is written in javascript that
+    doesn't support notion of ints.
+    """
+    with os.fdopen(int(fd), 'rt', encoding='utf-8') as stream:
+        return stream.read()

=== added file 'plainbox/plainbox/data/qml-shell/plainbox_qml_shell.qml'
--- plainbox/plainbox/data/qml-shell/plainbox_qml_shell.qml	1970-01-01 00:00:00 +0000
+++ plainbox/plainbox/data/qml-shell/plainbox_qml_shell.qml	2015-01-29 23:33:32 +0000
@@ -0,0 +1,111 @@
+/*
+ * This file is part of Checkbox
+ *
+ * Copyright 2014 Canonical Ltd.
+ *
+ * Authors:
+ * - Maciej Kisielewski <[e-mail]>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; version 3.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ */
+/*! \brief QML standalone shell
+
+    This component serves as QML shell that embeds native QML plainbox jobs.
+    The job it loads is specified by --job argument passed to qmlscene
+    launching this component.
+    Job to be run must have testingShell property that be used as a hook to
+    'outside world'. The job should also define testDone signal that gets
+    signalled once test is finished. The signal should pass result object
+    containing following fields:
+    'outcome' (mandatory): outcome of a test, e.g. 'pass', 'fail', 'undecided'.
+    'suggestedOutcome': if outcome is 'undecided', than this suggestion will be
+    presented to the user, letting them decide the final outcome of a test.
+*/
+import QtQuick 2.0
+import Ubuntu.Components 0.1
+import io.thp.pyotherside 1.2
+
+MainView {
+    id: mainView
+    width: units.gu(100)
+    height: units.gu(75)
+
+    Python {
+        id: py
+        Component.onCompleted: {
+            addImportPath(Qt.resolvedUrl('.'));
+            py.importModule('pipe_handler', function() {
+                console.log('pipe_handler.py imported');
+                py.readAndClose(args.values['fd-in'], function(job_repr) {
+                    testingShell.getTest = function() {
+                        return JSON.parse(job_repr);
+                    }
+                    loader.setSource(args.values.job,
+                                     {'testingShell': testingShell});
+                });
+            });
+        }
+
+        onError: console.error("python error: " + traceback)
+        onReceived: console.log("pyotherside.send: " + data)
+
+        function writeAndClose(str, fd, continuation) {
+            py.call('pipe_handler.write_and_close', [str, fd], continuation);
+        }
+        function readAndClose(fd, continuation) {
+            py.call('pipe_handler.read_and_close', [fd], continuation);
+        }
+    }
+
+    // information and functionality passed to qml job component
+    property var testingShell: {
+        "name": "Plainbox qml shell",
+        "pageStack": pageStack
+    }
+
+    Arguments {
+        id: args
+        Argument {
+            name: "job"
+            help: "QML-native job to run"
+            required: true
+            valueNames: ["PATH"]
+        }
+        Argument {
+            name: "fd-out"
+            help: "Descriptor number of pipe to write to"
+            required: false
+            valueNames: ["N"]
+        }
+        Argument {
+            name: "fd-in"
+            help: "Descriptor number of pipe to read from"
+            required: false
+            valueNames: ["N"]
+        }
+    }
+
+    Loader {
+        id: loader
+        visible: false
+        onLoaded: loader.item.testDone.connect(testDone)
+    }
+
+    function testDone(res) {
+        py.writeAndClose(JSON.stringify(res), args.values['fd-out'], Qt.quit);
+    }
+    PageStack {
+        id: pageStack
+    }
+
+}

=== modified file 'plainbox/plainbox/impl/ctrl.py'
--- plainbox/plainbox/impl/ctrl.py	2015-01-09 21:48:46 +0000
+++ plainbox/plainbox/impl/ctrl.py	2015-01-29 23:33:32 +0000
@@ -35,11 +35,13 @@
 
 import abc
 import contextlib
+import errno
 try:
     import grp
 except ImportError:
     grp = None
 import itertools
+import json
 import logging
 import os
 try:
@@ -54,6 +56,7 @@
 from plainbox.abc import IJobResult
 from plainbox.abc import ISessionStateController
 from plainbox.i18n import gettext as _
+from plainbox.impl import get_plainbox_dir
 from plainbox.impl.depmgr import DependencyDuplicateError
 from plainbox.impl.depmgr import DependencyMissingError
 from plainbox.impl.resource import ExpressionCannotEvaluateError
@@ -806,6 +809,160 @@
             return 1
 
 
+class QmlJobExecutionController(CheckBoxExecutionController):
+    """
+    An execution controller that is able to run jobs in QML shell.
+    """
+
+    QML_SHELL_PATH = os.path.join(get_plainbox_dir(), 'data', 'qml-shell',
+                                  'plainbox_qml_shell.qml')
+
+    def get_execution_command(self, job, job_state, config, session_dir,
+                              nest_dir, shell_out_fd, shell_in_fd):
+        """
+        Get the command to execute the specified job
+
+        :param job:
+            job definition with the command and environment definitions
+        :param job_state:
+            The JobState associated to the job to execute.
+        :param config:
+            A PlainBoxConfig instance which can be used to load missing
+            environment definitions that apply to all jobs. Ignored.
+        :param session_dir:
+            Base directory of the session this job will execute in.
+            This directory is used to co-locate some data that is unique to
+            this execution as well as data that is shared by all executions.
+            Ignored.
+        :param nest_dir:
+            A directory with a nest of symlinks to all executables required to
+            execute the specified job. Ingored.
+        :param shell_out_fd:
+            File descriptor number which is used to pipe through result object
+            from the qml shell to plainbox.
+        :param shell_in_fd:
+            File descriptor number which is used to pipe through test meta
+            information from plainbox to qml shell.
+        :returns:
+            List of command arguments
+
+        """
+        cmd = ['qmlscene', '--job', job.qml_file, '--fd-out', shell_out_fd,
+               '--fd-in', shell_in_fd, self.QML_SHELL_PATH]
+        return cmd
+
+    def get_checkbox_score(self, job):
+        """
+        Compute how applicable this controller is for the specified job.
+
+        :returns:
+            4 if the job is a qml job or -1 otherwise
+        """
+        if job.plugin == 'qml':
+            return 4
+        else:
+            return -1
+
+    def gen_job_repr(self, job):
+        """
+        Generate simplified job representation for use in qml shell
+        :returns:
+            dictionary with simplified job representation
+        """
+        logger.debug(_("Generating job repr for job: %r"), job)
+        return {
+            "id": job.id,
+            "summary": job.tr_summary(),
+            "description": job.tr_description(),
+        }
+
+    def execute_job(self, job, job_state, config, session_dir, extcmd_popen):
+        """
+        Execute the specified job using the specified subprocess-like object,
+        passing fd with opened pipe for qml-shell->plainbox communication.
+
+        :param job:
+            The JobDefinition to execute
+        :param config:
+            A PlainBoxConfig instance which can be used to load missing
+            environment definitions that apply to all jobs. It is used to
+            provide values for missing environment variables that are required
+            by the job (as expressed by the environ key in the job definition
+            file).
+        :param session_dir:
+            Base directory of the session this job will execute in.
+            This directory is used to co-locate some data that is unique to
+            this execution as well as data that is shared by all executions.
+        :param extcmd_popen:
+            A subprocess.Popen like object
+        :returns:
+            The return code of the command, as returned by subprocess.call()
+        """
+
+        class DuplexPipe:
+            """
+            Helper context creating two pipes, ensuring they are closed
+            properly
+            """
+            def __enter__(self):
+                self.a_read, self.b_write = os.pipe()
+                self.b_read, self.a_write = os.pipe()
+                return self.a_read, self.b_write, self.b_read, self.a_write
+
+            def __exit__(self, *args):
+                for pipe in (self.a_read, self.b_write,
+                             self.b_read, self.a_write):
+                    # typically those pipes are already closed; trying to
+                    # re-close them causes OSError (errno == 9) to be raised
+                    try:
+                        os.close(pipe)
+                    except OSError as exc:
+                        if exc.errno != errno.EBADF:
+                            raise
+        # CHECKBOX_DATA is where jobs can share output.
+        # It has to be an directory that scripts can assume exists.
+        if not os.path.isdir(self.get_CHECKBOX_DATA(session_dir)):
+            os.makedirs(self.get_CHECKBOX_DATA(session_dir))
+        # Setup the executable nest directory
+        with self.configured_filesystem(job, config) as nest_dir:
+            with DuplexPipe() as (plainbox_read, shell_write,
+                                  shell_read, plainbox_write):
+                # Get the command and the environment.
+                # of this execution controller
+                cmd = self.get_execution_command(
+                    job, job_state, config, session_dir, nest_dir,
+                    str(shell_write), str(shell_read))
+                env = self.get_execution_environment(
+                    job, job_state, config, session_dir, nest_dir)
+                with self.temporary_cwd(job, config) as cwd_dir:
+                    job_json = json.dumps(self.gen_job_repr(job))
+                    pipe_out = os.fdopen(plainbox_write, 'wt')
+                    pipe_out.write(job_json)
+                    pipe_out.close()
+                    # run the command
+                    logger.debug(_("job[%s] executing %r with"
+                                   "env %r in cwd %r"),
+                                 job.id, cmd, env, cwd_dir)
+                    ret = extcmd_popen.call(cmd, env=env, cwd=cwd_dir,
+                                            pass_fds=[shell_write, shell_read])
+                    os.close(shell_read)
+                    os.close(shell_write)
+                    pipe_in = os.fdopen(plainbox_read)
+                    res_object_json_string = pipe_in.read()
+                    pipe_in.close()
+                    if ret != 0:
+                        return ret
+                    try:
+                        result = json.loads(res_object_json_string)
+                        if result['outcome'] == "pass":
+                            return 0
+                        else:
+                            return 1
+                    except ValueError:
+                        # qml-job did not print proper json object
+                        return 1
+
+
 class CheckBoxDifferentialExecutionController(CheckBoxExecutionController):
     """
     A CheckBoxExecutionController subclass that uses differential environment.

=== added file 'plainbox/plainbox/impl/providers/stubbox/data/qml-navigation.qml'
--- plainbox/plainbox/impl/providers/stubbox/data/qml-navigation.qml	1970-01-01 00:00:00 +0000
+++ plainbox/plainbox/impl/providers/stubbox/data/qml-navigation.qml	2015-01-29 23:33:32 +0000
@@ -0,0 +1,68 @@
+import QtQuick 2.0
+import Ubuntu.Components 0.1
+import QtQuick.Layouts 1.1
+
+Item {
+    id: root
+    signal testDone(var test);
+    property var testingShell;
+
+    Component.onCompleted: testingShell.pageStack.push(mainPage)
+
+    Page {
+        id: mainPage
+        title: i18n.tr("A simple test")
+
+        ColumnLayout {
+            spacing: units.gu(10)
+            anchors {
+                margins: units.gu(5)
+                fill: parent
+            }
+
+            Button {
+                Layout.fillWidth: true; Layout.fillHeight: true
+                text: i18n.tr("Next screen")
+                color: "#38B44A"
+                onClicked: {
+                    testingShell.pageStack.push(subPage);
+                }
+            }
+        }
+    }
+
+
+    Page {
+        id: subPage
+        visible: false
+        ColumnLayout {
+            spacing: units.gu(10)
+            anchors {
+                margins: units.gu(5)
+                fill: parent
+            }
+
+            Text {
+                text: i18n.tr("You can use toolbar to nagivage back")
+            }
+
+            Button {
+                Layout.fillWidth: true; Layout.fillHeight: true
+                text: i18n.tr("Pass")
+                color: "#38B44A"
+                onClicked: {
+                    testDone({'outcome': 'pass'});
+                }
+            }
+
+            Button {
+                Layout.fillWidth: true; Layout.fillHeight: true
+                text: i18n.tr("Fail")
+                color: "#DF382C"
+                onClicked: {
+                    testDone({"outcome": "fail"});
+                }
+            }
+        }
+    }
+}

=== added file 'plainbox/plainbox/impl/providers/stubbox/data/qml-simple.qml'
--- plainbox/plainbox/impl/providers/stubbox/data/qml-simple.qml	1970-01-01 00:00:00 +0000
+++ plainbox/plainbox/impl/providers/stubbox/data/qml-simple.qml	2015-01-29 23:33:32 +0000
@@ -0,0 +1,40 @@
+import QtQuick 2.0
+import Ubuntu.Components 0.1
+import QtQuick.Layouts 1.1
+
+Item {
+    id: root
+    signal testDone(var test);
+    property var testingShell;
+
+    Component.onCompleted: testingShell.pageStack.push(testPage)
+
+    Page {
+        id: testPage
+        ColumnLayout {
+            spacing: units.gu(10)
+            anchors {
+                margins: units.gu(5)
+                fill: parent
+            }
+
+            Button {
+                Layout.fillWidth: true; Layout.fillHeight: true
+                text: i18n.tr("Pass")
+                color: "#38B44A"
+                onClicked: {
+                    testDone({'outcome': 'pass'});
+                }
+            }
+
+            Button {
+                Layout.fillWidth: true; Layout.fillHeight: true
+                text: i18n.tr("Fail")
+                color: "#DF382C"
+                onClicked: {
+                    testDone({"outcome": "fail"});
+                }
+            }
+        }
+    }
+}

=== modified file 'plainbox/plainbox/impl/providers/stubbox/units/jobs/categories.pxu'
--- plainbox/plainbox/impl/providers/stubbox/units/jobs/categories.pxu	2014-12-01 11:31:15 +0000
+++ plainbox/plainbox/impl/providers/stubbox/units/jobs/categories.pxu	2015-01-29 23:33:32 +0000
@@ -29,3 +29,7 @@
 id: overridden
 unit: category
 _name: Overridden Category
+
+id: qml-native
+unit: category
+_name: QML-native tests

=== modified file 'plainbox/plainbox/impl/providers/stubbox/units/jobs/stub.pxu'
--- plainbox/plainbox/impl/providers/stubbox/units/jobs/stub.pxu	2015-01-26 15:38:31 +0000
+++ plainbox/plainbox/impl/providers/stubbox/units/jobs/stub.pxu	2015-01-29 23:33:32 +0000
@@ -421,3 +421,25 @@
  dd if=/dev/zero bs=1M count=16384
 estimated_duration: 750
 category_id: stress
+
+id: stub/qml-simple
+_summary: A QML job that runs simple GUI
+_description:
+ This job displays a GUI that has two buttons determining outcome of the test.
+ It's similar to user-interact-verify, but this job is QML native.
+plugin: qml
+qml_file: qml-simple.qml
+flags: preserve-locale
+estimated_duration: 10
+category_id: qml-native
+
+id: stub/qml-navigation
+_summary: A QML job that has its own navigation
+_description:
+ This job displays a GUI with multiple screens using its own (independent) flow
+ control mechanism (page stack).
+plugin: qml
+qml_file: qml-navigation.qml
+flags: preserve-locale
+estimated_duration: 20
+category_id: qml-native

=== modified file 'plainbox/plainbox/impl/runner.py'
--- plainbox/plainbox/impl/runner.py	2015-01-26 15:38:31 +0000
+++ plainbox/plainbox/impl/runner.py	2015-01-29 23:33:32 +0000
@@ -296,12 +296,14 @@
                 from plainbox.impl.ctrl import RootViaPTL1ExecutionController
                 from plainbox.impl.ctrl import RootViaSudoExecutionController
                 from plainbox.impl.ctrl import UserJobExecutionController
+                from plainbox.impl.ctrl import QmlJobExecutionController
                 execution_ctrl_list = [
                     RootViaPTL1ExecutionController(provider_list),
                     RootViaPkexecExecutionController(provider_list),
                     # XXX: maybe this one should be only used on command line
                     RootViaSudoExecutionController(provider_list),
                     UserJobExecutionController(provider_list),
+                    QmlJobExecutionController(provider_list),
                 ]
             elif sys.platform == 'win32':
                 from plainbox.impl.ctrl import UserJobExecutionController
@@ -680,6 +682,77 @@
         result_cmd.outcome = IJobResult.OUTCOME_UNDECIDED
         return result_cmd
 
+    def run_qml_job(self, job, job_state, config):
+        """
+        Method called to run a job with plugin field equal to 'qml'
+
+        The 'qml' job implements the following scenario:
+
+        * Maybe display the description to the user
+        * Run qmlscene with provided test and wait for it to finish
+        * Decide on the outcome based on the result object returned by qml
+          shell
+        * The method ends here
+
+        .. note::
+            QML jobs are fully manual jobs with graphical user interface
+            implemented in QML. They implement proposal described in CEP-5.
+        """
+        if job.plugin != "qml":
+            # TRANSLATORS: please keep 'plugin' untranslated
+            raise ValueError(_("bad job plugin value"))
+        try:
+            ctrl = self._get_ctrl_for_job(job)
+        except LookupError:
+            return MemoryJobResult({
+                'outcome': IJobResult.OUTCOME_NOT_SUPPORTED,
+                'comment': _('No suitable execution controller is available)'),
+            })
+        # Run the embedded command
+        start_time = time.time()
+        delegate, io_log_gen = self._prepare_io_handling(job, config)
+        # Create a subprocess.Popen() like object that uses the delegate
+        # system to observe all IO as it occurs in real time.
+        delegate_cls = self._get_delegate_cls(config)
+        extcmd_popen = delegate_cls(delegate)
+        # Stream all IOLogRecord entries to disk
+        record_path = os.path.join(
+            self._jobs_io_log_dir, "{}.record.gz".format(
+                slugify(job.id)))
+        with gzip.open(record_path, mode='wb') as gzip_stream, \
+                io.TextIOWrapper(
+                    gzip_stream, encoding='UTF-8') as record_stream:
+            writer = IOLogRecordWriter(record_stream)
+            io_log_gen.on_new_record.connect(writer.write_record)
+            try:
+                # Start the process and wait for it to finish getting the
+                # result code. This will actually call a number of callbacks
+                # while the process is running. It will also spawn a few
+                # threads although all callbacks will be fired from a single
+                # thread (which is _not_ the main thread)
+                logger.debug(
+                    _("job[%s] starting qml shell: %s"), job.id, job.qml_file)
+                # Run the job command using extcmd
+                return_code = self._run_extcmd(job, job_state, config,
+                                               extcmd_popen, ctrl)
+                logger.debug(
+                    _("job[%s] shell return code: %r"), job.id, return_code)
+            finally:
+                io_log_gen.on_new_record.disconnect(writer.write_record)
+        execution_duration = time.time() - start_time
+        # Convert the return of the command to the outcome of the job
+        if return_code == 0:
+            outcome = IJobResult.OUTCOME_PASS
+        else:
+            outcome = IJobResult.OUTCOME_FAIL
+        # Create a result object and return it
+        return DiskJobResult({
+            'outcome': outcome,
+            'return_code': return_code,
+            'io_log_filename': record_path,
+            'execution_duration': execution_duration
+        })
+
     def _get_dry_run_result(self, job):
         """
         Internal method of JobRunner.

=== modified file 'plainbox/plainbox/impl/test_ctrl.py'
--- plainbox/plainbox/impl/test_ctrl.py	2015-01-09 21:48:46 +0000
+++ plainbox/plainbox/impl/test_ctrl.py	2015-01-29 23:33:32 +0000
@@ -34,6 +34,7 @@
 from plainbox.impl.applogic import PlainBoxConfig
 from plainbox.impl.ctrl import CheckBoxExecutionController
 from plainbox.impl.ctrl import CheckBoxSessionStateController
+from plainbox.impl.ctrl import QmlJobExecutionController
 from plainbox.impl.ctrl import RootViaPTL1ExecutionController
 from plainbox.impl.ctrl import RootViaPkexecExecutionController
 from plainbox.impl.ctrl import RootViaSudoExecutionController
@@ -1097,3 +1098,110 @@
         self.job.user = None
         # Ensure that we get a negative score for this controller
         self.assertEqual(self.ctrl.get_checkbox_score(self.job), -1)
+
+
+class QmlJobExecutionControllerTests(CheckBoxExecutionControllerTestsMixIn,
+                                     TestCase):
+    """
+    Tests for QmlJobExecutionController
+    """
+    CLS = QmlJobExecutionController
+
+    SHELL_OUT_FD = 6
+    SHELL_IN_FD = 7
+
+    def test_job_repr(self):
+        self.assertEqual(
+            self.ctrl.gen_job_repr(self.job),
+            {'id': self.job.id,
+             'summary': self.job.tr_summary(),
+             'description': self.job.tr_description()})
+
+    def test_get_execution_command(self):
+        """
+        Tests gluing of commandline arguments when running QML exec. ctrl.
+        """
+        self.assertEqual(
+            self.ctrl.get_execution_command(
+                self.job, self.job_state, self.config, self.SESSION_DIR,
+                self.NEST_DIR, self.SHELL_OUT_FD, self.SHELL_IN_FD),
+            ['qmlscene', '--job', self.job.qml_file, '--fd-out',
+             self.SHELL_OUT_FD, '--fd-in', self.SHELL_IN_FD,
+             self.ctrl.QML_SHELL_PATH])
+
+    @mock.patch('os.path.isdir')
+    @mock.patch('os.fdopen')
+    @mock.patch('os.pipe')
+    @mock.patch('os.write')
+    @mock.patch('os.close')
+    def test_execute_job(self, mock_os_close, mock_os_write, mock_os_pipe,
+                         mock_os_fdopen, mock_os_path_isdir):
+        """
+        Test if qml exec. ctrl. correctly runs piping
+        """
+        mock_os_pipe.side_effect = [("pipe0_r", "pipe0_w"),
+                                    ("pipe1_r", "pipe1_w")]
+        with mock.patch.object(self.ctrl, 'get_execution_command'), \
+                mock.patch.object(self.ctrl, 'get_execution_environment'), \
+                mock.patch.object(self.ctrl, 'configured_filesystem'), \
+                mock.patch.object(self.ctrl, 'temporary_cwd'), \
+                mock.patch.object(self.ctrl, 'gen_job_repr', return_value={}):
+            retval = self.ctrl.execute_job(
+                self.job, self.job_state, self.config, self.SESSION_DIR,
+                self.extcmd_popen)
+            # Ensure that call was invoked with command end environment (passed
+            # as keyword argument). Extract the return value of
+            # configured_filesystem() as nest_dir so that we can pass it to
+            # other calls to get their mocked return values.
+            # Urgh! is this doable somehow without all that?
+            nest_dir = self.ctrl.configured_filesystem().__enter__()
+            cwd_dir = self.ctrl.temporary_cwd().__enter__()
+            self.extcmd_popen.call.assert_called_with(
+                self.ctrl.get_execution_command(
+                    self.job, self.config, self.SESSION_DIR, nest_dir),
+                env=self.ctrl.get_execution_environment(
+                    self.job, self.config, self.SESSION_DIR, nest_dir),
+                cwd=cwd_dir,
+                pass_fds=["pipe0_w", "pipe1_r"])
+        # Ensure that execute_job() returns the return value of call()
+        self.assertEqual(retval, self.extcmd_popen.call())
+        # Ensure that presence of CHECKBOX_DATA directory was checked for
+        mock_os_path_isdir.assert_called_with(
+            self.ctrl.get_CHECKBOX_DATA(self.SESSION_DIR))
+        self.assertEqual(mock_os_pipe.call_count, 2)
+        self.assertEqual(mock_os_fdopen.call_count, 2)
+        self.assertEqual(mock_os_close.call_count, 6)
+
+    @mock.patch('os.path.isdir')
+    @mock.patch('os.fdopen')
+    @mock.patch('os.pipe')
+    @mock.patch('os.write')
+    @mock.patch('os.close')
+    def test_pipes_closed_when_cmd_raises(
+            self, mock_os_close, mock_os_write, mock_os_pipe, mock_os_fdopen,
+            mock_os_path_isdir):
+        """
+        Test if all pipes used by execute_job() are properly closed if
+        exception is raised during execution of command
+        """
+        mock_os_pipe.side_effect = [("pipe0_r", "pipe0_w"),
+                                    ("pipe1_r", "pipe1_w")]
+        with mock.patch.object(self.ctrl, 'get_execution_command'), \
+                mock.patch.object(self.ctrl, 'get_execution_environment'), \
+                mock.patch.object(self.ctrl, 'configured_filesystem'), \
+                mock.patch.object(self.ctrl, 'temporary_cwd'), \
+                mock.patch.object(self.ctrl, 'gen_job_repr', return_value={}), \
+                mock.patch.object(self.extcmd_popen, 'call',
+                                  side_effect=Exception('Boom')):
+            with self.assertRaises(Exception):
+                self.ctrl.execute_job(
+                    self.job, self.job_state, self.config, self.SESSION_DIR,
+                    self.extcmd_popen)
+        os.close.assert_any_call('pipe0_r')
+        os.close.assert_any_call('pipe1_r')
+        os.close.assert_any_call('pipe0_w')
+        os.close.assert_any_call('pipe1_w')
+
+    def test_get_checkbox_score_for_qml_job(self):
+        self.job.plugin = 'qml'
+        self.assertEqual(self.ctrl.get_checkbox_score(self.job), 4)

=== modified file 'plainbox/plainbox/impl/unit/job.py'
--- plainbox/plainbox/impl/unit/job.py	2015-01-07 14:55:36 +0000
+++ plainbox/plainbox/impl/unit/job.py	2015-01-29 23:33:32 +0000
@@ -24,6 +24,7 @@
 
 import logging
 import re
+import os
 
 from plainbox.abc import IJobDefinition
 from plainbox.i18n import gettext as _
@@ -103,6 +104,7 @@
     user_interact = "user-interact"
     user_interact_verify = "user-interact-verify"
     shell = 'shell'
+    qml = 'qml'
 
 
 class JobDefinition(UnitWithId, JobDefinitionLegacyAPI, IJobDefinition):
@@ -304,6 +306,22 @@
                 'category_id', '2013.com.canonical.plainbox::uncategorised'))
 
     @property
+    def qml_file(self):
+        """
+        path to a QML file that implements tests UI for this job
+
+        This property exposes a path to QML file that follows the Plainbox QML
+        Test Specification. The file will be loaded either in the native test
+        shell of the application using plainbox or with a helper, generic
+        loader for all command-line applications.
+
+        To use this property, the plugin type should be set to 'qml'.
+        """
+        qml_file = self.get_record_value('qml_file')
+        if qml_file is not None and self.provider is not None:
+            return os.path.join(self.provider.data_dir, qml_file)
+
+    @property
     def estimated_duration(self):
         """
         estimated duration of this job in seconds.
@@ -547,6 +565,7 @@
             purpose = 'purpose'
             steps = 'steps'
             verification = 'verification'
+            qml_file = 'qml_file'
 
         field_validators = {
             fields.name: [
@@ -599,11 +618,11 @@
                 # All jobs except for manual must have a command
                 PresentFieldValidator(
                     message=_("command is mandatory for non-manual jobs"),
-                    onlyif=lambda unit: unit.plugin != 'manual'),
+                    onlyif=lambda unit: unit.plugin not in ('manual', 'qml')),
                 # Manual jobs cannot have a command
                 UselessFieldValidator(
-                    message=_("command on a manual job makes no sense"),
-                    onlyif=lambda unit: unit.plugin == 'manual'),
+                    message=_("command on a manual or qml job makes no sense"),
+                    onlyif=lambda unit: unit.plugin in ('manual', 'qml')),
                 # We don't want to refer to CHECKBOX_SHARE anymore
                 CorrectFieldValueValidator(
                     lambda command: "CHECKBOX_SHARE" not in command,
@@ -792,5 +811,22 @@
                         ' non-C locale then set the preserve-locale flag'
                     ),
                     onlyif=lambda unit: unit.command),
+            ],
+            fields.qml_file: [
+                UntranslatableFieldValidator,
+                TemplateInvariantFieldValidator,
+                PresentFieldValidator(
+                    onlyif=lambda unit: unit.plugin == 'qml'),
+                CorrectFieldValueValidator(
+                    lambda value: value.endswith('.qml'),
+                    Problem.wrong, Severity.advice,
+                    message=_('use the .qml extension for all QML files'),
+                    onlyif=lambda unit: (unit.plugin == 'qml'
+                                         and unit.qml_file)),
+                CorrectFieldValueValidator(
+                    lambda value, unit: os.path.isfile(unit.qml_file),
+                    message=_('please point to an existing QML file'),
+                    onlyif=lambda unit: (unit.plugin == 'qml'
+                                         and unit.qml_file)),
             ]
         }

=== modified file 'plainbox/plainbox/impl/unit/test_job.py'
--- plainbox/plainbox/impl/unit/test_job.py	2015-01-07 14:53:47 +0000
+++ plainbox/plainbox/impl/unit/test_job.py	2015-01-29 23:33:32 +0000
@@ -126,6 +126,32 @@
         self.assertEqual(job.flags, "flags-value")
         self.assertEqual(job.category_id, "category_id-value")
 
+    def test_qml_file_property_none_when_missing_provider(self):
+        """
+        Ensure that qml_file property is set to None when provider is not set.
+        """
+        job = JobDefinition({
+            'qml_file': 'qml_file-value'
+        }, raw_data={
+            'qml_file': 'qml_file-raw'
+        })
+        self.assertEqual(job.qml_file, None)
+
+    def test_qml_file_property(self):
+        """
+        Ensure that qml_file property is properly constructed
+        """
+        mock_provider = mock.Mock()
+        type(mock_provider).data_dir = mock.PropertyMock(return_value='data')
+        job = JobDefinition({
+            'qml_file': 'qml_file-value'
+        }, raw_data={
+            'qml_file': 'qml_file-raw'
+        }, provider=mock_provider)
+        with mock.patch('os.path.join', return_value='path') as mock_join:
+            self.assertEqual(job.qml_file, 'path')
+            mock_join.assert_called_with('data', 'qml_file-value')
+
     def test_properties_default_values(self):
         """
         Ensure that all properties default to None
@@ -138,6 +164,7 @@
         self.assertEqual(job.shell, 'bash')
         self.assertEqual(job.flags, None)
         self.assertEqual(job.category_id, '2013.com.canonical.plainbox::uncategorised')
+        self.assertEqual(job.qml_file, None)
 
     def test_checksum_smoke(self):
         job1 = JobDefinition({'plugin': 'plugin', 'user': 'root'})
@@ -291,7 +318,7 @@
             'plugin': 'foo'
         }, provider=self.provider).check()
         message = ("field 'plugin', valid values are: attachment, local,"
-                   " manual, resource, shell, user-interact,"
+                   " manual, qml, resource, shell, user-interact,"
                    " user-interact-verify, user-verify")
         self.assertIssueFound(issue_list, self.unit_cls.Meta.fields.plugin,
                               Problem.wrong, Severity.error, message)
@@ -329,7 +356,7 @@
 
     def test_command__present__on_non_manual(self):
         for plugin in self.unit_cls.plugin.symbols.get_all_symbols():
-            if plugin == 'manual':
+            if plugin in ('manual', 'qml'):
                 continue
             # TODO: switch to subTest() once we depend on python3.4
             issue_list = self.unit_cls({
@@ -348,6 +375,15 @@
             issue_list, self.unit_cls.Meta.fields.command,
             Problem.useless, Severity.warning)
 
+    def test_command__useless__on_qml(self):
+        issue_list = self.unit_cls({
+            'plugin': 'qml',
+            'command': 'command'
+        }, provider=self.provider).check()
+        self.assertIssueFound(
+            issue_list, self.unit_cls.Meta.fields.command,
+            Problem.useless, Severity.warning)
+
     def test_command__not_using_CHECKBOX_SHARE(self):
         issue_list = self.unit_cls({
             'command': '$CHECKBOX_SHARE'

=== added directory 'plainbox/plainbox/qml_shell'
=== added file 'plainbox/plainbox/qml_shell/__init__.py'
=== added file 'plainbox/plainbox/qml_shell/qml_shell.py'
--- plainbox/plainbox/qml_shell/qml_shell.py	1970-01-01 00:00:00 +0000
+++ plainbox/plainbox/qml_shell/qml_shell.py	2015-01-29 23:33:32 +0000
@@ -0,0 +1,141 @@
+# This file is part of Checkbox.
+#
+# Copyright 2014 Canonical Ltd.
+# Written by:
+#  Maciej Kisielewski <[e-mail]>
+#
+# Checkbox is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 3,
+# as published by the Free Software Foundation.
+#
+# Checkbox is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with Checkbox.  If not, see <http://www.gnu.org/licenses/>.
+
+import json
+import os
+import subprocess
+import sys
+
+from plainbox import __version__ as plainbox_version
+from plainbox.i18n import docstring
+from plainbox.i18n import gettext as _
+from plainbox.i18n import gettext_noop as N_
+from plainbox.impl import get_plainbox_dir
+from plainbox.impl.applogic import PlainBoxConfig
+from plainbox.impl.clitools import SingleCommandToolMixIn
+from plainbox.impl.commands import PlainBoxCommand
+from plainbox.impl.commands import PlainBoxToolBase
+
+
+@docstring(
+    # TRANSLATORS: please leave various options (both long and short forms),
+    # environment variables and paths in their original form. Also keep the
+    # special @EPILOG@ string. The first line of the translation is special and
+    # is used as the help message. Please keep the pseudo-statement form and
+    # don't finish the sentence with a dot. Pay extra attention to whitespace.
+    # It must be correctly preserved or the result won't work. In particular
+    # the leading whitespace *must* be preserved and *must* have the same
+    # length on each line.
+    N_("""
+    run qml job in standalone shell
+
+    Runs specified file as it would be a plainbox' qml job.
+    Returns 0 if job returned 'pass', 1 if job returned 'fail', or
+    other value in case of an error.
+
+    @EPILOG@
+
+    General purpose of this command is to make development of qml-native jobs
+    faster, by making it easier to test qml file(s) that constitute to job
+    without resorting to installation of provider and running plainbox run.
+    Typical approach to the development of new qml job would be as follows:
+
+    - have an idea for a job
+
+    - create a qml file in Ubuntu-SDK or Your Favourite Editor
+
+    - hack on the file and iterate using qmlscene (or use plainbox-qml-shell
+      immediately if you start with next point)
+
+    - make it conformant to plainbox qml-native API described in CEP-5
+      (calling test-done at the end)
+
+    - copy qml file over to data dir of a provider and add a job unit to it
+
+    """))
+class QmlShellCommand(PlainBoxCommand):
+    def register_parser(self, subparsers):
+        parser = subparsers.add_parser(
+            "plainbox-qml-shell",
+            help=_("run qml-native test in a standalone shell"))
+
+        self.register_arguments(parser)
+
+    def register_arguments(self, parser):
+        parser.set_defaults(command=self)
+        parser.add_argument('QML_FILE', help=_("qml file with job to be run"),
+                            metavar='QML-FILE')
+
+    def invoked(self, ns):
+        QML_SHELL_PATH = os.path.join(get_plainbox_dir(), 'qml_shell',
+                                      'qml_shell.qml')
+
+        test_result_object_prefix = "qml: __test_result_object:"
+        test_res = None
+        p = subprocess.Popen(['qmlscene', '--job',
+                             os.path.abspath(ns.QML_FILE), QML_SHELL_PATH],
+                             stderr=subprocess.PIPE)
+        for line in iter(p.stderr.readline, ''):
+            line = line.decode(sys.stderr.encoding)
+            if not line:
+                break
+            if line.startswith(test_result_object_prefix):
+                obj_json = line[len(test_result_object_prefix):]
+                test_res = json.loads(obj_json)
+            else:
+                print(line)
+
+        if not test_res:
+            return _("Job did not return any result")
+
+        print(_("Test outcome: {}").format(test_res['outcome']))
+        return test_res['outcome'] != "pass"
+
+
+class PlainboxQmlShellTool(SingleCommandToolMixIn, PlainBoxToolBase):
+    def get_command(self):
+        return QmlShellCommand()
+
+    @classmethod
+    def get_exec_name(cls):
+        return "plainbox-qml-shell"
+
+    @classmethod
+    def get_exec_version(cls):
+        """
+        Get the version reported by this executable
+        """
+        return cls.format_version_tuple(plainbox_version)
+
+    @classmethod
+    def get_config_cls(cls):
+        """
+        Get the Config class that is used by this implementation.
+
+        This can be overridden by subclasses to use a different config
+        class that is suitable for the particular application.
+        """
+        return PlainBoxConfig
+
+
+def main(argv=None):
+    raise SystemExit(PlainboxQmlShellTool().main(argv))
+
+
+def get_parser_for_sphinx():
+    return PlainboxQmlShellTool().construct_parser()

=== added file 'plainbox/plainbox/qml_shell/qml_shell.qml'
--- plainbox/plainbox/qml_shell/qml_shell.qml	1970-01-01 00:00:00 +0000
+++ plainbox/plainbox/qml_shell/qml_shell.qml	2015-01-29 23:33:32 +0000
@@ -0,0 +1,75 @@
+/*
+ * This file is part of Checkbox
+ *
+ * Copyright 2014 Canonical Ltd.
+ *
+ * Authors:
+ * - Maciej Kisielewski <[e-mail]>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; version 3.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ */
+/*! \brief QML standalone shell
+
+    This component serves as QML shell that embeds native QML plainbox jobs.
+    The job it loads is specified by --job argument passed to qmlscene
+    launching this component.
+    Job to be run must have testingShell property that be used as a hook to
+    'outside world'. The job should also define testDone signal that gets
+    signalled once test is finished. The signal should pass result object
+    containing following fields:
+    'outcome' (mandatory): outcome of a test, e.g. 'pass', 'fail', 'undecided'.
+    'suggestedOutcome': if outcome is 'undecided', than this suggestion will be
+    presented to the user, letting them decide the final outcome of a test.
+*/
+import QtQuick 2.0
+import Ubuntu.Components 1.1
+
+MainView {
+    id: mainView
+    width: units.gu(100)
+    height: units.gu(75)
+
+    // information and functionality passed to qml job component
+    property var testingShell: {
+        "name": "Standalone testing shell",
+        "pageStack": pageStack
+    }
+
+    Arguments {
+        id: args
+        Argument {
+            name: "job"
+            help: "QML-native job to run"
+            required: true
+            valueNames: ["PATH"]
+        }
+    }
+
+    Loader {
+        id: loader
+        anchors.fill: parent
+        onLoaded: loader.item.testDone.connect(testDone)
+    }
+    PageStack {
+        id: pageStack
+    }
+
+    function testDone(res) {
+        console.error("__test_result_object:"+JSON.stringify(res));
+        Qt.quit();
+    }
+
+    Component.onCompleted: {
+        loader.setSource(args.values.job, {'testingShell': testingShell});
+    }
+}

=== modified file 'plainbox/plainbox/test_provider_manager.py'
--- plainbox/plainbox/test_provider_manager.py	2014-09-18 10:59:17 +0000
+++ plainbox/plainbox/test_provider_manager.py	2015-01-29 23:33:32 +0000
@@ -322,7 +322,7 @@
             test_io.stdout, (
                 "jobs/broken.txt:1-2: job '2014.com.example::broken', field"
                 " 'plugin': allowed values are: attachment, local, manual,"
-                " resource, shell, user-interact, user-interact-verify,"
+                " qml, resource, shell, user-interact, user-interact-verify,"
                 " user-verify\n"))
 
     def test_validate__broken_useless_field(self):

=== modified file 'plainbox/setup.py'
--- plainbox/setup.py	2015-01-29 01:55:49 +0000
+++ plainbox/setup.py	2015-01-29 23:33:32 +0000
@@ -81,6 +81,7 @@
             'stubbox=plainbox.impl.box:stubbox_main',
             ('plainbox-trusted-launcher-1='
              'plainbox.impl.secure.launcher1:main'),
+            'plainbox-qml-shell=plainbox.qml_shell.qml_shell:main',
         ],
         'plainbox.exporter': [
             'text=plainbox.impl.exporter.text:TextSessionStateExporter',

=== added directory 'providers/2015.com.canonical.certification:qml-tests'
=== added file 'providers/2015.com.canonical.certification:qml-tests/.bzrignore'
--- providers/2015.com.canonical.certification:qml-tests/.bzrignore	1970-01-01 00:00:00 +0000
+++ providers/2015.com.canonical.certification:qml-tests/.bzrignore	2015-01-29 23:33:32 +0000
@@ -0,0 +1,2 @@
+dist/*.tar.gz
+build/mo/*
\ No newline at end of file

=== added file 'providers/2015.com.canonical.certification:qml-tests/.gitignore'
--- providers/2015.com.canonical.certification:qml-tests/.gitignore	1970-01-01 00:00:00 +0000
+++ providers/2015.com.canonical.certification:qml-tests/.gitignore	2015-01-29 23:33:32 +0000
@@ -0,0 +1,2 @@
+dist/*.tar.gz
+build/mo/*
\ No newline at end of file

=== added file 'providers/2015.com.canonical.certification:qml-tests/README.md'
--- providers/2015.com.canonical.certification:qml-tests/README.md	1970-01-01 00:00:00 +0000
+++ providers/2015.com.canonical.certification:qml-tests/README.md	2015-01-29 23:33:32 +0000
@@ -0,0 +1,5 @@
+Qml Native Tests
+================
+
+This provider cointains jobs defined using qml.
+Its purpose is to demonstrate possible QML usage in system tests.

=== added directory 'providers/2015.com.canonical.certification:qml-tests/data'
=== added file 'providers/2015.com.canonical.certification:qml-tests/data/camera-test.qml'
--- providers/2015.com.canonical.certification:qml-tests/data/camera-test.qml	1970-01-01 00:00:00 +0000
+++ providers/2015.com.canonical.certification:qml-tests/data/camera-test.qml	2015-01-29 23:33:32 +0000
@@ -0,0 +1,101 @@
+/*
+ * This file is part of Checkbox.
+ *
+ * Copyright 2015 Canonical Ltd.
+ * Written by:
+ *   Maciej Kisielewski <[e-mail]>
+ *
+ * Checkbox is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 3,
+ * as published by the Free Software Foundation.
+ *
+ * Checkbox is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with Checkbox.  If not, see <http://www.gnu.org/licenses/>.
+ */
+import QtQuick 2.0
+import Ubuntu.Components 1.1
+import QtMultimedia 5.0
+import QtQuick.Layouts 1.1
+Item {
+    id: root
+    signal testDone(var test)
+    property var testingShell
+
+    Component.onCompleted: testingShell.pageStack.push(introPage)
+
+    Page {
+        id: introPage
+        title: i18n.tr("Camera test")
+        visible: false
+
+        ColumnLayout {
+            spacing: units.gu(5)
+            anchors.fill: parent
+            anchors.margins: units.gu(3)
+            Label {
+                fontSize: "large"
+                Layout.fillWidth: true
+                wrapMode: Text.WrapAtWordBoundaryOrAnywhere
+                text: i18n.tr("On the next screen you'll see video feed from \
+your camera. Decide whether camera is working correctly, and tap corresponding \
+button.")
+                font.bold: true
+            }
+            Button {
+                text: i18n.tr("Start the test")
+                Layout.fillWidth: true
+                onClicked: {
+                    testingShell.pageStack.push(subpage);
+                }
+            }
+        }
+    }
+
+    Page {
+        id: subpage
+        visible: false
+        title: i18n.tr("Camera test")
+        VideoOutput {
+            id: viewfinder
+            source: cam
+            anchors.fill: parent
+            orientation: 270
+        }
+
+        Camera {
+            id: cam
+        }
+
+        ColumnLayout {
+            spacing: units.gu(1)
+            anchors {
+                fill: parent
+            }
+            Button {
+                width: units.gu(10)
+                height: units.gu(5)
+                Layout.alignment: Qt.AlignHCenter
+                text: i18n.tr("Camera works")
+                color: UbuntuColors.green
+                onClicked: {
+                    testDone({'outcome': 'pass'});
+                }
+            }
+            Button {
+                width: units.gu(10)
+                height: units.gu(5)
+                Layout.alignment: Qt.AlignHCenter
+                text: i18n.tr("Camera doesn't work")
+                color: UbuntuColors.red
+                onClicked: {
+                    testDone({'outcome': 'fail'});
+                }
+            }
+        }
+    }
+}

=== added file 'providers/2015.com.canonical.certification:qml-tests/manage.py'
--- providers/2015.com.canonical.certification:qml-tests/manage.py	1970-01-01 00:00:00 +0000
+++ providers/2015.com.canonical.certification:qml-tests/manage.py	2015-01-29 23:33:32 +0000
@@ -0,0 +1,21 @@
+#!/usr/bin/env python3
+from plainbox.provider_manager import setup, N_
+
+# You can inject other stuff here but please don't go overboard.
+#
+# In particular, if you need comprehensive compilation support to get
+# your bin/ populated then please try to discuss that with us in the
+# upstream project IRC channel #checkbox on irc.freenode.net.
+
+# NOTE: one thing that you could do here, that makes a lot of sense,
+# is to compute version somehow. This may vary depending on the
+# context of your provider. Future version of PlainBox will offer git,
+# bzr and mercurial integration using the versiontools library
+# (optional)
+
+setup(
+    name='2015.com.canonical.certification:qml-tests',
+    version="1.0",
+    description=N_("The 2015.com.canonical.certification:qml-tests provider"),
+    gettext_domain="2015_com_canonical_certification_qml-tests",
+)
\ No newline at end of file

=== added directory 'providers/2015.com.canonical.certification:qml-tests/units'
=== added file 'providers/2015.com.canonical.certification:qml-tests/units/qml-tests.pxu'
--- providers/2015.com.canonical.certification:qml-tests/units/qml-tests.pxu	1970-01-01 00:00:00 +0000
+++ providers/2015.com.canonical.certification:qml-tests/units/qml-tests.pxu	2015-01-29 23:33:32 +0000
@@ -0,0 +1,14 @@
+
+unit: test plan
+_description:
+ Test plan containing tests in form of QML programs
+id: qml-tests
+_name: QML-native tests
+estimated_duration: 20
+include: .*
+
+id: qml-camera
+plugin: qml
+_description: Camera test
+qml_file: camera-test.qml
+estimated_duration: 20

=== modified file 'support/develop-providers'
--- support/develop-providers	2014-06-18 15:45:05 +0000
+++ support/develop-providers	2015-01-29 23:33:32 +0000
@@ -34,7 +34,7 @@
     exit 101
 fi
 
-for provider in $CHECKBOX_TOP/providers/plainbox-provider-*; do
+for provider in $CHECKBOX_TOP/providers/plainbox-provider-* $CHECKBOX_TOP/providers/2015.com.canonical.certification:qml-tests; do
     provider=$(basename "$provider")
     echo "I: running 'develop' on $provider"
     ( cd $CHECKBOX_TOP/providers/$provider && python3 manage.py develop --force --directory=$PROVIDERPATH )