Merge lp:~kissiel/checkbox/autopilot-testing into lp:checkbox

[Maciej Kisielewski]

This MR lays foundations for autopilot testing for checkbox touch.
Major changes/additions are:

* Addition of Utopic as a target in test-in-lxc
* Addition of per-release requirements provisioning
* Move from main.qml to checkbox-touch.qml as the 'entry point' to CBT
* Add option to run CBT in autopilot-y way

b660802 checkbox-touch: add basic autopilot test
fe8d162 checkbox-touch: add autopilot tests to container-tests
0cff0e5 test-in-lxc: make test-in-lxc run tests on utopic
0e2fb89 checkbox-touch: make autopilot tests fail if autopilot is not installed
15d5bec checkbox-touch: make `make autopilot` run ./get-libs first
1dba042 support: make install-*-dependencies support release-specific packages
86d9f2a checkbox-touch: use absolute paths for autopilot provider
90e5d25 checkbox-touch: make autopilot run CBT with '--autopilot option'
e0f9b84 checkbox-touch: remove old autopilot tests
1e53580 checkbox-touch: make checkbox-touch.qml entry point to the app
bfba415 checkbox-touch: add providersDir to generated settings.json
946e613 checkbox-touch: add --autopilot option to main.qml
37df775 checkbox-touch: make APP_DIR fall back to '.'
b113a33 checkbox-touch: be explicit about where to load providers from

[Sylvain Pineau (sylvain-pineau)]

Question: The test in lxc containers is what we're running when MR are proposed and this script is only called by the tarmac lander, right? I'm trying to make the difference between the test actually run when building a ppa in a build environment (ppa) from tests run with tarmac in containers.
I ask because I see that you're using lsb_release in a couple of places but IIRC such tool is not available in a build env.

And finally rev 3594, you're mentioning an autopilot provider, do we have such thing in the known providers?

review: Needs Information

[Maciej Kisielewski (kissiel)]

> Question: The test in lxc containers is what we're running when MR are
> proposed and this script is only called by the tarmac lander, right?
AFAIK yes. tarmac-verify runs test-in-lxc if TEST_RUNNER is not defined or set to lxc.

> I'm
> trying to make the difference between the test actually run when building a
> ppa in a build environment (ppa) from tests run with tarmac in containers.
> I ask because I see that you're using lsb_release in a couple of places but
> IIRC such tool is not available in a build env.
>
> And finally rev 3594, you're mentioning an autopilot provider, do we have such
> thing in the known providers?

Yeah, even empty autopilot provider is required. I just added it and repushed the branch. Sorry for missing it in original MR.

[Sylvain Pineau (sylvain-pineau)]

ok, thank for the quick fix.

I've tested it in a VM and it works well (Results here: http://pastebin.ubuntu.com/10579771/)

+1

review: Approve

[Daniel Manrique (roadmr)]

The attempt to merge lp:~kissiel/checkbox/autopilot-testing into lp:checkbox failed. Below is the output from the failed tests.

[precise] starting container
[precise] (timing) 0.07user 0.02system 0:04.25elapsed 2%CPU (0avgtext+0avgdata 7852maxresident)k
[precise] (timing) 0inputs+32outputs (0major+7006minor)pagefaults 0swaps
[precise] provisioning container
[precise] (timing) 39.58user 12.69system 1:31.80elapsed 56%CPU (0avgtext+0avgdata 51820maxresident)k
[precise] (timing) 0inputs+16424outputs (0major+4479699minor)pagefaults 0swaps
[precise-testing] Starting tests...
Found a test script: ./checkbox-gui/requirements/container-tests-checkbox-gui-build
[precise-testing] container-tests-checkbox-gui-build: PASS
[precise-testing] (timing) 29.34user 2.18system 0:31.93elapsed 98%CPU (0avgtext+0avgdata 116196maxresident)k
[precise-testing] (timing) 0inputs+4240outputs (0major+478802minor)pagefaults 0swaps
Found a test script: ./checkbox-ng/requirements/container-tests-checkbox-ng-unit
[precise-testing] container-tests-checkbox-ng-unit: PASS
[precise-testing] (timing) 0.67user 0.13system 0:00.83elapsed 96%CPU (0avgtext+0avgdata 40976maxresident)k
[precise-testing] (timing) 0inputs+3880outputs (0major+21245minor)pagefaults 0swaps
Found a test script: ./checkbox-support/requirements/container-tests-checkbox-support
[precise-testing] container-tests-checkbox-support: PASS
[precise-testing] (timing) 15.51user 0.18system 0:15.81elapsed 99%CPU (0avgtext+0avgdata 83228maxresident)k
[precise-testing] (timing) 0inputs+1032outputs (0major+31565minor)pagefaults 0swaps
Found a test script: ./checkbox-touch/requirements/container-tests-touch-autopilot
[precise-testing] container-tests-touch-autopilot: PASS
[precise-testing] (timing) 0.02user 0.01system 0:00.05elapsed 67%CPU (0avgtext+0avgdata 4696maxresident)k
[precise-testing] (timing) 0inputs+8outputs (0major+3881minor)pagefaults 0swaps
Found a test script: ./checkbox-touch/requirements/container-tests-touch-unit-tests
[precise-testing] container-tests-touch-unit-tests: PASS
[precise-testing] (timing) 0.00user 0.00system 0:00.02elapsed 28%CPU (0avgtext+0avgdata 2044maxresident)k
[precise-testing] (timing) 0inputs+8outputs (0major+2395minor)pagefaults 0swaps
Found a test script: ./plainbox/plainbox/impl/providers/categories/requirements/container-tests-provider-categories
[precise-testing] container-tests-provider-categories: PASS
[precise-testing] (timing) 0.22user 0.03system 0:00.28elapsed 93%CPU (0avgtext+0avgdata 15356maxresident)k
[precise-testing] (timing) 0inputs+176outputs (0major+6422minor)pagefaults 0swaps
Found a test script: ./plainbox/requirements/001-container-tests-plainbox-egg-info
[precise-testing] 001-container-tests-plainbox-egg-info: PASS
[precise-testing] (timing) 0.14user 0.04system 0:00.20elapsed 91%CPU (0avgtext+0avgdata 10544maxresident)k
[precise-testing] (timing) 0inputs+88outputs (0major+5043minor)pagefaults 0swaps
Found a test script: ./plainbox/requirements/container-tests-plainbox
[precise-testing] container-tests-plainbox: PASS
[precise-testing] (timing) 12.66user 0.82...

[Maciej Kisielewski (kissiel)]

Added 'thumbnailer-service' to utopic-deb-touch requirements.
Self aproving to retry the merge.

[Daniel Manrique (roadmr)]

The attempt to merge lp:~kissiel/checkbox/autopilot-testing into lp:checkbox failed. Below is the output from the failed tests.

[precise] starting container
[precise] (timing) 0.06user 0.03system 0:05.27elapsed 1%CPU (0avgtext+0avgdata 7852maxresident)k
[precise] (timing) 0inputs+32outputs (0major+7502minor)pagefaults 0swaps
[precise] provisioning container
[precise] (timing) 39.96user 12.85system 1:31.60elapsed 57%CPU (0avgtext+0avgdata 51824maxresident)k
[precise] (timing) 0inputs+16424outputs (0major+4488457minor)pagefaults 0swaps
[precise-testing] Starting tests...
Found a test script: ./checkbox-gui/requirements/container-tests-checkbox-gui-build
[precise-testing] container-tests-checkbox-gui-build: PASS
[precise-testing] (timing) 27.69user 2.23system 0:30.31elapsed 98%CPU (0avgtext+0avgdata 116196maxresident)k
[precise-testing] (timing) 0inputs+4240outputs (0major+478807minor)pagefaults 0swaps
Found a test script: ./checkbox-ng/requirements/container-tests-checkbox-ng-unit
[precise-testing] container-tests-checkbox-ng-unit: PASS
[precise-testing] (timing) 0.68user 0.12system 0:00.84elapsed 96%CPU (0avgtext+0avgdata 40976maxresident)k
[precise-testing] (timing) 0inputs+3880outputs (0major+21258minor)pagefaults 0swaps
Found a test script: ./checkbox-support/requirements/container-tests-checkbox-support
[precise-testing] container-tests-checkbox-support: PASS
[precise-testing] (timing) 17.37user 0.10system 0:18.26elapsed 95%CPU (0avgtext+0avgdata 83096maxresident)k
[precise-testing] (timing) 0inputs+1032outputs (0major+31454minor)pagefaults 0swaps
Found a test script: ./checkbox-touch/requirements/container-tests-touch-autopilot
[precise-testing] container-tests-touch-autopilot: PASS
[precise-testing] (timing) 0.01user 0.02system 0:00.05elapsed 69%CPU (0avgtext+0avgdata 4696maxresident)k
[precise-testing] (timing) 0inputs+8outputs (0major+3875minor)pagefaults 0swaps
Found a test script: ./checkbox-touch/requirements/container-tests-touch-unit-tests
[precise-testing] container-tests-touch-unit-tests: PASS
[precise-testing] (timing) 0.00user 0.00system 0:00.02elapsed 42%CPU (0avgtext+0avgdata 2044maxresident)k
[precise-testing] (timing) 0inputs+8outputs (0major+2390minor)pagefaults 0swaps
Found a test script: ./plainbox/plainbox/impl/providers/categories/requirements/container-tests-provider-categories
[precise-testing] container-tests-provider-categories: PASS
[precise-testing] (timing) 0.20user 0.05system 0:00.28elapsed 92%CPU (0avgtext+0avgdata 15356maxresident)k
[precise-testing] (timing) 0inputs+176outputs (0major+6420minor)pagefaults 0swaps
Found a test script: ./plainbox/requirements/001-container-tests-plainbox-egg-info
[precise-testing] 001-container-tests-plainbox-egg-info: PASS
[precise-testing] (timing) 0.15user 0.06system 0:00.23elapsed 92%CPU (0avgtext+0avgdata 10548maxresident)k
[precise-testing] (timing) 0inputs+88outputs (0major+5051minor)pagefaults 0swaps
Found a test script: ./plainbox/requirements/container-tests-plainbox
[precise-testing] container-tests-plainbox: PASS
[precise-testing] (timing) 12.78user 0.85...

[Maciej Kisielewski (kissiel)]

Disabled GLX for autopilot tests.

[Daniel Manrique (roadmr)]

The attempt to merge lp:~kissiel/checkbox/autopilot-testing into lp:checkbox failed. Below is the output from the failed tests.

[precise] starting container
[precise] (timing) 0.05user 0.03system 0:05.22elapsed 1%CPU (0avgtext+0avgdata 7860maxresident)k
[precise] (timing) 0inputs+32outputs (0major+7495minor)pagefaults 0swaps
[precise] provisioning container
[precise] (timing) 39.59user 11.16system 1:37.06elapsed 52%CPU (0avgtext+0avgdata 51820maxresident)k
[precise] (timing) 0inputs+16424outputs (0major+4503355minor)pagefaults 0swaps
[precise-testing] Starting tests...
Found a test script: ./checkbox-gui/requirements/container-tests-checkbox-gui-build
[precise-testing] container-tests-checkbox-gui-build: PASS
[precise-testing] (timing) 29.82user 2.02system 0:32.24elapsed 98%CPU (0avgtext+0avgdata 116196maxresident)k
[precise-testing] (timing) 0inputs+4240outputs (0major+478818minor)pagefaults 0swaps
Found a test script: ./checkbox-ng/requirements/container-tests-checkbox-ng-unit
[precise-testing] container-tests-checkbox-ng-unit: PASS
[precise-testing] (timing) 0.56user 0.13system 0:00.95elapsed 73%CPU (0avgtext+0avgdata 40976maxresident)k
[precise-testing] (timing) 0inputs+3880outputs (0major+21271minor)pagefaults 0swaps
Found a test script: ./checkbox-support/requirements/container-tests-checkbox-support
[precise-testing] container-tests-checkbox-support: PASS
[precise-testing] (timing) 15.48user 0.12system 0:16.07elapsed 97%CPU (0avgtext+0avgdata 83112maxresident)k
[precise-testing] (timing) 0inputs+1032outputs (0major+31565minor)pagefaults 0swaps
Found a test script: ./checkbox-touch/requirements/container-tests-touch-autopilot
[precise-testing] container-tests-touch-autopilot: PASS
[precise-testing] (timing) 0.02user 0.01system 0:00.05elapsed 66%CPU (0avgtext+0avgdata 4692maxresident)k
[precise-testing] (timing) 0inputs+8outputs (0major+3882minor)pagefaults 0swaps
Found a test script: ./checkbox-touch/requirements/container-tests-touch-unit-tests
[precise-testing] container-tests-touch-unit-tests: PASS
[precise-testing] (timing) 0.00user 0.00system 0:00.03elapsed 35%CPU (0avgtext+0avgdata 2044maxresident)k
[precise-testing] (timing) 0inputs+8outputs (0major+2388minor)pagefaults 0swaps
Found a test script: ./plainbox/plainbox/impl/providers/categories/requirements/container-tests-provider-categories
[precise-testing] container-tests-provider-categories: PASS
[precise-testing] (timing) 0.18user 0.06system 0:00.27elapsed 93%CPU (0avgtext+0avgdata 15360maxresident)k
[precise-testing] (timing) 0inputs+176outputs (0major+6416minor)pagefaults 0swaps
Found a test script: ./plainbox/requirements/001-container-tests-plainbox-egg-info
[precise-testing] 001-container-tests-plainbox-egg-info: PASS
[precise-testing] (timing) 0.14user 0.04system 0:00.19elapsed 90%CPU (0avgtext+0avgdata 10548maxresident)k
[precise-testing] (timing) 0inputs+88outputs (0major+5048minor)pagefaults 0swaps
Found a test script: ./plainbox/requirements/container-tests-plainbox
[precise-testing] container-tests-plainbox: PASS
[precise-testing] (timing) 12.13user 0.78...

[Zygmunt Krynicki (zyga)]

Please rebase, also look at two inline comments below.

[lp:~kissiel/checkbox/autopilot-testing]

3624. By Maciej Kisielewski on 2015-03-20

checkbox-touch: be explicit about where to load providers from

This patch makes all chain of calls responsible for loading providers in
Checkbox-Touch explicit about the path from which to load providers from.
This paves the way for autopilot provider to be loaded on the side of normal
providers.

Signed-off-by: Maciej Kisielewski <email address hidden>

3625. By Maciej Kisielewski on 2015-03-20

checkbox-touch: make APP_DIR fall back to '.'

This patch makes Checkbox-Touch expand APP_DIR envvar to '.' when it's absent.
This makes loading of local providers on desktop work as expected.
Signed-off-by: Maciej Kisielewski <email address hidden>

3626. By Maciej Kisielewski on 2015-03-20

checkbox-touch: add --autopilot option to main.qml

This patch adds an option to run Checkbox-Touch in autopilot mode, that changes
path from which providers are loaded and preselects autopilot testplan.

Signed-off-by: Maciej Kisielewski <email address hidden>

3627. By Maciej Kisielewski on 2015-03-20

checkbox-touch: add providersDir to generated settings.json

Signed-off-by: Maciej Kisielewski <email address hidden>

3628. By Maciej Kisielewski on 2015-03-20

checkbox-touch: remove checkbox-touch.qml symlink

Signed-off-by: Maciej Kisielewski <email address hidden>

3629. By Maciej Kisielewski on 2015-03-20

checkbox-touch: rename main.qml to checkbox-touch.qml

Signed-off-by: Maciej Kisielewski <email address hidden>

3630. By Maciej Kisielewski on 2015-03-20

checkbox-touch: make checkbox-touch.qml entry point to the app

This patch changes main file to be checkbox-touch.qml instead of main.
Except the obvious improvement, which is clarity, it makes it compliant with
the default behaviour autopilot takes when trying to launch app.

Signed-off-by: Maciej Kisielewski <email address hidden>

3631. By Maciej Kisielewski on 2015-03-20

checkbox-touch: remove old autopilot tests

Signed-off-by: Maciej Kisielewski <email address hidden>

3632. By Maciej Kisielewski on 2015-03-20

checkbox-touch: make autopilot run CBT with '--autopilot option'

This patch makes autopilot tests run Checkbox-Touch with '--autopilot' option
by default.

Signed-off-by: Maciej Kisielewski <email address hidden>

3633. By Maciej Kisielewski on 2015-03-20

checkbox-touch: use absolute paths for autopilot provider

This patch makes Checkbox-Touch use correct absolute path when building path to
autopilot provider files.

Signed-off-by: Maciej Kisielewski <email address hidden>

3634. By Maciej Kisielewski on 2015-03-20

checkbox-touch: make `make autopilot` run ./get-libs first

Signed-off-by: Maciej Kisielewski <email address hidden>

3635. By Maciej Kisielewski on 2015-03-20

checkbox-touch: make autopilot tests fail if autopilot is not installed

This patch fixes a false positive feedback when runnning autopilot without
having it installed.

Signed-off-by: Maciej Kisielewski <email address hidden>

3636. By Maciej Kisielewski on 2015-03-20

checkbox-touch: add autopilot provider

This patch adds empty provider that is used when running autopilot tests. All
future autopilot scenarios should base on jobs defined in this provider.

Signed-off-by: Maciej Kisielewski <email address hidden>

3637. By Maciej Kisielewski on 2015-03-20

checkbox-touch: add basic autopilot test

This patch adds an autopilot test that checks if app can be launched.

Signed-off-by: Maciej Kisielewski <email address hidden>

[Zygmunt Krynicki (zyga)]

+1, thanks, looking forward towards resolving infrastructure issues to see all your work in practice!

review: Approve

[Maciej Kisielewski (kissiel)]

self-approving to push this to merging

review: Approve

Unmerged revisions

3637. By Maciej Kisielewski on 2015-03-20

checkbox-touch: add basic autopilot test

This patch adds an autopilot test that checks if app can be launched.

Signed-off-by: Maciej Kisielewski <email address hidden>

3636. By Maciej Kisielewski on 2015-03-20

checkbox-touch: add autopilot provider

This patch adds empty provider that is used when running autopilot tests. All
future autopilot scenarios should base on jobs defined in this provider.

Signed-off-by: Maciej Kisielewski <email address hidden>

3635. By Maciej Kisielewski on 2015-03-20

checkbox-touch: make autopilot tests fail if autopilot is not installed

This patch fixes a false positive feedback when runnning autopilot without
having it installed.

Signed-off-by: Maciej Kisielewski <email address hidden>

3634. By Maciej Kisielewski on 2015-03-20

checkbox-touch: make `make autopilot` run ./get-libs first

Signed-off-by: Maciej Kisielewski <email address hidden>

3633. By Maciej Kisielewski on 2015-03-20

checkbox-touch: use absolute paths for autopilot provider

This patch makes Checkbox-Touch use correct absolute path when building path to
autopilot provider files.

Signed-off-by: Maciej Kisielewski <email address hidden>

3632. By Maciej Kisielewski on 2015-03-20

checkbox-touch: make autopilot run CBT with '--autopilot option'

This patch makes autopilot tests run Checkbox-Touch with '--autopilot' option
by default.

Signed-off-by: Maciej Kisielewski <email address hidden>

3631. By Maciej Kisielewski on 2015-03-20

checkbox-touch: remove old autopilot tests

Signed-off-by: Maciej Kisielewski <email address hidden>

3630. By Maciej Kisielewski on 2015-03-20

checkbox-touch: make checkbox-touch.qml entry point to the app

This patch changes main file to be checkbox-touch.qml instead of main.
Except the obvious improvement, which is clarity, it makes it compliant with
the default behaviour autopilot takes when trying to launch app.

Signed-off-by: Maciej Kisielewski <email address hidden>

3629. By Maciej Kisielewski on 2015-03-20

checkbox-touch: rename main.qml to checkbox-touch.qml

Signed-off-by: Maciej Kisielewski <email address hidden>

3628. By Maciej Kisielewski on 2015-03-20

checkbox-touch: remove checkbox-touch.qml symlink

Signed-off-by: Maciej Kisielewski <email address hidden>

=== modified file 'checkbox-touch/Makefile'
--- checkbox-touch/Makefile	2014-08-11 14:20:37 +0000
+++ checkbox-touch/Makefile	2015-03-20 13:15:31 +0000
@@ -11,6 +11,7 @@
 all:
 
 autopilot:
+	./get-libs
 	chmod +x tests/autopilot/run
 	tests/autopilot/run
 
@@ -18,5 +19,5 @@
 	qmltestrunner -input tests/unit
 
 run:
-	/usr/bin/qmlscene $@ main.qml
+	/usr/bin/qmlscene $@ checkbox-touch.qml
 

=== modified file 'checkbox-touch/README.rst'
--- checkbox-touch/README.rst	2014-12-03 15:05:34 +0000
+++ checkbox-touch/README.rst	2015-03-20 13:15:31 +0000
@@ -56,7 +56,7 @@
 Running Checkbox-Touch on a desktop
 -----------------------------------
 
-To run on a desktop run `qmlscene main.qml`
+To run on a desktop run `qmlscene checkbox-touch.qml`
 Note: Make sure you've ran `./get-libs` first.
 
 

=== modified file 'checkbox-touch/build-me'
--- checkbox-touch/build-me	2015-03-13 14:21:40 +0000
+++ checkbox-touch/build-me	2015-03-20 13:15:31 +0000
@@ -80,7 +80,8 @@
         "_comment": "file generated automatically with {0}"
                     .format(parser.prog),
         "revision": get_revision_string(),
-        "testplan": args.testplan
+        "testplan": args.testplan,
+        "providersDir": "providers"
     }
     settings_file = open('settings.json', 'w')
     settings_file.write(json.dumps(settings, sort_keys=True, indent=4))
@@ -115,7 +116,7 @@
     [Desktop Entry]
     Name=Checkbox
     Comment=System testing utility for Ubuntu
-    Exec=qmlscene {import_options} $@ main.qml
+    Exec=qmlscene {import_options} $@ checkbox-touch.qml
     Icon=checkbox-touch.svg
     Terminal=false
     Type=Application

=== removed symlink 'checkbox-touch/checkbox-touch.qml'
=== target was u'preview.qml'
=== renamed file 'checkbox-touch/main.qml' => 'checkbox-touch/checkbox-touch.qml'
--- checkbox-touch/main.qml	2015-02-09 14:19:48 +0000
+++ checkbox-touch/checkbox-touch.qml	2015-03-20 13:15:31 +0000
@@ -57,24 +57,41 @@
     property var appSettings: {
         "applicationName" : applicationName,
         "revision": "unknown revision",
-        "testplan": ""
+        "testplan": "",
+        "providersDir": "providers"
+    }
+
+    Arguments {
+        id: args
+        Argument {
+            name: "autopilot"
+            help: i18n.tr("Run Checkbox-Touch in autopilot-testing mode")
+            required: false
+        }
     }
 
     Component.onCompleted: {
-        var xhr = new XMLHttpRequest;
-        xhr.open("GET", "settings.json");
-        xhr.onreadystatechange = function() {
-            if (xhr.readyState == XMLHttpRequest.DONE) {
-                try {
-                    appSettings = JSON.parse(xhr.responseText);
-                } catch (x) {
-                    // if we cannot parse settings.json, we should leave
-                    // deafult values of appSettings
-                    console.log("Could not parse settings.json. Using default values")
+        if (args.values["autopilot"]) {
+            // autopilot-testing mode
+            appSettings["testplan"] = "2015.com.canonical.certification::checkbox-touch-autopilot"
+            appSettings["providersDir"] = "tests/autopilot/autopilot-provider"
+        } else {
+            // normal execution - load settings.json file
+            var xhr = new XMLHttpRequest;
+            xhr.open("GET", "settings.json");
+            xhr.onreadystatechange = function() {
+                if (xhr.readyState == XMLHttpRequest.DONE) {
+                    try {
+                        appSettings = JSON.parse(xhr.responseText);
+                    } catch (x) {
+                        // if we cannot parse settings.json, we should leave
+                        // deafult values of appSettings
+                        console.log("Could not parse settings.json. Using default values")
+                    }
                 }
             }
+            xhr.send();
         }
-        xhr.send();
     }
 
 
@@ -115,7 +132,7 @@
                 "checkbox_touch" : applicationVersion,
                 "plainbox" : plainboxVersion
             };
-            resumeOrStartSession();
+            resumeOrStartSession(appSettings["providersDir"]);
         }
         onSessionReady: {
             welcomePage.enableButton()
@@ -223,7 +240,7 @@
         onRestartSession: {
             pageStack.clear();
             pageStack.push(welcomePage);
-            app.startSession();
+            app.startSession(appSettings["providersDir"]);
         }
     }
 
@@ -310,10 +327,10 @@
             } else {
                 if (result.errors_encountered) {
                     ErrorLogic.showError(mainView, i18n.tr("Could not resume session."),
-                                         app.startSession(),
+                                         app.startSession(appSettings["providersDir"]),
                                          i18n.tr("Start new session"));
                 } else {
-                    app.startSession();
+                    app.startSession(appSettings["providersDir"]);
                 }
             }
         });
@@ -363,7 +380,7 @@
             resultsPage.endTesting.connect(function() {
                 pageStack.clear();
                 app.clearSession(function() {
-                    app.startSession();
+                    app.startSession(appSettings["providersDir"]);
                     pageStack.push(welcomePage);
                 });
             });

=== modified file 'checkbox-touch/checkbox-touch.qmlproject'
--- checkbox-touch/checkbox-touch.qmlproject	2014-08-12 18:58:43 +0000
+++ checkbox-touch/checkbox-touch.qmlproject	2015-03-20 13:15:31 +0000
@@ -1,7 +1,7 @@
 import QmlProject 1.1
 
 Project {
-    mainFile: "main.qml"
+    mainFile: "checkbox-touch.qml"
 
     /* Include .qml, .js, and image files from current directory and subdirectories */
     QmlFiles {

=== modified file 'checkbox-touch/components/CheckboxTouchApplication.qml'
--- checkbox-touch/components/CheckboxTouchApplication.qml	2015-03-13 14:21:45 +0000
+++ checkbox-touch/components/CheckboxTouchApplication.qml	2015-03-20 13:15:31 +0000
@@ -44,16 +44,16 @@
     // Starts session in plainbox and runs all necessary setup actions.
     // Calling this function will signal sessionReady() once it's finished
     // doing setup.
-    function startSession() {
-        request("start_session", [], function(result) {
+    function startSession(providersDir) {
+        request("start_session", [providersDir], function(result) {
             sessionDir = result['session_dir'];
             sessionReady();
         }, function(error) {
             console.error("Unable to start session: " + error);
         });
     }
-    function resumeSession(rerunLastTest, continuation) {
-        request("resume_session", [rerunLastTest], function(result) {
+    function resumeSession(rerunLastTest, providersDir, continuation) {
+        request("resume_session", [rerunLastTest, providersDir], function(result) {
             if (!result["session_id"]) {
                 pageStack.pop();
                 ErrorLogic.showError(mainView,

=== modified file 'checkbox-touch/py/checkbox_touch.py'
--- checkbox-touch/py/checkbox_touch.py	2015-03-13 14:21:45 +0000
+++ checkbox-touch/py/checkbox_touch.py	2015-03-20 13:15:31 +0000
@@ -351,7 +351,7 @@
         }
 
     @view
-    def start_session(self):
+    def start_session(self, providers_dir):
         if self.manager is not None:
             _logger.warning("start_session() should not be called twice!")
         else:
@@ -360,7 +360,7 @@
             self.manager.add_local_device_context()
             self.context = self.manager.default_device_context
             # Add some all providers into the context
-            for provider in self._get_default_providers():
+            for provider in self._get_default_providers(providers_dir):
                 self.context.add_provider(provider)
             # Fill in the meta-data
             self.context.state.metadata.app_id = 'checkbox-touch'
@@ -386,9 +386,10 @@
         }
 
     @view
-    def resume_session(self, rerun_last_test):
+    def resume_session(self, rerun_last_test, providers_dir):
         all_units = list(itertools.chain(
-            *[p.unit_list for p in self._get_default_providers()]))
+            *[p.unit_list for p in self._get_default_providers(
+                providers_dir)]))
         try:
             self.manager = SessionManager.load_session(
                 all_units, self.resume_candidate_storage)
@@ -772,12 +773,24 @@
         self.session_storage_repo = SessionStorageRepository(
             self._get_app_cache_directory())
 
-    def _get_default_providers(self):
+    def _get_default_providers(self, providers_dir):
+        """
+        Get providers
+
+        :param providers_dir:
+            Path within application tree from which to load providers
+        :returns:
+            list of loaded providers
+        """
         all_providers.load()
         provider_list = all_providers.get_all_plugin_objects()
         # when running on ubuntu-touch device, APP_DIR env var is present
         # and points to touch application top directory
-        path = os.path.join(os.path.expandvars('$APP_DIR'), 'providers')
+        app_root_dir = os.path.normpath(os.getenv(
+            'APP_DIR', os.path.join(os.path.dirname(__file__), '..')))
+        path = os.path.join(app_root_dir,
+                            os.path.normpath(providers_dir))
+        _logger.info("Loading all providers from %s", path)
         if os.path.exists(path):
             embedded_providers = EmbeddedProvider1PlugInCollection(path)
             provider_list += embedded_providers.get_all_plugin_objects()

=== added directory 'checkbox-touch/tests/autopilot/autopilot-provider'
=== added directory 'checkbox-touch/tests/autopilot/autopilot-provider/jobs'
=== added file 'checkbox-touch/tests/autopilot/autopilot-provider/jobs/autopilot.pxu'
--- checkbox-touch/tests/autopilot/autopilot-provider/jobs/autopilot.pxu	1970-01-01 00:00:00 +0000
+++ checkbox-touch/tests/autopilot/autopilot-provider/jobs/autopilot.pxu	2015-03-20 13:15:31 +0000
@@ -0,0 +1,7 @@
+unit: test plan
+_description:
+ Test plan that should be default one run by autopilot.
+id: checkbox-touch-autopilot
+_name: Checkbox-Touch autpoilot self-test
+estimated_duration: 60
+include: autopilot/.*

=== added file 'checkbox-touch/tests/autopilot/autopilot-provider/manage.py'
--- checkbox-touch/tests/autopilot/autopilot-provider/manage.py	1970-01-01 00:00:00 +0000
+++ checkbox-touch/tests/autopilot/autopilot-provider/manage.py	2015-03-20 13:15:31 +0000
@@ -0,0 +1,21 @@
+#!/usr/bin/env python3
+from plainbox.provider_manager import setup, N_
+
+# You can inject other stuff here but please don't go overboard.
+#
+# In particular, if you need comprehensive compilation support to get
+# your bin/ populated then please try to discuss that with us in the
+# upstream project IRC channel #checkbox on irc.freenode.net.
+
+# NOTE: one thing that you could do here, that makes a lot of sense,
+# is to compute version somehow. This may vary depending on the
+# context of your provider. Future version of PlainBox will offer git,
+# bzr and mercurial integration using the versiontools library
+# (optional)
+
+setup(
+    name='2015.com.canonical.certification:autopilot-touch',
+    version="1.0",
+    description=N_("The 2015.com.canonical.certification:autopilot-touch provider"),
+    gettext_domain="2015_com_canonical_certification_autopilot-touch",
+)
\ No newline at end of file

=== modified file 'checkbox-touch/tests/autopilot/checkbox_touch/__init__.py'
--- checkbox-touch/tests/autopilot/checkbox_touch/__init__.py	2014-08-11 12:36:57 +0000
+++ checkbox-touch/tests/autopilot/checkbox_touch/__init__.py	2015-03-20 13:15:31 +0000
@@ -47,6 +47,7 @@
                 base.get_qmlscene_launch_command(),
                 '-I' + _get_module_include_path(),
                 app_qml_source_location,
+                '--autopilot',
                 app_type='qt',
                 emulator_base=emulators.UbuntuUIToolkitEmulatorBase)
         else:

=== added file 'checkbox-touch/tests/autopilot/checkbox_touch/test_checkbox_touch.py'
--- checkbox-touch/tests/autopilot/checkbox_touch/test_checkbox_touch.py	1970-01-01 00:00:00 +0000
+++ checkbox-touch/tests/autopilot/checkbox_touch/test_checkbox_touch.py	2015-03-20 13:15:31 +0000
@@ -0,0 +1,12 @@
+# -*- Mode: Python; coding: utf-8; indent-tabs-mode: nil; tab-width: 4 -*-
+
+from testtools.matchers import Equals
+from autopilot.matchers import Eventually
+
+import checkbox_touch
+
+class TestCheckboxTouch(checkbox_touch.ClickAppTestCase):
+    """Test checking if app launches"""
+    def test_launches(self):
+        main_view = self.app.select_single(objectName='mainView')
+        self.assertThat(main_view.visible, Eventually(Equals(True)))

=== removed file 'checkbox-touch/tests/autopilot/checkbox_touch/test_latch_button.py'
--- checkbox-touch/tests/autopilot/checkbox_touch/test_latch_button.py	2014-09-11 08:09:35 +0000
+++ checkbox-touch/tests/autopilot/checkbox_touch/test_latch_button.py	1970-01-01 00:00:00 +0000
@@ -1,16 +0,0 @@
-# -*- Mode: Python; coding: utf-8; indent-tabs-mode: nil; tab-width: 4 -*-
-
-from testtools.matchers import Equals
-
-import checkbox_touch
-
-
-class LatchButtonTestCase(checkbox_touch.ClickAppTestCase):
-    """Tests for the LatchButton component"""
-    def test_latching(self):
-        latch_button = self.app.select_single(objectName="latchButton")
-        latch_button.swipe_into_view()
-        signal = latch_button.watch_signal("latchedClicked()")
-        self.pointing_device.click_object(latch_button)
-        self.pointing_device.click_object(latch_button)  # click again!
-        self.assertThat(signal.num_emissions, Equals(1))

=== removed file 'checkbox-touch/tests/autopilot/checkbox_touch/test_manual_intro_page.py'
--- checkbox-touch/tests/autopilot/checkbox_touch/test_manual_intro_page.py	2014-09-11 08:09:35 +0000
+++ checkbox-touch/tests/autopilot/checkbox_touch/test_manual_intro_page.py	1970-01-01 00:00:00 +0000
@@ -1,35 +0,0 @@
-# -*- Mode: Python; coding: utf-8; indent-tabs-mode: nil; tab-width: 4 -*-
-
-from testtools.matchers import Equals
-
-import checkbox_touch
-
-
-class ManualIntroPageTestCase(checkbox_touch.ClickAppTestCase):
-    """Tests for the Intro page to Manual test"""
-
-    def setUp(self):
-        super(ManualIntroPageTestCase, self).setUp()
-        # open ManualIntroPage by clicking on 'Manual test page' butotn
-        button = self.main_view.select_single(
-            objectName='manualIntroPageButton')
-        self.pointing_device.click_object(button)
-        self.page = self.app.select_single(objectName='manualIntroPage')
-
-    def test_default_content(self):
-        self.assertThat(self.page.title, Equals('Test Description'))
-
-    def test_continue_button(self):
-        continue_button = self.page.select_single(objectName='continueButton')
-        self.assertThat(continue_button.text, Equals('Continue'))
-        signal = self.page.watch_signal('continueClicked()')
-        self.pointing_device.click_object(continue_button)
-        self.assertThat(signal.was_emitted, Equals(True))
-
-    def test_goes_to_verification(self):
-        continue_button = self.page.select_single(objectName='continueButton')
-        self.pointing_device.click_object(continue_button)
-        next_page = self.app.select_single(objectName='testVerificationPage')
-        self.assertThat(next_page.title, Equals('Verification'))
-        page_stack = self.main_view.select_single(objectName='pageStack')
-        self.assertThat(page_stack.depth, Equals(3))

=== removed file 'checkbox-touch/tests/autopilot/checkbox_touch/test_resume_session_screen.py'
--- checkbox-touch/tests/autopilot/checkbox_touch/test_resume_session_screen.py	2014-09-11 08:09:35 +0000
+++ checkbox-touch/tests/autopilot/checkbox_touch/test_resume_session_screen.py	1970-01-01 00:00:00 +0000
@@ -1,52 +0,0 @@
-# -*- Mode: Python; coding: utf-8; indent-tabs-mode: nil; tab-width: 4 -*-
-
-"""Tests for the Hello World"""
-
-from testtools.matchers import Equals
-
-import checkbox_touch
-
-
-class ResumeSessionScreenTestCase(checkbox_touch.ClickAppTestCase):
-    """Tests for resume session screen"""
-
-    def setUp(self):
-        super(ResumeSessionScreenTestCase, self).setUp()
-        # click "resume session page"
-        button = self.main_view.select_single(
-            objectName='resumeSessionPageButton')
-        self.pointing_device.click_object(button)
-        self.page = self.app.select_single(objectName='resumeSessionPage')
-
-    def test_default_content(self):
-        self.assertThat(self.page.title, Equals('Resume session'))
-        button_texts = [('rerunButtonLabel', 'Rerun last'),
-                        ('continueButtonLabel', 'Continue'),
-                        ('restartButtonLabel', 'Restart')]
-        for obj_name, text in button_texts:
-            obj = self.main_view.select_single(objectName=obj_name)
-            self.assertThat(obj.text, Equals(text))
-
-    def test_signal_rerun_last(self):
-        signal_watcher = self.page.watch_signal('rerunLast()')
-        button = self.main_view.select_single(objectName='rerunButton')
-        self.pointing_device.click_object(button)
-        self.assertThat(signal_watcher.was_emitted, Equals(True))
-        page_stack = self.main_view.select_single(objectName='pageStack')
-        self.assertThat(page_stack.depth, Equals(1))
-
-    def test_signal_continue_session(self):
-        signal_watcher = self.page.watch_signal('continueSession()')
-        button = self.main_view.select_single(objectName='continueButton')
-        self.pointing_device.click_object(button)
-        self.assertThat(signal_watcher.was_emitted, Equals(True))
-        page_stack = self.main_view.select_single(objectName='pageStack')
-        self.assertThat(page_stack.depth, Equals(1))
-
-    def test_signal_restart_session(self):
-        signal_watcher = self.page.watch_signal('restartSession()')
-        button = self.main_view.select_single(objectName='restartButton')
-        self.pointing_device.click_object(button)
-        self.assertThat(signal_watcher.was_emitted, Equals(True))
-        page_stack = self.main_view.select_single(objectName='pageStack')
-        self.assertThat(page_stack.depth, Equals(1))

=== modified file 'checkbox-touch/tests/autopilot/run'
--- checkbox-touch/tests/autopilot/run	2014-08-11 12:36:57 +0000
+++ checkbox-touch/tests/autopilot/run	2015-03-20 13:15:31 +0000
@@ -2,11 +2,13 @@
 
 if [[ -z `which autopilot` ]]; then
   echo "Autopilot is not installed. Skip"
-  exit
+  exit 1
 fi
 
 SCRIPTPATH=`dirname $0`
 pushd ${SCRIPTPATH}
 autopilot run checkbox_touch
+ret=$?
 popd
+exit $ret