Merge lp:~canonical-platform-qa/ubuntu-clock-app/xvfb_and_qml_tests into lp:ubuntu-clock-app

[Leo Arias]

Copy the xvfb handling of QML tests from system apps.

[Ubuntu Phone Apps Jenkins Bot (ubuntu-phone-apps-jenkins-bot)]

FAILED: Continuous integration, rev:96
No commit message was specified in the merge proposal. Click on the following link and set the commit message (if you want a jenkins rebuild you need to trigger it yourself):
https://code.launchpad.net/~canonical-platform-qa/ubuntu-clock-app/xvfb_and_qml_tests/+merge/234421/+edit-commit-message

http://91.189.93.70:8080/job/ubuntu-clock-app-ci/457/
Executed test runs:
    SUCCESS: http://91.189.93.70:8080/job/generic-mediumtests-utopic-python3/303
        deb: http://91.189.93.70:8080/job/generic-mediumtests-utopic-python3/303/artifact/work/output/*zip*/output.zip
    SUCCESS: http://91.189.93.70:8080/job/ubuntu-clock-app-utopic-amd64-ci/104

Click here to trigger a rebuild:
http://91.189.93.70:8080/job/ubuntu-clock-app-ci/457/rebuild

review: Needs Fixing (continuous-integration)

[Ubuntu Phone Apps Jenkins Bot (ubuntu-phone-apps-jenkins-bot)]

PASSED: Continuous integration, rev:96
http://91.189.93.70:8080/job/ubuntu-clock-app-ci/460/
Executed test runs:
    SUCCESS: http://91.189.93.70:8080/job/generic-mediumtests-utopic-python3/306
        deb: http://91.189.93.70:8080/job/generic-mediumtests-utopic-python3/306/artifact/work/output/*zip*/output.zip
    SUCCESS: http://91.189.93.70:8080/job/ubuntu-clock-app-utopic-amd64-ci/107

Click here to trigger a rebuild:
http://91.189.93.70:8080/job/ubuntu-clock-app-ci/460/rebuild

review: Approve (continuous-integration)

[Nekhelesh Ramananthan (nik90)]

The test fails on my system as well. It is a known failure. I will fix asap. However jenkins at the moment is only configured to run AP tests. You will need to talk to fginther about enabling jenkins to run the qml tests as well.

[Nekhelesh Ramananthan (nik90)]

67 -if(NOT CLICK_MODE)
68 - add_subdirectory(tests)
69 -endif(NOT CLICK_MODE)

I believe this was added in the first place to ensure that the tests folder does not make its way into the clock app click package since the click package doesn't need to also include tests. I see that you added,

102 +add_subdirectory(unit)
103 +
104 +if(NOT CLICK_MODE)
105 + add_subdirectory(autopilot)
106 +endif(NOT CLICK_MODE)

But doesn't that mean that the units directory is still included in the click package?

review: Needs Information

[Leo Arias (elopio)]

I built the package with click-buddy, and then:

click contents com.ubuntu.clock_3.1.96_amd64.click | grep tst

There doesn't seem to be tst files added to the package. Do you know who added that if originally, to ask him?

[Ubuntu Phone Apps Jenkins Bot (ubuntu-phone-apps-jenkins-bot)]

FAILED: Continuous integration, rev:98
No commit message was specified in the merge proposal. Click on the following link and set the commit message (if you want a jenkins rebuild you need to trigger it yourself):
https://code.launchpad.net/~canonical-platform-qa/ubuntu-clock-app/xvfb_and_qml_tests/+merge/234529/+edit-commit-message

http://91.189.93.70:8080/job/ubuntu-clock-app-ci/464/
Executed test runs:
    SUCCESS: http://91.189.93.70:8080/job/generic-mediumtests-utopic-python3/310
        deb: http://91.189.93.70:8080/job/generic-mediumtests-utopic-python3/310/artifact/work/output/*zip*/output.zip
    SUCCESS: http://91.189.93.70:8080/job/ubuntu-clock-app-utopic-amd64-ci/111

Click here to trigger a rebuild:
http://91.189.93.70:8080/job/ubuntu-clock-app-ci/464/rebuild

review: Needs Fixing (continuous-integration)

[Ubuntu Phone Apps Jenkins Bot (ubuntu-phone-apps-jenkins-bot)]

FAILED: Continuous integration, rev:100
No commit message was specified in the merge proposal. Click on the following link and set the commit message (if you want a jenkins rebuild you need to trigger it yourself):
https://code.launchpad.net/~canonical-platform-qa/ubuntu-clock-app/xvfb_and_qml_tests/+merge/234529/+edit-commit-message

http://91.189.93.70:8080/job/ubuntu-clock-app-ci/466/
Executed test runs:
    SUCCESS: http://91.189.93.70:8080/job/generic-mediumtests-utopic-python3/313
        deb: http://91.189.93.70:8080/job/generic-mediumtests-utopic-python3/313/artifact/work/output/*zip*/output.zip
    SUCCESS: http://91.189.93.70:8080/job/ubuntu-clock-app-utopic-amd64-ci/113

Click here to trigger a rebuild:
http://91.189.93.70:8080/job/ubuntu-clock-app-ci/466/rebuild

review: Needs Fixing (continuous-integration)

[Ubuntu Phone Apps Jenkins Bot (ubuntu-phone-apps-jenkins-bot)]

PASSED: Continuous integration, rev:100
http://91.189.93.70:8080/job/ubuntu-clock-app-ci/471/
Executed test runs:
    SUCCESS: http://91.189.93.70:8080/job/generic-mediumtests-utopic-python3/344
        deb: http://91.189.93.70:8080/job/generic-mediumtests-utopic-python3/344/artifact/work/output/*zip*/output.zip
    SUCCESS: http://91.189.93.70:8080/job/ubuntu-clock-app-utopic-amd64-ci/118

Click here to trigger a rebuild:
http://91.189.93.70:8080/job/ubuntu-clock-app-ci/471/rebuild

review: Approve (continuous-integration)

[Ubuntu Phone Apps Jenkins Bot (ubuntu-phone-apps-jenkins-bot)]

FAILED: Continuous integration, rev:101
http://91.189.93.70:8080/job/ubuntu-clock-app-ci/472/
Executed test runs:
    FAILURE: http://91.189.93.70:8080/job/generic-mediumtests-utopic-python3/345/console
    FAILURE: http://91.189.93.70:8080/job/ubuntu-clock-app-utopic-amd64-ci/119/console

Click here to trigger a rebuild:
http://91.189.93.70:8080/job/ubuntu-clock-app-ci/472/rebuild

review: Needs Fixing (continuous-integration)

[Ubuntu Phone Apps Jenkins Bot (ubuntu-phone-apps-jenkins-bot)]

FAILED: Continuous integration, rev:103
http://91.189.93.70:8080/job/ubuntu-clock-app-ci/473/
Executed test runs:
    FAILURE: http://91.189.93.70:8080/job/generic-mediumtests-utopic-python3/346/console
    FAILURE: http://91.189.93.70:8080/job/ubuntu-clock-app-utopic-amd64-ci/120/console

Click here to trigger a rebuild:
http://91.189.93.70:8080/job/ubuntu-clock-app-ci/473/rebuild

review: Needs Fixing (continuous-integration)

[Ubuntu Phone Apps Jenkins Bot (ubuntu-phone-apps-jenkins-bot)]

PASSED: Continuous integration, rev:104
http://91.189.93.70:8080/job/ubuntu-clock-app-ci/490/
Executed test runs:
    SUCCESS: http://91.189.93.70:8080/job/generic-mediumtests-utopic-python3/498
        deb: http://91.189.93.70:8080/job/generic-mediumtests-utopic-python3/498/artifact/work/output/*zip*/output.zip
    SUCCESS: http://91.189.93.70:8080/job/ubuntu-clock-app-utopic-amd64-ci/137

Click here to trigger a rebuild:
http://91.189.93.70:8080/job/ubuntu-clock-app-ci/490/rebuild

review: Approve (continuous-integration)

[Francis Ginther (fginther)]

Leo, I think this is a great approach for running the qml tests. From my perspective, it's better to run the tests during the build, as it's then easier for the developers to run the tests as well and is more likely to happen all the time.

Implementing a jenkins only solution means that most developers won't run tests until they submit the MP, which is an unnecessary delay. Developers now have a template for adding QML tests that should work for other projects.

review: Approve

[Nekhelesh Ramananthan (nik90)]

Works as expected with jenkins. With my limited knowledge of CMake it looks okay. And most of the changes are within in the limits of the unit test directory. So it should be fine.

Thanks for the MP leo. We can now have qml tests run everything jenkins build the deb package.

review: Approve

[Ubuntu Phone Apps Jenkins Bot (ubuntu-phone-apps-jenkins-bot)]

review: Approve (continuous-integration)

=== modified file 'CMakeLists.txt'
--- CMakeLists.txt	2014-09-18 15:20:25 +0000
+++ CMakeLists.txt	2014-09-23 17:48:17 +0000
@@ -17,6 +17,9 @@
 
 option(INSTALL_TESTS "Install the tests on make install" on)
 option(CLICK_MODE "Build as a click package" on)
+option(USE_XVFB "Use XVFB to run qml tests" on)
+
+enable_testing()
 
 # Standard install paths 
 include(GNUInstallDirs) 
@@ -96,21 +99,13 @@
 add_subdirectory(app)
 add_subdirectory(backend)
 add_subdirectory(po)
-if(NOT CLICK_MODE)
-  add_subdirectory(tests)
-endif(NOT CLICK_MODE)
+add_subdirectory(tests)
 
 add_custom_target("autopilot" chmod +x ${CMAKE_SOURCE_DIR}/tests/autopilot/run
                     COMMAND ${CMAKE_SOURCE_DIR}/tests/autopilot/run
                     DEPENDS timezone timezone-qmldir alarmsettings alarmsettings-qmldir datetime datetime-qmldir
                     WORKING_DIRECTORY ./app)
 
-enable_testing()
-add_test(NAME basic
-  WORKING_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR}/app
-  COMMAND /usr/bin/qmltestrunner -input ${CMAKE_SOURCE_DIR}/tests/unit -import ${CMAKE_BINARY_DIR}/backend
-)
-
 add_custom_target("run" /usr/bin/qmlscene -I ${CMAKE_BINARY_DIR}/backend  ${CMAKE_SOURCE_DIR}/app/ubuntu-clock-app.qml
                     DEPENDS timezone timezone-qmldir alarmsettings alarmsettings-qmldir datetime datetime-qmldir
                     WORKING_DIRECTORY ./app)

=== modified file 'backend/CMakeLists.txt'
--- backend/CMakeLists.txt	2014-08-13 14:38:12 +0000
+++ backend/CMakeLists.txt	2014-09-23 17:48:17 +0000
@@ -1,3 +1,5 @@
+project(backend)
+
 include_directories(
     ${CMAKE_CURRENT_SOURCE_DIR}
 )

=== modified file 'debian/control'
--- debian/control	2014-08-19 19:11:35 +0000
+++ debian/control	2014-09-23 17:48:17 +0000
@@ -7,9 +7,13 @@
                intltool,
                python3 | python3-all | python3-dev | python3-all-dev,
                python3-minimal,
+               qml-module-qttest,
                qt5-default,
                qtbase5-dev,
                qtdeclarative5-dev,
+               qtdeclarative5-dev-tools,
+               qtdeclarative5-ubuntu-ui-toolkit-plugin,
+               xvfb,
 Standards-Version: 3.9.5
 Section: misc
 Homepage: https://launchpad.net/ubuntu-clock-app

=== modified file 'debian/rules'
--- debian/rules	2014-08-08 15:57:16 +0000
+++ debian/rules	2014-09-23 17:48:17 +0000
@@ -11,7 +11,7 @@
 	dh_auto_configure -- -DCLICK_MODE=OFF
 
 override_dh_auto_test:
-	echo
+	CTEST_OUTPUT_ON_FAILURE=1 make -C obj-$(DEB_HOST_GNU_TYPE) test
 
 override_dh_install:
 	dh_install --fail-missing

=== modified file 'tests/CMakeLists.txt'
--- tests/CMakeLists.txt	2014-08-08 15:57:16 +0000
+++ tests/CMakeLists.txt	2014-09-23 17:48:17 +0000
@@ -1,1 +1,6 @@
-add_subdirectory(autopilot)
+add_subdirectory(unit)
+
+if(NOT CLICK_MODE)
+  add_subdirectory(autopilot)
+endif(NOT CLICK_MODE)
+

=== added file 'tests/unit/CMakeLists.txt'
--- tests/unit/CMakeLists.txt	1970-01-01 00:00:00 +0000
+++ tests/unit/CMakeLists.txt	2014-09-23 17:48:17 +0000
@@ -0,0 +1,38 @@
+find_program(QMLTESTRUNNER_BIN
+    NAMES qmltestrunner
+    PATHS /usr/lib/*/qt5/bin
+    NO_DEFAULT_PATH
+)
+
+find_program(XVFB_RUN_BIN
+    NAMES xvfb-run
+)
+
+macro(DECLARE_QML_TEST TST_NAME TST_QML_FILE)
+    if(USE_XVFB)
+        set(COMMAND_PREFIX ${XVFB_RUN_BIN} -a -s "-screen 0 1024x768x24")
+    else()
+        set(COMMAND_PREFIX "")
+    endif()
+    add_test(NAME ${TST_NAME}
+        WORKING_DIRECTORY ${CMAKE_BINARY_DIR}
+        COMMAND ${COMMAND_PREFIX} ${QMLTESTRUNNER_BIN} -import ${backend_BINARY_DIR} -input ${CMAKE_CURRENT_SOURCE_DIR}/${TST_QML_FILE}
+    )
+endmacro()
+
+if(QMLTESTRUNNER_BIN AND XVFB_RUN_BIN)
+    declare_qml_test("AlarmLabel" tst_alarmLabel.qml)
+    declare_qml_test("AlarmRepeat" tst_alarmRepeat.qml)
+else()
+    if (NOT QMLTESTRUNNER_BIN)
+        message(WARNING "Qml tests disabled: qmltestrunner not found")
+    else()
+        message(WARNING "Qml tests disabled: xvfb-run not found")
+    endif()
+endif()
+
+set(QML_TST_FILES
+    tst_alarmLabel.qml
+    tst_alarmRepeat.qml
+)
+add_custom_target(tst_QmlFiles ALL SOURCES ${QML_TST_FILES})