Merge lp:~nik90/ubuntu-clock-app/checkbox-manual-tests into lp:ubuntu-clock-app

[Nekhelesh Ramananthan]

Added checkbox manual tests

Added checkbox manual tests. Do note that this is my first MP for this, so go easy on the review ;).

This MP doesn't cover all the manual tests. I just wanted to add a few and go about this one step at a time. In my next MP I will add debian packaging and the launcher for easy running of the tests.

To test this MP,
1. Add the checkbox devel ppa from https://launchpad.net/~checkbox-dev/+archive/ubuntu/ppa
2. Install plainbox and then navigate to the root-directory/test/manual/2014.com.ubuntu.clock:clock-tests/launcher folder
3. Run "sudo ../manage.py install"
4. Then run "./clock-test-cli"

Voila!

No worries this shoudl become way easier in the future.

[Ubuntu Phone Apps Jenkins Bot (ubuntu-phone-apps-jenkins-bot)]

FAILED: Continuous integration, rev:95
http://91.189.93.70:8080/job/ubuntu-clock-app-ci/455/
Executed test runs:
    FAILURE: http://91.189.93.70:8080/job/generic-mediumtests-utopic-python3/283/console
    FAILURE: http://91.189.93.70:8080/job/ubuntu-clock-app-utopic-amd64-ci/102/console

Click here to trigger a rebuild:
http://91.189.93.70:8080/job/ubuntu-clock-app-ci/455/rebuild

review: Needs Fixing (continuous-integration)

[Leo Arias (elopio)]

I think this is great. I would like to keep consistency on the ending periods at the end of the sentences. If you like them, add them everywhere. If you don't like them, remove them everywhere. But that's just because I'm a freak.

Please let me know the next time you are pushing a version to the store, and I will run all the tests to see how the process works.

[Leo Arias (elopio)]

review: Approve

[Zygmunt Krynicki (zyga)]

NOTE: instructions should include an extra step:

after step 2, you should do

3) Install the clock app test provider

./manage.py install

(this should be automated or handled in a repeatable way, perhaps through another package)

[Zygmunt Krynicki (zyga)]

Apart from a few comments down below I don't have anything to say other than that it looks OK.

If you have a CI loop for merging stuff into the clock-app you could teach it to run:

./manage.py validate -N

So that it can help everyone by catching various kind of problems. I'd merge this in and add a launcher so that everyone can run this easily.

[Nicholas Skaggs (nskaggs)]

I'd like to see you use this testcase format:

https://wiki.ubuntu.com/QATeam/TestCase

So something like:

1. Do something
    expect this
2. Do something else
    expect this
    expect that

review: Needs Fixing

[Nicholas Skaggs (nskaggs)]

It seems the ppa is also required for utopic; archive version must not be recent enough?

I managed to get it running by:

sudo add-apt-repository ppa:checkbox-dev/ppa
sudo apt-get update && sudo apt-get install plainbox
cd tests/manual/2014.com.ubuntu.clock:clock-tests
./manage develop
cd tests/manual/2014.com.ubuntu.clock:clock-tests/whitelist
plainbox run -w alarm.whitelist

Where are the test results written? What sort of gui can I use for this? How can this be packaged? Can I run this on the device?

[Zygmunt Krynicki (zyga)]

The version in Utopic is still the same as in Precise.

We're kind of neglecting non-LTS releases and with the current focus
on the touch + click that was okay at the time for our limited user
base. I'll refresh our packages in Debian and request a sync to Ubuntu
this week. I hope to have 0.11 which is both stable and featureful in
Utopic if I can.

Thanks
ZK

On Wed, Sep 10, 2014 at 11:38 PM, Nicholas Skaggs
<email address hidden> wrote:
> It seems the ppa is also required for utopic; archive version must not be recent enough?
>
> I managed to get it running by:
>
> sudo add-apt-repository ppa:checkbox-dev/ppa
> sudo apt-get update && sudo apt-get install plainbox
> cd tests/manual/2014.com.ubuntu.clock:clock-tests
> ./manage develop
> cd tests/manual/2014.com.ubuntu.clock:clock-tests/whitelist
> plainbox run -w alarm.whitelist
>
> Where are the test results written? What sort of gui can I use for this? How can this be packaged? Can I run this on the device?
> --
> https://code.launchpad.net/~nik90/ubuntu-clock-app/checkbox-manual-tests/+merge/234164
> You are requested to review the proposed merge of lp:~nik90/ubuntu-clock-app/checkbox-manual-tests into lp:ubuntu-clock-app.

[Zygmunt Krynicki (zyga)]

I meant Trusty :)

On Thu, Sep 11, 2014 at 2:23 AM, Zygmunt Krynicki
<email address hidden> wrote:
> The version in Utopic is still the same as in Precise.
>
> We're kind of neglecting non-LTS releases and with the current focus
> on the touch + click that was okay at the time for our limited user
> base. I'll refresh our packages in Debian and request a sync to Ubuntu
> this week. I hope to have 0.11 which is both stable and featureful in
> Utopic if I can.
>
> Thanks
> ZK
>
> On Wed, Sep 10, 2014 at 11:38 PM, Nicholas Skaggs
> <email address hidden> wrote:
>> It seems the ppa is also required for utopic; archive version must not be recent enough?
>>
>> I managed to get it running by:
>>
>> sudo add-apt-repository ppa:checkbox-dev/ppa
>> sudo apt-get update && sudo apt-get install plainbox
>> cd tests/manual/2014.com.ubuntu.clock:clock-tests
>> ./manage develop
>> cd tests/manual/2014.com.ubuntu.clock:clock-tests/whitelist
>> plainbox run -w alarm.whitelist
>>
>> Where are the test results written? What sort of gui can I use for this? How can this be packaged? Can I run this on the device?
>> --
>> https://code.launchpad.net/~nik90/ubuntu-clock-app/checkbox-manual-tests/+merge/234164
>> You are requested to review the proposed merge of lp:~nik90/ubuntu-clock-app/checkbox-manual-tests into lp:ubuntu-clock-app.

[lp:~nik90/ubuntu-clock-app/checkbox-manual-tests]

96. By Nekhelesh Ramananthan on 2014-09-10

updated whitelist with generic patterns

97. By Nekhelesh Ramananthan on 2014-09-12

Improved whitelist files

[Ubuntu Phone Apps Jenkins Bot (ubuntu-phone-apps-jenkins-bot)]

FAILED: Continuous integration, rev:97
http://91.189.93.70:8080/job/ubuntu-clock-app-ci/458/
Executed test runs:
    FAILURE: http://91.189.93.70:8080/job/generic-mediumtests-utopic-python3/304/console
    FAILURE: http://91.189.93.70:8080/job/ubuntu-clock-app-utopic-amd64-ci/105/console

Click here to trigger a rebuild:
http://91.189.93.70:8080/job/ubuntu-clock-app-ci/458/rebuild

review: Needs Fixing (continuous-integration)

[lp:~nik90/ubuntu-clock-app/checkbox-manual-tests]

98. By Nekhelesh Ramananthan on 2014-09-12

Removed gitignore and added newline

[Ubuntu Phone Apps Jenkins Bot (ubuntu-phone-apps-jenkins-bot)]

PASSED: Continuous integration, rev:98
http://91.189.93.70:8080/job/ubuntu-clock-app-ci/459/
Executed test runs:
    SUCCESS: http://91.189.93.70:8080/job/generic-mediumtests-utopic-python3/305
        deb: http://91.189.93.70:8080/job/generic-mediumtests-utopic-python3/305/artifact/work/output/*zip*/output.zip
    SUCCESS: http://91.189.93.70:8080/job/ubuntu-clock-app-utopic-amd64-ci/106

Click here to trigger a rebuild:
http://91.189.93.70:8080/job/ubuntu-clock-app-ci/459/rebuild

review: Approve (continuous-integration)

[lp:~nik90/ubuntu-clock-app/checkbox-manual-tests]

99. By Nekhelesh Ramananthan on 2014-09-12

Added a cli launcher

[Ubuntu Phone Apps Jenkins Bot (ubuntu-phone-apps-jenkins-bot)]

FAILED: Continuous integration, rev:99
http://91.189.93.70:8080/job/ubuntu-clock-app-ci/461/
Executed test runs:
    UNSTABLE: http://91.189.93.70:8080/job/generic-mediumtests-utopic-python3/307
        deb: http://91.189.93.70:8080/job/generic-mediumtests-utopic-python3/307/artifact/work/output/*zip*/output.zip
    SUCCESS: http://91.189.93.70:8080/job/ubuntu-clock-app-utopic-amd64-ci/108

Click here to trigger a rebuild:
http://91.189.93.70:8080/job/ubuntu-clock-app-ci/461/rebuild

review: Needs Fixing (continuous-integration)

[lp:~nik90/ubuntu-clock-app/checkbox-manual-tests]

100. By Nekhelesh Ramananthan on 2014-09-12

Improved the tests and launcher description

[Nekhelesh Ramananthan (nik90)]

> I'd like to see you use this testcase format:
>
> https://wiki.ubuntu.com/QATeam/TestCase
>
> So something like:
>
> 1. Do something
> expect this
> 2. Do something else
> expect this
> expect that

This has been addressed in the latest commit.

[Ubuntu Phone Apps Jenkins Bot (ubuntu-phone-apps-jenkins-bot)]

PASSED: Continuous integration, rev:100
http://91.189.93.70:8080/job/ubuntu-clock-app-ci/462/
Executed test runs:
    SUCCESS: http://91.189.93.70:8080/job/generic-mediumtests-utopic-python3/308
        deb: http://91.189.93.70:8080/job/generic-mediumtests-utopic-python3/308/artifact/work/output/*zip*/output.zip
    SUCCESS: http://91.189.93.70:8080/job/ubuntu-clock-app-utopic-amd64-ci/109

Click here to trigger a rebuild:
http://91.189.93.70:8080/job/ubuntu-clock-app-ci/462/rebuild

review: Approve (continuous-integration)

[lp:~nik90/ubuntu-clock-app/checkbox-manual-tests]

101. By Nekhelesh Ramananthan on 2014-09-12

Removed extra newlines

[Ubuntu Phone Apps Jenkins Bot (ubuntu-phone-apps-jenkins-bot)]

PASSED: Continuous integration, rev:101
http://91.189.93.70:8080/job/ubuntu-clock-app-ci/463/
Executed test runs:
    SUCCESS: http://91.189.93.70:8080/job/generic-mediumtests-utopic-python3/309
        deb: http://91.189.93.70:8080/job/generic-mediumtests-utopic-python3/309/artifact/work/output/*zip*/output.zip
    SUCCESS: http://91.189.93.70:8080/job/ubuntu-clock-app-utopic-amd64-ci/110

Click here to trigger a rebuild:
http://91.189.93.70:8080/job/ubuntu-clock-app-ci/463/rebuild

review: Approve (continuous-integration)

[Nicholas Skaggs (nskaggs)]

I approve the new test format, thanks!

Zygmunt, thanks for answering my questions about the version in utopic.

I still have a few questions:

What sort of gui can I use for this (on desktop)?
How can this be packaged?
Can I run this on the device (I know there's a touch package, but how does it work? how do the tests get on the device?)?

review: Approve

[Zygmunt Krynicki (zyga)]

Hey.

On the desktop you can use checkbox-gui (also from the PPA). The
launcher format is similar (but not identical). Have a look at the
lp:checkbox source tree at
providers/plainbox-providercertification-client/launchers/. The syntax
is an INI-style file with the fields that are documented on
http://checkbox.readthedocs.org/en/latest/profiles.html

On the device you still have to wait. The way this is going to work is
that you will create a similar launcher-like file but it will generate
a full click package instead. You can then submit that application to
the click store or side-load it on the device. I'll keep you updated
on the progress but we're making GREAT progress lately and it's all
coming together the way we wanted.

Meanwhile you can apt-get install plainbox, along with your provider
package, on the device if you make the filesystem writable.

Thanks
ZK

On Mon, Sep 15, 2014 at 4:33 PM, Nicholas Skaggs
<email address hidden> wrote:
> Review: Approve
>
> I approve the new test format, thanks!
>
> Zygmunt, thanks for answering my questions about the version in utopic.
>
> I still have a few questions:
>
> What sort of gui can I use for this (on desktop)?
> How can this be packaged?
> Can I run this on the device (I know there's a touch package, but how does it work? how do the tests get on the device?)?
> --
> https://code.launchpad.net/~nik90/ubuntu-clock-app/checkbox-manual-tests/+merge/234164
> You are requested to review the proposed merge of lp:~nik90/ubuntu-clock-app/checkbox-manual-tests into lp:ubuntu-clock-app.

[PS Jenkins bot (ps-jenkins)]

review: Approve (continuous-integration)

=== added directory 'tests/manual'
=== added directory 'tests/manual/2014.com.ubuntu.clock:clock-tests'
=== added file 'tests/manual/2014.com.ubuntu.clock:clock-tests/.bzrignore'
--- tests/manual/2014.com.ubuntu.clock:clock-tests/.bzrignore	1970-01-01 00:00:00 +0000
+++ tests/manual/2014.com.ubuntu.clock:clock-tests/.bzrignore	2014-09-12 17:33:17 +0000
@@ -0,0 +1,2 @@
+dist/*.tar.gz
+build/mo/*

=== added directory 'tests/manual/2014.com.ubuntu.clock:clock-tests/jobs'
=== added file 'tests/manual/2014.com.ubuntu.clock:clock-tests/jobs/alarms.pxu'
--- tests/manual/2014.com.ubuntu.clock:clock-tests/jobs/alarms.pxu	1970-01-01 00:00:00 +0000
+++ tests/manual/2014.com.ubuntu.clock:clock-tests/jobs/alarms.pxu	2014-09-12 17:33:17 +0000
@@ -0,0 +1,138 @@
+id: alarm/trigger-alarm
+plugin: manual
+_summary: Trigger alarm test 
+estimated_duration: 120
+_description:
+    This test checks if an alarm is triggered at the correct time by the datetime indicator.
+    1. Launch the clock app.
+        Clock app opens showing the current local time.
+    2. Swipe the bottom edge to open the alarms page.
+        Alarms page should appear showing a list of alarm. If empty then you should see a
+        message being displayed that "No saved alarms".
+    3. Press the plus icon to create a new alarm. Change the alarm time to ring in the next
+       minute. Do not change any other options. Save Alarm.
+        The saved alarm should be displayed in the alarms page.
+    4. Wait for the alarm to ring.
+        The alarm should ring (with haptic feedback) at the time set in the previous step and 
+        should show a notification with the buttons "Snooze" and "Ok".
+    5. Press "Ok" to dismiss the alarm.
+        The alarm should be dismissed.
+
+id: alarm/edit-alarm-sound
+plugin: manual
+depends: alarm/trigger-alarm
+_summary: Edit alarm sound test
+estimated_duration: 120
+_description:
+    This test checks if an updated alarm rings with the alarm sound selected by the user
+    as this functionality has regressed in the past.
+    1. Launch the clock app.
+        Clock app opens showing the current local time.
+    2. Swipe the bottom edge to open the alarms page.
+        Alarms page should appear showing a list of alarm. If empty then you should see a
+        message being displayed that "No saved alarms".
+    3. Press the plus icon to create a new alarm. Change the alarm time to ring in the next
+       minute. Do not change any other options. Save Alarm.
+        The saved alarm should be displayed in the alarms page.
+    4. Press the saved alarm to edit it.
+        The edit alarm page should be visible with the values set in the previous step.
+    5. Press the "Alarm Sound" list item to change the alarm sound.
+        The Sound page should be visible on pressing the "Alarm Sound" list item.
+    6. Choose a different alarm sound by pressing on any alarm sound.
+        On pressing any alarm sound, you should be able to hear a preview of the alarm sound
+        and also the checkbox is ticked to indicate it has been selected.
+    7. Press the back button and then save the alarm.
+        You should be taken back to the alarms page after you save the alarm.
+    8. Wait for the alarm to ring.
+        The alarm should ring (with haptic feedback) at the time set in the previous step and 
+        should show a notification with the buttons "Snooze" and "Ok".
+    9. Press "Ok" to dismiss the alarm.
+        The alarm should be dismissed.
+
+id: alarm/silence-setting
+plugin: manual
+depends: alarm/trigger-alarm
+_summary: Alarm Silence after setting test
+estimated_duration: 660
+_description:
+    This test checks if the alarm's "silence after" setting is obeyed.
+    1. Launch the clock app.
+        Clock app opens showing the current local time. 
+    2. Click the settings icon shown in the top right.
+        The settings page is visible showing the alarm settings.
+    3. Set the silence after setting to 10 minutes.
+        The alarm silence after setting should read "10 minutes" to indicate
+        it has been set.
+    4. Press the back button.
+        The settings page is closed and you are returned to the main clock page.
+    5. Swipe the bottom edge to open the alarms page.
+        Alarms page should appear showing a list of alarm. If empty then you should see a
+        message being displayed that "No saved alarms".
+    6. Press the plus icon to create a new alarm. Change the alarm time to ring in the next
+       minute. Do not change any other options. Save Alarm.
+        The saved alarm should be displayed in the alarms page. 
+    7. Wait for the alarm to ring.
+        The alarm should ring (with haptic feedback) at the time set in the previous step and 
+        should show a notification with the buttons "Snooze" and "Ok".
+    8. Allow the alarm to keep ringing.
+        The alarm should keep ringing and be silenced automatically by the phone only after
+        the duration set in step three.
+
+id: alarm/vibration-setting
+plugin: manual
+depends: alarm/trigger-alarm
+_summary: Alarm vibration setting test
+estimated_duration: 120
+_description:
+    This test checks if the alarm vibration settings enables/disables the alarm vibration.
+    1. Launch the clock app.
+        Clock app opens showing the current local time. 
+    2. Click the settings icon shown in the top right.
+        The settings page is visible showing the alarm settings.
+    3. Toggle the alarm vibration settings to off position.
+        The switch is discoloured to indicate that it has been disabled.
+    4. Press the back button.
+        The settings page is closed and you are returned to the main clock page.
+    5. Swipe the bottom edge to open the alarms page.
+        Alarms page should appear showing a list of alarm. If empty then you should see a
+        message being displayed that "No saved alarms".
+    6. Press the plus icon to create a new alarm. Change the alarm time to ring in the next
+       minute. Do not change any other options. Save Alarm.
+        The saved alarm should be displayed in the alarms page. 
+    7. Wait for the alarm to ring.
+        Wait for the alarm to ring.
+        The alarm should ring at the time set in the previous step and should show a
+        notification with the buttons "Snooze" and "Ok". However there should be no
+        vibration when the alarm rings.
+
+id: alarm/snooze-setting
+plugin: manual
+depends: alarm/trigger-alarm
+estimated_duration: 360
+_summary: Test the alarm snooze setting options
+_description:
+    This test checks if the alarm snooze settings works as expected.
+    1. Launch the clock app.
+        Clock app opens showing the current local time. 
+    2. Click the settings icon shown in the top right.
+        The settings page is visible showing the alarm settings.
+    3. Set the snooze duration to 2 minutes.
+        The alarm snooze duration setting should read "2 minutes" to indicate
+        it has been set.
+    4. Press the back button.
+        The settings page is closed and you are returned to the main clock page.
+    5. Swipe the bottom edge to open the alarms page.
+        Alarms page should appear showing a list of alarm. If empty then you should see a
+        message being displayed that "No saved alarms".
+    6. Press the plus icon to create a new alarm. Change the alarm time to ring in the next
+       minute. Do not change any other options. Save Alarm.
+        The saved alarm should be displayed in the alarms page. 
+    7. Wait for the alarm to ring.
+        The alarm should ring (with haptic feedback) at the time set in the previous step and 
+        should show a notification with the buttons "Snooze" and "Ok".
+    8. Press the "Snooze" button.
+        The alarm should be snoozed.
+    9. Wait for 2 minutes.
+        The alarm should ring again (with haptic feedback). Note that if an alarm is set to ring
+        originally at 10:00:00 and you snooze it the first time at 10:01:30, it should ring the next
+        time at 10:03:30 and not at 10:02:00.

=== added file 'tests/manual/2014.com.ubuntu.clock:clock-tests/jobs/setting.pxu'
--- tests/manual/2014.com.ubuntu.clock:clock-tests/jobs/setting.pxu	1970-01-01 00:00:00 +0000
+++ tests/manual/2014.com.ubuntu.clock:clock-tests/jobs/setting.pxu	2014-09-12 17:33:17 +0000
@@ -0,0 +1,22 @@
+id: setting/timezone-change
+plugin: manual
+_summary: Test to check if system timezone change is also done in the clock app
+estimated_duration: 60
+_description:
+    Test to check if system timezone change also updates the time in the clock app.
+    1. Launch the clock app.
+        Clock app opens showing the current local time. 
+    2. Click the settings icon shown in the top right.
+        The settings page is visible showing the alarm settings.
+    3. Press the "Change time and date" settings option.
+        The system settings app should be opened with the "Time & Date" page
+        shown.
+    4. Change the timezone.
+        The time shown in the top panel should reflect the new timezone to
+        confirm that it has been changed.
+    5. Return back to the clock app with switch right to left gesture.
+        The time shown below the "Change time and date" settings also shows
+        the new timezone time.
+    6. Press the back button.
+        The main clock page is shown and the main clock also shows the new
+        timezone time.   

=== added directory 'tests/manual/2014.com.ubuntu.clock:clock-tests/launcher'
=== added file 'tests/manual/2014.com.ubuntu.clock:clock-tests/launcher/clock-test-cli'
--- tests/manual/2014.com.ubuntu.clock:clock-tests/launcher/clock-test-cli	1970-01-01 00:00:00 +0000
+++ tests/manual/2014.com.ubuntu.clock:clock-tests/launcher/clock-test-cli	2014-09-12 17:33:17 +0000
@@ -0,0 +1,21 @@
+#!/usr/bin/env checkbox-launcher
+
+[welcome]
+text= Welcome to Clock App Manual Tests!
+    This application will guide you through some manual test cases which
+    confirm the workings of the Clock application (part of Ubuntu Core 
+    Applications) and its interaction with the system. The test suites are
+    split into various groups based on their functionality like alarm, 
+    settings etc.
+    
+    If all the actions produce the expected results listed, please submit
+    a 'passed' result.
+    
+    If an action fails, or produces an unexpected results, please submit a
+    'failed' result and file a bug. Please be sure to include the bug number
+    when you submit your result.
+
+[suite]
+whitelist_filter = clock-app-.*
+whitelist_selection=.*
+skip_test_selection = True

=== added file 'tests/manual/2014.com.ubuntu.clock:clock-tests/manage.py'
--- tests/manual/2014.com.ubuntu.clock:clock-tests/manage.py	1970-01-01 00:00:00 +0000
+++ tests/manual/2014.com.ubuntu.clock:clock-tests/manage.py	2014-09-12 17:33:17 +0000
@@ -0,0 +1,21 @@
+#!/usr/bin/env python3
+from plainbox.provider_manager import setup, N_
+
+# You can inject other stuff here but please don't go overboard.
+#
+# In particular, if you need comprehensive compilation support to get
+# your bin/ populated then please try to discuss that with us in the
+# upstream project IRC channel #checkbox on irc.freenode.net.
+
+# NOTE: one thing that you could do here, that makes a lot of sense,
+# is to compute version somehow. This may vary depending on the
+# context of your provider. Future version of PlainBox will offer git,
+# bzr and mercurial integration using the versiontools library
+# (optional)
+
+setup(
+    name='2014.com.ubuntu.clock:clock-tests',
+    version="1.0",
+    description=N_("The 2014.com.ubuntu.clock:clock-tests provider"),
+    gettext_domain="2014_com_ubuntu_clock_clock-tests",
+)

=== added directory 'tests/manual/2014.com.ubuntu.clock:clock-tests/whitelists'
=== added file 'tests/manual/2014.com.ubuntu.clock:clock-tests/whitelists/clock-app-alarm.whitelist'
--- tests/manual/2014.com.ubuntu.clock:clock-tests/whitelists/clock-app-alarm.whitelist	1970-01-01 00:00:00 +0000
+++ tests/manual/2014.com.ubuntu.clock:clock-tests/whitelists/clock-app-alarm.whitelist	2014-09-12 17:33:17 +0000
@@ -0,0 +1,2 @@
+alarm/.*
+

=== added file 'tests/manual/2014.com.ubuntu.clock:clock-tests/whitelists/clock-app-setting.whitelist'
--- tests/manual/2014.com.ubuntu.clock:clock-tests/whitelists/clock-app-setting.whitelist	1970-01-01 00:00:00 +0000
+++ tests/manual/2014.com.ubuntu.clock:clock-tests/whitelists/clock-app-setting.whitelist	2014-09-12 17:33:17 +0000
@@ -0,0 +1,1 @@
+setting/.*