Merge lp:~nskaggs/music-app/manual-tests into lp:music-app

[Nicholas Skaggs]

* Add manual tests in checkbox format to repo

* Add manual tests in checkbox format to repo

This has been directly copied from the old manual tests [0]

Note that the durations are all 600 and that any notes/prerequisites I've currently put in the 1.

0 - http://pad.ubuntu.com/MusicAppManualTesting

---

Andrew, this is the version that shipped with Pilot. The ideas about inline images, etc, still stands. Something to take more about with the checkbox team if we can't get it to work. For now however, these work well.

[Ubuntu Phone Apps Jenkins Bot (ubuntu-phone-apps-jenkins-bot)]

PASSED: Continuous integration, rev:918
http://91.189.93.70:8080/job/music-app-ci/1381/
Executed test runs:
    SUCCESS: http://91.189.93.70:8080/job/music-app-vivid-amd64-ci/233

Click here to trigger a rebuild:
http://91.189.93.70:8080/job/music-app-ci/1381/rebuild

review: Approve (continuous-integration)

[Victor Thompson (vthompson)]

I noticed a few small things, but otherwise this lgtm.

review: Needs Fixing

[Andrew Hayzen (ahayzen)]

* Modify tests that use SD card to not mention krillin and include the 'only applicable'
* Add capitalisation

Please retest :-)

[Ubuntu Phone Apps Jenkins Bot (ubuntu-phone-apps-jenkins-bot)]

PASSED: Continuous integration, rev:919
http://91.189.93.70:8080/job/music-app-ci/1383/
Executed test runs:
    SUCCESS: http://91.189.93.70:8080/job/music-app-vivid-amd64-ci/235

Click here to trigger a rebuild:
http://91.189.93.70:8080/job/music-app-ci/1383/rebuild

review: Approve (continuous-integration)

[Victor Thompson (vthompson)]

This looks good to me. I'd like to get balloon's final opinion prior to merging, however.

review: Approve

[Nicholas Skaggs (nskaggs)]

These are fine, but will need a provider and folder layout to make them plug and play. I'll bundle them and try them before providing a diff.

It's not crystal clear if we do want to include the provider in here, so perhaps these will be a-ok.

[Nicholas Skaggs (nskaggs)]

Prefix the tests file names with 'music-'

You are missing a colon
s/_description/_description:/g

Still deciding on the provider / no provider bit.

review: Needs Fixing

[Nicholas Skaggs (nskaggs)]

Ohh my, I didn't notice the embedded links for results! Those won't play nice on the phone. Not sure what to do with them right now, other than to ensure there is text to go along with them.

It is possible to embed the image, but we are limited to text because of how QA stores the tests and processes them. It may be possible to embed them anyway though :p

review: Needs Fixing

[Nicholas Skaggs (nskaggs)]

Leaving a note on how the images would have to be embedded. We would have to change the job type, as well as package the images and reference them.

use qml type of job
include $your_file.png in the 'data' dir of the provider
reference it as Qt.resolvedUrl("$your_file.png")

[Victor Thompson (vthompson)]

I'm going to hold off on reviewing this for now. Andrew, could you look into balloon's concerns?

[Jenkins Bot (ubuntu-core-apps-jenkins-bot)]

FAILED: Continuous integration, rev:919
https://core-apps-jenkins.ubuntu.com/job/music-app-ci/9/
Executed test runs:
    None: https://core-apps-jenkins.ubuntu.com/job/generic-update-mp/142/console

Click here to trigger a rebuild:
https://core-apps-jenkins.ubuntu.com/job/music-app-ci/9/rebuild

review: Needs Fixing (continuous-integration)

[Jenkins Bot (ubuntu-core-apps-jenkins-bot)]

PASSED: Continuous integration, rev:956
https://core-apps-jenkins.ubuntu.com/job/music-app-ci/30/
Executed test runs:
    None: https://core-apps-jenkins.ubuntu.com/job/generic-update-mp/261/console

Click here to trigger a rebuild:
https://core-apps-jenkins.ubuntu.com/job/music-app-ci/30/rebuild

review: Approve (continuous-integration)

[Andrew Hayzen (ahayzen)]

LGTM! As discussed for now we'll merge this with the image links so that Pilot can be 'in-sync' :-)

review: Approve

[Victor Thompson (vthompson)]

Lgtm!

review: Approve

[Jenkins Bot (ubuntu-core-apps-jenkins-bot)]

review: Approve (continuous-integration)

=== added directory 'tests/manual'
=== added directory 'tests/manual/2015.com.ubuntu.music:music-tests'
=== added directory 'tests/manual/2015.com.ubuntu.music:music-tests/jobs'
=== added file 'tests/manual/2015.com.ubuntu.music:music-tests/jobs/music-external.pxu'
--- tests/manual/2015.com.ubuntu.music:music-tests/jobs/music-external.pxu	1970-01-01 00:00:00 +0000
+++ tests/manual/2015.com.ubuntu.music:music-tests/jobs/music-external.pxu	2015-12-16 22:01:27 +0000
@@ -0,0 +1,62 @@
+plugin: resource
+id: music-external/system-image
+_summary: [Automated] gather system information
+command: system-image-cli -i
+_description:
+ Gather system info
+
+id: music-external/uri-handler
+plugin: manual
+_summary: Test music appears in scope and can be launched
+estimated_duration: 600
+_description:
+    Test music appears in scope and can be launched
+    1. Switch to music scope
+        Expect to see artists:-  http://people.canonical.com/~alan/screenshots/device-2014-11-17-101609.png
+        Sometimes the search in the scope doesn't seem to load.  I'll recreate and file that later since it's the scope and not the app
+    2. Tap artist
+        Expect to see zoomed in view of artist and below it, some albums:- http://people.canonical.com/~alan/screenshots/device-2014-11-17-101722.png
+    3. Tap album
+        Expect to see a zoomed in view of the album including a list of tracks and button to play in music app:- http://people.canonical.com/~alan/screenshots/device-2014-11-17-101752.png
+    4. Tap "Play in music app"
+        Expect to see music app launch and a track to start playing:- http://people.canonical.com/~alan/screenshots/device-2014-11-17-101818.png
+
+id: music-external/content-hub
+plugin: manual
+_summary: Test music can be imported via content-hub
+estimated_duration: 600
+_description:
+    Test music can be imported via content-hub
+    1. Open the browser, navigate to jamendo.com
+    2. Find a song to download
+    3. Select to "Open with" the Music app
+    4. When the download is finished, select "Open"
+        Expect the music app to open
+        Expect the track to be imported into ~/Music/Imported
+        Expect the current queue to be cleared and replaced by the item(s) imported via content-hub
+        Expect the first song in the import to start playing
+
+id: music-external/headphones
+plugin: manual
+_summary: Test music plays in headphones
+estimated_duration: 600
+_description:
+    Test music plays in headphones
+    1. Play music in app
+    2. Attach headphones
+        Expect music to play through headphones
+    3. Detach headphones
+        Expect the music to pause
+    4. Select play
+        Expect music to play through internal speaker
+
+id: music-external/infographics
+plugin: manual
+_summary: Test welcome screen stats
+estimated_duration: 600
+_description:
+    Test welcome screen stats
+    1. Note: This should be done after the above tests have been completed, and music has been played successfully
+    2. Tap power button to lock screen
+    3. Tap power button to wake phone
+        Expect to see user metrics of songs played (may need to tap-tap this screen to loop round to "songs played today"):- http://people.canonical.com/~alan/screenshots/device-2014-11-17-210613.png

=== added file 'tests/manual/2015.com.ubuntu.music:music-tests/jobs/music-library.pxu'
--- tests/manual/2015.com.ubuntu.music:music-tests/jobs/music-library.pxu	1970-01-01 00:00:00 +0000
+++ tests/manual/2015.com.ubuntu.music:music-tests/jobs/music-library.pxu	2015-12-16 22:01:27 +0000
@@ -0,0 +1,131 @@
+plugin: resource
+id: music-library/system-image
+_summary: [Automated] gather system information
+command: system-image-cli -i
+_description:
+ Gather system info
+
+id: music-library/empty-state
+plugin: manual
+_summary: Test to check that the empty state for the library is shown
+estimated_duration: 600
+_description:
+    Test to check that the empty state for the library is shown
+    1. Launch music app with no music on device (and no or empty SD card if applicable)
+    2. Expected empty state screen
+        http://people.canonical.com/~alan/screenshots/device-2015-01-23-094554.png
+
+id: music-library/music-appears
+plugin: manual
+_summary: Test music appearing when on device
+estimated_duration: 600
+_description:
+    Test music appearing when on device
+    1. Launch music app
+    2. Unlock device
+    3. On desktop navigate to Devices -> Device name -> Music
+    4. Copy music over USB cable via MTP to Music folder
+        Note: Only MP3 and FLAC files.
+    5. Expect to see music appearing inside the application with appropriate artwork (this may take a while as mediascanner looks for music)
+
+id: music-library/music-appears-views
+plugin: manual
+_summary: Test music appears appropriately in each view
+estimated_duration: 600
+_description:
+    Test music appears appropriately in each view
+    1. Launch music app - expect populated views
+    2. Switch to Artists view
+        Expect to see artists listed in card view:- http://people.canonical.com/~alan/screenshots/device-2014-11-17-101312.png
+    3. Switch to Albums view
+        Expect to see album covers:- http://people.canonical.com/~alan/screenshots/device-2014-11-17-101331.png
+    4. Switch to Genres view
+        Expect to see (multiple if copied) genres with (multiple if appropriate) album cover art:- http://people.canonical.com/~alan/screenshots/device-2014-11-17-101348.png
+    5. Switch to Songs view
+        Expect to see a list of all songs on the device including album art:- http://people.canonical.com/~alan/screenshots/device-2014-11-17-101402.png
+    6. Expect recent to not exist in the tabs
+    7. Switch to Playlists view
+        Expect empty view Playlists:- http://people.canonical.com/~alan/screenshots/device-2015-01-23-094234.png
+
+id: music-library/music-appears-sd-card
+plugin: manual
+_summary: Test music appearing when on sdcard
+estimated_duration: 600
+_description:
+    Test music appearing when on sdcard
+    1. (only applicable on devices with sdcard support)
+    2. Unlock device
+        then wait a few moments for the SD card to show up
+    3. On desktop navigate to Devices -> Device name -> SDCARD -> Music
+    4. Copy music over USB cable via MTP to Music folder
+    5. Launch Music app
+    6. Expect music to show up in Songs, Artists and Albums view in Music app
+
+id: music-library/deletion-of-music
+plugin: manual
+_summary: Test deletion of music
+estimated_duration: 600
+_description:
+    Test deletion of music
+    1. Open music app, observe music in Albums view
+    2. Ensure phone is unlocked
+    3. Connect phone via USB
+    4. On desktop navigate to Devices -> Device name -> Music and delete an entire album folder
+        Cannot delete via mtp, will file separately (right click delete in nautilus)
+    5. Expect the album disappears from the music player shortly afterwards
+
+id: music-library/search-albums
+plugin: manual
+_summary: Test Search Albums
+estimated_duration: 600
+_description:
+    Test Search Albums
+    1. Navigate to Albums view
+    2. Press search button in toolbar
+    3. Type a partial search term
+        Expect to see results:- http://people.canonical.com/~alan/screenshots/device-2015-01-12-102618.png
+
+id: music-library/search-artists
+plugin: manual
+_summary: Test Search Artists
+estimated_duration: 600
+_description:
+    Test Search Artists
+    1. Navigate to Artists view
+    2. Press search button in toolbar
+    3. Type a partial search term
+        Expect to see results:- http://people.canonical.com/~alan/screenshots/device-2015-01-12-102729.png
+
+id: music-library/search-songs
+plugin: manual
+_summary: Test Search Songs
+estimated_duration: 600
+_description:
+    Test Search Songs
+    1. Navigate to Songs view
+    2. Press search button in toolbar
+    3. Type a partial search term
+        Expect to see results:- http://people.canonical.com/~alan/screenshots/device-2015-01-12-102836.png
+
+id: music-library/search-genres
+plugin: manual
+_summary: Test Search Genres
+estimated_duration: 600
+_description:
+    Test Search Genres
+    1. Navigate to Genres view
+    2. Press search button in toolbar
+    3. Type a partial search term
+        Expect to see results:- http://people.canonical.com/~alan/screenshots/device-2015-01-12-102927.png
+
+id: music-library/search-playlists
+plugin: manual
+_summary: Test Search Genres
+estimated_duration: 600
+_description:
+    Test Search Genres
+    1. Note: This relies on an earlier test where a playlist was created
+    2. Navigate to Playlist view
+    3. Press search button in toolbar
+    4. Type a partial search term
+        Expect to see results:- http://people.canonical.com/~alan/screenshots/device-2015-01-12-103017.png

=== added file 'tests/manual/2015.com.ubuntu.music:music-tests/jobs/music-playlists.pxu'
--- tests/manual/2015.com.ubuntu.music:music-tests/jobs/music-playlists.pxu	1970-01-01 00:00:00 +0000
+++ tests/manual/2015.com.ubuntu.music:music-tests/jobs/music-playlists.pxu	2015-12-16 22:01:27 +0000
@@ -0,0 +1,73 @@
+plugin: resource
+id: music-playlists/system-image
+_summary: [Automated] gather system information
+command: system-image-cli -i
+_description:
+ Gather system info
+
+id: music-playlists/create-playlists
+plugin: manual
+_summary: Test create playlist
+estimated_duration: 600
+_description:
+    Test create playlist
+    1. Open music app
+    2. Navigate to playlist view
+        Expect empty playlist:- http://people.canonical.com/~alan/screenshots/device-2014-11-17-101416.png
+    3. Tap "+" button in top right
+        Expect "New Playlist" dialog:- http://people.canonical.com/~alan/screenshots/device-2014-11-17-204544.png
+    4. Tap in the input field
+        Expect to see the OSK appear:- http://people.canonical.com/~alan/screenshots/device-2014-11-17-205144.png
+    5. Type "Best songs"
+    6. Press Create
+        Expect to see empty playlist:- http://people.canonical.com/~alan/screenshots/device-2014-11-17-205248.png
+
+id: music-playlists/add-tracks-to-playlist
+plugin: manual
+_summary: Test add tracks to playlist
+estimated_duration: 600
+_description:
+    Test add tracks to playlist
+    1. Navigate to Songs view
+    2. Slide left a song
+        Expect to see buttons revealed:- http://people.canonical.com/~alan/screenshots/device-2014-11-17-205354.png
+    3. Tap last icon (add to playlist)
+        Expect to be taken to "Select playlist" view:- http://people.canonical.com/~alan/screenshots/device-2014-11-17-205514.png
+    4. Select previously created playlist "Best songs"
+        Expect to be taken back to previous view:- http://people.canonical.com/~alan/screenshots/device-2014-11-17-205552.png
+    5. Navigate to albums view
+    6. Tap on an album
+        Expect to see the individual album view:- http://people.canonical.com/~alan/screenshots/device-2014-11-17-205713.png
+    7. Long press on a track
+        Expect multi-select boxes and buttons to appear:- http://people.canonical.com/~alan/screenshots/device-2014-11-17-205819.png
+    8. Tap multiple songs
+    9. Tap 'burger' menu in the top right, choose "Add to playlist"
+        Expect to be taken to the "Select playlist" view:- http://people.canonical.com/~alan/screenshots/device-2014-11-17-205923.png
+    10. Tap previously create playlist
+         Expect to be taken back to individual album view:- http://people.canonical.com/~alan/screenshots/device-2014-11-17-210010.png
+    11. Navigate to Playlist view
+    12. Tap previously created playlist
+         Expect to see songs previously selected:- http://people.canonical.com/~alan/screenshots/device-2014-11-17-210333.png
+
+id: music-playlists/remove-playlist
+plugin: manual
+_summary: Test removing playlist
+estimated_duration: 600
+_description:
+    Test removing playlist
+    1. Navigate to Playlist view
+    2. Tap previously created playlist
+    3. Tap rubbish bin in top toolbar
+        Expect to get a confirmation prompt:- http://people.canonical.com/~alan/screenshots/device-2014-11-17-210424.png
+    4. Tap 'Remove' button
+        Expect playlist to be removed:- http://people.canonical.com/~alan/screenshots/device-2014-11-17-210520.png
+
+id: music-playlists/tab-persist
+plugin: manual
+_summary: Test tab index is persists a restart
+estimated_duration: 600
+_description:
+    Test tab index is persists a restart
+    1. Start the app and switch to a tab (eg playlists)
+    2. Close the app and restart it
+        Expect the app to start on the same tab

=== added file 'tests/manual/2015.com.ubuntu.music:music-tests/jobs/music-queue.pxu'
--- tests/manual/2015.com.ubuntu.music:music-tests/jobs/music-queue.pxu	1970-01-01 00:00:00 +0000
+++ tests/manual/2015.com.ubuntu.music:music-tests/jobs/music-queue.pxu	2015-12-16 22:01:27 +0000
@@ -0,0 +1,44 @@
+plugin: resource
+id: music-queue/system-image
+_summary: [Automated] gather system information
+command: system-image-cli -i
+_description:
+ Gather system info
+
+id: music-queue/deletion-music
+plugin: manual
+_summary: Test deletion of music from queue
+estimated_duration: 600
+_description:
+    Test deletion of music from queue
+    1. Open music app
+    2. Select an album and play all
+        Expect the album to start playing and that single album to be in the queue
+    3. Delete the entire album from the device
+        Expect the view to be 'popped' back to the Albums view
+        Expect the music to be stopped and the queue to be empty
+
+id: music-queue/sd-card-remove
+plugin: manual
+_summary: Test removing SD Card during playback
+estimated_duration: 600
+_description:
+    Test removing SD Card during playback
+    1. (only applicable on devices with sdcard support)
+    2. Add some songs to an SD card
+    3. Queue up an album which is on the SD card
+    4. Start playing tracks from the queue
+    5. Remove the SD Card
+        Expect tracks which are no longer accessible should disappear from the queue
+
+id: music-queue/restart
+plugin: manual
+_summary: Test queue persists a restart
+estimated_duration: 600
+_description:
+    Test queue persists a restart
+    1. Start the app
+    2. Add a few albums to the queue
+    3. Take note of the queue and current track
+    4. Restart the app
+        Expect the queue to be restored and the same track to be selected

=== added file 'tests/manual/2015.com.ubuntu.music:music-tests/jobs/music-recent.pxu'
--- tests/manual/2015.com.ubuntu.music:music-tests/jobs/music-recent.pxu	1970-01-01 00:00:00 +0000
+++ tests/manual/2015.com.ubuntu.music:music-tests/jobs/music-recent.pxu	2015-12-16 22:01:27 +0000
@@ -0,0 +1,28 @@
+plugin: resource
+id: music-recent/system-image
+_summary: [Automated] gather system information
+command: system-image-cli -i
+_description:
+ Gather system info
+
+id: music-recent/recent-appears
+plugin: manual
+_summary: Test recent tab appears
+estimated_duration: 600
+_description:
+    Test recent tab appears
+    1. Goto the album view
+    2. Select an album and play all
+        Expect the recent tab to appear
+    3. Navigate to the recent tab ensure that the album is correct
+
+id: music-recent/recent-disappears
+plugin: manual
+_summary: Test recent tab disappears
+estimated_duration: 600
+_description:
+    Test recent tab disappears
+    1. Ensure that the recent tab is visible and switch to the tab
+    2. Select clear all action
+        Expect to be switched to the albums tab
+        Expect the recent tab to have been removed

=== added file 'tests/manual/2015.com.ubuntu.music:music-tests/manage.py'
--- tests/manual/2015.com.ubuntu.music:music-tests/manage.py	1970-01-01 00:00:00 +0000
+++ tests/manual/2015.com.ubuntu.music:music-tests/manage.py	2015-12-16 22:01:27 +0000
@@ -0,0 +1,21 @@
+#!/usr/bin/env python3
+from plainbox.provider_manager import setup, N_
+
+# You can inject other stuff here but please don't go overboard.
+#
+# In particular, if you need comprehensive compilation support to get
+# your bin/ populated then please try to discuss that with us in the
+# upstream project IRC channel #checkbox on irc.freenode.net.
+
+# NOTE: one thing that you could do here, that makes a lot of sense,
+# is to compute version somehow. This may vary depending on the
+# context of your provider. Future version of PlainBox will offer git,
+# bzr and mercurial integration using the versiontools library
+# (optional)
+
+setup(
+    name='2015.com.ubuntu.music:music-tests',
+    version="1.0",
+    description=N_("The 2015.com.ubuntu.music:music-tests provider"),
+    gettext_domain="2015_com_ubuntu_music_music-tests",
+)

=== added directory 'tests/manual/2015.com.ubuntu.music:music-tests/whitelists'
=== added file 'tests/manual/2015.com.ubuntu.music:music-tests/whitelists/music-external.whitelist'
--- tests/manual/2015.com.ubuntu.music:music-tests/whitelists/music-external.whitelist	1970-01-01 00:00:00 +0000
+++ tests/manual/2015.com.ubuntu.music:music-tests/whitelists/music-external.whitelist	2015-12-16 22:01:27 +0000
@@ -0,0 +1,1 @@
+music-external/.*
\ No newline at end of file

=== added file 'tests/manual/2015.com.ubuntu.music:music-tests/whitelists/music-library.whitelist'
--- tests/manual/2015.com.ubuntu.music:music-tests/whitelists/music-library.whitelist	1970-01-01 00:00:00 +0000
+++ tests/manual/2015.com.ubuntu.music:music-tests/whitelists/music-library.whitelist	2015-12-16 22:01:27 +0000
@@ -0,0 +1,1 @@
+music-library/.*
\ No newline at end of file

=== added file 'tests/manual/2015.com.ubuntu.music:music-tests/whitelists/music-playlists.whitelist'
--- tests/manual/2015.com.ubuntu.music:music-tests/whitelists/music-playlists.whitelist	1970-01-01 00:00:00 +0000
+++ tests/manual/2015.com.ubuntu.music:music-tests/whitelists/music-playlists.whitelist	2015-12-16 22:01:27 +0000
@@ -0,0 +1,1 @@
+music-playlists/.*
\ No newline at end of file

=== added file 'tests/manual/2015.com.ubuntu.music:music-tests/whitelists/music-queue.whitelist'
--- tests/manual/2015.com.ubuntu.music:music-tests/whitelists/music-queue.whitelist	1970-01-01 00:00:00 +0000
+++ tests/manual/2015.com.ubuntu.music:music-tests/whitelists/music-queue.whitelist	2015-12-16 22:01:27 +0000
@@ -0,0 +1,1 @@
+music-queue/.*
\ No newline at end of file

=== added file 'tests/manual/2015.com.ubuntu.music:music-tests/whitelists/music-recent.whitelist'
--- tests/manual/2015.com.ubuntu.music:music-tests/whitelists/music-recent.whitelist	1970-01-01 00:00:00 +0000
+++ tests/manual/2015.com.ubuntu.music:music-tests/whitelists/music-recent.whitelist	2015-12-16 22:01:27 +0000
@@ -0,0 +1,1 @@
+music-recent/.*
\ No newline at end of file