Merge lp:~canonical-platform-qa/ubuntu-system-tests/mtp-backup-local-media into lp:ubuntu-system-tests

[Richard Huddie]

Add BackupRestoreLocalMediaFixture and update MTP test_copy_music to backup and restore any local media before running the test.

Update MTP test_copy_music to backup and restore any local media before running the test.

Create a new fixture for backing up and restoring all local media files on the device. Update the backup and restore helper so that when it moves a folder, it leaves the top-level folder in place. This is required for MTP where it can cause problems if the Music, Videos or Pictures folders get deleted and re-added.

This mp also creates a helper to restart the media scanner after moving the local media files to ensure the scopes are updated.

[PS Jenkins bot (ps-jenkins)]

PASSED: Continuous integration, rev:158
http://s-jenkins.ubuntu-ci:8080/job/ubuntu-sanity-tests-ci/544/
Executed test runs:
    SUCCESS: http://s-jenkins.ubuntu-ci:8080/job/ubuntu-sanity-tests-wily-amd64-ci/272
    SUCCESS: http://s-jenkins.ubuntu-ci:8080/job/ubuntu-sanity-tests-wily-armhf-ci/273
    SUCCESS: http://s-jenkins.ubuntu-ci:8080/job/ubuntu-sanity-tests-wily-i386-ci/272

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/ubuntu-sanity-tests-ci/544/rebuild

review: Approve (continuous-integration)

[Sergio Cazzolato (sergio-j-cazzolato)]

I added a card to make a Fixture that makes the media backup, do you want to include this Fixture as part of this branch?

The card is:
https://trello.com/c/DtVWrbMj/112-update-mtp-copy-music-test-to-backup-all-the-music-media-on-setup

[lp:~canonical-platform-qa/ubuntu-system-tests/mtp-backup-local-media]

159. By Richard Huddie on 2015-07-27

Create and use a fixture for backing up and restoring local media files during testing.

[Richard Huddie (rhuddie)]

> I added a card to make a Fixture that makes the media backup, do you want to
> include this Fixture as part of this branch?
>
> The card is:
> https://trello.com/c/DtVWrbMj/112-update-mtp-copy-music-test-to-backup-all-
> the-music-media-on-setup

Thanks Sergio for suggestion. I've now added the BackupRestoreLocalMediaFixture to this.

[PS Jenkins bot (ps-jenkins)]

PASSED: Continuous integration, rev:159
http://s-jenkins.ubuntu-ci:8080/job/ubuntu-sanity-tests-ci/550/
Executed test runs:
    SUCCESS: http://s-jenkins.ubuntu-ci:8080/job/ubuntu-sanity-tests-wily-amd64-ci/278
    SUCCESS: http://s-jenkins.ubuntu-ci:8080/job/ubuntu-sanity-tests-wily-armhf-ci/279
    SUCCESS: http://s-jenkins.ubuntu-ci:8080/job/ubuntu-sanity-tests-wily-i386-ci/278

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/ubuntu-sanity-tests-ci/550/rebuild

review: Approve (continuous-integration)

[Sergio Cazzolato (sergio-j-cazzolato)]

Getting this error when I execute the test http://paste.ubuntu.com/11953459/

[Sergio Cazzolato (sergio-j-cazzolato)]

review: Needs Fixing

[lp:~canonical-platform-qa/ubuntu-system-tests/mtp-backup-local-media]

160. By Richard Huddie on 2015-07-29

Merge from trunk.

[PS Jenkins bot (ps-jenkins)]

PASSED: Continuous integration, rev:160
http://s-jenkins.ubuntu-ci:8080/job/ubuntu-sanity-tests-ci/566/
Executed test runs:
    SUCCESS: http://s-jenkins.ubuntu-ci:8080/job/ubuntu-sanity-tests-wily-amd64-ci/294
    SUCCESS: http://s-jenkins.ubuntu-ci:8080/job/ubuntu-sanity-tests-wily-armhf-ci/295
    SUCCESS: http://s-jenkins.ubuntu-ci:8080/job/ubuntu-sanity-tests-wily-i386-ci/294

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/ubuntu-sanity-tests-ci/566/rebuild

review: Approve (continuous-integration)

[Sergio Cazzolato (sergio-j-cazzolato)]

still failing, this is the media dir that I have

phablet@ubuntu-phablet:~$ ll /media/phablet/5AC0-A72A/Music/
total 288
drwxr-xr-x 3 phablet phablet 32768 jul 28 12:52 ./
drwxr-xr-x 8 phablet phablet 32768 jul 29 18:27 ../
-rw-r--r-- 1 phablet phablet 11977 jul 28 12:21 aaa1111.mp3
-rw-r--r-- 1 phablet phablet 18358 jul 28 12:21 aaaaaa.ogg
-rw-r--r-- 1 phablet phablet 18358 jul 28 12:21 hello (copy).ogg
-rw-r--r-- 1 phablet phablet 11977 jul 28 12:21 hello.mp3
drwxr-xr-x 2 phablet phablet 32768 jul 28 12:52 mymusic/
-rw-r--r-- 1 phablet phablet 18358 jul 28 12:21 sssaaa.ogg
-rw-r--r-- 1 phablet phablet 10240 jul 17 20:06 XATTKXKDJV.mp3

[lp:~canonical-platform-qa/ubuntu-system-tests/mtp-backup-local-media]

161. By Richard Huddie on 2015-07-30

Create backup destination folder if path does not exist.

[PS Jenkins bot (ps-jenkins)]

PASSED: Continuous integration, rev:161
http://s-jenkins.ubuntu-ci:8080/job/ubuntu-sanity-tests-ci/568/
Executed test runs:
    SUCCESS: http://s-jenkins.ubuntu-ci:8080/job/ubuntu-sanity-tests-wily-amd64-ci/296
    SUCCESS: http://s-jenkins.ubuntu-ci:8080/job/ubuntu-sanity-tests-wily-armhf-ci/297
    SUCCESS: http://s-jenkins.ubuntu-ci:8080/job/ubuntu-sanity-tests-wily-i386-ci/296

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/ubuntu-sanity-tests-ci/568/rebuild

review: Approve (continuous-integration)

[Richard Huddie (rhuddie)]

Ok, I've made an update that should fix this. Please could you re-test.

[Sergio Cazzolato (sergio-j-cazzolato)]

The Fixture is working for me. The test test_copy_music is failing in my device I'll take a look to that.

review: Approve

=== modified file 'ubuntu_system_tests/helpers/backup_restore_fixture.py'
--- ubuntu_system_tests/helpers/backup_restore_fixture.py	2015-05-21 18:15:40 +0000
+++ ubuntu_system_tests/helpers/backup_restore_fixture.py	2015-07-30 09:41:10 +0000
@@ -21,6 +21,8 @@
 import os
 import shutil
 
+from ubuntu_system_tests.helpers.file_system import move_folder_contents
+
 DEFAULT_STORE_ROOT = os.path.expanduser('~/.tmp_backup')
 
 
@@ -48,7 +50,7 @@
         shutil.rmtree(self.storage_dir.path)
         if os.path.exists(self.backup_dir):
             if self.start_clean:
-                shutil.move(self.backup_dir, self.storage_dir.path)
+                move_folder_contents(self.backup_dir, self.storage_dir.path)
             else:
                 shutil.copytree(self.backup_dir, self.storage_dir.path)
 

=== modified file 'ubuntu_system_tests/helpers/file_system.py'
--- ubuntu_system_tests/helpers/file_system.py	2015-07-24 11:13:22 +0000
+++ ubuntu_system_tests/helpers/file_system.py	2015-07-30 09:41:10 +0000
@@ -110,6 +110,25 @@
             remove_dir(obj_path)
 
 
+def move_folder_contents(src_root, dst_root):
+    """
+    Move each top-level item individually from src folder to dest folder
+    leaving the top-level folder in place. Result will be an empty src folder
+    with all items moved to dest folder.
+
+    :param src_folder: The folder containing content to move
+    :parap dst_folder: The destination folder for all content
+
+    """
+    for item in os.listdir(src_root):
+        src = os.path.join(src_root, item)
+        dst = os.path.join(dst_root, item)
+        dst_folder = os.path.dirname(dst)
+        if not os.path.exists(dst_folder):
+            os.makedirs(dst_folder)
+        shutil.move(src, dst)
+
+
 def calculate_file_sha1(file_path):
     """Calculates the sha1 digest for the file
     :param file_path: The path to the file to calculate the sha1 digest

=== added file 'ubuntu_system_tests/helpers/media.py'
--- ubuntu_system_tests/helpers/media.py	1970-01-01 00:00:00 +0000
+++ ubuntu_system_tests/helpers/media.py	2015-07-30 09:41:10 +0000
@@ -0,0 +1,67 @@
+# -*- Mode: Python; coding: utf-8; indent-tabs-mode: nil; tab-width: 4 -*-
+#
+# Ubuntu System Tests
+# Copyright (C) 2015 Canonical
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+
+import fixtures
+import os
+
+from unity8.process_helpers import start_job, stop_job, is_job_running
+
+from ubuntu_system_tests.helpers import backup_restore_fixture as brf
+from ubuntu_system_tests.helpers.file_system import get_media_folder_list
+from ubuntu_system_tests.helpers.unity8.dash import get_dash
+
+MEDIA_SCANNER = 'mediascanner-2.0'
+
+
+def restart_media_scanner():
+    """Restart the media scanner process."""
+    if is_job_running(MEDIA_SCANNER):
+        stop_job(MEDIA_SCANNER)
+    start_job(MEDIA_SCANNER)
+    dash = get_dash()
+    dash.wait_for_dash_loaded()
+    dash.wait_for_processing_to_complete()
+
+
+class BackupRestoreLocalMediaFixture(fixtures.Fixture):
+    """Generic fixture class to backup and restore all local media files."""
+
+    def __init__(self, start_clean=True):
+        """
+        :param start_clean: Whether to clean media directories after backup.
+
+        """
+        self.start_clean = start_clean
+
+    def setUp(self):
+        """Backup the required directory and register restore actions."""
+        super().setUp()
+        self._backup_local_media()
+
+    def _backup_local_media(self):
+        """Backup the local media files which are removed during testing."""
+        for folder in get_media_folder_list():
+            if folder.startswith('/media/'):
+                backup_path = os.path.join(
+                    os.path.dirname(folder), '.tmp_backup')
+            else:
+                backup_path = os.path.expanduser('~/.tmp_backup')
+            self.useFixture(brf.BackupRestoreFixture(
+                folder, storage_root_dir=backup_path,
+                start_clean=self.start_clean))

=== modified file 'ubuntu_system_tests/tests/test_mtp.py'
--- ubuntu_system_tests/tests/test_mtp.py	2015-07-23 15:03:58 +0000
+++ ubuntu_system_tests/tests/test_mtp.py	2015-07-30 09:41:10 +0000
@@ -26,6 +26,7 @@
 from ubuntu_system_tests.helpers import file_system as fs
 from ubuntu_system_tests.helpers.mtp import MTPObjFactory, MTPHelper
 from ubuntu_system_tests.helpers import ssh
+from ubuntu_system_tests.helpers import media
 from ubuntu_system_tests.helpers.scopes import music
 from ubuntu_system_tests.helpers.unity8.dash import get_dash
 
@@ -115,6 +116,11 @@
 
         return albums_dir, album_mp3_dir, album_ogg_dir
 
+    def backup_local_media(self):
+        """Backup the local media files which are removed during testing."""
+        self.useFixture(media.BackupRestoreLocalMediaFixture())
+        media.restart_media_scanner()
+
     def verify_copied_files(self, seed_file, original_files, copied_dir):
         """ Verify all the files have been copied correctly to the dst
         :param seed_file: this is the test data file that is used to generate
@@ -147,7 +153,8 @@
         albums have been copied properly to the device and the music appears
         in the music scope.
         """
-
+        # Remove any existing local media from device
+        self.backup_local_media()
         # Prepare the albums structure in the remote machine previous to be
         # copied to the device
         mp3_file = self.factory.create_file(

=== modified file 'ubuntu_system_tests/tests/test_scopes.py'
--- ubuntu_system_tests/tests/test_scopes.py	2015-07-21 21:02:07 +0000
+++ ubuntu_system_tests/tests/test_scopes.py	2015-07-30 09:41:10 +0000
@@ -26,11 +26,9 @@
 from ubuntu_system_tests.helpers import autopilot
 from ubuntu_system_tests.helpers import file_system
 from ubuntu_system_tests.helpers import scopes
-from ubuntu_system_tests.helpers.backup_restore_fixture import (
-    BackupRestoreFixture)
+from ubuntu_system_tests.helpers import media
 from ubuntu_system_tests.helpers.unity8.dash import get_dash
 
-from unity8.process_helpers import start_job, stop_job, is_job_running
 from unity8 import UnityException
 
 SAMPLE_VIDEO_FILE = 'video1.ogv'
@@ -78,7 +76,7 @@
             len(expected_scopes) > 0,
             'No scopes defined for channel: {c} device: {d}'.format(
                 c=channel, d=device))
-        self._backup_local_media()
+        self.useFixture(media.BackupRestoreLocalMediaFixture())
         self._dismiss_today_scope_setup()
         self.addCleanup(scopes.go_to_scope, expected_scopes[0].url)
 
@@ -89,45 +87,25 @@
         except UnityException:
             pass
 
-    def _backup_local_media(self):
-        """Backup the local media files which are removed during testing."""
-        for folder in file_system.get_media_folder_list():
-            if folder.startswith('/media/'):
-                backup_path = os.path.join(
-                    os.path.dirname(folder), '.tmp_backup')
-            else:
-                backup_path = os.path.expanduser('~/.tmp_backup')
-            self.useFixture(BackupRestoreFixture(
-                folder, storage_root_dir=backup_path, start_clean=True))
-            os.makedirs(folder)
-
-    def restart_media_scanner(self):
-        MEDIA_SCANNER = 'mediascanner-2.0'
-        if is_job_running(MEDIA_SCANNER):
-            stop_job(MEDIA_SCANNER)
-        start_job(MEDIA_SCANNER)
-        self.dash.wait_for_dash_loaded()
-        self.dash.wait_for_processing_to_complete()
-
     def _apply_condition_local_video(self):
         """Method to apply 'local_video' condition."""
         shutil.copy2(SRC_VIDEO_FILE, LOCAL_VIDEO_FILE)
-        self.restart_media_scanner()
+        media.restart_media_scanner()
 
     def _apply_condition_no_local_video(self):
         """Method to apply 'no_local_video' condition."""
         file_system.clean_dir(file_system.DIR_HOME_VIDEOS)
-        self.restart_media_scanner()
+        media.restart_media_scanner()
 
     def _apply_condition_local_music(self):
         """Method to apply 'local_music' condition."""
         shutil.copy2(SRC_AUDIO_FILE, LOCAL_AUDIO_FILE)
-        self.restart_media_scanner()
+        media.restart_media_scanner()
 
     def _apply_condition_no_local_music(self):
         """Method to apply 'no_local_music' condition."""
         file_system.clean_dir(file_system.DIR_HOME_MUSIC)
-        self.restart_media_scanner()
+        media.restart_media_scanner()
 
     def _apply_condition_none(self):
         """Method to apply 'none' condition."""