Merge lp:~canonical-platform-qa/ubuntu-system-tests/refine-depends-map into lp:ubuntu-system-tests

[Richard Huddie]

Refine the dependencies mapping file so that only required dependencies are used for tests being run.

Changes:
- Update the dependencies mappings for specific tests so they match the imports required.
- Replace DummyObject with DummyImportObject, which only works during test import. If a DummyImportObject is used during a test then it will raise exception. This is done to remove un-neccessary error logging during test setup, because only the minimum dependencies are now used.
- Add a self test to make sure the dependencies.json file is correctly formatted.

To test try any sequence of individual tests or groups of tests. The tests should run normally without import errors using only the dependencies required for the tests. - You may see some import warnings in the test log before the test runs, but this is expected. Once the test is running there should be no import errors shown.

[platform-qa-bot (platform-qa-bot)]

PASSED: Continuous integration, rev:437
https://platform-qa-jenkins.ubuntu.com/job/ubuntu-system-tests-ci/362/
Executed test runs:
    SUCCESS: https://platform-qa-jenkins.ubuntu.com/job/build-wily-amd64-package/2976
        deb: https://platform-qa-jenkins.ubuntu.com/job/build-wily-amd64-package/2976/artifact/work/output/*zip*/output.zip
    SUCCESS: https://platform-qa-jenkins.ubuntu.com/job/build-wily-i386-package/2976
        deb: https://platform-qa-jenkins.ubuntu.com/job/build-wily-i386-package/2976/artifact/work/output/*zip*/output.zip
    None: https://platform-qa-jenkins.ubuntu.com/job/generic-update-mp/490/console

Click here to trigger a rebuild:
https://platform-qa-jenkins.ubuntu.com/job/ubuntu-system-tests-ci/362/rebuild

review: Approve (continuous-integration)

[lp:~canonical-platform-qa/ubuntu-system-tests/refine-depends-map]

438. By Richard Huddie on 2016-08-01

Merge from trunk.

[platform-qa-bot (platform-qa-bot)]

PASSED: Continuous integration, rev:438
https://platform-qa-jenkins.ubuntu.com/job/ubuntu-system-tests-ci/365/
Executed test runs:
    SUCCESS: https://platform-qa-jenkins.ubuntu.com/job/build-wily-amd64-package/2988
        deb: https://platform-qa-jenkins.ubuntu.com/job/build-wily-amd64-package/2988/artifact/work/output/*zip*/output.zip
    SUCCESS: https://platform-qa-jenkins.ubuntu.com/job/build-wily-i386-package/2988
        deb: https://platform-qa-jenkins.ubuntu.com/job/build-wily-i386-package/2988/artifact/work/output/*zip*/output.zip
    None: https://platform-qa-jenkins.ubuntu.com/job/generic-update-mp/501/console

Click here to trigger a rebuild:
https://platform-qa-jenkins.ubuntu.com/job/ubuntu-system-tests-ci/365/rebuild

review: Approve (continuous-integration)

[lp:~canonical-platform-qa/ubuntu-system-tests/refine-depends-map]

439. By Richard Huddie on 2016-08-01

Use DummyImportObject which only works during import and remove un-needed logging.

[platform-qa-bot (platform-qa-bot)]

FAILED: Continuous integration, rev:439
https://platform-qa-jenkins.ubuntu.com/job/ubuntu-system-tests-ci/366/
Executed test runs:
    FAILURE: https://platform-qa-jenkins.ubuntu.com/job/build-wily-amd64-package/2989/console
    FAILURE: https://platform-qa-jenkins.ubuntu.com/job/build-wily-i386-package/2989/console
    None: https://platform-qa-jenkins.ubuntu.com/job/generic-update-mp/502/console

Click here to trigger a rebuild:
https://platform-qa-jenkins.ubuntu.com/job/ubuntu-system-tests-ci/366/rebuild

review: Needs Fixing (continuous-integration)

[lp:~canonical-platform-qa/ubuntu-system-tests/refine-depends-map]

440. By Richard Huddie on 2016-08-01

Fix flake8.

[platform-qa-bot (platform-qa-bot)]

PASSED: Continuous integration, rev:440
https://platform-qa-jenkins.ubuntu.com/job/ubuntu-system-tests-ci/367/
Executed test runs:
    SUCCESS: https://platform-qa-jenkins.ubuntu.com/job/build-wily-amd64-package/2990
        deb: https://platform-qa-jenkins.ubuntu.com/job/build-wily-amd64-package/2990/artifact/work/output/*zip*/output.zip
    SUCCESS: https://platform-qa-jenkins.ubuntu.com/job/build-wily-i386-package/2990
        deb: https://platform-qa-jenkins.ubuntu.com/job/build-wily-i386-package/2990/artifact/work/output/*zip*/output.zip
    None: https://platform-qa-jenkins.ubuntu.com/job/generic-update-mp/503/console

Click here to trigger a rebuild:
https://platform-qa-jenkins.ubuntu.com/job/ubuntu-system-tests-ci/367/rebuild

review: Approve (continuous-integration)

[Sergio Cazzolato (sergio-j-cazzolato)]

Code looks good to me, still testing to validate all the dependency groups.

[lp:~canonical-platform-qa/ubuntu-system-tests/refine-depends-map]

441. By Richard Huddie on 2016-08-03

Merge from trunk.

[platform-qa-bot (platform-qa-bot)]

PASSED: Continuous integration, rev:441
https://platform-qa-jenkins.ubuntu.com/job/ubuntu-system-tests-ci/375/
Executed test runs:
    SUCCESS: https://platform-qa-jenkins.ubuntu.com/job/build-wily-amd64-package/3018
        deb: https://platform-qa-jenkins.ubuntu.com/job/build-wily-amd64-package/3018/artifact/work/output/*zip*/output.zip
    SUCCESS: https://platform-qa-jenkins.ubuntu.com/job/build-wily-i386-package/3018
        deb: https://platform-qa-jenkins.ubuntu.com/job/build-wily-i386-package/3018/artifact/work/output/*zip*/output.zip
    None: https://platform-qa-jenkins.ubuntu.com/job/generic-update-mp/527/console

Click here to trigger a rebuild:
https://platform-qa-jenkins.ubuntu.com/job/ubuntu-system-tests-ci/375/rebuild

review: Approve (continuous-integration)

[Sergio Cazzolato (sergio-j-cazzolato)]

I see this error with the last build:

Unpacking lxc-android-config (0.230+15.04.20160729-0ubuntu1) over (0.230+15.04.20160602-0ubuntu1) ...
dpkg: error processing archive /var/cache/apt/archives/lxc-android-config_0.230+15.04.20160729-0ubuntu1_all.deb (--unpack):
 unable to make backup link of `./lib/udev/rules.d/70-android.rules' before installing new version: Invalid cross-device link
Processing triggers for dbus (1.8.12-1ubuntu5) ...
Processing triggers for ureadahead (0.100.0-19) ...
ureadahead will be reprofiled on next reboot
Errors were encountered while processing:
 /var/cache/apt/archives/lxc-android-config_0.230+15.04.20160729-0ubuntu1_all.deb
E: Sub-process /usr/bin/dpkg returned an error code (1)
Exit request sent.

[Richard Huddie (rhuddie)]

Just to confirm the above seems to be a problem with the current image/archive state. Same issue is reproduced with trunk version. So it should be resolved when the next image is produced.

[Sergio Cazzolato (sergio-j-cazzolato)]

> I see this error with the last build:
>
> Unpacking lxc-android-config (0.230+15.04.20160729-0ubuntu1) over
> (0.230+15.04.20160602-0ubuntu1) ...
> dpkg: error processing archive /var/cache/apt/archives/lxc-android-
> config_0.230+15.04.20160729-0ubuntu1_all.deb (--unpack):
> unable to make backup link of `./lib/udev/rules.d/70-android.rules' before
> installing new version: Invalid cross-device link
> Processing triggers for dbus (1.8.12-1ubuntu5) ...
> Processing triggers for ureadahead (0.100.0-19) ...
> ureadahead will be reprofiled on next reboot
> Errors were encountered while processing:
> /var/cache/apt/archives/lxc-android-
> config_0.230+15.04.20160729-0ubuntu1_all.deb
> E: Sub-process /usr/bin/dpkg returned an error code (1)
> Exit request sent.

As this error seems to be an archive problem, I'll approve this branch.

review: Approve

=== modified file 'debian/tests/dependencies.json'
--- debian/tests/dependencies.json	2016-07-18 16:39:49 +0000
+++ debian/tests/dependencies.json	2016-08-03 11:19:31 +0000
@@ -82,13 +82,6 @@
                 "ubuntu_system_tests"
             ],
             "depends": [
-                "address-book-app-autopilot",
-                "camera-app-autopilot",
-                "dialer-app-autopilot",
-                "exiftool",
-                "gallery-app-autopilot",
-                "mediaplayer-app-autopilot",
-                "messaging-app-autopilot",
                 "python3-autopilot",
                 "python3-dbus",
                 "python3-debian",
@@ -97,31 +90,153 @@
                 "python3-pexpect",
                 "python3-requests",
                 "python3-retrying",
-                "python3-selenium",
                 "python3-testtools",
                 "python3-warlock",
                 "subunit",
                 "ubuntu-keyboard-autopilot",
-                "ubuntuone-credentials-autopilot",
-                "ubuntu-system-settings-autopilot",
                 "ubuntu-system-tests-helpers",
                 "ubuntu-system-tests-suite",
                 "unity8-autopilot",
                 "unity-scope-click-autopilot",
-                "usbutils",
+                "usbutils"
+            ]
+        },
+        {
+            "ids": [
+                "ubuntu_system_tests.perftests.app_startup",
+                "ubuntu_system_tests.tests.test_messaging_sms",
+                "ubuntu_system_tests.tests.test_messaging_mms",
+                "ubuntu_system_tests.tests.test_calls",
+                "ubuntu_system_tests.tests.test_app_switcher",
+                "ubuntu_system_tests.tests.test_add_contact"
+            ],
+            "depends": [
+                "address-book-app-autopilot"
+            ]
+        },
+        {
+            "ids": [
+                "ubuntu_system_tests.perftests.app_startup",
+                "ubuntu_system_tests.tests.test_messaging_sms",
+                "ubuntu_system_tests.tests.test_messaging_mms",
+                "ubuntu_system_tests.tests.test_app_switcher",
+                "ubuntu_system_tests.tests.test_camera"
+            ],
+            "depends": [
+                "camera-app-autopilot"
+            ]
+        },
+        {
+            "ids": [
+                "ubuntu_system_tests.perftests.app_startup",
+                "ubuntu_system_tests.tests.test_app_switcher",
+                "ubuntu_system_tests.tests.test_calls",
+                "ubuntu_system_tests.tests.test_add_contact",
+                "ubuntu_system_tests.tests.test_with_dialer"
+            ],
+            "depends": [
+                "dialer-app-autopilot"
+            ]
+        },
+        {
+            "ids": [
+                "ubuntu_system_tests.perftests.scalability",
+                "ubuntu_system_tests.tests.test_camera"
+            ],
+            "depends": [
+                "exiftool"
+            ]
+        },
+        {
+            "ids": [
+                "ubuntu_system_tests.tests.webapps"
+            ],
+            "depends": [
+                "chromium-chromedriver",
+                "python3-selenium"
+            ]
+        },
+        {
+            "ids": [
+                "ubuntu_system_tests.perftests.scalability.test_photos",
+                "ubuntu_system_tests.perftests.app_startup",
+                "ubuntu_system_tests.tests.test_messaging_sms",
+                "ubuntu_system_tests.tests.test_messaging_mms",
+                "ubuntu_system_tests.tests.scopes.test_photos_scope",
+                "ubuntu_system_tests.tests.test_sdcard",
+                "ubuntu_system_tests.tests.test_with_webbrowser"
+            ],
+            "depends": [
+                "gallery-app-autopilot"
+            ]
+        },
+        {
+            "ids": [
+                "ubuntu_system_tests.tests.scopes.test_video_scope",
+                "ubuntu_system_tests.tests.test_calls",
+                "ubuntu_system_tests.tests.test_sdcard",
+                "ubuntu_system_tests.tests.test_camera",
+                "ubuntu_system_tests.tests.test_with_webbrowser"
+            ],
+            "depends": [
+                "mediaplayer-app-autopilot"
+            ]
+        },
+        {
+            "ids": [
+                "ubuntu_system_tests.perftests.app_startup",
+                "ubuntu_system_tests.tests.test_ubuntu_keyboard",
+                "ubuntu_system_tests.tests.test_messaging_sms",
+                "ubuntu_system_tests.tests.test_messaging_mms",
+                "ubuntu_system_tests.tests.test_app_switcher",
+                "ubuntu_system_tests.tests.test_calls",
+                "ubuntu_system_tests.tests.test_with_webbrowser"
+            ],
+            "depends": [
+                "messaging-app-autopilot"
+            ]
+        },
+        {
+            "ids": [
+                "ubuntu_system_tests.perftests.app_startup",
+                "ubuntu_system_tests.tests.test_about_this_phone",
+                "ubuntu_system_tests.tests.test_messaging_sms",
+                "ubuntu_system_tests.tests.test_messaging_mms",
+                "ubuntu_system_tests.tests.test_rotation",
+                "ubuntu_system_tests.tests.test_bluetooth",
+                "ubuntu_system_tests.tests.test_calls",
+                "ubuntu_system_tests.tests.test_brightness",
+                "ubuntu_system_tests.tests.indicators",
+                "ubuntu_system_tests.tests.test_sim_pin",
+                "ubuntu_system_tests.tests.test_system_settings"
+            ],
+            "depends": [
+                "ubuntu-system-settings-autopilot"
+            ]
+        },
+        {
+            "ids": [
+                "ubuntu_system_tests.tests.test_app_store"
+            ],
+            "depends": [
+                "ubuntuone-credentials-autopilot"
+            ]
+        },
+        {
+            "ids": [
+                "ubuntu_system_tests.perftests.app_startup",
+                "ubuntu_system_tests.tests.test_messaging_sms",
+                "ubuntu_system_tests.tests.test_messaging_mms",
+                "ubuntu_system_tests.tests.scopes.test_video_scope",
+                "ubuntu_system_tests.tests.test_calls",
+                "ubuntu_system_tests.tests.test_with_webbrowser"
+            ],
+            "depends": [
                 "webbrowser-app-autopilot"
             ]
         },
         {
             "ids": [
-                "ubuntu_system_tests.tests.webapps"
-            ],
-            "depends": [
-                "chromium-chromedriver"
-            ]
-        },
-        {
-            "ids": [
                 "unity8"
             ],
             "depends": [

=== modified file 'ubuntu_system_tests/helpers/utils.py'
--- ubuntu_system_tests/helpers/utils.py	2016-07-15 10:02:49 +0000
+++ ubuntu_system_tests/helpers/utils.py	2016-08-03 11:19:31 +0000
@@ -27,10 +27,14 @@
 
 logger = logging.getLogger(__name__)
 
+DUMMY_IMPORT_ERR = ('DummyImportObject being used because try_import failed. '
+                    'Check all required dependencies are installed.')
+
 
 def try_import(name, package=None):
     """
-    Import named module from package and return it or DummyObject if not found.
+    Import named module from package and return it or DummyImportObject if
+    not found.
 
     This method is used in cases where an import is made but should not raise
     an ImportError if the module is not found. The main usage for this is
@@ -39,22 +43,19 @@
     module must import successfully, but won't actually be executed so it is
     not neccessary to have the dependency installed.
 
-    If a DummyObject is returned this allows any attributes or methods to be
-    imported from the DummyObject without error.
+    If a DummyImportObject is returned this allows any attributes or methods
+    to be imported from the DummyImportObject without error.
 
     :param name: Absolute or relative name of module to import.
     :param package: If name is relative, package must specify the name of the
                     package to import from.
-    :return: Imported module or DummyObject if it doesn't exist.
+    :return: Imported module or DummyImportObject if it doesn't exist.
 
     """
     try:
         return import_module(name, package)
-    except ImportError as e:
-        logger.warn(e.msg)
-        logger.warn('DummyObject created for module: {m} at: {i}'.format(
-            m=name, i=_get_caller_info()))
-        return DummyObject()
+    except ImportError:
+        return DummyImportObject()
 
 
 def wait_until(predicate, *args, timeout=10, period=1, stop_event=None,
@@ -139,14 +140,14 @@
         return bool(result)
 
 
-def _get_caller_info(stack_level=2):
-    """Return info string about the caller of the calling function.
-
-    :param stack_level: Number of level up the call stack to check.
-    :return: String info showing file path and line number.
-    """
-    caller = stack()[stack_level]
-    return '{path}:{line}'.format(path=caller[1], line=caller[2])
+def _is_import():
+    """Return True if being imported during test discovery."""
+    stack_trace = stack()
+    stack_trace.reverse()
+    for frame in stack_trace:
+        if frame[3] == 'load_test_suite_from_name':
+            return True
+    return False
 
 
 class AttrDict(dict):
@@ -157,13 +158,14 @@
         self.__dict__ = self
 
 
-class DummyObject:
+class DummyImportObject:
     """
-    Class used to create objects with dummy attributes and methods.
+    Class used to create objects with dummy attributes and methods that can
+    be accessed during import operations only.
 
     This is used when an object is required to return a dummy attribute value
-    or method result rather than raise a AttributeError or TypeError
-    exception.
+    or method result during import rather than raise a AttributeError or
+    TypeError exception.
 
     This class can be used to create objects that support nested attributes
     and methods.
@@ -176,12 +178,13 @@
     def __getattribute__(self, name):
         """Return a refernce to self for every attribute value to support
         nested attributes."""
-        logger.warn('Accessing DummyObject attribute: {n} from: {f}'.format(
-            n=name, f=_get_caller_info()))
-        return self
+        if _is_import():
+            return self
+        raise AttributeError(DUMMY_IMPORT_ERR)
 
     def __call__(self, *args, **kwargs):
         """Return a reference to self for every function call to support
         nested function calls."""
-        logger.warn('DummyObject call from: {}'.format(_get_caller_info()))
-        return self
+        if _is_import():
+            return self
+        raise AttributeError(DUMMY_IMPORT_ERR)

=== modified file 'ubuntu_system_tests/selftests/test_debian.py'
--- ubuntu_system_tests/selftests/test_debian.py	2016-07-21 08:34:46 +0000
+++ ubuntu_system_tests/selftests/test_debian.py	2016-08-03 11:19:31 +0000
@@ -18,7 +18,9 @@
 # along with this program.  If not, see <http://www.gnu.org/licenses/>.
 #
 
+import unittest
 import json
+import os
 import testscenarios
 
 from fixtures import TempDir
@@ -340,3 +342,11 @@
         dep = {'id': 'dep1', 'condition': 'installed', 'tmpinstall': 'true'}
         self.assertFalse(debian.is_dependency_required(dep, tmp_install=False))
         mock_is_installed.assert_not_called()
+
+
+class ReadDependenciesJSONTestCase(unittest.TestCase):
+
+    def test_dependencies_json_file_valid(self):
+        deps_path = os.path.realpath('debian/tests/dependencies.json')
+        with open(deps_path) as deps_file:
+            json.loads(deps_file.read())