Merge lp:~om26er/ubuntu-system-image/add_upgrade_test into lp:~registry/ubuntu-system-image/client

[Omer Akram]

Add a test runner that flashes the touch device and then runs the upgrade test on the Device under test.

Add a test runner that flashes the touch device and then runs the upgrade test on the Device under test.

Needs packaging work so that we have a separate package system-image-autopilot ?

depends on: lp:~om26er/ubuntu-system-settings/upgrade_testing_prerequisite

For Barry regarding packaging, the newly created test package that you create out of these tests should depend on.
  * ubuntu-system-settings-autopilot
  * unity8-autopilot

[lp:~om26er/ubuntu-system-image/add_upgrade_test]

259. By Omer Akram on 2014-04-09

remove hard-coded package list

Unmerged revisions

259. By Omer Akram on 2014-04-09

remove hard-coded package list

258. By Omer Akram on 2014-04-09

add docstrings to elaborate helpers

257. By Omer Akram on 2014-04-09

update header, arrange import

256. By Omer Akram on 2014-04-09

minor cleaning, make some methods private, remove unsued variables

255. By Omer Akram on 2014-04-09

don't try to push tests to the device if the test runner is installed system-wide

254. By Omer Akram on 2014-04-09

fix pep8

253. By Omer Akram on 2014-04-09

adapt to helper class changes

252. By Omer Akram on 2014-04-09

import update helpers from system-settings

251. By Omer Akram on 2014-04-09

clean code a bit

250. By Omer Akram on 2014-04-08

bring in upgrade tests.

=== added directory 'systemimage/tests/autopilot'
=== added directory 'systemimage/tests/autopilot/system_upgrade'
=== added file 'systemimage/tests/autopilot/system_upgrade/__init__.py'
--- systemimage/tests/autopilot/system_upgrade/__init__.py	1970-01-01 00:00:00 +0000
+++ systemimage/tests/autopilot/system_upgrade/__init__.py	2014-04-09 20:44:26 +0000
@@ -0,0 +1,6 @@
+# -*- Mode: Python; coding: utf-8; indent-tabs-mode: nil; tab-width: 4 -*-
+# Copyright 2014 Canonical
+#
+# This program is free software: you can redistribute it and/or modify it
+# under the terms of the GNU General Public License version 3, as published
+# by the Free Software Foundation.

=== added directory 'systemimage/tests/autopilot/system_upgrade/helpers'
=== added file 'systemimage/tests/autopilot/system_upgrade/helpers/__init__.py'
--- systemimage/tests/autopilot/system_upgrade/helpers/__init__.py	1970-01-01 00:00:00 +0000
+++ systemimage/tests/autopilot/system_upgrade/helpers/__init__.py	2014-04-09 20:44:26 +0000
@@ -0,0 +1,6 @@
+# -*- Mode: Python; coding: utf-8; indent-tabs-mode: nil; tab-width: 4 -*-
+# Copyright 2014 Canonical
+#
+# This program is free software: you can redistribute it and/or modify it
+# under the terms of the GNU General Public License version 3, as published
+# by the Free Software Foundation.

=== added file 'systemimage/tests/autopilot/system_upgrade/helpers/system_image_setup.py'
--- systemimage/tests/autopilot/system_upgrade/helpers/system_image_setup.py	1970-01-01 00:00:00 +0000
+++ systemimage/tests/autopilot/system_upgrade/helpers/system_image_setup.py	2014-04-09 20:44:26 +0000
@@ -0,0 +1,167 @@
+# -*- Mode: Python; coding: utf-8; indent-tabs-mode: nil; tab-width: 4 -*-
+# Copyright 2014 Canonical
+#
+# This program is free software: you can redistribute it and/or modify it
+# under the terms of the GNU General Public License version 3, as published
+# by the Free Software Foundation.
+
+import logging
+import subprocess
+import time
+
+logger = logging.getLogger(__name__)
+
+
+class DeviceImageFlash(object):
+
+    def setup_device(self, ppa=None, package_list=None, **kwargs):
+        """Sets up the device for upgrade testing
+
+        :param ppa: A launchpad ppa to add to the DUT before
+        apt-get update.
+
+        :param package_list: Names of the packages to be installed
+        on the DUT before running tests.
+
+        :param channel: Ubuntu system image update channel like
+        ubuntu-touch/devel.
+
+        :param revision_number: System image revision number.
+
+        :param bootstrap: clear everything on DUT before flashing
+        the image.
+
+        """
+        self.flash_device(**kwargs)
+        self.wait_for_device()
+        self.set_device_read_write()
+        self.reboot_device()
+        self.disable_intro_screen()
+        self.reboot_device()
+
+        self.setup_network()
+
+        if ppa is not None:
+            self.add_apt_repository(ppa)
+
+        self.apt_get_update()
+
+        if package_list is not None:
+            self.apt_get_install(package_list)
+
+    def str_to_lst(self, string):
+        result = []
+        for char in string.split():
+            result.append(char)
+
+        return result
+
+    def run_command_on_host(self, command, str_to_list=True, **kwargs):
+        logger.info('Running command: {}'.format(command))
+        if str_to_list is True:
+            command = self.str_to_lst(command)
+
+        subprocess.call(command, **kwargs)
+
+    def flash_device(
+            self,
+            channel='ubuntu-touch/trusty-proposed',
+            revision_number=None,
+            bootstrap=False
+            ):
+
+        command = []
+        command.append('phablet-flash')
+        command.append('ubuntu-system')
+        command.append('--serial')
+        command.append(self.dut_serial)
+        command.append('--channel')
+        command.append(channel)
+        if revision_number is not None:
+            command.append('--revision')
+            command.append(revision_number)
+        if bootstrap is True:
+            command.append('--bootstrap')
+
+        self.run_command_on_host(command, str_to_list=False)
+
+    def run_command_on_target(self, command, user=None, shell=False):
+        if user == 'phablet':
+            command = 'adb -s {} shell sudo -iu {} bash -ic "{}"'.format(
+                self.dut_serial, user, command
+            )
+            self.run_command_on_host(command)
+        elif user is None:
+            command = 'adb -s {} shell {}'.format(self.dut_serial, command)
+            self.run_command_on_host(command)
+
+    def wait_for_device(self, timeout=60):
+        """wait for the DUT to restart
+
+        :param timeout: number of seconds to wait for the system
+        to settle.
+        """
+        command = 'adb -s {} wait-for-device'.format(self.dut_serial)
+        self.run_command_on_host(command)
+        logger.info(
+            'Waiting {} seconds for the device to settle before '
+            'proceeding further.'.format(timeout)
+        )
+        time.sleep(timeout)
+        logger.info('Just to be sure')
+        self.run_command_on_host(command)
+
+    def set_device_read_write(self):
+        command = 'touch /userdata/.writable_image'
+        logger.info('Changing system to read/write.')
+        self.run_command_on_target(command)
+
+    def reboot_device(self):
+        wait = 10
+        self.run_command_on_target('reboot')
+        logger.info(
+            'Wating {} seconds to ensure target goes away from adb before '
+            'waiting for it to come back'.format(wait)
+        )
+        time.sleep(wait)
+        self.wait_for_device()
+
+    def disable_intro_screen(self):
+        self.run_command_on_target(
+            'dbus-send --system --print-reply '
+            '--dest=org.freedesktop.Accounts '
+            '/org/freedesktop/Accounts/User32011 '
+            'org.freedesktop.DBus.Properties.Set '
+            'string:com.canonical.unity.AccountsService '
+            'string:demo-edges variant:boolean:false')
+        self.run_command_on_target('touch /.intro_off')
+
+    def setup_network(self):
+        if self.running_in_ci:
+            command = ('adb -s {} shell nmcli dev wifi '
+                       'connect ubuntu-qa-g-wpa-d password '
+                       'qalabwireless'.format(self.dut_serial))
+            logger.info('Setting up network on target.')
+            self.run_command_on_host(command)
+        else:
+            command = 'phablet-network -s {}'.format(self.dut_serial)
+            logger.warning(
+                'Setting up network on target. you may need to '
+                'enter your password for "{}" to complete.'.format(command)
+            )
+            self.run_command_on_host(command)
+
+    def add_apt_repository(self, ppa):
+        command = 'add-apt-repository -y {}'.format(ppa)
+        logger.info('Adding ppa {} on target.'.format(ppa))
+        self.run_command_on_target(command)
+
+    def apt_get_update(self):
+        command = 'apt-get update'
+        logger.info('Updating apt database.')
+        self.run_command_on_target(command)
+
+    def apt_get_install(self, packages):
+        command = 'apt-get -y -f --force-yes install {}'.format(packages)
+        logger.info('Installing required packages on target.')
+        self.run_command_on_target(command)

=== added file 'systemimage/tests/autopilot/system_upgrade/helpers/unlock_screen.py'
--- systemimage/tests/autopilot/system_upgrade/helpers/unlock_screen.py	1970-01-01 00:00:00 +0000
+++ systemimage/tests/autopilot/system_upgrade/helpers/unlock_screen.py	2014-04-09 20:44:26 +0000
@@ -0,0 +1,55 @@
+#!/usr/bin/env python
+
+import logging
+import sys
+
+from unity8 import process_helpers as helpers
+
+import dbus
+
+logging.basicConfig(level=logging.INFO)
+
+
+class UnlockScreen(object):
+
+    def __init__(self):
+        self.powerd = self._get_powerd_interface()
+
+    def _get_powerd_interface(self):
+        bus = dbus.SystemBus()
+        return bus.get_object(
+            'com.canonical.powerd', '/com/canonical/powerd'
+        )
+
+    def _powerd_request_state_active(self):
+        logging.info('Setting system state "Active"')
+        return self.powerd.requestSysState(
+            'autopilot-lock', 1, dbus_interface='com.canonical.powerd'
+        )
+
+    def _powerd_release_state_active(self):
+        logging.info('Releasing system state "Active"')
+        self.powerd.clearSysState(
+            self.active_cookie, dbus_interface='com.canonical.powerd'
+        )
+
+    def restart_with_testability(self):
+        self.active_cookie = self._powerd_request_state_active()
+        helpers.restart_unity_with_testability()
+        self._powerd_release_state_active()
+
+    def unlock_screen(self):
+        self.restart_with_testability()
+        helpers.unlock_unity()
+
+
+def help():
+    print("Usage:")
+    print("Run the script without any argument to unlock with assertion.")
+
+
+if len(sys.argv) >= 2 and sys.argv[1] == '-h':
+    help()
+else:
+    unlocker = UnlockScreen()
+    unlocker.unlock_screen()

=== added directory 'systemimage/tests/autopilot/system_upgrade/tests'
=== added file 'systemimage/tests/autopilot/system_upgrade/tests/__init__.py'
=== added file 'systemimage/tests/autopilot/system_upgrade/tests/test_system_updates.py'
--- systemimage/tests/autopilot/system_upgrade/tests/test_system_updates.py	1970-01-01 00:00:00 +0000
+++ systemimage/tests/autopilot/system_upgrade/tests/test_system_updates.py	2014-04-09 20:44:26 +0000
@@ -0,0 +1,46 @@
+# -*- Mode: Python; coding: utf-8; indent-tabs-mode: nil; tab-width: 4 -*-
+# Copyright 2014 Canonical
+#
+# This program is free software: you can redistribute it and/or modify it
+# under the terms of the GNU General Public License version 3, as published
+# by the Free Software Foundation.
+
+import logging
+
+from autopilot.matchers import Eventually
+from testtools.matchers import Equals
+
+from ubuntu_system_settings.emulators import UpdatesPage
+from ubuntu_system_settings.tests import SystemUpdatesBaseTestCase
+
+logging.basicConfig(level=logging.INFO)
+
+
+class SystemUpdatesTestCases(SystemUpdatesBaseTestCase):
+
+    """Tests for System Updates."""
+
+    def setUp(self):
+        self.patch_environment('IGNORE_CREDENTIALS', 'True')
+        super(SystemUpdatesTestCases, self).setUp()
+
+    @property
+    def updates_page(self):
+        return UpdatesPage(self.app, self.pointer)
+
+    def test_image_update_from_ui_works(self):
+        """Install latest system update available."""
+        self.updates_page.wait_for_state('UPDATE')
+        logging.info('Waiting for update to download.')
+        self.updates_page.wait_for_updates_to_download()
+        logging.info('finished waiting for update')
+        self.updates_page.click_install_and_restart_button()
+        installing_screen = self.updates_page.get_installing_update_screen()
+        self.assertThat(
+            installing_screen.visible, Eventually(Equals(True))
+        )
+
+    def test_state_noupdates(self):
+        """Check if system is fully updated."""
+        state = self.updates_page.wait_for_state('NOUPDATES')
+        self.assertThat(state, Equals('NOUPDATES'))

=== added file 'systemimage/tests/autopilot/system_upgrade/variables.py'
--- systemimage/tests/autopilot/system_upgrade/variables.py	1970-01-01 00:00:00 +0000
+++ systemimage/tests/autopilot/system_upgrade/variables.py	2014-04-09 20:44:26 +0000
@@ -0,0 +1,14 @@
+# -*- Mode: Python; coding: utf-8; indent-tabs-mode: nil; tab-width: 4 -*-
+# Copyright 2014 Canonical
+#
+# This program is free software: you can redistribute it and/or modify it
+# under the terms of the GNU General Public License version 3, as published
+# by the Free Software Foundation.
+
+
+TEST1_PART1 = 'system_upgrade.tests.test_system_updates.SystemUpdatesTestCases\
+.test_image_update_from_ui_works'
+
+
+TEST1_PART2 = 'system_upgrade.tests.test_system_updates.SystemUpdatesTestCases\
+.test_state_noupdates'

=== added file 'systemimage/tests/autopilot/upgrade_test_runner'
--- systemimage/tests/autopilot/upgrade_test_runner	1970-01-01 00:00:00 +0000
+++ systemimage/tests/autopilot/upgrade_test_runner	2014-04-09 20:44:26 +0000
@@ -0,0 +1,142 @@
+#!/usr/bin/python3
+
+import argparse
+import logging
+import os
+import subprocess
+import sys
+import tempfile
+import time
+import unittest
+
+from system_upgrade.helpers.system_image_setup import DeviceImageFlash
+from system_upgrade.variables import TEST1_PART1, TEST1_PART2
+
+
+class SystemImageUpgrader(unittest.TestCase, DeviceImageFlash):
+
+    source = 'system_upgrade'
+    test_suite = 'system_upgrade'
+    device_test_path = '/home/phablet/{}'.format(test_suite)
+    unlocker = 'unlock_screen.py'
+    unlocker_location = 'system_upgrade/helpers/' + unlocker
+
+    def setUp(self):
+        self.dut_serial = arguments.serial
+        self.running_in_ci = arguments.ci
+        self.setup_device(
+            ppa=arguments.ppa,
+            package_list=arguments.packages,
+            bootstrap=True,
+            revision_number='-1'
+        )
+
+    def test_system_update(self):
+        if self.installed_system_wide():
+            self.run_tests(test_suite=TEST1_PART1)
+        else:
+            self.push_and_run_tests(test_suite=TEST1_PART1)
+        log_file = self.open_log_file()
+        self.assertTrue(log_file.endswith('OK\n'))
+
+        self.wait_for_device(timeout=150)
+
+        self.run_tests(test_suite=TEST1_PART2)
+        log_file = self.open_log_file()
+        self.assertTrue(log_file.endswith('OK\n'))
+
+    def open_log_file(self, retries=10):
+        for retry in range(retries):
+            if os.path.exists(self.result_log):
+                break
+            time.sleep(1)
+            retry = retry - 1
+        else:
+            raise IOError('file {} not found'.format(self.result_log))
+
+        log_file = open(self.result_log, 'r')
+        self.addCleanup(log_file.close)
+        return log_file.read()
+
+    def push_and_run_tests(self, test_suite):
+        self._push_files_to_device(self.source, self.device_test_path)
+        self.run_tests(test_suite)
+
+    def run_tests(self, test_suite):
+        self._unlock_device_screen()
+        self._run_autopilot_test_on_target(test_suite)
+
+    def _push_files_to_device(self, source, location):
+        self.addCleanup(self._delete_autopilot_test)
+        command = 'adb -s {} push {} {}'.format(
+            self.dut_serial, source, location
+        )
+
+        logger.info('Pushing files to target with command:')
+        logger.info(command)
+        self.run_command_on_host(command)
+
+    def _delete_autopilot_test(self):
+        command = 'rm -rf {}'.format(self.device_test_path)
+        self.run_command_on_target(command)
+
+    def _unlock_device_screen(self):
+        command = 'python3 -m system_upgrade.helpers.unlock_screen'
+        self.run_command_on_target(command, user='phablet')
+
+    def _run_autopilot_test_on_target(self, test_suite, user='phablet'):
+        self.result_log = tempfile.mkstemp()[1]
+        autopilot_log_target = '/tmp/upgrade.log'
+        test_command = \
+            "autopilot run -f text -o {} -v {}".format(
+                autopilot_log_target, test_suite)
+        command = 'adb -s {} shell sudo -iu {} bash -ic "{}"'.format(
+            self.dut_serial, user, test_command)
+
+        subprocess.call(command, shell=True)
+
+        self.run_command_on_host('adb -s {} pull {} {}'.format(
+            self.dut_serial, autopilot_log_target, self.result_log)
+        )
+        self.addCleanup(os.remove, self.result_log)
+
+    def installed_system_wide(self):
+        """Returns true if the test runner is installed in /usr
+        or any of its subdirectories.
+        """
+        return os.path.abspath('.').startswith('/usr')
+
+
+def _parse_command_line_arguments():
+    parser = argparse.ArgumentParser(
+        description='Ubuntu system update test runner'
+    )
+    parser.add_argument(
+        'serial', metavar='SERIAL_NUMBER', type=str,
+        help='Serial numer of the device to run tests on.'
+    )
+    parser.add_argument(
+        '--ppa', type=str,
+        help='Launchpad ppa to add to the device before installing '
+             'test packages. example: ppa:test/ppa'
+    )
+    parser.add_argument(
+        '--packages', type=str,
+        help='Names of extra packages to be installed on the test device '
+             'before running the tests.'
+    )
+    parser.add_argument(
+        '--ci', metavar='BOOLEAN', default=False,
+        help='Specify whether the test suite is running in the Canonical '
+             'CI lab so that the network connection could be established '
+             'with the appropriate Wi-Fi network.'
+    )
+
+    return parser.parse_args()
+
+
+if __name__ == '__main__':
+    arguments = _parse_command_line_arguments()
+    logging.basicConfig(level=logging.INFO)
+    logger = logging.getLogger(__name__)
+    unittest.main(argv=[sys.argv[0]])