Merge lp:~canonical-platform-qa/ubuntu-test-cases/dep8-app-startup into lp:ubuntu-test-cases/touch

[Leo Arias]

Added dep8 tests for app startup.

There is room for a lot of improvements, but the goal of this branch was to copy the old utah test into dep8. More changes will come later.

[Christopher Lee (veebers)]

Looking good so far.

[lp:~canonical-platform-qa/ubuntu-test-cases/dep8-app-startup]

365. By Leo Arias on 2014-11-13

Fixed the README command.

366. By Leo Arias on 2014-11-13

Fixed the README command.

367. By Leo Arias on 2014-11-13

Moved the setup to a test.

368. By Leo Arias on 2014-11-13

Fixed the call to datetime.

[Ted Gould (ted)]

You won't get the start message callback events doing this because you haven't send the environment variable before unity starts. Which is required for the version of libual that's loaded into its memory region. That tracepoint is important to know how long Unity is delaying the startup of apps.

[Allan LeSage (allanlesage)]

I'm seeing that babeltrace isn't available on the phone image? http://pastebin.ubuntu.com/9009103/

review: Needs Fixing

[Leo Arias (elopio)]

Allan, that's weird. This is from my krillin:

phablet@ubuntu-phablet:~$ apt-cache search babeltrace
libbabeltrace-ctf-dev - Common Trace Format (CTF) development files
libbabeltrace-ctf1 - Common Trace Format (CTF) library
libbabeltrace-dev - Babeltrace development files
libbabeltrace1 - Babeltrace conversion libraries
babeltrace - Trace conversion program
python3-babeltrace - Babeltrace conversion libraries

In what phone and version are you running it?

[Allan LeSage (allanlesage)]

I'm on mako, maybe that explains?

phablet@ubuntu-phablet:~$ apt-cache search babeltrace
phablet@ubuntu-phablet:~$

[Leo Arias (elopio)]

> You won't get the start message callback events doing this because you haven't
> send the environment variable before unity starts. Which is required for the
> version of libual that's loaded into its memory region. That tracepoint is
> important to know how long Unity is delaying the startup of apps.

<elopio> ted: thanks for looking at it. But didn't you tell me that if after writing the config file I rebooted the machine, the right env vars will be set?
<ted> elopio, Ah, okay. Was looking at app-startup. Why is that script setting them?
<elopio> ted: I'm not sure. I thought something might read the env vars and not the initctl vars.
<elopio> maybe it's not needed and I can remove it.

Unmerged revisions

368. By Leo Arias on 2014-11-13

Fixed the call to datetime.

367. By Leo Arias on 2014-11-13

Moved the setup to a test.

366. By Leo Arias on 2014-11-13

Fixed the README command.

365. By Leo Arias on 2014-11-13

Fixed the README command.

364. By Leo Arias on 2014-11-13

Copied the generator script from veebers.

363. By Leo Arias on 2014-11-13

Added a backlist.

362. By Leo Arias on 2014-11-13

Added a timetout to the app launch.

361. By Leo Arias on 2014-11-13

Print the unittests results to stdout.

360. By Leo Arias on 2014-11-13

Moved the root commands to setup.

359. By Leo Arias on 2014-11-13

Checkpoing for the debian packages working.

=== added directory 'tests/app-startup'
=== added file 'tests/app-startup/README.md'
--- tests/app-startup/README.md	1970-01-01 00:00:00 +0000
+++ tests/app-startup/README.md	2014-11-13 18:49:53 +0000
@@ -0,0 +1,37 @@
+# Ubuntu applications startup test
+
+This test launches the applications and measure the time it takes for them to
+be fully opened, by listening to the lttng traces.
+
+## Resources
+
+This test should be run on an Ubuntu Touch device.
+
+## To run the test
+
+1. Flash an Ubuntu Touch device.
+2. Enable adb access on the device.
+3. Get the test code:
+
+    $ bzr branch lp:ubuntu-test-cases/touch
+
+4. Connect the device to the runner machine with an USB cable.
+5. Run the tests with adt-run:
+
+    $ adt-run -B --built-tree=touch/tests/app-startup --setup-commands=touch/tests/app-startup/debian/tests/setup --output-dir=output --- ssh -s adb -- -p <device password>
+
+## Test results
+
+The test executable exits with 0 if successful, non-zero otherwise.
+
+## Non-functional stats data
+
+The NFSS json data file will be stored at the output/artifacts/ directory.
+
+## Triggering
+
+This test should be run every time there is a new Ubuntu Touch image.
+
+## Monitoring
+
+work in progress.
 
=== added directory 'tests/app-startup/debian'
=== added file 'tests/app-startup/debian/changelog'
--- tests/app-startup/debian/changelog	1970-01-01 00:00:00 +0000
+++ tests/app-startup/debian/changelog	2014-11-13 18:49:53 +0000
@@ -0,0 +1,5 @@
+app-startup-tests (0.1-1) unstable; urgency=low
+
+  * Initial release.
+
+ -- Leonardo Arias Fonseca <[e-mail]>  Wed, 12 Nov 2014 09:44:55 -0600
 
=== added directory 'tests/app-startup/debian/tests'
=== added file 'tests/app-startup/debian/tests/app-startup'
--- tests/app-startup/debian/tests/app-startup	1970-01-01 00:00:00 +0000
+++ tests/app-startup/debian/tests/app-startup	2014-11-13 18:49:53 +0000
@@ -0,0 +1,18 @@
+#!/bin/sh
+
+
+# XXX ask why is there a first run error. --elopio - 2014-11-12
+echo "Creating dummy session ignoring the error to get around first run error."
+
+lttng create dummy 2> /dev/null || echo 'error'
+lttng destroy dummy
+
+echo "Setting environment variables for tracing."
+export LTTNG_UST_REGISTER_TIMEOUT=-1
+export UBUNTU_APP_LAUNCH_LTTNG_ENABLED=TRUE
+
+# Run the tests and generate the lttng files.
+python3 debian/tests/test_app_startup.py
+
+# Parse the lttng files and save the restuls to the nfss data file.
+python3 debian/tests/app-startup-nfss-datafile-generator.py $ADT_ARTIFACTS/lttng_traces $ADT_ARTIFACTS
 
=== added file 'tests/app-startup/debian/tests/app-startup-nfss-datafile-generator.py'
--- tests/app-startup/debian/tests/app-startup-nfss-datafile-generator.py	1970-01-01 00:00:00 +0000
+++ tests/app-startup/debian/tests/app-startup-nfss-datafile-generator.py	2014-11-13 18:49:53 +0000
@@ -0,0 +1,175 @@
+#!/usr/bin/env python3
+
+# Ubuntu Test Cases
+# Copyright 2014 Canonical Ltd.
+#
+# This program is free software: you can redistribute it and/or modify it
+# under the terms of the GNU Affero General Public License version 3, as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranties of
+# MERCHANTABILITY, SATISFACTORY QUALITY, or FITNESS FOR A PARTICULAR
+# PURPOSE.  See the GNU Affero General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+"""Export application trace events for app startup."""
+
+import datetime
+import json
+import os
+import re
+import subprocess
+import sys
+
+import babeltrace
+
+
+def usage():
+    print(
+        'Usage:\n {} <source directory> <destination directory>'.format(
+            sys.argv[0]
+        )
+    )
+
+
+def main(source_directory, destination_directory):
+    """source_directory contains all the resulting directories from the test
+    run (i.e. com.ubuntu.calculator-startup-20141030-021839)
+
+    destination_directory is where the resulting nfss-data.json file will be
+    written.
+
+    """
+    all_applications_data = []
+    for directory_name in os.listdir(source_directory):
+        full_directory_path = os.path.abspath(
+            os.path.join(source_directory , directory_name)
+        )
+        package_name = _get_package_name(directory_name)
+        app_id = _get_app_id(full_directory_path)
+
+        traces_path = os.path.join(
+            full_directory_path, app_id, 'ust/uid/32011/32-bit')
+        tracepoints = parse_traces(traces_path),
+        doc = generate_doc(tracepoints, app_id, package_name)
+        all_applications_data.append(dict(
+            project='app-startup',
+            test=package_name,
+            data=doc
+        ))
+
+    if device_is_krillin():
+        data_file_name = 'private-nfss-data.json'
+    else:
+        data_file_name = 'nfss-data.json'
+
+    dest_filepath = os.path.join(destination_directory, data_file_name)
+    print("Writing results to %s" % (dest_filepath))
+    with open(dest_filepath, "w") as f:
+        json.dump(all_applications_data, f)
+
+
+def parse_traces(trace_path):
+    """Convert binary traces to dict of tracepoint names and timestamps.
+
+    Timestamp format is: epoch secs + nanosecs.
+
+    """
+    # Tracepoints to include for app startup
+    tracepoints = {
+        'libual_start': {},
+        'libual_start_message_sent': {},
+        'libual_start_message_callback': {},
+        'handshake_wait': {},
+        'handshake_complete': {},
+        'exec_pre_exec': {},
+    }
+
+    # Build up the traces collection
+    traces = babeltrace.TraceCollection()
+    ret = traces.add_trace(trace_path, 'ctf')
+    if ret is None:
+        raise IOError('Error adding trace')
+
+    # Record the timestamps
+    timestamps = {}
+    for event in traces.events:
+        name = event.name.split(':')[1]
+        # We only want the event if it's the first occurance to
+        # handle possibly messy traces.
+        if name not in timestamps:
+            # Keep only the timestamps we care about by event name
+            if name in tracepoints:
+                if len(tracepoints[name]) == 0:
+                    timestamps[name] = event.timestamp
+                else:
+                    fieldname = name
+                    for field in tracepoints[name]:
+                        if not event[field] == tracepoints[name][field]:
+                            fieldname = None
+                            break
+                        else:
+                            fieldname += ":{}={}".format(
+                                field, tracepoints[name][field])
+
+                    if fieldname is not None and fieldname not in timestamps:
+                        timestamps[fieldname] = event.timestamp
+
+    return timestamps
+
+
+def generate_doc(tracepoints, app_id, package_name):
+    return dict(
+        package_name=package_name,
+        app_id=app_id,
+        android_serial=os.environ.get('ANDROID_SERIAL'),
+        image_type=os.environ.get('IMAGE_TYPE'),
+        build_num=os.environ.get('BUILD_NUMBER'),
+        build_id=os.environ.get('BUILD_ID'),
+        job_name=os.environ.get('JOB_NAME'),
+        node_name=os.environ.get('NODE_NAME'),
+        datetime=datetime.datetime.utcnow().isoformat(),
+        tracepoints=tracepoints,
+    )
+
+
+def _get_package_name(directory):
+    # com.ubuntu.calculator-startup-20141030-021839 -> com.ubuntu.calculator
+    directory_name = os.path.basename(directory)
+    package_name_search = re.search(
+        '(.*)-startup.*',
+        directory_name
+    )
+    try:
+        return package_name_search.group(1)
+    except IndexError:
+        return "Unknown Package"
+
+
+def _get_app_id(directory):
+    return os.listdir(directory)[0]
+
+
+def device_is_krillin():
+    """Returns True if the device executing this code is a Krillin."""
+    try:
+        prop_output = subprocess.check_output(
+            ['getprop', 'ro.product.device'],
+            universal_newlines=True
+        ).strip('\n')
+        return prop_output == 'krillin'
+    except FileNotFoundError:
+        return False
+
+
+if __name__ == '__main__':
+    try:
+        source_directory, destination_directory = sys.argv[1:]
+    except (IndexError, ValueError):
+        usage()
+        exit(1)
+
+    main(source_directory, destination_directory)
 
=== added file 'tests/app-startup/debian/tests/app-startup-root-setup'
--- tests/app-startup/debian/tests/app-startup-root-setup	1970-01-01 00:00:00 +0000
+++ tests/app-startup/debian/tests/app-startup-root-setup	2014-11-13 18:49:53 +0000
@@ -0,0 +1,33 @@
+#!/bin/sh
+
+set -e
+
+case "$ADT_REBOOT_MARK" in
+  "") echo "Beginning test setup.";
+
+echo "Allowing LTTng events from confined applications."
+sed 's/deny \(.*shm\/lttng-ust-.*\)/owner @{HOME}\/.lttng\/ rw,\nowner @{HOME}\/.lttng\/\* rwk,\n\/run\/lttng\/ rw,\n\/run\/lttng\/** rwk,/' -i /usr/share/apparmor/easyprof/templates/ubuntu/*/ubuntu-*
+
+echo "Regenerating AppArmor profiles for installed click applications."
+aa-clickhook -f
+
+echo "Setting environment variables for session."
+mkdir -p /home/$SUDO_USER/.config/upstart
+cat > /home/$SUDO_USER/.config/upstart/lttng-trace.conf <<EOF
+start on starting dbus
+script
+initctl set-env --global LTTNG_UST_REGISTER_TIMEOUT=-1
+initctl set-env --global UBUNTU_APP_LAUNCH_LTTNG_ENABLED=TRUE
+end script
+EOF
+
+chown $SUDO_USER:$SUDO_USER /home/$SUDO_USER/.config/upstart/lttng-trace.conf
+
+echo "Rebooting."
+/tmp/autopkgtest-reboot rebooted ;;
+
+  rebooted) echo "Rebooted." ;;
+
+esac
+
+echo "Setup ended."
 
=== added file 'tests/app-startup/debian/tests/control'
--- tests/app-startup/debian/tests/control	1970-01-01 00:00:00 +0000
+++ tests/app-startup/debian/tests/control	2014-11-13 18:49:53 +0000
@@ -0,0 +1,11 @@
+Tests: app-startup-root-setup
+Restrictions: needs-root
+
+Tests: app-startup
+Depends: babeltrace,
+         lttng-tools,
+         python3-apt,
+         python3-babeltrace,
+         python3-fixtures,
+         python3-testscenarios,
+         python3-testtools
 
=== added file 'tests/app-startup/debian/tests/test_app_startup.py'
--- tests/app-startup/debian/tests/test_app_startup.py	1970-01-01 00:00:00 +0000
+++ tests/app-startup/debian/tests/test_app_startup.py	2014-11-13 18:49:53 +0000
@@ -0,0 +1,145 @@
+#!/usr/bin/env python3
+
+# Ubuntu Test Cases
+# Copyright 2014 Canonical Ltd.
+#
+# This program is free software: you can redistribute it and/or modify it
+# under the terms of the GNU Affero General Public License version 3, as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranties of
+# MERCHANTABILITY, SATISFACTORY QUALITY, or FITNESS FOR A PARTICULAR
+# PURPOSE.  See the GNU Affero General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+import datetime
+import os
+import subprocess
+import sys
+import time
+import unittest
+
+import apt
+import fixtures
+import testscenarios
+import testtools
+from gi.repository import Click
+
+
+debian_packages = [
+    'address-book-app', 'dialer-app', 'messaging-app',
+    'ubuntu-system-settings', 'webbrowser-app'
+]
+black_list = [
+    'com.canonical.payui', 'com.ubuntu.reminders',
+    'com.ubuntu.scopes.youtube'
+]
+
+
+def get_click_packages():
+    click_db = Click.DB()
+    click_db.read()
+    registry = Click.User.for_user(click_db, name=os.environ.get('USER'))
+    return registry.get_package_names()
+
+
+class LTTngStarted(fixtures.Fixture):
+
+    def __init__(self, name, output_directory):
+        super().__init__()
+        self.name = name
+        self.output_directory = output_directory
+
+    def setUp(self):
+        super().setUp()
+        self.addCleanup(self._destroy_tracing_session)
+        self._create_tracing_session()
+        self._enable_all_userspace_events()
+        self.addCleanup(self._stop_tracing)
+        self._start_tracing()
+
+    def _create_tracing_session(self):
+        subprocess.check_call(
+            ['lttng', 'create', self.name, '-o', self.output_directory])
+
+    def _destroy_tracing_session(self):
+        subprocess.check_call(['lttng', 'destroy', self.name])
+
+    def _enable_all_userspace_events(self):
+        subprocess.check_call(['lttng', 'enable-event', '-u', '-a'])
+
+    def _start_tracing(self):
+        subprocess.check_call(['lttng', 'start'])
+
+    def _stop_tracing(self):
+        subprocess.check_call(['lttng', 'stop'])
+
+
+class AppStartupTestCase(testscenarios.TestWithScenarios, testtools.TestCase):
+
+    click_packages = get_click_packages()
+    scenarios = [
+        ('Startup test for {}'.format(package_name), {
+            'package_name': package_name
+        })
+        for package_name in click_packages + debian_packages
+        if package_name not in black_list
+    ]
+
+    def get_app_id(self, package_name):
+        if package_name in debian_packages:
+            version = self.get_debian_package_version(package_name)
+            return '{}_{}'.format(package_name, version)
+        else:
+            return self.get_click_app_id(package_name)
+
+    def get_debian_package_version(self, package_name):
+        return apt.Cache()[package_name].versions.get(0).version
+
+    def get_click_app_id(self, package_name):
+        return subprocess.check_output(
+            ['ubuntu-app-triplet', package_name],
+            universal_newlines=True).strip()
+
+    def get_output_directory(self, app_id):
+        artifacts_directory = os.environ.get('ADT_ARTIFACTS')
+        package_directory = '{}-startup-{}'.format(
+            self.package_name, datetime.datetime.now().strftime(
+                '%Y%m%d-%H%M%S'))
+        return os.path.join(
+            artifacts_directory, 'lttng_traces', package_directory, app_id)
+
+    def start_app(self, launch_argument):
+        subprocess.check_call(
+            ['ubuntu-app-launch', launch_argument], timeout=15)
+
+    def stop_app(self, launch_argument):
+        subprocess.check_call(['ubuntu-app-stop', launch_argument])
+
+    def setUp(self):
+        super().setUp()
+        app_id = self.get_app_id(self.package_name)
+        output_directory = self.get_output_directory(app_id)
+        self.useFixture(LTTngStarted(self.package_name, output_directory))
+
+    def test_app_startup(self):
+        if self.package_name in debian_packages:
+            launch_argument = self.package_name
+        else:
+            launch_argument = self.get_click_app_id(self.package_name)
+
+        self.addCleanup(self.stop_app, launch_argument)
+        self.start_app(launch_argument)
+        # XXX it would be better to exit as soon as the trace point is found.
+        # Ask if that is possible. --elopio - 2014-11-12
+        time.sleep(30)
+
+
+if __name__ == '__main__':
+    # We can not just use the default discover because it prints the results
+    # in stderr.
+    unittest.main(
+        testRunner=unittest.TextTestRunner(stream=sys.stdout, verbosity=2))