Merge lp:~nuclearbob/ubuntu-test-cases/touch-trace-export into lp:ubuntu-test-cases/touch

[Max Brustkern]

This branch adds app startup benchmarking tests. It now relies on Paul's branch that adds the device type to the available variables.

[Max Brustkern (nuclearbob)]

I'd like to get a review of this so we can get app startup benchmarking running on each image. I'm not at all sure I'm doing the right thing for the job templates, so guidance on that would be appreciated.

[Francis Ginther (fginther)]

A few comments inline, mostly questions.

review: Needs Information

[Max Brustkern (nuclearbob)]

I'm pushing a couple of changes to address two of these.

[Paul Larson (pwlars)]

A couple of comments/questions below. What happens if that app-trace-export.py or insertion script fails? We need to be cautious about adding something that if it goes wrong, will cause the subsequent jobs to fail. I'd like to run this through some testing, but don't think I can at the moment due to the missing insertion script.

review: Needs Fixing

[Max Brustkern (nuclearbob)]

The data generation and insertion scripts run on the host and don't affect the device, so I don't think they have a high probability of failing unless we're making changes to the adb host. If they do fail, they won't create or write data on the adb host, but that should have no effect on future runs.

I've replied to diff comments individually. It seems like between the package installations and the aa containment changes, this may not run well in sequence with other smoke jobs. Is there a better way to have something run once per image after the other jobs have completed?

[lp:~nuclearbob/ubuntu-test-cases/touch-trace-export]

208. By Max Brustkern on 2014-08-06

Added nfss insertion script

209. By Max Brustkern on 2014-08-07

Merged changes

210. By Ted Gould on 2014-08-11

Make it so that we can have an Upstart config file for our env vars

211. By Ted Gould on 2014-08-11

Dropping setting env per test

212. By Ted Gould on 2014-08-11

Make sure to install UTAH too

213. By Ted Gould on 2014-08-11

Ensure the LTTng tools are installed

214. By Ted Gould on 2014-08-11

Switch to using UAL completely

215. By Ted Gould on 2014-08-11

Since the tests reboot anyway, settling is pretty useless

216. By Ted Gould on 2014-08-11

Use the reboot-and-unlock from autopilot to ensure we're in a good Unity

217. By Ted Gould on 2014-08-11

Fix command name

218. By Ted Gould on 2014-08-11

Ensure we're engaged

219. By Ted Gould on 2014-08-11

Don't think we need the update

220. By Ted Gould on 2014-08-11

Quiet

221. By Ted Gould on 2014-08-11

Give two secs for the display to come up, to ensure we're stable

222. By Ted Gould on 2014-08-12

Different tracepoitns

223. By Ted Gould on 2014-08-12

Cleanup the PIDs to turn on the screen

224. By Ted Gould on 2014-08-12

Give the dash more time to settle

225. By Ted Gould on 2014-08-12

Stop the dash so it can't screw things up

226. By Ted Gould on 2014-08-15

Make sure to trap on error

227. By Ted Gould on 2014-10-20

Change the tracking of tracepoints

228. By Ted Gould on 2014-10-20

Merge trunk

229. By Max Brustkern on 2014-10-20

Skipping actual launch

230. By Max Brustkern on 2014-10-20

Merged latest changes

231. By Max Brustkern on 2014-10-20

Adding local krillin

232. By Max Brustkern on 2014-10-20

Launch back in

233. By Max Brustkern on 2014-10-20

Cleaning up leftovers from iteration

234. By Max Brustkern on 2014-10-20

Updating nfss variable reading

235. By Max Brustkern on 2014-10-21

Job configuration updates

236. By Max Brustkern on 2014-10-21

Restored old name

237. By Max Brustkern on 2014-10-21

Re-applying Ted's updates

238. By Max Brustkern on 2014-10-21

Using sudo for apt-get

239. By Max Brustkern on 2014-10-21

Being -ex(plicit) about setup

240. By Max Brustkern on 2014-10-21

Updated nfss insert path

241. By Max Brustkern on 2014-10-21

Reverted some bootspeed changes that should go in a different branch

242. By Max Brustkern on 2014-10-21

Multiple runs of all click packages

243. By Max Brustkern on 2014-10-21

Removed testing configuration

244. By Max Brustkern on 2014-10-21

Updated installation instructions to use phase 0 PPA

245. By Max Brustkern on 2014-10-21

Adding application whitelist and blacklist and blacklisting file manager

246. By Max Brustkern on 2014-10-21

Removed file manager from blacklist

247. By Max Brustkern on 2014-10-21

Raised timeout

248. By Max Brustkern on 2014-10-21

Fixed blacklist

249. By Max Brustkern on 2014-10-22

Added blacklist apps

250. By Max Brustkern on 2014-10-22

Merged upstream changes

251. By Max Brustkern on 2014-10-23

Fixed environment variables

252. By Max Brustkern on 2014-10-23

Removed uneeded environment variables

253. By Max Brustkern on 2014-10-23

Integrated more tracepoints

254. By Max Brustkern on 2014-10-24

Giving a little extra time before timeout

[Max Brustkern (nuclearbob)]

This should be ready to review again. The comment change to setup_jenkins.py was for pep8 compliance. The change to provision.sh was to support package installation. That's no longer used in this branch, so I can take it out, but it seems like it is still the right way to do things now that a password is required? Let me know what questions you have.

review: Needs Resubmitting

[lp:~nuclearbob/ubuntu-test-cases/touch-trace-export]

255. By Max Brustkern on 2014-11-14

Merged changes

[Leo Arias (elopio)]

I copied Max's code and turned it into dep8 tests in https://code.launchpad.net/~canonical-platform-qa/ubuntu-test-cases/dep8-app-startup/+merge/241613

Max, it would be nice if you could take a look at it to confirm I didn't misunderstood anything.

review: Disapprove

Unmerged revisions

255. By Max Brustkern on 2014-11-14

Merged changes

254. By Max Brustkern on 2014-10-24

Giving a little extra time before timeout

253. By Max Brustkern on 2014-10-23

Integrated more tracepoints

252. By Max Brustkern on 2014-10-23

Removed uneeded environment variables

251. By Max Brustkern on 2014-10-23

Fixed environment variables

250. By Max Brustkern on 2014-10-22

Merged upstream changes

249. By Max Brustkern on 2014-10-22

Added blacklist apps

248. By Max Brustkern on 2014-10-21

Fixed blacklist

247. By Max Brustkern on 2014-10-21

Raised timeout

246. By Max Brustkern on 2014-10-21

Removed file manager from blacklist

=== modified file 'README-cli.rst'
--- README-cli.rst	2014-11-10 19:28:56 +0000
+++ README-cli.rst	2014-11-14 15:22:02 +0000
@@ -64,7 +64,7 @@
 
   sudo add-apt-repository ppa:utah/stable
   sudo apt-get update
-  sudo apt-get install utah-client
+  sudo apt-get install utah utah-client
 
 With that package installed UTAH tests can be run with::
 

=== modified file 'jenkins/templates/touch-base.xml.jinja2'
--- jenkins/templates/touch-base.xml.jinja2	2014-10-31 03:02:46 +0000
+++ jenkins/templates/touch-base.xml.jinja2	2014-11-14 15:22:02 +0000
@@ -73,6 +73,22 @@
 python3 ${BZRDIR}/tests/memevent/nfss_upload_results.py ./clientlogs/ nfss_insert.py ${NFSS_CONFIG_FILE}
 {% endif %}
 {% endif %}
+{% if test == 'app-startup-benchmark' %}
+NFSS_CONFIG_FILE=${WORKSPACE}/nfss_config
+cat - > ${NFSS_CONFIG_FILE} <<EOD
+{{nfss_config}}
+EOD
+adb pull /home/phablet/lttng-traces ./clientlogs/lttng-traces
+cd ./clientlogs/lttng-traces
+for DIR in $(find -mindepth 1 -maxdepth 1 -type d)
+do
+    export PACKAGE_NAME=$(echo "$DIR" | sed 's/\.\/\(.*\)-startup.*/\1/')
+    export APP_ID=$(cat $DIR/app_id)
+    export JSON_FILE=$DIR/results.json
+    ${BZRDIR}/scripts/app-trace-export.py ${PACKAGE_NAME} ${APP_ID} ${DIR}/ust/uid/32011/32-bit/
+    cat $JSON_FILE | nfss_insert.py ${NFSS_CONFIG_FILE} app-startup $PACKAGE_NAME
+done
+{% endif %}
       ]]></command>
     </hudson.tasks.Shell>
   </builders>

=== modified file 'jenkins/testconfig.py'
--- jenkins/testconfig.py	2014-11-12 03:15:23 +0000
+++ jenkins/testconfig.py	2014-11-14 15:22:02 +0000
@@ -94,6 +94,10 @@
     Test('bootspeed',
          fmt='{prefix}{testname}-{series}-{imagetype}-{type}-{type}',
          smoke=False),
+    Test('app-startup-benchmark',
+         fmt='{prefix}{testname}-{series}-{imagetype}-{type}-{type}',
+         smoke=False,
+         timeout=24300),
 ]
 
 

=== added file 'scripts/app-trace-export.py'
--- scripts/app-trace-export.py	1970-01-01 00:00:00 +0000
+++ scripts/app-trace-export.py	2014-11-14 15:22:02 +0000
@@ -0,0 +1,101 @@
+#!/usr/bin/env python3
+
+"""
+Export application trace events for app startup.
+"""
+
+import argparse
+import json
+import os
+
+from datetime import datetime
+from pprint import pprint
+
+from babeltrace import TraceCollection
+# Install Python bindings for libbabeltrace:
+#     $ sudo add-apt-repository -y \
+#     $ ppa:canonical-ci-engineering/ci-airline-phase-0
+#     $ sudo apt-get update
+#     $ sudo apt-get -qy install babeltrace python3-babeltrace
+
+
+def parse_traces(trace_path):
+    """
+    Read binary traces and convert to dict of
+    tracepoint names and timestamps.  Timestamp format is:
+    epoch secs + nanosecs.
+    """
+
+    # Tracepoints to include for app startup
+    tracepoints = {
+        'libual_start': {},
+        'libual_start_message_sent': {},
+        'libual_start_message_callback': {},
+        'handshake_wait': {},
+        'handshake_complete': {},
+        'exec_pre_exec': {},
+    }
+
+    # Build up the traces collection
+    traces = TraceCollection()
+    ret = traces.add_trace(trace_path, 'ctf')
+    if ret is None:
+        raise IOError('Error adding trace')
+
+    # Record the timestamps
+    timestamps = {}
+    for event in traces.events:
+        name = event.name.split(':')[1]
+        # We only want the event if it's the first occurance to
+        # handle possibly messy traces.
+        if name not in timestamps:
+            # Keep only the timestamps we care about by event name
+            if name in tracepoints:
+                if len(tracepoints[name]) == 0:
+                    timestamps[name] = event.timestamp
+                else:
+                    fieldname = name
+                    for field in tracepoints[name]:
+                        if not event[field] == tracepoints[name][field]:
+                            fieldname = None
+                            break
+                        else:
+                            fieldname += ":{}={}".format(
+                                field, tracepoints[name][field])
+
+                    if fieldname is not None and fieldname not in timestamps:
+                        timestamps[fieldname] = event.timestamp
+
+    return timestamps
+
+
+def generate_doc(tracepoints, app_id):
+    return dict(
+        package_name=args.package_name,
+        app_id=app_id,
+        android_serial=os.environ.get('ANDROID_SERIAL'),
+        image_type=os.environ.get('IMAGE_TYPE'),
+        build_num=os.environ.get('BUILD_NUMBER'),
+        build_id=os.environ.get('BUILD_ID'),
+        job_name=os.environ.get('JOB_NAME'),
+        node_name=os.environ.get('NODE_NAME'),
+        datetime=datetime.utcnow().isoformat(),
+        tracepoints=tracepoints,
+    )
+
+
+if __name__ == '__main__':
+    parser = argparse.ArgumentParser()
+    parser.add_argument('package_name', help='click package name')
+    parser.add_argument('app_id', help='application identifier')
+    parser.add_argument('path', help='path to trace')
+    args = parser.parse_args()
+
+    tracepoints = parse_traces(args.path),
+    doc = generate_doc(tracepoints, args.app_id)
+
+    pprint(doc)
+
+    if os.environ.get('JSON_FILE'):
+        with open(os.environ.get('JSON_FILE'), 'w') as jsonfile:
+            json.dump(doc, jsonfile)

=== added directory 'tests/app-startup-benchmark'
=== added file 'tests/app-startup-benchmark/app-startup.sh'
--- tests/app-startup-benchmark/app-startup.sh	1970-01-01 00:00:00 +0000
+++ tests/app-startup-benchmark/app-startup.sh	2014-11-14 15:22:02 +0000
@@ -0,0 +1,107 @@
+#!/bin/bash
+
+## This is the script jenkins should run for app startup benchmarks
+
+set -e
+
+CMD_PREFIX=""
+BENCHMARK_CMD="app-startup-benchmark.sh"
+
+
+usage() {
+	cat <<EOF
+usage: $0 -a APP [-s ANDROID_SERIAL] [-Q] [-t TIMEOUT]
+
+Benchmark the startup of APP
+
+OPTIONS:
+  -h	Show this message
+  -s    Specify the serial of the device to install
+  -a    The name of the application to start
+  -Q    "Quick" don't do a reboot of the device before running the test
+  -t	Time to wait for application startup (Default is 60)
+  -r    Number of times to repeat (Default is 1)
+
+EOF
+}
+
+PIDS=""
+
+cleanup() {
+	set +e
+	echo "killing child pids: $PIDS"
+	for p in $PIDS ; do
+	kill $p
+	done
+}
+
+main() {
+	PIDS=""
+
+	set -x
+	if [ -z $QUICK ] ; then
+		reboot-and-unlock.sh
+		adb-shell ${CMD_PREFIX} initctl stop unity8-dash
+		adb shell powerd-cli active &
+		PIDS="$PIDS $!"
+		adb shell powerd-cli display on &
+		PIDS="$PIDS $!"
+		sleep 5
+	else
+		echo "SKIPPING phone reboot..."
+	fi
+
+	echo "launching benchmark...."
+	[ -z $ANDROID_SERIAL ] || ADBOPTS="-s $ANDROID_SERIAL"
+	adb-shell ${CMD_PREFIX} ${BENCHMARK_CMD} $APP $TIMEOUT
+
+	cleanup
+}
+
+export REPEAT=1
+
+while getopts p:s:a:Qt:r:h opt; do
+    case $opt in
+    h)
+        usage
+        exit 0
+        ;;
+    s)
+        export ANDROID_SERIAL=$OPTARG
+        ;;
+    a)
+        APP=$OPTARG
+        ;;
+    Q)
+        QUICK=1
+        ;;
+    t)
+	export TIMEOUT=$OPTARG
+	;;
+    r)
+	export REPEAT=$OPTARG
+	;;
+  esac
+done
+
+if [ -z $ANDROID_SERIAL ] ; then
+    # ensure we only have one device attached
+    lines=$(adb devices | wc -l)
+    if [ $lines -gt 3 ] ; then
+        echo "ERROR: More than one device attached, please use -s option"
+	echo
+        usage
+        exit 1
+    fi
+fi
+if [ -z $APP ] ; then
+    echo "ERROR: No app specified"
+    usage
+    exit 1
+fi
+
+trap cleanup TERM INT EXIT
+
+for X in $(seq $REPEAT) ; do
+    main
+done

=== added file 'tests/app-startup-benchmark/blacklist'
--- tests/app-startup-benchmark/blacklist	1970-01-01 00:00:00 +0000
+++ tests/app-startup-benchmark/blacklist	2014-11-14 15:22:02 +0000
@@ -0,0 +1,3 @@
+com.canonical.payui
+com.ubuntu.reminders
+com.ubuntu.scopes.youtube

=== added file 'tests/app-startup-benchmark/master.run'
--- tests/app-startup-benchmark/master.run	1970-01-01 00:00:00 +0000
+++ tests/app-startup-benchmark/master.run	2014-11-14 15:22:02 +0000
@@ -0,0 +1,5 @@
+---
+testsuites:
+  - name: app-startup-benchmark
+    fetch_method: dev
+    fetch_location: ./

=== added file 'tests/app-startup-benchmark/setup.sh'
--- tests/app-startup-benchmark/setup.sh	1970-01-01 00:00:00 +0000
+++ tests/app-startup-benchmark/setup.sh	2014-11-14 15:22:02 +0000
@@ -0,0 +1,23 @@
+#!/bin/sh
+
+set -ex
+
+echo "Installing LTTng"
+adb-shell "sudo apt-get install -yq lttng-modules-dkms lttng-tools"
+
+echo "Removing old traces"
+adb-shell "rm -rf /home/phablet/lttng-traces"
+
+echo "Allowing LTTng events from confined applications"
+adb-shell "sudo sed 's/deny \\(.*shm\\/lttng-ust-.*\\)/\\1\nowner @{HOME}\/.lttng\/ rw,\nowner @{HOME}\/.lttng\/\* rwk,\n\/run\/lttng\/ rw,\n\/run\/lttng\/** rwk,/' -i /usr/share/apparmor/easyprof/templates/ubuntu/*/ubuntu-*"
+
+echo "Regenerating AppArmor profiles for installed click applications"
+adb-shell "sudo aa-clickhook -f"
+
+echo "Setting environment variables for session"
+adb-shell "mkdir -p /home/phablet/.config/upstart"
+adb-shell "echo \"start on starting dbus\" > /home/phablet/.config/upstart/lttng-trace.conf"
+adb-shell "echo \"script\" >> /home/phablet/.config/upstart/lttng-trace.conf"
+adb-shell "echo \"initctl set-env --global LTTNG_UST_REGISTER_TIMEOUT=-1\" >> /home/phablet/.config/upstart/lttng-trace.conf"
+adb-shell "echo \"initctl set-env --global UBUNTU_APP_LAUNCH_LTTNG_ENABLED=TRUE\" >> /home/phablet/.config/upstart/lttng-trace.conf"
+adb-shell "echo \"end script\" >> /home/phablet/.config/upstart/lttng-trace.conf"

=== added file 'tests/app-startup-benchmark/ts_control'
--- tests/app-startup-benchmark/ts_control	1970-01-01 00:00:00 +0000
+++ tests/app-startup-benchmark/ts_control	2014-11-14 15:22:02 +0000
@@ -0,0 +1,1 @@
+ts_setup: ./setup.sh

=== added file 'tests/app-startup-benchmark/tslist.auto'
--- tests/app-startup-benchmark/tslist.auto	1970-01-01 00:00:00 +0000
+++ tests/app-startup-benchmark/tslist.auto	2014-11-14 15:22:02 +0000
@@ -0,0 +1,4 @@
+-
+  discovery_cmd: adb-shell click list | grep -v "ADB_RC" | awk '{print $1}' | grep -vf app-startup-benchmark/blacklist | cat - app-startup-benchmark/whitelist | sort | uniq
+  test_cmd: ../app-startup.sh -a {} -r 10 -t 10
+

=== added file 'tests/app-startup-benchmark/whitelist'
=== added file 'utils/target/app-startup-benchmark.sh'
--- utils/target/app-startup-benchmark.sh	1970-01-01 00:00:00 +0000
+++ utils/target/app-startup-benchmark.sh	2014-11-14 15:22:02 +0000
@@ -0,0 +1,35 @@
+#!/bin/bash
+
+set -x
+
+echo "Setting environment variables for tracing"
+export LTTNG_UST_REGISTER_TIMEOUT=-1
+export UBUNTU_APP_LAUNCH_LTTNG_ENABLED=TRUE
+
+echo "Creating dummy session without -e to get around first run error"
+lttng create dummy
+lttng destroy dummy
+
+set -e
+
+if [ -z ${1} ]
+then
+    echo "Usage:" >&2
+    echo "$0 APP [TIMEOUT]" >&2
+    exit 1
+fi
+
+PACKAGE=${1}
+APP_ID=$(ubuntu-app-triplet ${PACKAGE})
+
+DIR=$(lttng create ${PACKAGE}-startup | grep "Traces will be written in" | awk '{print $NF}')
+
+lttng enable-event -u -a
+lttng start
+
+ubuntu-app-launch ${APP_ID}
+
+sleep ${2-30}
+lttng stop
+lttng destroy ${PACKAGE}-startup
+echo $APP_ID > $DIR/app_id