Merge lp:~canonical-platform-qa/ubuntu-system-tests/device_monitoring into lp:ubuntu-system-tests

[Sergio Cazzolato]

Enable the device monitoring used to visualize the resources usage in the device during tests execution

[Brendan Donegan (brendan-donegan)]

Do you think the plotting of the data should be coupled to ubuntu-system-tests? I suppose it's a nice to have but really the plotting should be done externally so we have a choice of which tool to use, how to configure it etc.

review: Needs Information

[PS Jenkins bot (ps-jenkins)]

FAILED: Continuous integration, rev:287
http://s-jenkins.ubuntu-ci:8080/job/ubuntu-sanity-tests-ci/1193/
Executed test runs:
    FAILURE: http://s-jenkins.ubuntu-ci:8080/job/ubuntu-sanity-tests-wily-amd64-ci/921/console
    FAILURE: http://s-jenkins.ubuntu-ci:8080/job/ubuntu-sanity-tests-wily-armhf-ci/922/console
    FAILURE: http://s-jenkins.ubuntu-ci:8080/job/ubuntu-sanity-tests-wily-i386-ci/921/console

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/ubuntu-sanity-tests-ci/1193/rebuild

review: Needs Fixing (continuous-integration)

[lp:~canonical-platform-qa/ubuntu-system-tests/device_monitoring]

288. By Sergio Cazzolato on 2015-12-11

Adding test case to open 100 time several apps

[PS Jenkins bot (ps-jenkins)]

FAILED: Continuous integration, rev:288
http://s-jenkins.ubuntu-ci:8080/job/ubuntu-sanity-tests-ci/1194/
Executed test runs:
    FAILURE: http://s-jenkins.ubuntu-ci:8080/job/ubuntu-sanity-tests-wily-amd64-ci/922/console
    FAILURE: http://s-jenkins.ubuntu-ci:8080/job/ubuntu-sanity-tests-wily-armhf-ci/923/console
    FAILURE: http://s-jenkins.ubuntu-ci:8080/job/ubuntu-sanity-tests-wily-i386-ci/922/console

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/ubuntu-sanity-tests-ci/1194/rebuild

review: Needs Fixing (continuous-integration)

[lp:~canonical-platform-qa/ubuntu-system-tests/device_monitoring]

289. By Sergio Cazzolato on 2015-12-11

Adding new test to open all the apps

290. By Sergio Cazzolato on 2015-12-14

Adding new test case

[PS Jenkins bot (ps-jenkins)]

FAILED: Continuous integration, rev:290
http://s-jenkins.ubuntu-ci:8080/job/ubuntu-sanity-tests-ci/1198/
Executed test runs:
    FAILURE: http://s-jenkins.ubuntu-ci:8080/job/ubuntu-sanity-tests-wily-amd64-ci/926/console
    FAILURE: http://s-jenkins.ubuntu-ci:8080/job/ubuntu-sanity-tests-wily-armhf-ci/927/console
    FAILURE: http://s-jenkins.ubuntu-ci:8080/job/ubuntu-sanity-tests-wily-i386-ci/926/console

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/ubuntu-sanity-tests-ci/1198/rebuild

review: Needs Fixing (continuous-integration)

[lp:~canonical-platform-qa/ubuntu-system-tests/device_monitoring]

291. By Sergio Cazzolato on 2015-12-14

Flake8 fix

[PS Jenkins bot (ps-jenkins)]

FAILED: Continuous integration, rev:291
http://s-jenkins.ubuntu-ci:8080/job/ubuntu-sanity-tests-ci/1201/
Executed test runs:
    FAILURE: http://s-jenkins.ubuntu-ci:8080/job/ubuntu-sanity-tests-wily-amd64-ci/929/console
    FAILURE: http://s-jenkins.ubuntu-ci:8080/job/ubuntu-sanity-tests-wily-armhf-ci/930/console
    FAILURE: http://s-jenkins.ubuntu-ci:8080/job/ubuntu-sanity-tests-wily-i386-ci/929/console

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/ubuntu-sanity-tests-ci/1201/rebuild

review: Needs Fixing (continuous-integration)

[lp:~canonical-platform-qa/ubuntu-system-tests/device_monitoring]

292. By Sergio Cazzolato on 2015-12-14

Improving webapps start

[PS Jenkins bot (ps-jenkins)]

FAILED: Continuous integration, rev:292
http://s-jenkins.ubuntu-ci:8080/job/ubuntu-sanity-tests-ci/1205/
Executed test runs:
    FAILURE: http://s-jenkins.ubuntu-ci:8080/job/ubuntu-sanity-tests-wily-amd64-ci/933/console
    FAILURE: http://s-jenkins.ubuntu-ci:8080/job/ubuntu-sanity-tests-wily-armhf-ci/934/console
    FAILURE: http://s-jenkins.ubuntu-ci:8080/job/ubuntu-sanity-tests-wily-i386-ci/933/console

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/ubuntu-sanity-tests-ci/1205/rebuild

review: Needs Fixing (continuous-integration)

[Sergio Cazzolato (sergio-j-cazzolato)]

> Do you think the plotting of the data should be coupled to ubuntu-system-
> tests? I suppose it's a nice to have but really the plotting should be done
> externally so we have a choice of which tool to use, how to configure it etc.

Agree with that, I just added the visualization as part of the branch to make easy to see if the results are what we need, after that I could remove it.

[lp:~canonical-platform-qa/ubuntu-system-tests/device_monitoring]

293. By Sergio Cazzolato on 2015-12-14

Adding config parameter for nmon

[PS Jenkins bot (ps-jenkins)]

FAILED: Continuous integration, rev:293
http://s-jenkins.ubuntu-ci:8080/job/ubuntu-sanity-tests-ci/1207/
Executed test runs:
    FAILURE: http://s-jenkins.ubuntu-ci:8080/job/ubuntu-sanity-tests-wily-amd64-ci/935/console
    FAILURE: http://s-jenkins.ubuntu-ci:8080/job/ubuntu-sanity-tests-wily-armhf-ci/936/console
    FAILURE: http://s-jenkins.ubuntu-ci:8080/job/ubuntu-sanity-tests-wily-i386-ci/935/console

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/ubuntu-sanity-tests-ci/1207/rebuild

review: Needs Fixing (continuous-integration)

[lp:~canonical-platform-qa/ubuntu-system-tests/device_monitoring]

294. By Sergio Cazzolato on 2015-12-14

Removing clock app

[PS Jenkins bot (ps-jenkins)]

FAILED: Continuous integration, rev:294
http://s-jenkins.ubuntu-ci:8080/job/ubuntu-sanity-tests-ci/1208/
Executed test runs:
    FAILURE: http://s-jenkins.ubuntu-ci:8080/job/ubuntu-sanity-tests-wily-amd64-ci/936/console
    FAILURE: http://s-jenkins.ubuntu-ci:8080/job/ubuntu-sanity-tests-wily-armhf-ci/937/console
    FAILURE: http://s-jenkins.ubuntu-ci:8080/job/ubuntu-sanity-tests-wily-i386-ci/936/console

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/ubuntu-sanity-tests-ci/1208/rebuild

review: Needs Fixing (continuous-integration)

[lp:~canonical-platform-qa/ubuntu-system-tests/device_monitoring]

295. By Sergio Cazzolato on 2015-12-15

Adding new test case to maname apps in a cicle

296. By Sergio Cazzolato on 2015-12-16

Minor improvements for new test

297. By Sergio Cazzolato on 2015-12-16

Adding fixtures for test

[PS Jenkins bot (ps-jenkins)]

FAILED: Continuous integration, rev:297
http://s-jenkins.ubuntu-ci:8080/job/ubuntu-sanity-tests-ci/1218/
Executed test runs:
    FAILURE: http://s-jenkins.ubuntu-ci:8080/job/ubuntu-sanity-tests-wily-amd64-ci/946/console
    FAILURE: http://s-jenkins.ubuntu-ci:8080/job/ubuntu-sanity-tests-wily-armhf-ci/947/console
    FAILURE: http://s-jenkins.ubuntu-ci:8080/job/ubuntu-sanity-tests-wily-i386-ci/946/console

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/ubuntu-sanity-tests-ci/1218/rebuild

review: Needs Fixing (continuous-integration)

[lp:~canonical-platform-qa/ubuntu-system-tests/device_monitoring]

298. By Sergio Cazzolato on 2015-12-16

Minor fixes

[PS Jenkins bot (ps-jenkins)]

FAILED: Continuous integration, rev:298
http://s-jenkins.ubuntu-ci:8080/job/ubuntu-sanity-tests-ci/1220/
Executed test runs:
    FAILURE: http://s-jenkins.ubuntu-ci:8080/job/ubuntu-sanity-tests-wily-amd64-ci/948/console
    FAILURE: http://s-jenkins.ubuntu-ci:8080/job/ubuntu-sanity-tests-wily-armhf-ci/949/console
    FAILURE: http://s-jenkins.ubuntu-ci:8080/job/ubuntu-sanity-tests-wily-i386-ci/948/console

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/ubuntu-sanity-tests-ci/1220/rebuild

review: Needs Fixing (continuous-integration)

[lp:~canonical-platform-qa/ubuntu-system-tests/device_monitoring]

299. By Sergio Cazzolato on 2015-12-18

Moving test configuration to json file

300. By Sergio Cazzolato on 2015-12-18

Merge with trunk

301. By Sergio Cazzolato on 2015-12-18

Adding comments

302. By Sergio Cazzolato on 2015-12-18

Adding missing file

303. By Sergio Cazzolato on 2015-12-18

Minor fix in name

[PS Jenkins bot (ps-jenkins)]

FAILED: Continuous integration, rev:303
http://s-jenkins.ubuntu-ci:8080/job/ubuntu-sanity-tests-ci/1226/
Executed test runs:
    FAILURE: http://s-jenkins.ubuntu-ci:8080/job/ubuntu-sanity-tests-wily-amd64-ci/954/console
    FAILURE: http://s-jenkins.ubuntu-ci:8080/job/ubuntu-sanity-tests-wily-armhf-ci/955/console
    FAILURE: http://s-jenkins.ubuntu-ci:8080/job/ubuntu-sanity-tests-wily-i386-ci/954/console

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/ubuntu-sanity-tests-ci/1226/rebuild

review: Needs Fixing (continuous-integration)

[lp:~canonical-platform-qa/ubuntu-system-tests/device_monitoring]

304. By Sergio Cazzolato on 2015-12-19

Fixing camera actions

[PS Jenkins bot (ps-jenkins)]

FAILED: Continuous integration, rev:304
http://s-jenkins.ubuntu-ci:8080/job/ubuntu-sanity-tests-ci/1227/
Executed test runs:
    FAILURE: http://s-jenkins.ubuntu-ci:8080/job/ubuntu-sanity-tests-wily-amd64-ci/955/console
    FAILURE: http://s-jenkins.ubuntu-ci:8080/job/ubuntu-sanity-tests-wily-armhf-ci/956/console
    FAILURE: http://s-jenkins.ubuntu-ci:8080/job/ubuntu-sanity-tests-wily-i386-ci/955/console

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/ubuntu-sanity-tests-ci/1227/rebuild

review: Needs Fixing (continuous-integration)

[lp:~canonical-platform-qa/ubuntu-system-tests/device_monitoring]

305. By Sergio Cazzolato on 2015-12-20

Adding config file with plotting directives

[PS Jenkins bot (ps-jenkins)]

FAILED: Continuous integration, rev:305
http://s-jenkins.ubuntu-ci:8080/job/ubuntu-sanity-tests-ci/1228/
Executed test runs:
    FAILURE: http://s-jenkins.ubuntu-ci:8080/job/ubuntu-sanity-tests-wily-amd64-ci/956/console
    FAILURE: http://s-jenkins.ubuntu-ci:8080/job/ubuntu-sanity-tests-wily-armhf-ci/957/console
    FAILURE: http://s-jenkins.ubuntu-ci:8080/job/ubuntu-sanity-tests-wily-i386-ci/956/console

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/ubuntu-sanity-tests-ci/1228/rebuild

review: Needs Fixing (continuous-integration)

[Sergio Cazzolato (sergio-j-cazzolato)]

Getting this error from jenkins http://paste.ubuntu.com/14120530/

[lp:~canonical-platform-qa/ubuntu-system-tests/device_monitoring]

306. By Sergio Cazzolato on 2015-12-21

Moving dependencies and increasing iterations

[PS Jenkins bot (ps-jenkins)]

PASSED: Continuous integration, rev:306
http://s-jenkins.ubuntu-ci:8080/job/ubuntu-sanity-tests-ci/1231/
Executed test runs:
    SUCCESS: http://s-jenkins.ubuntu-ci:8080/job/ubuntu-sanity-tests-wily-amd64-ci/959
    SUCCESS: http://s-jenkins.ubuntu-ci:8080/job/ubuntu-sanity-tests-wily-armhf-ci/960
    SUCCESS: http://s-jenkins.ubuntu-ci:8080/job/ubuntu-sanity-tests-wily-i386-ci/959

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/ubuntu-sanity-tests-ci/1231/rebuild

review: Approve (continuous-integration)

[Richard Huddie (rhuddie)]

Looks good, some minor comments below. I had this problem when running test_open_apps: http://pastebin.ubuntu.com/14161943/

The other test was running, but I've not had a chance to run it until completion yet.

review: Needs Fixing

[lp:~canonical-platform-qa/ubuntu-system-tests/device_monitoring]

307. By Sergio Cazzolato on 2015-12-23

Fixes based on PM comments

308. By Sergio Cazzolato on 2015-12-23

Minor fix

[PS Jenkins bot (ps-jenkins)]

PASSED: Continuous integration, rev:308
http://s-jenkins.ubuntu-ci:8080/job/ubuntu-sanity-tests-ci/1238/
Executed test runs:
    SUCCESS: http://s-jenkins.ubuntu-ci:8080/job/ubuntu-sanity-tests-wily-amd64-ci/966
    SUCCESS: http://s-jenkins.ubuntu-ci:8080/job/ubuntu-sanity-tests-wily-armhf-ci/967
    SUCCESS: http://s-jenkins.ubuntu-ci:8080/job/ubuntu-sanity-tests-wily-i386-ci/966

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/ubuntu-sanity-tests-ci/1238/rebuild

review: Approve (continuous-integration)

[Richard Huddie (rhuddie)]

I got the following error when running test_open_apps: http://pastebin.ubuntu.com/14419424/

I was able to create an HTML graph for statistics using nmonchart (http://nmon.sourceforge.net/pmwiki.php?n=Site.Nmonchart), is this correct usage? It would be useful to add a section to README to explain how to generate graphs from the .nmon result file.

Other question, when I ran on Arale with no SD card, the exception is raised which gets reported as error: http://pastebin.ubuntu.com/14419431/
This is not an error, but just unsupported feature. Could this just be logged, rather than raising exception?

Also, it would be useful if the the error message ("testtools.matchers._impl.MismatchError: 0 != 5: Errors found 5") could give a summary of the reported errors. Otherwise it could be long job to go through the logs to understand what has happened.

review: Needs Fixing

[Selene ToyKeeper (toykeeper)]

To get this working at all, I had to rename a couple functions:
--- ubuntu_system_tests/tests/data/perf/metadata.json 2015-12-23 21:54:32 +0000
+++ ubuntu_system_tests/tests/data/perf/metadata.json 2016-01-15 07:53:46 +0000
@@ -2,7 +2,7 @@
     "test_open_apps": {
         "iterations": 100,
         "apps": ["camera-app", "webbrowser-app", "messaging-app",
- "address-book-app", "ubuntu-calculator-app", "dialer-app"]
+ "address-book", "calculator-app", "dialer-app"]
     },
     "test_do_actions": {
         "iterations": 2,

... and to run on arale, I had to disable _validate_sd_card() in ubuntu_system_tests/tests/perf/__init__.py .

I'm still getting 4 errors in test_do_actions() though, which I haven't finished tracing.

What is the recommended way to get pyNmonAnalyzer?

[Sergio Cazzolato (sergio-j-cazzolato)]

> To get this working at all, I had to rename a couple functions:
> --- ubuntu_system_tests/tests/data/perf/metadata.json 2015-12-23 21:54:32
> +0000
> +++ ubuntu_system_tests/tests/data/perf/metadata.json 2016-01-15 07:53:46
> +0000
> @@ -2,7 +2,7 @@
> "test_open_apps": {
> "iterations": 100,
> "apps": ["camera-app", "webbrowser-app", "messaging-app",
> - "address-book-app", "ubuntu-calculator-app", "dialer-app"]
> + "address-book", "calculator-app", "dialer-app"]
> },
> "test_do_actions": {
> "iterations": 2,

I am working on that, I think I am going to remove this test and will add actions to close the apps so we will be able to manage everything in the test test_do_actions
>
>
>
> ... and to run on arale, I had to disable _validate_sd_card() in
> ubuntu_system_tests/tests/perf/__init__.py .

This is fixed in the next commit. When you want to use the sdcard and it is not inserted in the device, this action will be skipped and logged as a warning

>
> I'm still getting 4 errors in test_do_actions() though, which I haven't
> finished tracing.
>
> What is the recommended way to get pyNmonAnalyzer?

You need to have installed pip as a prerequisite. Then:
  sudo apt-get install python-numpy python-matplotlib
  sudo pip install pyNmonAnalyzer

In the meantime I am analyzing other tools to plot the nmon information. Here you have an example graphic that I took https://drive.google.com/open?id=0B2R7lqeEPdMUU1p0Mlhhc2FxbTA

Thanks for the review

[Sergio Cazzolato (sergio-j-cazzolato)]

> I got the following error when running test_open_apps:
> http://pastebin.ubuntu.com/14419424/

I am going to remove this test and adding some actions to close apps so this test will be included in the test_do_actions.

>
> I was able to create an HTML graph for statistics using nmonchart
> (http://nmon.sourceforge.net/pmwiki.php?n=Site.Nmonchart), is this correct
> usage? It would be useful to add a section to README to explain how to
> generate graphs from the .nmon result file.

Nice, taking a look now

>
> Other question, when I ran on Arale with no SD card, the exception is raised
> which gets reported as error: http://pastebin.ubuntu.com/14419431/
> This is not an error, but just unsupported feature. Could this just be logged,
> rather than raising exception?

Done, When you want to use the sdcard and it is not inserted in the device, this action will be skipped and logged as a warning
>
> Also, it would be useful if the the error message
> ("testtools.matchers._impl.MismatchError: 0 != 5: Errors found 5") could give
> a summary of the reported errors. Otherwise it could be long job to go through
> the logs to understand what has happened.

WIP

[lp:~canonical-platform-qa/ubuntu-system-tests/device_monitoring]

309. By Sergio Cazzolato on 2016-01-15

Changes based on comments in review

[PS Jenkins bot (ps-jenkins)]

FAILED: Continuous integration, rev:309
http://s-jenkins.ubuntu-ci:8080/job/ubuntu-sanity-tests-ci/1272/
Executed test runs:
    FAILURE: http://s-jenkins.ubuntu-ci:8080/job/ubuntu-sanity-tests-wily-amd64-ci/998/console
    FAILURE: http://s-jenkins.ubuntu-ci:8080/job/ubuntu-sanity-tests-wily-armhf-ci/1001/console
    FAILURE: http://s-jenkins.ubuntu-ci:8080/job/ubuntu-sanity-tests-wily-i386-ci/998/console

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/ubuntu-sanity-tests-ci/1272/rebuild

review: Needs Fixing (continuous-integration)

[lp:~canonical-platform-qa/ubuntu-system-tests/device_monitoring]

310. By Sergio Cazzolato on 2016-01-15

adding wait action

[PS Jenkins bot (ps-jenkins)]

FAILED: Continuous integration, rev:310
http://s-jenkins.ubuntu-ci:8080/job/ubuntu-sanity-tests-ci/1273/
Executed test runs:
    FAILURE: http://s-jenkins.ubuntu-ci:8080/job/ubuntu-sanity-tests-wily-amd64-ci/999/console
    FAILURE: http://s-jenkins.ubuntu-ci:8080/job/ubuntu-sanity-tests-wily-armhf-ci/1002/console
    FAILURE: http://s-jenkins.ubuntu-ci:8080/job/ubuntu-sanity-tests-wily-i386-ci/999/console

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/ubuntu-sanity-tests-ci/1273/rebuild

review: Needs Fixing (continuous-integration)

[lp:~canonical-platform-qa/ubuntu-system-tests/device_monitoring]

311. By Sergio Cazzolato on 2016-01-20

Merge from trunk

[PS Jenkins bot (ps-jenkins)]

PASSED: Continuous integration, rev:311
http://s-jenkins.ubuntu-ci:8080/job/ubuntu-sanity-tests-ci/1283/
Executed test runs:
    SUCCESS: http://s-jenkins.ubuntu-ci:8080/job/ubuntu-sanity-tests-wily-amd64-ci/1009
    SUCCESS: http://s-jenkins.ubuntu-ci:8080/job/ubuntu-sanity-tests-wily-armhf-ci/1012
    SUCCESS: http://s-jenkins.ubuntu-ci:8080/job/ubuntu-sanity-tests-wily-i386-ci/1009

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/ubuntu-sanity-tests-ci/1283/rebuild

review: Approve (continuous-integration)

[Brendan Donegan (brendan-donegan)]

From a code point of view this is fine, but I would look for some early feedback from jibel to make sure this is what is wanted. In my opinion this solution could do with being a bit more flexible. Rather than writing tests specifically for performance, it might be a better idea to re-use existing tests and have a way to get the metrics at certain points (in between tests, after all the tests are completed, etc). It might also be possible to granularise the test approach, so rather than gathering all the metrics for all tests, have a specific set of tests for memory usage, a specific set for CPU usage and a specific set for network usage. It really depends on what the intent is - is it to emulate real usage or is it to provoke specific bugs?

review: Needs Information

[Richard Huddie (rhuddie)]

Showing some conflicts on this branch:
Text conflict in ubuntu_system_tests/config.py
Text conflict in ubuntu_system_tests/tests/base.py
2 conflicts encountered.

Could you update?

[platform-qa-bot (platform-qa-bot)]

FAILED: Continuous integration, rev:311
https://platform-qa-jenkins.ubuntu.com/job/ubuntu-system-tests-ci/6/
Executed test runs:
    FAILURE: https://platform-qa-jenkins.ubuntu.com/job/build-wily-amd64-package/28/console
    FAILURE: https://platform-qa-jenkins.ubuntu.com/job/build-wily-i386-package/29/console
    None: https://platform-qa-jenkins.ubuntu.com/job/generic-update-mp/33/console

Click here to trigger a rebuild:
https://platform-qa-jenkins.ubuntu.com/job/ubuntu-system-tests-ci/6/rebuild

review: Needs Fixing (continuous-integration)

[platform-qa-bot (platform-qa-bot)]

FAILED: Continuous integration, rev:311
https://platform-qa-jenkins.ubuntu.com/job/ubuntu-system-tests-ci/1140/
Executed test runs:
    FAILURE: https://platform-qa-jenkins.ubuntu.com/job/build-xenial-amd64-package/416/console
    FAILURE: https://platform-qa-jenkins.ubuntu.com/job/build-xenial-i386-package/416/console
    None: https://platform-qa-jenkins.ubuntu.com/job/generic-update-mp/1913/console

Click here to trigger a rebuild:
https://platform-qa-jenkins.ubuntu.com/job/ubuntu-system-tests-ci/1140/rebuild

review: Needs Fixing (continuous-integration)

Unmerged revisions

311. By Sergio Cazzolato on 2016-01-20

Merge from trunk

310. By Sergio Cazzolato on 2016-01-15

adding wait action

309. By Sergio Cazzolato on 2016-01-15

Changes based on comments in review

308. By Sergio Cazzolato on 2015-12-23

Minor fix

307. By Sergio Cazzolato on 2015-12-23

Fixes based on PM comments

306. By Sergio Cazzolato on 2015-12-21

Moving dependencies and increasing iterations

305. By Sergio Cazzolato on 2015-12-20

Adding config file with plotting directives

304. By Sergio Cazzolato on 2015-12-19

Fixing camera actions

303. By Sergio Cazzolato on 2015-12-18

Minor fix in name

302. By Sergio Cazzolato on 2015-12-18

Adding missing file

=== modified file 'debian/control'
--- debian/control	2015-06-26 22:41:14 +0000
+++ debian/control	2016-01-20 15:40:48 +0000
@@ -22,7 +22,10 @@
 Architecture: all
 Depends: ${misc:Depends},
          ${python3:Depends},
+         pyNmonAnalyzer,
          python3-dbus,
+         python3-matplotlib,
+         python3-numpy,
          python3-testtools,
          python3-uci-config,
          subunit,

=== modified file 'debian/tests/control'
--- debian/tests/control	2015-11-25 15:04:04 +0000
+++ debian/tests/control	2016-01-20 15:40:48 +0000
@@ -14,6 +14,7 @@
          mediaplayer-app-autopilot,
          messaging-app,
          messaging-app-autopilot, # TMPINSTALL
+         nmon,
          oxideqt-chromedriver,
          python3-autopilot,
          python3-dbus,

=== modified file 'ubuntu_system_tests/command_line.py'
--- ubuntu_system_tests/command_line.py	2015-12-03 20:15:55 +0000
+++ ubuntu_system_tests/command_line.py	2016-01-20 15:40:48 +0000
@@ -456,6 +456,11 @@
 def generate_user_report(config_stack):
     """Create a human readable report from the subunit results."""
     artifacts_directory = results.get_artifacts_directory(config_stack)
+    if config_stack.get('profiling_statistics'):
+        serial = config_stack.get('device_serial')
+        data_file = results.copy_monitor_data(artifacts_directory, serial)
+        if config_stack.get('plot_statistics'):
+            results.generate_statistics_report(artifacts_directory, data_file)
     results.convert_subunit_results_to_pyunit(artifacts_directory)
     results.print_results(artifacts_directory)
 

=== modified file 'ubuntu_system_tests/config.py'
--- ubuntu_system_tests/config.py	2015-11-05 17:35:49 +0000
+++ ubuntu_system_tests/config.py	2016-01-20 15:40:48 +0000
@@ -109,6 +109,17 @@
         'max_unity8_retry_delay', default=30000,
         help_string='The boundary for how long we should be retrying until '
                     'unity8 is started'),
+    options.Option(
+        'profiling_statistics',
+        help_string="Enable performance profiling. CPU, Disk usage, Memory, "
+                    "Network and other statistics will be measured and stored "
+                    "as part of the test results. If empty, statistics won't "
+                    "be taken from the device."),
+    options.Option(
+        'plot_statistics',
+        help_string="Enable performance plotting. A report will be generated "
+                    "as part of the test results. If empty, statistics won't "
+                    "be plotted."),
 ]
 """Options for tests in external debian and click packages."""
 

=== modified file 'ubuntu_system_tests/helpers/data.py'
--- ubuntu_system_tests/helpers/data.py	2015-09-23 21:12:39 +0000
+++ ubuntu_system_tests/helpers/data.py	2016-01-20 15:40:48 +0000
@@ -28,7 +28,7 @@
     """ Class used to decode a dict into a AttrDict object """
 
     def __init__(self, *args, **kargs):
-        JSONDecoder.__init__(self, object_hook=dict_to_attr_dict)
+        JSONDecoder.__init__(self, object_hook=_dict_to_attr_dict)
 
 
 class AttrDict(dict):
@@ -51,7 +51,7 @@
     return file
 
 
-def dict_to_attr_dict(d):
+def _dict_to_attr_dict(d):
     try:
         return AttrDict(**d)
     except:
@@ -91,3 +91,15 @@
 
     _full_dict = _load_metadata_list(fs.DIR_TEST_DATA)
     return _full_dict
+
+
+def load_data_file(file):
+    """ Load in a AttrDict the data stored in the specific json file
+    :param file: the path to the json file
+    :return: A AttrDict with the data stored in the specific json file
+    """
+    if not path.isfile(file):
+        raise RuntimeError('Json data file does not exist')
+
+    with open(file) as data_file:
+        return load(data_file, cls=AttrDictDecoder)

=== modified file 'ubuntu_system_tests/helpers/file_system.py'
--- ubuntu_system_tests/helpers/file_system.py	2015-12-22 23:21:51 +0000
+++ ubuntu_system_tests/helpers/file_system.py	2016-01-20 15:40:48 +0000
@@ -91,6 +91,7 @@
 DIR_TEST_DATA_VIDEO = os.path.join(DIR_TEST_DATA, 'video')
 DIR_TEST_DATA_DEVICES = os.path.join(DIR_TEST_DATA, 'devices')
 DIR_TEST_DATA_DATABASES = os.path.join(DIR_TEST_DATA, 'databases')
+DIR_TEST_DATA_PERF = os.path.join(DIR_TEST_DATA, 'perf')
 
 # Media dirs
 DIR_MEDIA = '/media'

=== added file 'ubuntu_system_tests/helpers/monitoring.py'
--- ubuntu_system_tests/helpers/monitoring.py	1970-01-01 00:00:00 +0000
+++ ubuntu_system_tests/helpers/monitoring.py	2016-01-20 15:40:48 +0000
@@ -0,0 +1,61 @@
+
+# -*- Mode: Python; coding: utf-8; indent-tabs-mode: nil; tab-width: 4 -*-
+
+# Ubuntu System Tests
+# Copyright (C) 2015 Canonical
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+import logging
+import os
+import subprocess
+
+from ubuntu_system_tests import config
+from ubuntu_system_tests.helpers import file_system as fs
+
+logger = logging.getLogger(__name__)
+
+
+MONITOR = 'nmon'
+NAME = 'monitoring_data.nmon'
+OUTPUT_DIR = '/tmp/monitor_data'
+
+
+def is_monitoring_enabled():
+    """ Determine if the monitoring was requested in the config file """
+    return bool(config.get_device_config_stack().get('profiling_statistics'))
+
+
+def get_output_file():
+    """ Retrieve the .nmon file for current/last execution """
+    return os.path.join(OUTPUT_DIR, NAME)
+
+
+def start(interval=2, counts=50000):
+    """ Start the monitoring tool with defined parameters
+    :param interval: time in seconds between snap shots
+    :param counts: number of snapshots before nmon stops
+    """
+    fs.create_directory_if_not_exists(OUTPUT_DIR)
+    fs.clean_files(OUTPUT_DIR)
+    cmd = '{mon} -F {name} -s {interval} -t -c {counts}'
+    vars = dict(mon=MONITOR, interval=interval, name=get_output_file(),
+                counts=counts)
+
+    subprocess.call(cmd.format(**vars), shell=True)
+
+
+def stop():
+    """ Stop the nmon process """
+    subprocess.call('pkill {}'.format(MONITOR), shell=True)

=== modified file 'ubuntu_system_tests/helpers/sdcard.py'
--- ubuntu_system_tests/helpers/sdcard.py	2015-11-03 21:03:37 +0000
+++ ubuntu_system_tests/helpers/sdcard.py	2016-01-20 15:40:48 +0000
@@ -106,3 +106,8 @@
     :param dst_name: The filename in the sd card
     """
     return copy_file(src_path, fs.DIR_PICTURES, dst_name)
+
+
+def is_sdcard_mounted():
+    """ Indicate if the sd card is mounted in the device """
+    return fs.is_media_folder_dir(fs.DIR_PICTURES)

=== modified file 'ubuntu_system_tests/results.py'
--- ubuntu_system_tests/results.py	2015-05-21 18:15:40 +0000
+++ ubuntu_system_tests/results.py	2016-01-20 15:40:48 +0000
@@ -18,9 +18,16 @@
 # along with this program.  If not, see <http://www.gnu.org/licenses/>.
 #
 
+import logging
 import os
 import subprocess
 
+from ubuntu_system_tests.helpers import data
+from ubuntu_system_tests.helpers import file_system as fs
+from ubuntu_system_tests.helpers import monitoring
+
+logger = logging.getLogger(__name__)
+
 
 def convert_subunit_results_to_pyunit(artifacts_directory):
     subunit_source_file_path = os.path.join(
@@ -48,3 +55,47 @@
     """Return the path to the directory in which artifacts are stored."""
     output_dir = config_stack.get('output_dir')
     return os.path.join(output_dir, 'artifacts')
+
+
+def copy_monitor_data(artifacts_directory, device=None):
+    """ Copy the monitoring data from the device to the test results dir
+    :param artifacts_directory: the path where the artifacts are stored
+    :param device: the serial for the desired device
+    """
+    from_file = monitoring.get_output_file()
+    to_file = os.path.join(artifacts_directory, os.path.basename(from_file))
+    if device:
+        cmd = 'adb -s {} pull {} {}'.format(device, from_file, to_file)
+    else:
+        cmd = 'adb pull {} {}'.format(from_file, to_file)
+    subprocess.call(cmd, shell=True)
+    return to_file
+
+
+def _write_statistics_config_file(config_file, directives):
+    """ Create the config file used to plot the monitoring data """
+    fs.delete_file(config_file)
+    with open(config_file, 'w') as f:
+        for directive in directives:
+            f.write(directive + '\n')
+
+
+def generate_statistics_report(artifacts_directory, data_file):
+    """ Generate the html report with the statistics usage
+    :param artifacts_directory: the path where the artifacts are stored
+    :param data_file: file with monitoring raw data copied from the device
+    """
+    plot_data = data.load_data_file(os.path.join(fs.DIR_TEST_DATA_PERF,
+                                                 'plot.json'))
+
+    output_report_dir = os.path.join(artifacts_directory, plot_data.output_dir)
+    config_file = os.path.join(artifacts_directory, plot_data.config_file_name)
+    _write_statistics_config_file(config_file, plot_data.directives)
+
+    cmd = 'pyNmonAnalyzer -x -b -o {} -i {} -t static -r {}'.format(
+        output_report_dir, data_file, config_file)
+    try:
+        subprocess.call(cmd, shell=True)
+    except OSError:
+        logger.error("Statistics not generated. Error to call "
+                     "pyNmonAnalyzer")

=== modified file 'ubuntu_system_tests/tests/base.py'
--- ubuntu_system_tests/tests/base.py	2016-01-11 11:06:16 +0000
+++ ubuntu_system_tests/tests/base.py	2016-01-20 15:40:48 +0000
@@ -48,6 +48,7 @@
 from ubuntu_system_tests.helpers import click
 from ubuntu_system_tests.helpers import images
 from ubuntu_system_tests.helpers import mir
+from ubuntu_system_tests.helpers import monitoring
 from ubuntu_system_tests.helpers import processes
 from ubuntu_system_tests.helpers import screen
 from ubuntu_system_tests.helpers import unity8
@@ -61,9 +62,21 @@
 # Any test name ending with this will leave device locked
 TEST_ID_DEVICE_LOCKED = 'device_locked'
 
+logger = logging.getLogger(__name__)
+
+# Apps and cleanup names
+ADDRESS_BOOK_APP = 'address-book-app'
 CALCULATOR_APP = 'ubuntu-calculator-app'
-
-logger = logging.getLogger(__name__)
+CAMERA_APP = 'camera-app'
+CLOCK_APP = 'ubuntu-clock-app'
+DIALER_APP = 'dialer-app'
+GALLERY_APP = 'gallery-app'
+MEDIA_PLAYER_APP = 'mediaplayer-app'
+MESSAGING_APP = 'messaging-app'
+SYSTEM_SETTINGS = 'system-settings'
+SYSTEM_SETTINGS_CLEANUP = 'ubuntu-system-settings'
+UNITY_8 = 'unity8'
+WEB_BROWSER_APP = 'webbrowser-app'
 
 
 class BaseUbuntuSystemTestCase(testcase.AutopilotTestCase):
@@ -72,11 +85,15 @@
     def setUpClass(cls):
         unity8.ensure_unity_stopped()
         super().setUpClass()
+        if monitoring.is_monitoring_enabled():
+            monitoring.start()
 
     @classmethod
     def tearDownClass(cls):
         super().tearDownClass()
         unity8.ensure_unity_running_and_greeter_hidden()
+        if monitoring.is_monitoring_enabled():
+            monitoring.stop()
 
     def setUp(self):
         super().setUp()
@@ -106,7 +123,7 @@
         else:
             self._start_unity_with_testability()
 
-        unity = self.get_job_proxy_object('unity8')
+        unity = self.get_job_proxy_object(UNITY_8)
         unity.wait_for_greeter_content()
         self._reset_keyboard()
         # Workaround for bug lp:1474444
@@ -115,7 +132,7 @@
         return unity
 
     def _start_unity_with_testability(self):
-        process_helpers.start_job('unity8', 'QT_LOAD_TESTABILITY=1')
+        process_helpers.start_job(UNITY_8, 'QT_LOAD_TESTABILITY=1')
 
     # Retry added until fix for bug 1427946 in QtMir is landed
     @retry(stop_max_delay=30000,
@@ -153,7 +170,7 @@
     def _create_sensors(self):
         """Create fake sensors for testing"""
         Eventually(Equals(True)).match(
-            lambda: process_helpers.is_job_running('unity8')
+            lambda: process_helpers.is_job_running(UNITY_8)
         )
         unity_pid = process_helpers._get_unity_pid()
         fifo_path = '/tmp/sensor-fifo-{}'.format(unity_pid)
@@ -200,7 +217,7 @@
 
     def _get_screenshot_size(self, x, y, width, height):
         """ Calculate the screenshot size """
-        unity = self.get_job_proxy_object('unity8')
+        unity = self.get_job_proxy_object(UNITY_8)
         if width and height:
             return width, height
         else:
@@ -240,7 +257,7 @@
         :param app_name: Name of the displayed application.
 
         """
-        unity = self.get_job_proxy_object('unity8')
+        unity = self.get_job_proxy_object(UNITY_8)
         unity.main_window.launch_application(app_name)
 
     def _get_proxy_object_for_existing_app(
@@ -332,7 +349,7 @@
 
         """
         from ubuntu_system_tests.helpers.addressbook import _cpo  # NOQA
-        proxy = self._get_proxy_object_for_existing_app('address-book-app')
+        proxy = self._get_proxy_object_for_existing_app(ADDRESS_BOOK_APP)
         return proxy.main_window
 
     def launch_camera_app(self):
@@ -360,7 +377,7 @@
         """
         from ubuntu_system_tests.helpers.camera._cpo import MainWindow
         proxy = self._get_proxy_object_for_existing_app(
-            'camera-app',
+            CAMERA_APP,
             cleanup_process=click.get_click_app_identifier('com.ubuntu.camera')
         )
         return MainWindow(proxy)
@@ -383,7 +400,7 @@
 
         """
         from ubuntu_system_tests.helpers.dialer_app import _cpo  # NOQA
-        proxy = self._get_proxy_object_for_existing_app('dialer-app')
+        proxy = self._get_proxy_object_for_existing_app(DIALER_APP)
         return proxy.main_view
 
     def launch_gallery_app(self):
@@ -404,9 +421,9 @@
         :return: Proxy object for gallery application.
 
         """
-        proxy = self._get_proxy_object_for_existing_app('gallery-app')
+        proxy = self._get_proxy_object_for_existing_app(GALLERY_APP)
         # ubuntu-app-stop does not work with gallery app, use process helper
-        self.addCleanup(processes.stop_process, 'gallery-app')
+        self.addCleanup(processes.stop_process, GALLERY_APP)
         return proxy
 
     def get_gallery_events_view(self):
@@ -427,7 +444,7 @@
 
         """
         from ubuntu_system_tests.helpers.mediaplayer._cpo import MainWindow
-        proxy = self._get_proxy_object_for_existing_app('mediaplayer-app')
+        proxy = self._get_proxy_object_for_existing_app(MEDIA_PLAYER_APP)
         return MainWindow(proxy)
 
     def launch_messaging_app(self):
@@ -453,7 +470,7 @@
                                                            base=base)
 
         from ubuntu_system_tests.helpers.messaging._cpo import MainView
-        proxy = self._get_proxy_object_for_existing_app('messaging-app')
+        proxy = self._get_proxy_object_for_existing_app(MESSAGING_APP)
         return proxy.wait_select_single(MainView)
 
     def launch_system_settings(self):
@@ -475,7 +492,7 @@
         """
         from ubuntu_system_tests.helpers.system_settings import _cpo  # NOQA
         proxy = self._get_proxy_object_for_existing_app(
-            'system-settings', cleanup_process='ubuntu-system-settings')
+            SYSTEM_SETTINGS, cleanup_process=SYSTEM_SETTINGS_CLEANUP)
         return proxy.main_view
 
     def launch_webbrowser_app(self):
@@ -496,7 +513,7 @@
 
         """
         from ubuntu_system_tests.helpers.webbrowser import _cpo  # NOQA
-        proxy = self._get_proxy_object_for_existing_app('webbrowser-app')
+        proxy = self._get_proxy_object_for_existing_app(WEB_BROWSER_APP)
         return proxy.main_window
 
     def launch_calendar_app(self):
@@ -530,14 +547,9 @@
         """
         self._launch_application_from_apps_scope('Clock')
         wait_until(processes.is_qmlscene_running_with_qmlfile,
-                   'ubuntu-clock-app')
+                   CLOCK_APP)
         return self.get_clock_app_proxy()
 
-    def launch_webbrowser_from_launcher(self):
-        """Start webbrowser from the launcher."""
-        self._launch_application_from_launcher('webbrowser-app')
-        return self.get_webbrowser_app_proxy()
-
     def get_clock_app_proxy(self):
         """
         Return clock app proxy object from existing process.
@@ -546,18 +558,22 @@
 
         """
         from ubuntu_system_tests.helpers.clock import _cpo  # NOQA
-        proxy = self._get_proxy_object_for_existing_qmlscene_process(
-            'ubuntu-clock-app')
+        proxy = self._get_proxy_object_for_existing_qmlscene_process(CLOCK_APP)
         return proxy.main_view
 
+    def launch_webbrowser_from_launcher(self):
+        """Start webbrowser from the launcher."""
+        self._launch_application_from_launcher(WEB_BROWSER_APP)
+        return self.get_webbrowser_app_proxy()
+
     def launch_messaging_app_from_launcher(self):
         """Drag out the launcher and tap the messaging app icon to start it."""
-        self._launch_application_from_launcher('messaging-app')
+        self._launch_application_from_launcher(MESSAGING_APP)
         return self.get_messaging_app_proxy()
 
     def launch_dialer_app_from_launcher(self):
         """Drag out the launcher and tap the dialer app icon to start it."""
-        self._launch_application_from_launcher('dialer-app')
+        self._launch_application_from_launcher(DIALER_APP)
         return self.get_dialer_app_proxy()
 
     def get_webapp_proxy(self, web_app_package):

=== added directory 'ubuntu_system_tests/tests/data/perf'
=== added file 'ubuntu_system_tests/tests/data/perf/metadata.json'
--- ubuntu_system_tests/tests/data/perf/metadata.json	1970-01-01 00:00:00 +0000
+++ ubuntu_system_tests/tests/data/perf/metadata.json	2016-01-20 15:40:48 +0000
@@ -0,0 +1,8 @@
+{
+    "test_perform_actions": {
+        "setup_actions": ["start_all_webapps"],
+        "iterations": 2,
+        "test_actions": ["take_photo", "open_two_urls", "add_random_contact",
+            "add_random_numbers", "remove_add_sdcard", "record_video"]
+    }
+}

=== added file 'ubuntu_system_tests/tests/data/perf/plot.json'
--- ubuntu_system_tests/tests/data/perf/plot.json	1970-01-01 00:00:00 +0000
+++ ubuntu_system_tests/tests/data/perf/plot.json	2016-01-20 15:40:48 +0000
@@ -0,0 +1,12 @@
+{
+    "output_dir": "monitoring_report",
+    "config_file_name": "report.config",
+    "directives": [
+        "CPU_ALL={stackedGraph: true, fillGraph: true}",
+        "MEM={}",
+        "NET=wlan0{}",
+        "DISKBUSY=mmcblk0,mmcblk0p1,mmcblk0p2,mmcblk0p3,mmcblk0p4,mmcblk0p5,mmcblk0p6,mmcblk0p7,mmcblk0boot1,mmcblk0boot0,mmcblk1,mmcblk1p1{}",
+        "DISKREAD=mmcblk0,mmcblk0p1,mmcblk0p2,mmcblk0p3,mmcblk0p4,mmcblk0p5,mmcblk0p6,mmcblk0p7,mmcblk0boot1,mmcblk0boot0,mmcblk1,mmcblk1p1{}",
+        "DISKWRITE=mmcblk0,mmcblk0p1,mmcblk0p2,mmcblk0p3,mmcblk0p4,mmcblk0p5,mmcblk0p6,mmcblk0p7,mmcblk0boot1,mmcblk0boot0,mmcblk1,mmcblk1p1{}"
+    ]
+}

=== added directory 'ubuntu_system_tests/tests/perf'
=== added file 'ubuntu_system_tests/tests/perf/__init__.py'
--- ubuntu_system_tests/tests/perf/__init__.py	1970-01-01 00:00:00 +0000
+++ ubuntu_system_tests/tests/perf/__init__.py	2016-01-20 15:40:48 +0000
@@ -0,0 +1,91 @@
+# -*- Mode: Python; coding: utf-8; indent-tabs-mode: nil; tab-width: 4 -*-
+
+#
+# Ubuntu System Tests
+# Copyright (C) 2015 Canonical
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+
+import logging
+import traceback
+
+from ubuntu_system_tests.helpers.addressbook import (
+    fixture_setup as addressbook_fixtures)
+from ubuntu_system_tests.helpers import autopilot
+from ubuntu_system_tests.helpers.backup_restore_fixture import (
+    BackupRestoreFixture)
+from ubuntu_system_tests.helpers.camera.fixture_setup import (
+    SetCameraAccessRequests)
+from ubuntu_system_tests.helpers.clock.fixture_setup import (
+    SetClockAccessRequests)
+from ubuntu_system_tests.helpers import file_system as fs
+from ubuntu_system_tests.helpers.location.fixture_setup import (
+    SetLocationAccessRequests)
+from ubuntu_system_tests.helpers import media
+from ubuntu_system_tests.helpers.scopes import go_to_apps_scope
+from ubuntu_system_tests.helpers.webbrowser.fixture_setup import (
+    WebbrowserTestEnvironment)
+
+from ubuntu_system_tests.tests import base
+
+logger = logging.getLogger(__name__)
+
+WEB_APPS = ['Gmail', 'Facebook', 'Twitter', 'Amazon', 'eBay', 'HERE Maps']
+
+
+class BasePerfTestCase(base.BaseUbuntuSystemTestCase):
+
+    def setUp(self):
+        super().setUp()
+
+        self.useFixture(SetCameraAccessRequests())
+        self.useFixture(SetClockAccessRequests())
+        self.useFixture(SetLocationAccessRequests('com.nokia.heremaps_here'))
+
+        self.useFixture(BackupRestoreFixture(fs.DIR_HOME_CAMERA_CONFIG))
+        self.useFixture(BackupRestoreFixture(fs.DIR_HOME_CAMERA_THUMBNAILS))
+        self.useFixture(media.BackupRestoreLocalMediaFixture())
+
+        self.useFixture(addressbook_fixtures.AddressBookTestEnvironment())
+        self.useFixture(WebbrowserTestEnvironment())
+
+        self.errors = 0
+        self.errors_msg = []
+        self.addCleanup(self._print_errors)
+
+    def _manage_app(self, method, *args):
+        try:
+            method(*args)
+        except Exception as e:
+            self.errors += 1
+            self.errors_msg.append(traceback.format_exc())
+            logger.error(e)
+
+    def _start_webapp(self, app_name):
+        apps_scope = go_to_apps_scope()
+        apps_scope.click_scope_item('local', app_name)
+
+    def _validate_camera_app(self):
+        if not autopilot.is_camera_supported():
+            raise RuntimeError("Camera-app required")
+
+    def _print_errors(self):
+        print('*' * 100)
+        print('ERRORS FOUND')
+        print('*' * 100)
+        for error in self.errors_msg:
+            print(error)
+            print('-' * 50)
+        print('*' * 100)

=== added file 'ubuntu_system_tests/tests/perf/test_apps.py'
--- ubuntu_system_tests/tests/perf/test_apps.py	1970-01-01 00:00:00 +0000
+++ ubuntu_system_tests/tests/perf/test_apps.py	2016-01-20 15:40:48 +0000
@@ -0,0 +1,197 @@
+# -*- Mode: Python; coding: utf-8; indent-tabs-mode: nil; tab-width: 4 -*-
+
+#
+# Ubuntu System Tests
+# Copyright (C) 2015 Canonical
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+
+import logging
+import subprocess
+import random
+import os
+from testtools import skipUnless
+
+from address_book_app.address_book.data import Phone, Contact
+
+from ubuntu_system_tests.helpers import camera
+from ubuntu_system_tests.helpers.data import load_test_metadata
+from ubuntu_system_tests.helpers import monitoring
+from ubuntu_system_tests.helpers import processes
+from ubuntu_system_tests.helpers import sdcard
+from ubuntu_system_tests.helpers import wait_until
+
+from ubuntu_system_tests.tests import base
+from ubuntu_system_tests.tests import perf
+
+logger = logging.getLogger(__name__)
+
+
+class AppsTestCase(perf.BasePerfTestCase):
+    """ This Class tests performance aspects open and closing apps """
+
+    @skipUnless(monitoring.is_monitoring_enabled(), 'Perf test skipped')
+    def setUp(self):
+        super().setUp()
+        self.unity_proxy = self.restart_unity()
+        self.test_metadata = load_test_metadata().perf
+        self.pic = 0
+
+    def _start_action(self, action):
+        getattr(self, '_action_{}'.format(action))()
+
+    def _action_open_webbrowser(self):
+        self.launch_webbrowser_app()
+
+    def _action_close_webbrowser(self):
+        self.ensure_application_closed(base.WEB_BROWSER_APP)
+
+    def _action_open_dialer(self):
+        self.launch_dialer_app()
+
+    def _action_close_dialer(self):
+        self.ensure_application_closed(base.DIALER_APP)
+
+    def _action_open_messaging(self):
+        self.launch_messaging_app()
+
+    def _action_close_messaging(self):
+        self.ensure_application_closed(base.MESSAGING_APP)
+
+    def _action_open_clock(self):
+        self.launch_clock_app()
+
+    def _action_close_clock(self):
+        self.ensure_application_closed(base.CLOCK_APP)
+
+    def _action_open_address_book(self):
+        self.launch_address_book()
+
+    def _action_close_address_book(self):
+        self.ensure_application_closed(base.ADDRESS_BOOK_APP)
+
+    def _action_open_camera(self):
+        self.launch_calendar_app()
+
+    def _action_close_camera(self):
+        self.ensure_application_closed(base.CAMERA_APP)
+
+    def _action_open_calculator(self):
+        self.launch_calculator_app()
+
+    def _action_close_calculator(self):
+        processes.stop_qmlscene_process(base.CALCULATOR_APP)
+
+    def _action_open_system_settings(self):
+        self.launch_system_settings()
+
+    def _action_close_system_settings(self):
+        processes.ensure_application_closed(base.SYSTEM_SETTINGS_CLEANUP)
+
+    def _action_start_all_webapps(self):
+        # Open apps that will be open during the whole tests
+        self._manage_app(self._start_webapp, 'Gmail')
+        self._manage_app(self._start_webapp, 'Facebook')
+        self._manage_app(self._start_webapp, 'Twitter')
+        self._manage_app(self._start_webapp, 'Amazon')
+        self._manage_app(self._start_webapp, 'eBay')
+        self._manage_app(self._start_webapp, 'HERE Maps')
+        self.addCleanup(self._action_close_all_webapps)
+
+    def _action_close_all_webapps(self):
+        cmd = 'pkill -ef --signal SIGKILL webapp-container'
+        subprocess.call(cmd, shell=True)
+
+    def _action_open_two_urls(self):
+        """ Open two pages """
+        browser = self.launch_webbrowser_app()
+        browser.go_to_url('launchpad.net')
+        browser.wait_page_loaded()
+        browser.go_to_url('canonical.com')
+        browser.wait_page_loaded()
+
+    def _action_add_random_contact(self):
+        # Add a random contact
+        address_book = self.launch_address_book()
+
+        first_name = 'first_name_{}'.format(random.randint(0, 999999))
+        last_name = 'last_name_{}'.format(random.randint(0, 999999))
+        phone = Phone(type_='Mobile', number='818616313')
+        test_contact = Contact(first_name=first_name, last_name=last_name,
+                               phones=[phone])
+        contact_editor = address_book.go_to_add_contact()
+        contact_editor.fill_form(test_contact)
+        address_book.save()
+
+    def _action_add_random_numbers(self):
+        """ Add two numbers """
+        calculator = self.launch_calculator_app()
+        calculator.calculate('{}+{}').format(random.randint(0, 999),
+                                             random.randint(0, 999))
+
+    def _action_remove_add_sdcard(self):
+        # Remove and insert the sd card
+        if sdcard.is_sdcard_mounted():
+            device = sdcard.remove()
+            sdcard.insert(device)
+        else:
+            logging.warning('Action Remove Add sdcard skipped')
+
+    def _action_take_photo(self):
+        self._validate_camera_app()
+        camera_app = self.launch_camera_app()
+        config = camera.get_config(mode='camera')
+        camera.config_camera(camera_app, config)
+        photoroll_hint = camera_app.take_photo(self.pic == 0)
+        self.pic += 1
+        if photoroll_hint:
+            gallery = photoroll_hint.swipe_to_dismiss()
+            gallery.go_back()
+
+    def _action_record_video(self):
+        self._validate_camera_app()
+        camera_app = self.launch_camera_app()
+        config = camera.get_config(mode='video')
+        camera.config_camera(camera_app, config)
+        camera_app.record_video(2, random_focus=False)
+        if self.pic == 0:
+            camera_app.swipe_to_gallery(self)
+            camera_app.start_playback()
+
+    def _action_wait_for_stop_test_file(self):
+        file = os.path.join('/', 'home', 'stop_test')
+        timeout = 60 * 60 * 12
+        if wait_until(os.path.isfile, file, timeout=timeout, period=5):
+            os.remove(file)
+
+    def test_perform_actions(self):
+        """ Perform all the actions defined in the input test data repeatedly.
+         The test first read the test_perform_actions test data and based on
+         that execute all the setup actions first, then it performs
+         iteratively based on the iterations parameter the test_actions.
+         The cleanup has to be set through the standard cleanup method.
+         The test passes when all the actions finished successfully. The
+         exceptions raise during the execution are printed in stdout.
+         """
+        test_data = self.test_metadata.test_perform_actions
+
+        for action in test_data.setup_actions:
+            self._manage_app(self._start_action, action)
+
+        for i in range(test_data.iterations):
+            for action in test_data.test_actions:
+                self._manage_app(self._start_action, action)
+
+        self.assertEqual(0, self.errors, 'Errors found {}'.format(self.errors))

=== modified file 'ubuntu_system_tests/tests/test_sdcard.py'
--- ubuntu_system_tests/tests/test_sdcard.py	2015-12-23 13:30:00 +0000
+++ ubuntu_system_tests/tests/test_sdcard.py	2016-01-20 15:40:48 +0000
@@ -62,7 +62,7 @@
 class SDCardSystemTest(base.BaseUbuntuSystemTestCase):
     """This Class tests the sd card"""
 
-    @skipUnless(fs.is_media_folder_dir(fs.DIR_PICTURES), SKIP_REASON)
+    @skipUnless(sdcard.is_sdcard_mounted(), SKIP_REASON)
     def setUp(self):
         super().setUp()
         self.useFixture(ScopesHints())