Merge lp:~canonical-platform-qa/qa-jenkins-jobs/performance-jobs into lp:qa-jenkins-jobs

[Sergio Cazzolato]

jobs for performance tests execution

[platform-qa-bot (platform-qa-bot)]

PASSED: Continuous integration, rev:205
https://platform-qa-jenkins.ubuntu.com/job/qa-jenkins-jobs-ci/76/
Executed test runs:
    SUCCESS: https://platform-qa-jenkins.ubuntu.com/job/build-xenial-amd64-package/242
        deb: https://platform-qa-jenkins.ubuntu.com/job/build-xenial-amd64-package/242/artifact/work/output/*zip*/output.zip
    SUCCESS: https://platform-qa-jenkins.ubuntu.com/job/build-xenial-i386-package/240
        deb: https://platform-qa-jenkins.ubuntu.com/job/build-xenial-i386-package/240/artifact/work/output/*zip*/output.zip
    None: https://platform-qa-jenkins.ubuntu.com/job/generic-update-mp/1555/console

Click here to trigger a rebuild:
https://platform-qa-jenkins.ubuntu.com/job/qa-jenkins-jobs-ci/76/rebuild

review: Approve (continuous-integration)

[lp:~canonical-platform-qa/qa-jenkins-jobs/performance-jobs]

206. By Sergio Cazzolato on 2017-02-07

Adding triggers and wrappers

[platform-qa-bot (platform-qa-bot)]

PASSED: Continuous integration, rev:206
https://platform-qa-jenkins.ubuntu.com/job/qa-jenkins-jobs-ci/77/
Executed test runs:
    SUCCESS: https://platform-qa-jenkins.ubuntu.com/job/build-xenial-amd64-package/243
        deb: https://platform-qa-jenkins.ubuntu.com/job/build-xenial-amd64-package/243/artifact/work/output/*zip*/output.zip
    SUCCESS: https://platform-qa-jenkins.ubuntu.com/job/build-xenial-i386-package/241
        deb: https://platform-qa-jenkins.ubuntu.com/job/build-xenial-i386-package/241/artifact/work/output/*zip*/output.zip
    None: https://platform-qa-jenkins.ubuntu.com/job/generic-update-mp/1556/console

Click here to trigger a rebuild:
https://platform-qa-jenkins.ubuntu.com/job/qa-jenkins-jobs-ci/77/rebuild

review: Approve (continuous-integration)

[lp:~canonical-platform-qa/qa-jenkins-jobs/performance-jobs]

207. By Sergio Cazzolato on 2017-02-07

Fix for device ip in job

[platform-qa-bot (platform-qa-bot)]

PASSED: Continuous integration, rev:207
https://platform-qa-jenkins.ubuntu.com/job/qa-jenkins-jobs-ci/78/
Executed test runs:
    SUCCESS: https://platform-qa-jenkins.ubuntu.com/job/build-xenial-amd64-package/244
        deb: https://platform-qa-jenkins.ubuntu.com/job/build-xenial-amd64-package/244/artifact/work/output/*zip*/output.zip
    SUCCESS: https://platform-qa-jenkins.ubuntu.com/job/build-xenial-i386-package/242
        deb: https://platform-qa-jenkins.ubuntu.com/job/build-xenial-i386-package/242/artifact/work/output/*zip*/output.zip
    None: https://platform-qa-jenkins.ubuntu.com/job/generic-update-mp/1557/console

Click here to trigger a rebuild:
https://platform-qa-jenkins.ubuntu.com/job/qa-jenkins-jobs-ci/78/rebuild

review: Approve (continuous-integration)

[Max Brustkern (nuclearbob)]

Some diff questions, but in general I think this will work well.

[Sergio Cazzolato (sergio-j-cazzolato)]

Answers and questions in diff

[Max Brustkern (nuclearbob)]

Responded!

[lp:~canonical-platform-qa/qa-jenkins-jobs/performance-jobs]

208. By Sergio Cazzolato on 2017-02-07

reusing grafana setup between perf and tests

[platform-qa-bot (platform-qa-bot)]

PASSED: Continuous integration, rev:208
https://platform-qa-jenkins.ubuntu.com/job/qa-jenkins-jobs-ci/79/
Executed test runs:
    SUCCESS: https://platform-qa-jenkins.ubuntu.com/job/build-xenial-amd64-package/245
        deb: https://platform-qa-jenkins.ubuntu.com/job/build-xenial-amd64-package/245/artifact/work/output/*zip*/output.zip
    SUCCESS: https://platform-qa-jenkins.ubuntu.com/job/build-xenial-i386-package/243
        deb: https://platform-qa-jenkins.ubuntu.com/job/build-xenial-i386-package/243/artifact/work/output/*zip*/output.zip
    None: https://platform-qa-jenkins.ubuntu.com/job/generic-update-mp/1558/console

Click here to trigger a rebuild:
https://platform-qa-jenkins.ubuntu.com/job/qa-jenkins-jobs-ci/79/rebuild

review: Approve (continuous-integration)

[Max Brustkern (nuclearbob)]

I get an error when attempting a deployment:

./scripts/deploy-jobs.sh
+ dirname ./scripts/deploy-jobs.sh
+ cd ./scripts
+ pwd
+ JOB_PATH=/home/max/bzr/merges/performance-jobs/scripts/../jobs
+ ARGS=update
+ make -C /home/max/bzr/merges/performance-jobs/scripts/../jobs
make: Entering directory '/home/max/bzr/merges/performance-jobs/jobs'
make -C auto-upgrade-testing
make[1]: Entering directory '/home/max/bzr/merges/performance-jobs/jobs/auto-upgrade-testing'
../../scripts/generate-upgrade-testing-profiles-from-branch.sh > profiles
make[1]: Leaving directory '/home/max/bzr/merges/performance-jobs/jobs/auto-upgrade-testing'
make: Leaving directory '/home/max/bzr/merges/performance-jobs/jobs'
+ python -m jenkins_jobs.cmd update -r /home/max/bzr/merges/performance-jobs/scripts/../jobs
INFO:root:Updating jobs in ['/home/max/bzr/merges/performance-jobs/jobs', '/home/max/bzr/merges/performance-jobs/jobs/performance', '/home/max/bzr/merges/performance-jobs/jobs/iso-testing', '/home/max/bzr/merges/performance-jobs/jobs/mp-ci-autoland', '/home/max/bzr/merges/performance-jobs/jobs/ubiquity', '/home/max/bzr/merges/performance-jobs/jobs/auto-upgrade-testing', '/home/max/bzr/merges/performance-jobs/jobs/ubuntu-system-tests', '/home/max/bzr/merges/performance-jobs/jobs/vis'] ([])
/usr/lib/python2.7/dist-packages/jenkins/__init__.py:527: DeprecationWarning: get_plugins_info() is deprecated, use get_plugins()
  DeprecationWarning)
Traceback (most recent call last):
  File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/usr/lib/python2.7/dist-packages/jenkins_jobs/cmd.py", line 390, in <module>
    main()
  File "/usr/lib/python2.7/dist-packages/jenkins_jobs/cmd.py", line 191, in main
    execute(options, config)
  File "/usr/lib/python2.7/dist-packages/jenkins_jobs/cmd.py", line 372, in execute
    n_workers=options.n_workers)
  File "/usr/lib/python2.7/dist-packages/jenkins_jobs/builder.py", line 348, in update_jobs
    self.load_files(input_fn)
  File "/usr/lib/python2.7/dist-packages/jenkins_jobs/builder.py", line 293, in load_files
    self.parser.parse(in_file)
  File "/usr/lib/python2.7/dist-packages/jenkins_jobs/parser.py", line 128, in parse
    self.parse_fp(fp)
  File "/usr/lib/python2.7/dist-packages/jenkins_jobs/parser.py", line 97, in parse_fp
    data = local_yaml.load(utils.wrap_stream(fp), search_path=self.path)
  File "/usr/lib/python2.7/dist-packages/jenkins_jobs/local_yaml.py", line 325, in load
    return yaml.load(stream, functools.partial(LocalLoader, **kwargs))
  File "/usr/lib/python2.7/dist-packages/yaml/__init__.py", line 71, in load
    return loader.get_single_data()
  File "/usr/lib/python2.7/dist-packages/yaml/constructor.py", line 37, in get_single_data
    node = self.get_single_node()
  File "/usr/lib/python2.7/dist-packages/yaml/composer.py", line 36, in get_single_node
    document = self.compose_document()
  File "/usr/lib/python2.7/dist-packages/jenkins_jobs/local_yaml.py", line 166, in compose_document
    node = self.compose_node(None, None)
  File "/usr/lib/python2.7/...

review: Needs Fixing

[lp:~canonical-platform-qa/qa-jenkins-jobs/performance-jobs]

209. By Sergio Cazzolato on 2017-02-07

Minor fixes

[platform-qa-bot (platform-qa-bot)]

PASSED: Continuous integration, rev:209
https://platform-qa-jenkins.ubuntu.com/job/qa-jenkins-jobs-ci/80/
Executed test runs:
    SUCCESS: https://platform-qa-jenkins.ubuntu.com/job/build-xenial-amd64-package/246
        deb: https://platform-qa-jenkins.ubuntu.com/job/build-xenial-amd64-package/246/artifact/work/output/*zip*/output.zip
    SUCCESS: https://platform-qa-jenkins.ubuntu.com/job/build-xenial-i386-package/244
        deb: https://platform-qa-jenkins.ubuntu.com/job/build-xenial-i386-package/244/artifact/work/output/*zip*/output.zip
    None: https://platform-qa-jenkins.ubuntu.com/job/generic-update-mp/1559/console

Click here to trigger a rebuild:
https://platform-qa-jenkins.ubuntu.com/job/qa-jenkins-jobs-ci/80/rebuild

review: Approve (continuous-integration)

[lp:~canonical-platform-qa/qa-jenkins-jobs/performance-jobs]

210. By Sergio Cazzolato on 2017-02-07

Fixing issues

[platform-qa-bot (platform-qa-bot)]

PASSED: Continuous integration, rev:210
https://platform-qa-jenkins.ubuntu.com/job/qa-jenkins-jobs-ci/81/
Executed test runs:
    SUCCESS: https://platform-qa-jenkins.ubuntu.com/job/build-xenial-amd64-package/247
        deb: https://platform-qa-jenkins.ubuntu.com/job/build-xenial-amd64-package/247/artifact/work/output/*zip*/output.zip
    SUCCESS: https://platform-qa-jenkins.ubuntu.com/job/build-xenial-i386-package/245
        deb: https://platform-qa-jenkins.ubuntu.com/job/build-xenial-i386-package/245/artifact/work/output/*zip*/output.zip
    None: https://platform-qa-jenkins.ubuntu.com/job/generic-update-mp/1560/console

Click here to trigger a rebuild:
https://platform-qa-jenkins.ubuntu.com/job/qa-jenkins-jobs-ci/81/rebuild

review: Approve (continuous-integration)

[lp:~canonical-platform-qa/qa-jenkins-jobs/performance-jobs]

211. By Sergio Cazzolato on 2017-02-08

minor fix on parameters

[platform-qa-bot (platform-qa-bot)]

PASSED: Continuous integration, rev:211
https://platform-qa-jenkins.ubuntu.com/job/qa-jenkins-jobs-ci/82/
Executed test runs:
    SUCCESS: https://platform-qa-jenkins.ubuntu.com/job/build-xenial-amd64-package/248
        deb: https://platform-qa-jenkins.ubuntu.com/job/build-xenial-amd64-package/248/artifact/work/output/*zip*/output.zip
    SUCCESS: https://platform-qa-jenkins.ubuntu.com/job/build-xenial-i386-package/246
        deb: https://platform-qa-jenkins.ubuntu.com/job/build-xenial-i386-package/246/artifact/work/output/*zip*/output.zip
    None: https://platform-qa-jenkins.ubuntu.com/job/generic-update-mp/1561/console

Click here to trigger a rebuild:
https://platform-qa-jenkins.ubuntu.com/job/qa-jenkins-jobs-ci/82/rebuild

review: Approve (continuous-integration)

[Max Brustkern (nuclearbob)]

If you've tested the deployment of these, they should be ready to go.

review: Approve

[Sergio Cazzolato (sergio-j-cazzolato)]

I tested the deployment and there are no errors now.

=== modified file 'jobs/macros.yaml'
--- jobs/macros.yaml	2017-02-03 18:18:46 +0000
+++ jobs/macros.yaml	2017-02-08 00:13:12 +0000
@@ -20,21 +20,29 @@
             cat media_info
 
 - builder:
-    name: grafana-upload
+    name: grafana-setup
     builders:
         - shell: |
-            if [ -z "$BUILDSTAMP" ] ; then
-                BUILDSTAMP=$(grep -o '([0-9.]\+)' media_info | tr -d '()')
-            fi
             bzr branch lp:custom-grafana-integration $WORKSPACE/grafana-integration
-            export GRAFANA_INTEGRATION_PATH=$WORKSPACE/grafana-integration
+            cat > $WORKSPACE/grafana.conf <<EOF
             export MONGO_HOST=10.25.236.10
             export MONGO_PORT=27017
             export CARBON_HOST=10.25.236.71
             export CARBON_PORT=2003
             export MSG_CHUNK_SIZE=500
             export MSG_WAIT=30
-            ### Add global_info
+            export GRAFANA_INTEGRATION_PATH=$WORKSPACE/grafana-integration
+            EOF
+
+- builder:
+    name: grafana-upload
+    builders:
+        - grafana-setup:
+        - shell: |
+            source $WORKSPACE/grafana.conf
+            if [ -z "$BUILDSTAMP" ] ; then
+                BUILDSTAMP=$(grep -o '([0-9.]\+)' media_info | tr -d '()')
+            fi
             cat > global_info.json <<EOF
             {{
                 "device": "{device}",
@@ -46,7 +54,6 @@
                 "build": "$BUILDSTAMP"
             }}
             EOF
-            ### Run grafana integration and send data to grafana server ###
             $GRAFANA_INTEGRATION_PATH/run_script sync-tests -rt xunit -rf {xunit_file} -if global_info.json -m tests_01 -t $(date +%s)
 
 - publisher: # This publisher archives all job artifacts

=== added directory 'jobs/performance'
=== added file 'jobs/performance/jobs.yaml'
--- jobs/performance/jobs.yaml	1970-01-01 00:00:00 +0000
+++ jobs/performance/jobs.yaml	2017-02-08 00:13:12 +0000
@@ -0,0 +1,156 @@
+# vim: sw=4 ts=4 et
+
+# QA Jenkins Jobs
+# Copyright 2017 Canonical Ltd.
+
+# This program is free software: you can redistribute it and/or modify it
+# under the terms of the GNU General Public License version 3, as published
+# by the Free Software Foundation.
+
+# This program is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranties of
+# MERCHANTABILITY, SATISFACTORY QUALITY, or FITNESS FOR A PARTICULAR
+# PURPOSE.  See the GNU General Public License for more details.
+
+# You should have received a copy of the GNU General Public License along
+# with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+# The project stanza describes all jobs and parameters
+- project:
+    name: performance-desktop
+    device: 'desktop'
+    model:
+        - amd-gfx
+        - intel-gfx
+        - nvidia-gfx
+    release:
+        - xenial
+        - zesty
+    arch: 'amd64'
+    unity8-mode:
+        - deb
+        - snap
+    jobs:
+        - 'kpi-tests-{device}-{model}-{release}-{arch}-unity8-{unity8-mode}'
+
+- project:
+    name: performance-board
+    device: 'board'
+    model: 'dragonboard'
+    arch: 'arm64'
+    release: 'core_16'
+    unity8-mode: 'snap'
+    jobs:
+        - 'kpi-tests-{device}-{model}-{release}-{arch}-unity8-{unity8-mode}'
+
+# builder stanzas describe reusable build steps
+- builder:
+    name: setup-env
+    builders:
+        - shell: |
+            rm -rf *
+            mkdir $WORKSPACE/results
+
+- builder:
+    name: branch-bzr-project
+    builders:
+        - shell: |
+            bzr branch {project-url} {project-name}
+
+- builder:
+    name: run-performance-tests
+    builders:
+        - shell: |
+            set -ex
+            BUILDID=$(date +%Y%m%d)
+            cat > job.json <<EOF
+            {{
+                "job_queue": "$DEVICE_QUEUE",
+                "provision_data": {{
+                    "distro": "{release}"
+                }},
+                 "test_data": {{
+                    "test_cmds": [
+                        "bzr branch $PERFORMANCE_TESTS_BRANCH ubuntu-performance-tests",
+                        "mkdir artifacts",
+                        "cd ubuntu-performance-tests && KPI_OS_BUILD=$BUILDID KPI_DEVICE_IP={{device_ip}} $ITER_PARAMS $EXEC_PARAMS $ENV_PARAMS ./run_tests desktop",
+                        "cp ubuntu-performance-tests/tests/tests.log artifacts"
+                    ]
+                }}
+            }}
+            EOF
+            python3 qakit/qakit/testflinger/runner.py -d job.json -o testflinger.log -t 7200 -a artifacts.tgz
+            tar -xzf artifacts.tgz
+            cp artifacts/tests.log $WORKSPACE/results/tests.log
+
+- builder:
+    name: sync-grafana
+    builders:
+        - grafana-setup
+        - shell: |
+            set -ex
+            source $WORKSPACE/grafana.conf
+            $GRAFANA_INTEGRATION_PATH/run_script sync-perf -l $WORKSPACE/results/tests.log -m performance_01 -t $(date +%s)
+
+- job-template:
+    name: kpi-tests-{device}-{model}-{release}-{arch}-unity8-{unity8-mode}
+    description: |
+        Run performance tests
+        Defined in lp:qa-jenkins-jobs
+    node: 'jenkins-slave-2'
+    triggers:
+        - timed: 'H H * * *'
+    wrappers:
+        - timeout:
+            timeout: 180
+            fail: true
+    parameters:
+        - string:
+            name: PERFORMANCE_TESTS_BRANCH
+            default: lp:ubuntu-performance-tests
+            description: "Launchpad branch of ubuntu-performance-tests from which to run tests."
+        - string:
+            name: QAKIT_BRANCH
+            default: lp:qakit
+            description: Launchpad branch of qakit project to use testflinger.
+        - string:
+            name: GRAFANA_INTEGRATION_BRANCH
+            default: lp:custom-grafana-integration
+            description: "Launchpad branch of custom-grafana-integration project to process the performnace results."
+        - string:
+            name: RELEASE
+            default: '{release}'
+            description: "The given alternate distro will be used if specified."
+        - string:
+            name: DEVICE_QUEUE
+            default: '{model}'
+            description: "The device where it is going to be executed."
+        - string:
+            name: ITER_PARAMS
+            default: KPI_DEFAULT_RETRIES=1 KPI_APPLAUNCH_RETRIES=0 KPI_WEBBROWSER_RETRIES=0 KPI_DROMAEO_RETRIES=0
+            description: "Iterations for tests"
+        - string:
+            name: EXECUTION_PARAMETERS
+            default: KPI_SSH_TIMEOUT=600000  KPI_USER=ubuntu
+            description: "Parametes used to execute the tests"
+        - string:
+            name: ENV_PARAMS
+            default: KPI_DEVICE_TYPE={device} KPI_DEVICE_MODEL={model} KPI_OS_RELEASE={release} KPI_OS_ARCH={arch} KPI_UNITY8_MODE={unity8-mode}
+            description: "Environment parameters"
+    builders:
+        - setup-env:
+        - branch-bzr-project:
+            project-url: '$QAKIT_BRANCH'
+            project-name: 'qakit'
+        - branch-bzr-project:
+            project-url: '$PERFORMANCE_TESTS_BRANCH'
+            project-name: 'ubuntu-performance'
+        - run-performance-tests:
+            device: '{device}'
+            model: '{model}'
+            release: '{release}'
+            arch: '{arch}'
+            unity8-mode: '{unity8-mode}'
+        - sync-grafana:
+    publishers:
+        - archive-artifacts

=== removed directory 'scripts/performance'
=== removed file 'scripts/performance/run_performance_qakit.sh'
--- scripts/performance/run_performance_qakit.sh	2016-09-21 22:09:51 +0000
+++ scripts/performance/run_performance_qakit.sh	1970-01-01 00:00:00 +0000
@@ -1,21 +0,0 @@
-#!/bin/bash
-
-# QA Jenkins Jobs
-# Copyright 2016 Canonical Ltd.
-#
-# This program is free software: you can redistribute it and/or modify it
-# under the terms of the GNU General Public License version 3, as published
-# by the Free Software Foundation.
-#
-# This program is distributed in the hope that it will be useful, but
-# WITHOUT ANY WARRANTY; without even the implied warranties of
-# MERCHANTABILITY, SATISFACTORY QUALITY, or FITNESS FOR A PARTICULAR
-# PURPOSE.  See the GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License along
-# with this program.  If not, see <http://www.gnu.org/licenses/>.
-
-set -ex
-
-python3 $QAKIT_PATH/qakit/prodenv/perf/sync_perf.py save $KPI_LOG
-python3 $QAKIT_PATH/qakit/prodenv/perf/sync_perf.py update -b $BUILD_NUMBER -c $CHANNEL -d $DEVICE

=== removed file 'scripts/performance/run_performance_tests.sh'
--- scripts/performance/run_performance_tests.sh	2016-08-25 19:48:42 +0000
+++ scripts/performance/run_performance_tests.sh	1970-01-01 00:00:00 +0000
@@ -1,31 +0,0 @@
-#!/bin/bash
-
-# QA Jenkins Jobs
-# Copyright 2016 Canonical Ltd.
-#
-# This program is free software: you can redistribute it and/or modify it
-# under the terms of the GNU General Public License version 3, as published
-# by the Free Software Foundation.
-#
-# This program is distributed in the hope that it will be useful, but
-# WITHOUT ANY WARRANTY; without even the implied warranties of
-# MERCHANTABILITY, SATISFACTORY QUALITY, or FITNESS FOR A PARTICULAR
-# PURPOSE.  See the GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License along
-# with this program.  If not, see <http://www.gnu.org/licenses/>.
-
-set -ex
-
-cd $PERFORMANCE_TESTS_PATH
-export PYTHONPATH=$PERFORMANCE_TESTS_PATH
-
-if [ $RUN_TESTS = true ]; then
-	python3 kpi/runner.py -k applaunch -s $ANDROID_SERIAL -i $APPLAUNCH_ITERS -w 15
-	python3 kpi/runner.py -k reboot -s $ANDROID_SERIAL -i $REBOOT_ITERS
-    python3 kpi/runner.py -k idlecputime -s $ANDROID_SERIAL -i $CPU_ITERS -t 120
-	python3 kpi/runner.py -k webbrowser -s $ANDROID_SERIAL -i $WEB_BROWSER_ITERS
-	python3 kpi/runner.py -k dashqmlmetrics -s $ANDROID_SERIAL -i $DASH_ITERS -t
-else
-	python3 kpi/runner.py -k fake -s $ANDROID_SERIAL
-fi