snappy-hwe-snaps

Merge ~morphis/snappy-hwe-snaps/+git/jenkins-jobs:f/import-jobs into ~snappy-hwe-team/snappy-hwe-snaps/+git/jenkins-jobs:master

  1. Git
  2. lp:~morphis/snappy-hwe-snaps/+git/jenkins-jobs
  3. f/import-jobs
  4. Merge into master
Proposed by Simon Fels
Status: Merged
Merged at revision: 33165787bfe19d008b348128b397caaf7eb9f328
Proposed branch: ~morphis/snappy-hwe-snaps/+git/jenkins-jobs:f/import-jobs
Merge into: ~snappy-hwe-team/snappy-hwe-snaps/+git/jenkins-jobs:master
Diff against target: 3299 lines (+3017/-0)
46 files modified
README.md (+48/-0)
docker/Dockerfile (+59/-0)
docker/initial-setup.groovy (+34/-0)
docker/jenkins.sh (+24/-0)
docker/plugins.sh (+28/-0)
docker/plugins.txt (+29/-0)
jobs/infrastructure/credentials-0-ssh.sh (+58/-0)
jobs/infrastructure/credentials-0-ssh.yaml (+30/-0)
jobs/infrastructure/credentials-1-launchpad.py (+86/-0)
jobs/infrastructure/credentials-1-launchpad.yaml (+22/-0)
jobs/infrastructure/credentials-2-launchpad-plugin.sh (+108/-0)
jobs/infrastructure/credentials-2-launchpad-plugin.yaml (+25/-0)
jobs/infrastructure/prepare-0-install.sh (+38/-0)
jobs/infrastructure/prepare-0-install.yaml (+28/-0)
jobs/snap/common-job-prepare.sh (+19/-0)
jobs/snap/snap-build-worker.sh (+120/-0)
jobs/snap/snap-build-worker.yaml (+73/-0)
jobs/snap/snap-build.yaml (+85/-0)
jobs/snap/snap-cleanup.sh (+34/-0)
jobs/snap/snap-cleanup.yaml (+33/-0)
jobs/snap/snap-nightly.yaml (+41/-0)
jobs/snap/snap-project-jobs.yaml (+11/-0)
jobs/snap/snap-release.sh (+90/-0)
jobs/snap/snap-release.yaml (+59/-0)
jobs/snap/snap-test.sh (+116/-0)
jobs/snap/snap-test.yaml (+65/-0)
jobs/snap/snap-trigger-ci.sh (+25/-0)
jobs/snap/snap-trigger-ci.yaml (+23/-0)
jobs/snap/snap-update-mp.sh (+25/-0)
jobs/snap/snap-update-mp.yaml (+33/-0)
local.conf (+11/-0)
local.yaml (+40/-0)
run-tests.sh (+2/-0)
system-enablement.conf (+9/-0)
system-enablement.yaml (+44/-0)
tools/automerge-snap-mp.py (+128/-0)
tools/build-rootfs-create (+26/-0)
tools/common.sh (+93/-0)
tools/delete-ci-repo.py (+45/-0)
tools/hardware-test.sh (+112/-0)
tools/se_utils/__init__.py (+122/-0)
tools/snapbuild.sh (+97/-0)
tools/test-snap.sh (+100/-0)
tools/trigger-lp-build.py (+230/-0)
tools/trigger-snap-builds.py (+218/-0)
tools/vote-on-merge-proposal.py (+271/-0)
Reviewer Review Type Date Requested Status
System Enablement Bot continuous-integration Approve
Review via email: mp+318603@code.launchpad.net

Description of the change

Initial import of all our existing jenkins jobs with their definition.

This is still WIP and needs further integration with our build-scripts repository.

Overall goal is to have this as a framework which can be easily reused by other teams.

To post a comment you must log in.
Revision history for this message
System Enablement Bot (system-enablement-ci-bot) wrote :

FAILED: Continuous integration, rev:9f01f6d43331bce39b4f49b8b228aeb89dec6d16
https://jenkins.canonical.com/system-enablement/job/generic-build-snap/1074/
Executed test runs:
    FAILURE: https://jenkins.canonical.com/system-enablement/job/generic-build-snap-worker/75/console
    None: https://jenkins.canonical.com/system-enablement/job/generic-update-snap-mp/982/console
    FAILURE: https://jenkins.canonical.com/system-enablement/job/generic-test-snap/782/console

Click here to trigger a rebuild:
https://jenkins.canonical.com/system-enablement/job/generic-build-snap/1074/rebuild

review: Needs Fixing (continuous-integration)
Revision history for this message
System Enablement Bot (system-enablement-ci-bot) wrote :

FAILED: Continuous integration, rev:4e1b21ecf459168018a3c39bd12e1af8bf6b5216
https://jenkins.canonical.com/system-enablement/job/generic-build-snap/1075/
Executed test runs:
    FAILURE: https://jenkins.canonical.com/system-enablement/job/generic-build-snap-worker/76/console
    None: https://jenkins.canonical.com/system-enablement/job/generic-update-snap-mp/983/console
    FAILURE: https://jenkins.canonical.com/system-enablement/job/generic-test-snap/783/console

Click here to trigger a rebuild:
https://jenkins.canonical.com/system-enablement/job/generic-build-snap/1075/rebuild

review: Needs Fixing (continuous-integration)
Revision history for this message
System Enablement Bot (system-enablement-ci-bot) wrote :

FAILED: Continuous integration, rev:cfa8d931cc1552ea7a05a6cb60c1f5b0c2c1a3f6
https://jenkins.canonical.com/system-enablement/job/generic-build-snap/1079/
Executed test runs:
    FAILURE: https://jenkins.canonical.com/system-enablement/job/generic-build-snap-worker/80/console
    None: https://jenkins.canonical.com/system-enablement/job/generic-update-snap-mp/987/console
    FAILURE: https://jenkins.canonical.com/system-enablement/job/generic-test-snap/789/console

Click here to trigger a rebuild:
https://jenkins.canonical.com/system-enablement/job/generic-build-snap/1079/rebuild

review: Needs Fixing (continuous-integration)
Revision history for this message
System Enablement Bot (system-enablement-ci-bot) wrote :

FAILED: Continuous integration, rev:6304aeefe4a254b032c38234b2bd6790e3b35b39
https://jenkins.canonical.com/system-enablement/job/generic-build-snap/1081/
Executed test runs:
    FAILURE: https://jenkins.canonical.com/system-enablement/job/generic-build-snap-worker/82/console
    None: https://jenkins.canonical.com/system-enablement/job/generic-update-snap-mp/989/console
    FAILURE: https://jenkins.canonical.com/system-enablement/job/generic-test-snap/792/console

Click here to trigger a rebuild:
https://jenkins.canonical.com/system-enablement/job/generic-build-snap/1081/rebuild

review: Needs Fixing (continuous-integration)
Revision history for this message
System Enablement Bot (system-enablement-ci-bot) wrote :

FAILED: Continuous integration, rev:5e5c338a3931c15a7a32de319aaedb65a8ce08d1
https://jenkins.canonical.com/system-enablement/job/generic-build-snap/1082/
Executed test runs:
    FAILURE: https://jenkins.canonical.com/system-enablement/job/generic-build-snap-worker/83/console
    None: https://jenkins.canonical.com/system-enablement/job/generic-update-snap-mp/990/console
    FAILURE: https://jenkins.canonical.com/system-enablement/job/generic-test-snap/793/console

Click here to trigger a rebuild:
https://jenkins.canonical.com/system-enablement/job/generic-build-snap/1082/rebuild

review: Needs Fixing (continuous-integration)
Revision history for this message
System Enablement Bot (system-enablement-ci-bot) wrote :

FAILED: Continuous integration, rev:0fe2d5cf69a853e0686e48b14d2a681680f19cfe
https://jenkins.canonical.com/system-enablement/job/generic-build-snap/1083/
Executed test runs:
    FAILURE: https://jenkins.canonical.com/system-enablement/job/generic-build-snap-worker/84/console
    None: https://jenkins.canonical.com/system-enablement/job/generic-update-snap-mp/991/console
    FAILURE: https://jenkins.canonical.com/system-enablement/job/generic-test-snap/794/console

Click here to trigger a rebuild:
https://jenkins.canonical.com/system-enablement/job/generic-build-snap/1083/rebuild

review: Needs Fixing (continuous-integration)
Revision history for this message
System Enablement Bot (system-enablement-ci-bot) wrote :

FAILED: Continuous integration, rev:c1fc8538728e79fbd50c4b256675a937583304ef
https://jenkins.canonical.com/system-enablement/job/generic-build-snap/1084/
Executed test runs:
    FAILURE: https://jenkins.canonical.com/system-enablement/job/generic-build-snap-worker/85/console
    None: https://jenkins.canonical.com/system-enablement/job/generic-update-snap-mp/992/console
    FAILURE: https://jenkins.canonical.com/system-enablement/job/generic-test-snap/795/console

Click here to trigger a rebuild:
https://jenkins.canonical.com/system-enablement/job/generic-build-snap/1084/rebuild

review: Needs Fixing (continuous-integration)
Revision history for this message
System Enablement Bot (system-enablement-ci-bot) wrote :

FAILED: Continuous integration, rev:32f5fb3821bda0a22b1b014b2c8ec17accaf5fc1
https://jenkins.canonical.com/system-enablement/job/generic-build-snap/1086/
Executed test runs:
    FAILURE: https://jenkins.canonical.com/system-enablement/job/generic-build-snap-worker/87/console
    None: https://jenkins.canonical.com/system-enablement/job/generic-update-snap-mp/993/console
    FAILURE: https://jenkins.canonical.com/system-enablement/job/generic-test-snap/798/console

Click here to trigger a rebuild:
https://jenkins.canonical.com/system-enablement/job/generic-build-snap/1086/rebuild

review: Needs Fixing (continuous-integration)
Revision history for this message
System Enablement Bot (system-enablement-ci-bot) wrote :

PASSED: Continuous integration, rev:4c18a86460a121bf5fba094a90d6807d110d7b47
https://jenkins.canonical.com/system-enablement/job/generic-build-snap/1087/
Executed test runs:
    SUCCESS: https://jenkins.canonical.com/system-enablement/job/generic-build-snap-worker/88
    SUCCESS: https://jenkins.canonical.com/system-enablement/job/generic-test-snap/800
    None: https://jenkins.canonical.com/system-enablement/job/generic-update-snap-mp/995/console
    SUCCESS: https://jenkins.canonical.com/system-enablement/job/generic-test-snap/799

Click here to trigger a rebuild:
https://jenkins.canonical.com/system-enablement/job/generic-build-snap/1087/rebuild

review: Approve (continuous-integration)
Revision history for this message
System Enablement Bot (system-enablement-ci-bot) wrote :

PASSED: Continuous integration, rev:302777290cadd46bed769f42b12cda0c8627e526
https://jenkins.canonical.com/system-enablement/job/generic-build-snap/1091/
Executed test runs:
    SUCCESS: https://jenkins.canonical.com/system-enablement/job/generic-build-snap-worker/92
    None: https://jenkins.canonical.com/system-enablement/job/generic-update-snap-mp/999/console
    SUCCESS: https://jenkins.canonical.com/system-enablement/job/generic-test-snap/805

Click here to trigger a rebuild:
https://jenkins.canonical.com/system-enablement/job/generic-build-snap/1091/rebuild

review: Approve (continuous-integration)
Revision history for this message
System Enablement Bot (system-enablement-ci-bot) wrote :

PASSED: Continuous integration, rev:1e0f0449cbaeaffb59289e600adea7441bfc97f1
https://jenkins.canonical.com/system-enablement/job/generic-build-snap/1092/
Executed test runs:
    SUCCESS: https://jenkins.canonical.com/system-enablement/job/generic-build-snap-worker/93
    None: https://jenkins.canonical.com/system-enablement/job/generic-update-snap-mp/1000/console
    SUCCESS: https://jenkins.canonical.com/system-enablement/job/generic-test-snap/806

Click here to trigger a rebuild:
https://jenkins.canonical.com/system-enablement/job/generic-build-snap/1092/rebuild

review: Approve (continuous-integration)
Revision history for this message
System Enablement Bot (system-enablement-ci-bot) wrote :

PASSED: Continuous integration, rev:d2043c008bbaeb80251a38b82e7bdbc114df9378
https://jenkins.canonical.com/system-enablement/job/generic-build-snap/1093/
Executed test runs:
    SUCCESS: https://jenkins.canonical.com/system-enablement/job/generic-build-snap-worker/94
    None: https://jenkins.canonical.com/system-enablement/job/generic-update-snap-mp/1001/console
    SUCCESS: https://jenkins.canonical.com/system-enablement/job/generic-test-snap/807

Click here to trigger a rebuild:
https://jenkins.canonical.com/system-enablement/job/generic-build-snap/1093/rebuild

review: Approve (continuous-integration)
Revision history for this message
System Enablement Bot (system-enablement-ci-bot) wrote :

PASSED: Continuous integration, rev:1db17d13cc2ff01e712b780429d635c3045f0b80
https://jenkins.canonical.com/system-enablement/job/generic-build-snap/1094/
Executed test runs:
    SUCCESS: https://jenkins.canonical.com/system-enablement/job/generic-build-snap-worker/95
    None: https://jenkins.canonical.com/system-enablement/job/generic-update-snap-mp/1002/console
    SUCCESS: https://jenkins.canonical.com/system-enablement/job/generic-test-snap/808

Click here to trigger a rebuild:
https://jenkins.canonical.com/system-enablement/job/generic-build-snap/1094/rebuild

review: Approve (continuous-integration)
Revision history for this message
System Enablement Bot (system-enablement-ci-bot) wrote :

PASSED: Continuous integration, rev:cfea0ce16bbc011dafcc955d7c6871c5f0f6cf2f
https://jenkins.canonical.com/system-enablement/job/generic-build-snap/1095/
Executed test runs:
    SUCCESS: https://jenkins.canonical.com/system-enablement/job/generic-build-snap-worker/96
    None: https://jenkins.canonical.com/system-enablement/job/generic-update-snap-mp/1003/console
    SUCCESS: https://jenkins.canonical.com/system-enablement/job/generic-test-snap/809

Click here to trigger a rebuild:
https://jenkins.canonical.com/system-enablement/job/generic-build-snap/1095/rebuild

review: Approve (continuous-integration)
Revision history for this message
System Enablement Bot (system-enablement-ci-bot) wrote :

PASSED: Continuous integration, rev:7e1f8fa55c341e3951cd9db169cfee0ade2942c8
https://jenkins.canonical.com/system-enablement/job/generic-build-snap/1096/
Executed test runs:
    SUCCESS: https://jenkins.canonical.com/system-enablement/job/generic-build-snap-worker/97
    None: https://jenkins.canonical.com/system-enablement/job/generic-update-snap-mp/1004/console
    SUCCESS: https://jenkins.canonical.com/system-enablement/job/generic-test-snap/810

Click here to trigger a rebuild:
https://jenkins.canonical.com/system-enablement/job/generic-build-snap/1096/rebuild

review: Approve (continuous-integration)
Revision history for this message
System Enablement Bot (system-enablement-ci-bot) wrote :

PASSED: Continuous integration, rev:b7786e41c545b503849758970fd497c4843fc4fd
https://jenkins.canonical.com/system-enablement/job/generic-build-snap/1097/
Executed test runs:
    SUCCESS: https://jenkins.canonical.com/system-enablement/job/generic-build-snap-worker/98
    None: https://jenkins.canonical.com/system-enablement/job/generic-update-snap-mp/1005/console
    SUCCESS: https://jenkins.canonical.com/system-enablement/job/generic-test-snap/811

Click here to trigger a rebuild:
https://jenkins.canonical.com/system-enablement/job/generic-build-snap/1097/rebuild

review: Approve (continuous-integration)
Revision history for this message
System Enablement Bot (system-enablement-ci-bot) wrote :

PASSED: Continuous integration, rev:4bdbbfdf0ebf00660f740df03194663c4a60ee1c
https://jenkins.canonical.com/system-enablement/job/generic-build-snap/1098/
Executed test runs:
    SUCCESS: https://jenkins.canonical.com/system-enablement/job/generic-build-snap-worker/99
    None: https://jenkins.canonical.com/system-enablement/job/generic-update-snap-mp/1006/console
    SUCCESS: https://jenkins.canonical.com/system-enablement/job/generic-test-snap/812

Click here to trigger a rebuild:
https://jenkins.canonical.com/system-enablement/job/generic-build-snap/1098/rebuild

review: Approve (continuous-integration)
Revision history for this message
System Enablement Bot (system-enablement-ci-bot) wrote :

FAILED: Continuous integration, rev:bb2432a2aa0e89d23d2030b4157e4db1b99c205d
https://jenkins.canonical.com/system-enablement/job/generic-build-snap/1120/
Executed test runs:
    FAILURE: https://jenkins.canonical.com/system-enablement/job/generic-build-snap-worker/123/console
    None: https://jenkins.canonical.com/system-enablement/job/generic-update-snap-mp/1028/console
    FAILURE: https://jenkins.canonical.com/system-enablement/job/generic-test-snap/823/console

Click here to trigger a rebuild:
https://jenkins.canonical.com/system-enablement/job/generic-build-snap/1120/rebuild

review: Needs Fixing (continuous-integration)
Revision history for this message
System Enablement Bot (system-enablement-ci-bot) wrote :

FAILED: Continuous integration, rev:bbd591b50561791a77c7dc9a74a4feada92956cd
https://jenkins.canonical.com/system-enablement/job/generic-build-snap/1121/
Executed test runs:
    FAILURE: https://jenkins.canonical.com/system-enablement/job/generic-build-snap-worker/124/console
    None: https://jenkins.canonical.com/system-enablement/job/generic-update-snap-mp/1029/console
    FAILURE: https://jenkins.canonical.com/system-enablement/job/generic-test-snap/824/console

Click here to trigger a rebuild:
https://jenkins.canonical.com/system-enablement/job/generic-build-snap/1121/rebuild

review: Needs Fixing (continuous-integration)
Revision history for this message
System Enablement Bot (system-enablement-ci-bot) wrote :

FAILED: Continuous integration, rev:c9e1b6ae09bfb22417acb6ef92f861104e366b49
https://jenkins.canonical.com/system-enablement/job/generic-build-snap/1127/
Executed test runs:
    FAILURE: https://jenkins.canonical.com/system-enablement/job/generic-build-snap-worker/130/console
    None: https://jenkins.canonical.com/system-enablement/job/generic-update-snap-mp/1035/console
    FAILURE: https://jenkins.canonical.com/system-enablement/job/generic-test-snap/829/console

Click here to trigger a rebuild:
https://jenkins.canonical.com/system-enablement/job/generic-build-snap/1127/rebuild

review: Needs Fixing (continuous-integration)
Revision history for this message
System Enablement Bot (system-enablement-ci-bot) wrote :

FAILED: Continuous integration, rev:2483f1f63d6f370defebb9006bd6f0b1607acb53
https://jenkins.canonical.com/system-enablement/job/generic-build-snap/1130/
Executed test runs:
    FAILURE: https://jenkins.canonical.com/system-enablement/job/generic-build-snap-worker/133/console
    None: https://jenkins.canonical.com/system-enablement/job/generic-update-snap-mp/1038/console
    FAILURE: https://jenkins.canonical.com/system-enablement/job/generic-test-snap/831/console

Click here to trigger a rebuild:
https://jenkins.canonical.com/system-enablement/job/generic-build-snap/1130/rebuild

review: Needs Fixing (continuous-integration)
Revision history for this message
System Enablement Bot (system-enablement-ci-bot) wrote :

PASSED: Continuous integration, rev:c2ccdf29e332e7e7920969ede9de82d6a374a521
https://jenkins.canonical.com/system-enablement/job/generic-build-snap/1536/
Executed test runs:
    SUCCESS: https://jenkins.canonical.com/system-enablement/job/generic-build-snap-worker/1816
    None: https://jenkins.canonical.com/system-enablement/job/generic-update-snap-mp/1444/console
    SUCCESS: https://jenkins.canonical.com/system-enablement/job/generic-test-snap/2198

Click here to trigger a rebuild:
https://jenkins.canonical.com/system-enablement/job/generic-build-snap/1536/rebuild

review: Approve (continuous-integration)
Revision history for this message
System Enablement Bot (system-enablement-ci-bot) wrote :

PASSED: Continuous integration, rev:32a95f97e6feb5a35b846713f6bf7f99bfe21c09
https://jenkins.canonical.com/system-enablement/job/generic-build-snap/1537/
Executed test runs:
    SUCCESS: https://jenkins.canonical.com/system-enablement/job/generic-build-snap-worker/1817
    None: https://jenkins.canonical.com/system-enablement/job/generic-update-snap-mp/1445/console
    SUCCESS: https://jenkins.canonical.com/system-enablement/job/generic-test-snap/2199

Click here to trigger a rebuild:
https://jenkins.canonical.com/system-enablement/job/generic-build-snap/1537/rebuild

review: Approve (continuous-integration)
Revision history for this message
System Enablement Bot (system-enablement-ci-bot) wrote :

PASSED: Continuous integration, rev:e3b89eaf92b7f5d307a44345118269f3f78bbfd0
https://jenkins.canonical.com/system-enablement/job/generic-build-snap/1545/
Executed test runs:
    SUCCESS: https://jenkins.canonical.com/system-enablement/job/generic-build-snap-worker/1835
    None: https://jenkins.canonical.com/system-enablement/job/generic-update-snap-mp/1453/console
    SUCCESS: https://jenkins.canonical.com/system-enablement/job/generic-test-snap/2217

Click here to trigger a rebuild:
https://jenkins.canonical.com/system-enablement/job/generic-build-snap/1545/rebuild

review: Approve (continuous-integration)
Revision history for this message
System Enablement Bot (system-enablement-ci-bot) wrote :

PASSED: Continuous integration, rev:26da714e2afb31a134020ad26672050bb0a3df64
https://jenkins.canonical.com/system-enablement/job/generic-build-snap/1546/
Executed test runs:
    SUCCESS: https://jenkins.canonical.com/system-enablement/job/generic-build-snap-worker/1836
    None: https://jenkins.canonical.com/system-enablement/job/generic-update-snap-mp/1454/console
    SUCCESS: https://jenkins.canonical.com/system-enablement/job/generic-test-snap/2218

Click here to trigger a rebuild:
https://jenkins.canonical.com/system-enablement/job/generic-build-snap/1546/rebuild

review: Approve (continuous-integration)
Revision history for this message
System Enablement Bot (system-enablement-ci-bot) wrote :

PASSED: Continuous integration, rev:764f0a66dcb1010692095f24924e7a615daa4b91
https://jenkins.canonical.com/system-enablement/job/generic-build-snap/1599/
Executed test runs:
    SUCCESS: https://jenkins.canonical.com/system-enablement/job/generic-build-snap-worker/1989
    None: https://jenkins.canonical.com/system-enablement/job/generic-update-snap-mp/1507/console
    SUCCESS: https://jenkins.canonical.com/system-enablement/job/generic-test-snap/2362

Click here to trigger a rebuild:
https://jenkins.canonical.com/system-enablement/job/generic-build-snap/1599/rebuild

review: Approve (continuous-integration)
Revision history for this message
System Enablement Bot (system-enablement-ci-bot) wrote :

PASSED: Continuous integration, rev:7154f95586ca5e9877b4e65e364e6868a3801e59
https://jenkins.canonical.com/system-enablement/job/generic-build-snap/1600/
Executed test runs:
    SUCCESS: https://jenkins.canonical.com/system-enablement/job/generic-build-snap-worker/1990
    None: https://jenkins.canonical.com/system-enablement/job/generic-update-snap-mp/1508/console
    SUCCESS: https://jenkins.canonical.com/system-enablement/job/generic-test-snap/2363

Click here to trigger a rebuild:
https://jenkins.canonical.com/system-enablement/job/generic-build-snap/1600/rebuild

review: Approve (continuous-integration)
Revision history for this message
System Enablement Bot (system-enablement-ci-bot) wrote :

PASSED: Continuous integration, rev:356106d4e5012fce219dc923c33da611fe9b6dc0
https://jenkins.canonical.com/system-enablement/job/generic-build-snap/1602/
Executed test runs:
    SUCCESS: https://jenkins.canonical.com/system-enablement/job/generic-build-snap-worker/1992
    None: https://jenkins.canonical.com/system-enablement/job/generic-update-snap-mp/1509/console
    SUCCESS: https://jenkins.canonical.com/system-enablement/job/generic-test-snap/2365

Click here to trigger a rebuild:
https://jenkins.canonical.com/system-enablement/job/generic-build-snap/1602/rebuild

review: Approve (continuous-integration)
Revision history for this message
System Enablement Bot (system-enablement-ci-bot) wrote :

PASSED: Continuous integration, rev:76ac3670a762d610fd5aaed73610344e9f93bce2
https://jenkins.canonical.com/system-enablement/job/generic-build-snap/1603/
Executed test runs:
    SUCCESS: https://jenkins.canonical.com/system-enablement/job/generic-build-snap-worker/1993
    None: https://jenkins.canonical.com/system-enablement/job/generic-update-snap-mp/1511/console
    SUCCESS: https://jenkins.canonical.com/system-enablement/job/generic-test-snap/2366

Click here to trigger a rebuild:
https://jenkins.canonical.com/system-enablement/job/generic-build-snap/1603/rebuild

review: Approve (continuous-integration)
Revision history for this message
System Enablement Bot (system-enablement-ci-bot) wrote :

PASSED: Continuous integration, rev:d73916bfb82d2a989b0349ba88af835440526220
https://jenkins.canonical.com/system-enablement/job/generic-build-snap/1606/
Executed test runs:
    SUCCESS: https://jenkins.canonical.com/system-enablement/job/generic-build-snap-worker/1996
    None: https://jenkins.canonical.com/system-enablement/job/generic-update-snap-mp/1514/console
    SUCCESS: https://jenkins.canonical.com/system-enablement/job/generic-test-snap/2369

Click here to trigger a rebuild:
https://jenkins.canonical.com/system-enablement/job/generic-build-snap/1606/rebuild

review: Approve (continuous-integration)
Revision history for this message
System Enablement Bot (system-enablement-ci-bot) wrote :

PASSED: Continuous integration, rev:92ed16f7bafb8f13667dc5ed9c9b8875f95ba162
https://jenkins.canonical.com/system-enablement/job/generic-build-snap/1625/
Executed test runs:
    SUCCESS: https://jenkins.canonical.com/system-enablement/job/generic-build-snap-worker/2065
    None: https://jenkins.canonical.com/system-enablement/job/generic-update-snap-mp/1532/console
    SUCCESS: https://jenkins.canonical.com/system-enablement/job/generic-test-snap/2436

Click here to trigger a rebuild:
https://jenkins.canonical.com/system-enablement/job/generic-build-snap/1625/rebuild

review: Approve (continuous-integration)
Revision history for this message
System Enablement Bot (system-enablement-ci-bot) wrote :

PASSED: Continuous integration, rev:886cafccda52c477f602a0b336b8eb6eaaa1b8d1
https://jenkins.canonical.com/system-enablement/job/generic-build-snap/1630/
Executed test runs:
    SUCCESS: https://jenkins.canonical.com/system-enablement/job/generic-build-snap-worker/2070
    None: https://jenkins.canonical.com/system-enablement/job/generic-update-snap-mp/1536/console
    SUCCESS: https://jenkins.canonical.com/system-enablement/job/generic-test-snap/2440

Click here to trigger a rebuild:
https://jenkins.canonical.com/system-enablement/job/generic-build-snap/1630/rebuild

review: Approve (continuous-integration)
Revision history for this message
System Enablement Bot (system-enablement-ci-bot) wrote :

PASSED: Continuous integration, rev:1296968056f293dd896f6b9ab744ecc2200b4024
https://jenkins.canonical.com/system-enablement/job/generic-build-snap/1631/
Executed test runs:
    SUCCESS: https://jenkins.canonical.com/system-enablement/job/generic-build-snap-worker/2071
    None: https://jenkins.canonical.com/system-enablement/job/generic-update-snap-mp/1538/console
    SUCCESS: https://jenkins.canonical.com/system-enablement/job/generic-test-snap/2442

Click here to trigger a rebuild:
https://jenkins.canonical.com/system-enablement/job/generic-build-snap/1631/rebuild

review: Approve (continuous-integration)
Revision history for this message
System Enablement Bot (system-enablement-ci-bot) wrote :

PASSED: Continuous integration, rev:33165787bfe19d008b348128b397caaf7eb9f328
https://jenkins.canonical.com/system-enablement/job/generic-build-snap/1633/
Executed test runs:
    SUCCESS: https://jenkins.canonical.com/system-enablement/job/generic-build-snap-worker/2073
    None: https://jenkins.canonical.com/system-enablement/job/generic-update-snap-mp/1539/console
    SUCCESS: https://jenkins.canonical.com/system-enablement/job/generic-test-snap/2443

Click here to trigger a rebuild:
https://jenkins.canonical.com/system-enablement/job/generic-build-snap/1633/rebuild

review: Approve (continuous-integration)

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1diff --git a/README.md b/README.md
2index eba5d73..cf42275 100644
3--- a/README.md
4+++ b/README.md
5@@ -3,3 +3,51 @@
6 This is a repository collecting a set of jenkins job definitions
7 used to build a contineous integrate pipeline for snap based
8 components.
9+
10+## Required Jenkins Plugins
11+
12+ * build-name-setter
13+ * conditional-buildstep
14+ * description-setter
15+ * dynamic-axis
16+ * matrix-combinations-parameter
17+ * matrix-project
18+ * nodelabelparameter
19+ * parameterized-trigger
20+ * rebuild
21+ * timestamper
22+ * ws-cleanup
23+
24+## Test locally
25+
26+Build the docker container which comes with this repository:
27+
28+```
29+$ cd docker
30+$ sudo docker build -t se-jenkins .
31+```
32+
33+Spawn up a docker container with jenkins
34+
35+```
36+$ sudo docker run -p 8080:8080 -p 50000:50000 --name jenkins se-jenkins
37+```
38+
39+You can now login into jenkins on http://localhost:8080 with user 'system-enablement-ci-bot'
40+and password 'jenkins'.
41+
42+Install jenkins-job-builder on your host via
43+
44+```
45+$ sudo apt install python3-pip
46+$ pip3 install jenkins-job-builder
47+```
48+
49+Now you should have everything setup and ready for a first deployment:
50+
51+```
52+$ jenkins-jobs --conf local.conf update local.yaml:jobs/
53+```
54+
55+This will create all configured jobs on the jenkins instance or update
56+them if already available.
57diff --git a/docker/Dockerfile b/docker/Dockerfile
58new file mode 100644
59index 0000000..a898d1e
60--- /dev/null
61+++ b/docker/Dockerfile
62@@ -0,0 +1,59 @@
63+FROM ubuntu:latest
64+
65+ENV DEBIAN_FRONTEND noninteractive
66+ENV INITRD No
67+ENV LANG en_US.UTF-8
68+
69+RUN apt-get update && \
70+ apt-get upgrade -y && \
71+ apt-get install -y --no-install-recommends \
72+ vim.tiny wget curl sudo net-tools pwgen \
73+ git-core logrotate software-properties-common locales openssh-client && \
74+ locale-gen en_US en_US.UTF-8 && \
75+ apt-get clean && \
76+ rm -rf /var/lib/apt/lists/*
77+
78+RUN apt-get update && \
79+ apt-get install --no-install-recommends -y openjdk-8-jre-headless && \
80+ apt-get clean && \
81+ rm -rf /var/lib/apt/lists/*
82+
83+RUN wget -qO - http://pkg.jenkins-ci.org/debian/jenkins-ci.org.key | sudo apt-key add - && \
84+ echo 'deb http://pkg.jenkins-ci.org/debian binary/' \
85+ | tee /etc/apt/sources.list.d/jenkins.list && \
86+ apt-get update && \
87+ apt-get install --no-install-recommends -y jenkins && \
88+ apt-get clean && \
89+ rm -rf /var/lib/apt/lists/* && \
90+ update-rc.d -f jenkins disable
91+
92+RUN usermod -d /var/jenkins jenkins
93+
94+# Passthrough all sudo requests without requiring a password for our jenkins user
95+RUN echo "jenkins ALL = NOPASSWD: ALL" > /etc/sudoers.d/jenkins
96+
97+RUN apt-get update && apt-get install --yes unzip zip wget
98+
99+ENV JENKINS_UC https://updates.jenkins-ci.org
100+ENV JENKINS_HOME /var/jenkins
101+ENV COPY_REFERENCE_FILE_LOG $JENKINS_HOME/copy_reference_file.log
102+ENV JENKINS_USER system-enablement-ci-bot
103+ENV JENKINS_PASS jenkins
104+
105+COPY plugins.sh /usr/local/bin/plugins.sh
106+RUN chmod +x /usr/local/bin/plugins.sh
107+COPY plugins.txt /tmp/plugins.txt
108+RUN /usr/local/bin/plugins.sh /tmp/plugins.txt
109+
110+COPY jenkins.sh /usr/local/bin/jenkins.sh
111+RUN chmod +x /usr/local/bin/jenkins.sh
112+
113+COPY initial-setup.groovy /usr/share/jenkins/ref/init.groovy.d/
114+
115+ENV JAVA_OPTS -Djenkins.install.runSetupWizard=false
116+
117+EXPOSE 8080 50000
118+
119+USER jenkins
120+
121+CMD ["/usr/local/bin/jenkins.sh"]
122diff --git a/docker/initial-setup.groovy b/docker/initial-setup.groovy
123new file mode 100644
124index 0000000..5a9598d
125--- /dev/null
126+++ b/docker/initial-setup.groovy
127@@ -0,0 +1,34 @@
128+import jenkins.model.*
129+import jenkins.security.*
130+import hudson.security.*
131+
132+def env = System.getenv()
133+
134+def jenkins = Jenkins.getInstance()
135+
136+jenkins.setLabelString("snap build test release misc monitor")
137+
138+jenkins.setSecurityRealm(new HudsonPrivateSecurityRealm(false))
139+jenkins.setAuthorizationStrategy(new GlobalMatrixAuthorizationStrategy())
140+
141+def user = jenkins.getSecurityRealm().createAccount(env.JENKINS_USER, env.JENKINS_PASS)
142+ApiTokenProperty t = user.getProperty(ApiTokenProperty.class)
143+def token = t.getApiTokenInsecure()
144+
145+println ""
146+println "########################################################################"
147+println "API token for user " + env.JENKINS_USER + " is " + token
148+println "########################################################################"
149+println ""
150+
151+user.save()
152+
153+jenkins.getAuthorizationStrategy().add(Jenkins.ADMINISTER, env.JENKINS_USER)
154+jenkins.save()
155+
156+for (slave in jenkins.model.Jenkins.instance.slaves) {
157+ println "Slave: " + slave.getNodeName() + "\n";
158+ println "Label: " + slave.getLabelString() + "\n\n";
159+ slave.setLabelString("snap build test monitor release misc")
160+}
161+jenkins.save()
162diff --git a/docker/jenkins.sh b/docker/jenkins.sh
163new file mode 100644
164index 0000000..5d095be
165--- /dev/null
166+++ b/docker/jenkins.sh
167@@ -0,0 +1,24 @@
168+#! /bin/bash
169+
170+set -e
171+
172+# Copy files from /usr/share/jenkins/ref into $JENKINS_HOME
173+# So the initial JENKINS-HOME is set with expected content.
174+# Don't override, as this is just a reference setup, and use from UI
175+# can then change this, upgrade plugins, etc.
176+copy_reference_file() {
177+ f=${1%/}
178+ rel=${f:23}
179+ dir=$(dirname ${f})
180+ if [[ ! -e $JENKINS_HOME/${rel} ]]
181+ then
182+ mkdir -p $JENKINS_HOME/${dir:23}
183+ cp -r /usr/share/jenkins/ref/${rel} $JENKINS_HOME/${rel};
184+ # pin plugins on initial copy
185+ [[ ${rel} == plugins/*.jpi ]] && touch $JENKINS_HOME/${rel}.pinned
186+ fi;
187+}
188+export -f copy_reference_file
189+find /usr/share/jenkins/ref/ -type f -exec bash -c "copy_reference_file '{}'" \;
190+
191+exec /usr/bin/java $JAVA_OPTS -jar /usr/share/jenkins/jenkins.war
192diff --git a/docker/plugins.sh b/docker/plugins.sh
193new file mode 100755
194index 0000000..ec8e8e5
195--- /dev/null
196+++ b/docker/plugins.sh
197@@ -0,0 +1,28 @@
198+#! /bin/bash
199+
200+# Parse a support-core plugin -style txt file as specification for jenkins plugins to be installed
201+# in the reference directory, so user can define a derived Docker image with just :
202+#
203+# FROM jenkins
204+# COPY plugins.txt /plugins.txt
205+# RUN /usr/local/bin/plugins.sh /plugins.txt
206+#
207+
208+set -e
209+
210+REF=/usr/share/jenkins/ref/plugins
211+mkdir -p $REF
212+
213+while read spec || [ -n "$spec" ]; do
214+ plugin=(${spec//:/ });
215+ [[ ${plugin[0]} =~ ^# ]] && continue
216+ [[ ${plugin[0]} =~ ^\s*$ ]] && continue
217+ [[ -z ${plugin[1]} ]] && plugin[1]="latest"
218+ echo "Downloading ${plugin[0]}:${plugin[1]}"
219+
220+ if [ -z "$JENKINS_UC_DOWNLOAD" ]; then
221+ JENKINS_UC_DOWNLOAD=$JENKINS_UC/download
222+ fi
223+ curl -sSL -f ${JENKINS_UC_DOWNLOAD}/plugins/${plugin[0]}/${plugin[1]}/${plugin[0]}.hpi -o $REF/${plugin[0]}.jpi
224+ unzip -qqt $REF/${plugin[0]}.jpi
225+done < $1
226diff --git a/docker/plugins.txt b/docker/plugins.txt
227new file mode 100644
228index 0000000..5020d33
229--- /dev/null
230+++ b/docker/plugins.txt
231@@ -0,0 +1,29 @@
232+ build-name-setter
233+ conditional-buildstep
234+ description-setter
235+ dynamic-axis
236+ matrix-combinations-parameter
237+ matrix-project
238+ nodelabelparameter
239+ parameterized-trigger
240+ rebuild
241+ timestamper
242+ ws-cleanup
243+ maven-plugin
244+ token-macro
245+ junit
246+ script-security
247+ jquery
248+ workflow-basic-steps
249+ structs
250+ javadoc
251+ workflow-api
252+ workflow-step-api
253+ mailer
254+ matrix-auth
255+ scm-api
256+ display-url-api
257+ scm-api
258+ icon-shim
259+ resource-disposer
260+ run-condition
261diff --git a/jobs/infrastructure/credentials-0-ssh.sh b/jobs/infrastructure/credentials-0-ssh.sh
262new file mode 100644
263index 0000000..d72fd49
264--- /dev/null
265+++ b/jobs/infrastructure/credentials-0-ssh.sh
266@@ -0,0 +1,58 @@
267+#!/bin/sh -ex
268+#
269+# Copyright (C) 2016 Canonical Ltd
270+#
271+# This program is free software: you can redistribute it and/or modify
272+# it under the terms of the GNU General Public License version 3 as
273+# published by the Free Software Foundation.
274+#
275+# This program is distributed in the hope that it will be useful,
276+# but WITHOUT ANY WARRANTY; without even the implied warranty of
277+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
278+# GNU General Public License for more details.
279+#
280+# You should have received a copy of the GNU General Public License
281+# along with this program. If not, see <http://www.gnu.org/licenses/>.
282+
283+# put it in $JENKINS_HOME so it doesn't get purged with the workspace
284+SSH_PATH="${{JENKINS_HOME}}/.ssh/"
285+
286+# this is so that this key's only used with git
287+SSH_KEY_PATH="${{SSH_PATH}}/git.launchpad.net/{bot_username}"
288+
289+if [ ! -d "${{SSH_KEY_PATH}}" ]; then
290+ mkdir -p "${{SSH_KEY_PATH}}"
291+fi
292+
293+# don't care about host keys, but use the one id key for launchpad
294+# TODO: use http://pad.lv/p/canonical-sshebang instead
295+cat > "${{SSH_PATH}}/config" << EOF
296+Host *
297+ StrictHostKeyChecking no
298+
299+Host git.launchpad.net
300+ User {bot_username}
301+ IdentityFile ~/.ssh/%h/%r/id_rsa
302+EOF
303+
304+# only use keys if both were uploaded, otherwise fail
305+if [ -f "keys/id_rsa" -o -f "keys/id_rsa.pub" ]; then
306+ [ -f "keys/id_rsa" -a -f "keys/id_rsa.pub" ] || \
307+ (echo "ERROR: You need to upload both keys, or none of them"; exit 1)
308+ mv "keys/id_rsa" "keys/id_rsa.pub" "${{SSH_KEY_PATH}}/"
309+ chmod 600 ${{SSH_KEY_PATH}}/*
310+fi
311+
312+# generate the keypair if none was uploaded or pre-existing
313+if [ ! -f "${{SSH_KEY_PATH}}/id_rsa" ]; then
314+ ssh-keygen -f "${{SSH_KEY_PATH}}/id_rsa"
315+fi
316+
317+# display the public key
318+echo "The public key for this jenkins user is:"
319+echo "----------"
320+cat "${{SSH_KEY_PATH}}/id_rsa.pub"
321+echo "----------"
322+
323+# Probe ssh public ssh key for relevant hosts and add it to our known_hosts file
324+ssh-keyscan -t rsa,dsa git.launchpad.net >> ${{SSH_PATH}}/known_hosts
325diff --git a/jobs/infrastructure/credentials-0-ssh.yaml b/jobs/infrastructure/credentials-0-ssh.yaml
326new file mode 100644
327index 0000000..c422660
328--- /dev/null
329+++ b/jobs/infrastructure/credentials-0-ssh.yaml
330@@ -0,0 +1,30 @@
331+- job-template:
332+ name: '{name}-credentials-0-ssh'
333+ project-type: matrix
334+ description: |
335+ This job will generate or store the supplied RSA keypair
336+ and display the public key for use with Launchpad.
337+ node: monitor
338+ properties:
339+ - build-discarder:
340+ num-to-keep: 1
341+ - rebuild
342+ parameters:
343+ - matrix-combinations:
344+ name: nodes
345+ - file:
346+ name: 'keys/id_rsa'
347+ description: Private RSA key
348+ - file:
349+ name: 'keys/id_rsa.pub'
350+ description: Public RSA key
351+ axes:
352+ - axis:
353+ type: slave
354+ name: node
355+ values: '{obj:build_slaves}'
356+ wrappers:
357+ - timestamps
358+ builders:
359+ - shell:
360+ !include-raw: credentials-0-ssh.sh
361diff --git a/jobs/infrastructure/credentials-1-launchpad.py b/jobs/infrastructure/credentials-1-launchpad.py
362new file mode 100644
363index 0000000..59dd706
364--- /dev/null
365+++ b/jobs/infrastructure/credentials-1-launchpad.py
366@@ -0,0 +1,86 @@
367+#!/usr/bin/env python
368+#
369+# Copyright (C) 2016 Canonical Ltd
370+#
371+# This program is free software: you can redistribute it and/or modify
372+# it under the terms of the GNU General Public License version 3 as
373+# published by the Free Software Foundation.
374+#
375+# This program is distributed in the hope that it will be useful,
376+# but WITHOUT ANY WARRANTY; without even the implied warranty of
377+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
378+# GNU General Public License for more details.
379+#
380+# You should have received a copy of the GNU General Public License
381+# along with this program. If not, see <http://www.gnu.org/licenses/>.
382+
383+import sys
384+import time
385+
386+from launchpadlib.credentials import RequestTokenAuthorizationEngine
387+from launchpadlib.credentials import UnencryptedFileCredentialStore
388+from launchpadlib.launchpad import Launchpad
389+from lazr.restfulclient.errors import HTTPError
390+
391+
392+ACCESS_TOKEN_POLL_TIME = 1
393+WAITING_FOR_USER = """Open this link:
394+{{}}
395+to authorize this program to access Launchpad on your behalf.
396+Waiting to hear from Launchpad about your decision. . . ."""
397+
398+
399+class AuthorizeRequestTokenWithConsole(RequestTokenAuthorizationEngine):
400+ """Authorize a token in a server environment (with no browser).
401+
402+ Print a link for the user to copy-and-paste into his/her browser
403+ for authentication.
404+ """
405+
406+ def __init__(self, *args, **kwargs):
407+ # as implemented in AuthorizeRequestTokenWithBrowser
408+ kwargs['consumer_name'] = None
409+ kwargs.pop('allow_access_levels', None)
410+ super(AuthorizeRequestTokenWithConsole, self).__init__(*args, **kwargs)
411+
412+ def make_end_user_authorize_token(self, credentials, request_token):
413+ """Ask the end-user to authorize the token in their browser.
414+
415+ """
416+ authorization_url = self.authorization_url(request_token)
417+ print(WAITING_FOR_USER.format(authorization_url))
418+ # if we don't flush we may not see the message
419+ sys.stdout.flush()
420+ while credentials.access_token is None:
421+ time.sleep(ACCESS_TOKEN_POLL_TIME)
422+ try:
423+ credentials.exchange_request_token_for_access_token(
424+ self.web_root)
425+ break
426+ except HTTPError as e:
427+ if e.response.status == 403:
428+ # The user decided not to authorize this
429+ # application.
430+ raise e
431+ elif e.response.status == 401:
432+ # The user has not made a decision yet.
433+ pass
434+ else:
435+ # There was an error accessing the server.
436+ raise e
437+
438+
439+def get_launchpad(cred_path, launchpadlib_dir=None):
440+ """ return a launchpad API class. In case launchpadlib_dir is
441+ specified used that directory to store launchpadlib cache instead of
442+ the default """
443+ store = UnencryptedFileCredentialStore(cred_path)
444+ authorization_engine = AuthorizeRequestTokenWithConsole(
445+ 'production', 'ci-jenkins-slave')
446+ return Launchpad.login_with('ci-jenkins-slave', 'production',
447+ credential_store=store,
448+ authorization_engine=authorization_engine,
449+ launchpadlib_dir=launchpadlib_dir)
450+
451+if __name__ == '__main__':
452+ get_launchpad("{credentials_path}")
453diff --git a/jobs/infrastructure/credentials-1-launchpad.yaml b/jobs/infrastructure/credentials-1-launchpad.yaml
454new file mode 100644
455index 0000000..3c63207
456--- /dev/null
457+++ b/jobs/infrastructure/credentials-1-launchpad.yaml
458@@ -0,0 +1,22 @@
459+- job-template:
460+ name: '{name}-credentials-1-launchpad'
461+ project-type: matrix
462+ description: This job creates or validates OAuth tokens to allow launchpadlib to work.
463+ node: monitor
464+ properties:
465+ - build-discarder:
466+ num-to-keep: 1
467+ - rebuild
468+ parameters:
469+ - matrix-combinations:
470+ name: nodes
471+ axes:
472+ - axis:
473+ type: slave
474+ name: node
475+ values: '{obj:build_slaves}'
476+ wrappers:
477+ - timestamps
478+ builders:
479+ - shell:
480+ !include-raw: credentials-1-launchpad.py
481diff --git a/jobs/infrastructure/credentials-2-launchpad-plugin.sh b/jobs/infrastructure/credentials-2-launchpad-plugin.sh
482new file mode 100644
483index 0000000..d47164f
484--- /dev/null
485+++ b/jobs/infrastructure/credentials-2-launchpad-plugin.sh
486@@ -0,0 +1,108 @@
487+#!/bin/bash -ex
488+#
489+# Copyright (C) 2016 Canonical Ltd
490+#
491+# This program is free software: you can redistribute it and/or modify
492+# it under the terms of the GNU General Public License version 3 as
493+# published by the Free Software Foundation.
494+#
495+# This program is distributed in the hope that it will be useful,
496+# but WITHOUT ANY WARRANTY; without even the implied warranty of
497+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
498+# GNU General Public License for more details.
499+#
500+# You should have received a copy of the GNU General Public License
501+# along with this program. If not, see <http://www.gnu.org/licenses/>.
502+
503+ # Apply the configuration file.
504+CONFIG_DIR="${{JENKINS_HOME}}/.jlp"
505+CONFIG_PATH="${{CONFIG_DIR}}/jlp.config"
506+
507+if [ ! -d "${{CONFIG_DIR}}" ]; then
508+ mkdir -p "${{CONFIG_DIR}}"
509+fi
510+
511+cat > "${{CONFIG_PATH}}" << EOF
512+#You must explicitely allow users to trigger the jobs on your jenkins
513+#Otherwise anybody can run arbitrary code on your jenkins servers.
514+allowed_users: [{allowed_users}]
515+
516+#path to your credentials file. The first time you run one of these scripts,
517+#launchpad will ask you to authenticate (via a provided URL). Once you do so
518+#(in launchpad) you won't need to do this again.
519+#If your jenkins "lives" in /var/lib/jenkins you probably don't need to change
520+#this
521+credential_store_path: {credentials_path}
522+
523+# When doing a dput into ppa (in autoland.py) a new changelog entry is
524+# generated. DEBEMAIL and DEBFULLNAME are used to generate the entry correctly.
525+# Please note that the gpg keys of the user specified here must be available
526+# on the host where autoland.py is running
527+DEBEMAIL:
528+DEBFULLNAME:
529+
530+#user and password for accessing jenkins. This is needed as we need to find
531+#out if a job is being published to public jenkins or not. The user needs to be
532+#able to see the job configuration
533+jenkins_user: "{bot_username}"
534+jenkins_password: "${{jenkins_api_token}}"
535+
536+#Actual URL of your jenkins (e.g. the jenkins backend URL)
537+jenkins_url: "{backend_url}"
538+
539+#Proxy URL of your jenkins (e.g. the URL accessed by users)
540+jenkins_proxy_url: "${{JENKINS_URL}}"
541+
542+#Token to pass when triggering a jenkins build (leave blank for none)
543+jenkins_build_token: "BUILD_ME"
544+
545+# console output from the following jobs will not be printed to the
546+# affected merge proposal (in the "Executed test runs:" section)
547+jobs_blacklisted_from_messages:
548+{blacklisted_jobs}
549+
550+#message that is used for "testing in progress" comment
551+launchpad_build_in_progress_message: "Jenkins: testing in progress"
552+
553+#login of the launchpad user you will be using for this plugin
554+#ideally this user is part of your project group
555+launchpad_login: {bot_username}
556+
557+#Review type that is used for voting on merge proposals.
558+#Usually you don't need to change this
559+launchpad_review_type: continuous-integration
560+
561+# directory containing lockfiles for Launchpad merge proposals
562+launchpadlocks_dir: /tmp/jenkins-launchpad-plugin/locks
563+
564+#lock file that is being used to limit the number of parallel launchpad
565+#connections
566+lock_name: launchpad-trigger-lock
567+
568+#you don't need to change this
569+lp_app: launchpad-trigger
570+
571+#which launchpad are you using (production/staging)
572+#you don't need to change this
573+lp_env: production
574+
575+#URL of your public jenkins in case you are publishing your jobs to some
576+#other jenkins
577+public_jenkins_url:
578+
579+#in case you are running jenkins in a private infrastructure you probably don't
580+#want to expose your private IPs in public merge proposals
581+#the following defines (IP, replacement) pairs. Your URLs in merge proposals
582+#are then replaced by the replacement (and you can e.g. edit your /etc/hosts
583+#so the links still work for you). The form to specify a replacement is:
584+#urls_to_hide:
585+# - ['http://1.2.3.4:8080','http://jenkins:8080']
586+#
587+#To specify no replacement:
588+#urls_to_hide: []
589+urls_to_hide: []
590+
591+# verbosity of the commands
592+# one of: debug, info, warning, error, critical
593+log_level: debug
594+EOF
595diff --git a/jobs/infrastructure/credentials-2-launchpad-plugin.yaml b/jobs/infrastructure/credentials-2-launchpad-plugin.yaml
596new file mode 100644
597index 0000000..9de7756
598--- /dev/null
599+++ b/jobs/infrastructure/credentials-2-launchpad-plugin.yaml
600@@ -0,0 +1,25 @@
601+- job-template:
602+ name: '{name}-credentials-2-launchpad-plugin'
603+ project-type: matrix
604+ description: This job configures Launchpad integration.
605+ node: monitor
606+ properties:
607+ - build-discarder:
608+ num-to-keep: 1
609+ - rebuild
610+ parameters:
611+ - matrix-combinations:
612+ name: nodes
613+ - password:
614+ name: jenkins_api_token
615+ description: Jenkins API key of the "{bot_username}" account
616+ axes:
617+ - axis:
618+ type: slave
619+ name: node
620+ values: '{obj:build_slaves}'
621+ wrappers:
622+ - timestamps
623+ builders:
624+ - shell:
625+ !include-raw: credentials-2-launchpad-plugin.sh
626diff --git a/jobs/infrastructure/prepare-0-install.sh b/jobs/infrastructure/prepare-0-install.sh
627new file mode 100644
628index 0000000..e0fa677
629--- /dev/null
630+++ b/jobs/infrastructure/prepare-0-install.sh
631@@ -0,0 +1,38 @@
632+#!/bin/sh -ex
633+#
634+# Copyright (C) 2016 Canonical Ltd
635+#
636+# This program is free software: you can redistribute it and/or modify
637+# it under the terms of the GNU General Public License version 3 as
638+# published by the Free Software Foundation.
639+#
640+# This program is distributed in the hope that it will be useful,
641+# but WITHOUT ANY WARRANTY; without even the implied warranty of
642+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
643+# GNU General Public License for more details.
644+#
645+# You should have received a copy of the GNU General Public License
646+# along with this program. If not, see <http://www.gnu.org/licenses/>.
647+
648+sudo apt-get install --yes software-properties-common
649+
650+# build tools as used in Launchpad
651+sudo add-apt-repository --yes ppa:launchpad/buildd-staging
652+sudo add-apt-repository --yes ppa:jenkaas-hackers/tools
653+sudo add-apt-repository --yes ppa:snappy-hwe-team/ci-tools
654+
655+sudo apt-get update
656+
657+sudo apt-get install --yes \
658+ git \
659+ python \
660+ python-launchpadlib \
661+ python-bzrlib \
662+ python-lockfile \
663+ python-yaml \
664+ python-jenkins \
665+ tarmac \
666+ jenkins-launchpad-plugin \
667+ openssh-client \
668+ debootstrap \
669+ {install_packages}
670diff --git a/jobs/infrastructure/prepare-0-install.yaml b/jobs/infrastructure/prepare-0-install.yaml
671new file mode 100644
672index 0000000..82f6f41
673--- /dev/null
674+++ b/jobs/infrastructure/prepare-0-install.yaml
675@@ -0,0 +1,28 @@
676+- job-template:
677+ name: '{name}-prepare-0-install'
678+ install_packages: ''
679+ project-type: matrix
680+ description: |
681+ This job adds all the needed repositories and installs dependencies needed
682+ on build slaves.
683+ node: monitor
684+ properties:
685+ - build-discarder:
686+ num-to-keep: 10
687+ - rebuild
688+ parameters:
689+ - matrix-combinations:
690+ name: configurations
691+ description: Which slaves to install packages on
692+ - string:
693+ name: CLEANUP_WORKSPACE
694+ default: "0"
695+ axes:
696+ - axis:
697+ type: slave
698+ name: node
699+ values: '{obj:build_slaves}'
700+ builders:
701+ - shell:
702+ !include-raw:
703+ - prepare-0-install.sh
704diff --git a/jobs/snap/common-job-prepare.sh b/jobs/snap/common-job-prepare.sh
705new file mode 100644
706index 0000000..39810d8
707--- /dev/null
708+++ b/jobs/snap/common-job-prepare.sh
709@@ -0,0 +1,19 @@
710+JENKINS_JOBS_GIT_REPO="{jobs-git-repo}"
711+JENKINS_JOBS_GIT_REPO_BRANCH="{jobs-git-repo-branch}"
712+
713+if [ -n "${{CLEANUP_WORKSPACE}}" ] && [ "${{CLEANUP_WORKSPACE}}" -eq 1 ]; then
714+ rm -rf ${{WORKSPACE}}/*
715+fi
716+if [ -e jenkins-jobs ] ; then
717+ (cd jenkins-jobs ; git clean -fdx . ; git fetch origin ; git reset --hard origin/${{JENKINS_JOBS_GIT_REPO_BRANCH}})
718+else
719+ git clone -b ${{JENKINS_JOBS_GIT_REPO_BRANCH}} ${{JENKINS_JOBS_GIT_REPO}}
720+fi
721+
722+cat << EOF > $WORKSPACE/.build_env
723+BOT_USERNAME={bot_username}
724+LAUNCHPAD_PROJECT={launchpad_project}
725+SNAP_BUILD_JOB={name}-snap-build
726+BUILD_SCRIPTS=$WORKSPACE/jenkins-jobs
727+BUILD_ON_LAUNCHPAD={build_on_launchpad}
728+EOF
729diff --git a/jobs/snap/snap-build-worker.sh b/jobs/snap/snap-build-worker.sh
730new file mode 100644
731index 0000000..cd612ca
732--- /dev/null
733+++ b/jobs/snap/snap-build-worker.sh
734@@ -0,0 +1,120 @@
735+#!/bin/sh
736+#
737+# Copyright (C) 2016 Canonical Ltd
738+#
739+# This program is free software: you can redistribute it and/or modify
740+# it under the terms of the GNU General Public License version 3 as
741+# published by the Free Software Foundation.
742+#
743+# This program is distributed in the hope that it will be useful,
744+# but WITHOUT ANY WARRANTY; without even the implied warranty of
745+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
746+# GNU General Public License for more details.
747+#
748+# You should have received a copy of the GNU General Public License
749+# along with this program. If not, see <http://www.gnu.org/licenses/>.
750+
751+set -x
752+
753+. "$WORKSPACE/.build_env"
754+
755+rm -rf $WORKSPACE/src $WORKSPACE/results $WORKSPACE/build-props
756+
757+git clone --no-checkout $TARGET_GIT_REPO $WORKSPACE/src
758+cd $WORKSPACE/src
759+for remote in `git branch -r | grep -v origin/master`; do git checkout --track $remote ; done
760+
761+git checkout $TARGET_GIT_REPO_BRANCH
762+
763+git config user.name "System Enablement CI Bot"
764+git config user.email "ce-system-enablement@lists.canonical.com"
765+
766+if [ -n "$SOURCE_GIT_REPO" ]; then
767+ git remote add other $SOURCE_GIT_REPO
768+ git fetch other
769+ git merge \
770+ --no-ff \
771+ -m "Merge remote tracking branch other/$SOURCE_GIT_REPO_BRANCH" \
772+ $REVISION
773+fi
774+
775+# Try to find the correct branch we need to build from. In the case that
776+# $SOURCE_GIT_REPO_BRANCH points us to an upstream component branch we
777+# will take master as the next suitable candidate.
778+CI_BRANCH=
779+for branch in $SOURCE_GIT_REPO_BRANCH master ; do
780+ git checkout $branch
781+ if [ -e snapcraft.yaml ]; then
782+ CI_BRANCH=$branch
783+ break
784+ fi
785+done
786+
787+if [ -z "$CI_BRANCH" ]; then
788+ echo "WARNING: Can't build snap as no snapcraft.yaml exists!"
789+ exit 0
790+fi
791+
792+REPO_NAME=$(awk -v a="$TARGET_GIT_REPO" 'BEGIN{print substr(a, index(a, "+git/") + 5)}')
793+# We rely on the snapcraft.yaml to have the snap name in the first five lines
794+# which is the case for all our snaps. This is a bit lazy but the best way to
795+# ensure we don't fetch any other name: fields which might be present in the file.
796+SNAP_NAME=$(cat snapcraft.yaml | grep -v ^\# | head -n 5 | grep "^name:" | awk '{print $2}')
797+SNAP_REV=$(git rev-parse --short HEAD)
798+CI_REPO=$REPO_NAME-$BUILD_ID-$SNAP_REV
799+
800+sed -i "s/~snappy-hwe-team\/snappy-hwe-snaps\/+git\/$REPO_NAME/~snappy-hwe-team\/snappy-hwe-snaps\/+git\/$CI_REPO/g" snapcraft.yaml
801+git add snapcraft.yaml
802+git commit -m "jenkins-ci: Fix paths"
803+
804+# The project as two different options of how snaps can be build:
805+#
806+# 1. Locally in a chroot but only for the host architecture
807+# 2. On launchpad for any architecture
808+#
809+# Which of both options will be used is configured in the
810+# $WORKSPACE/.build_env file.
811+
812+if [ "$BUILD_ON_LAUNCHPAD" = False ]; then
813+ sudo $BUILD_SCRIPTS/tools/snapbuild.sh \
814+ --source-dir=$WORKSPACE/src \
815+ --results-dir=$WORKSPACE/results
816+else
817+ git remote add jenkins-ci git+ssh://system-enablement-ci-bot@git.launchpad.net/~snappy-hwe-team/snappy-hwe-snaps/+git/$CI_REPO
818+ git push jenkins-ci --all
819+ git push jenkins-ci --tags
820+
821+ # Save repo name as soon as it gets created so it can be deleted by the cleanup
822+ # job even if this job fails.
823+ echo "CI_REPO=$CI_REPO" >> $WORKSPACE/build-props
824+ echo "CI_BRANCH=$CI_BRANCH" >> $WORKSPACE/build-props
825+
826+ EXTRA_ARGS=
827+ if [ -n "$ARCHITECTURES" ]; then
828+ EXTRA_ARGS="$EXTRA_ARGS --architectures=$ARCHITECTURES"
829+ fi
830+
831+ $BUILD_SCRIPTS/tools/trigger-lp-build.py \
832+ -s $SNAP_NAME -n \
833+ --git-repo=https://git.launchpad.net/~snappy-hwe-team/snappy-hwe-snaps/+git/$CI_REPO \
834+ --git-repo-branch=$CI_BRANCH \
835+ --results-dir=$WORKSPACE/results \
836+ $EXTRA_ARGS
837+fi
838+
839+SSH_PATH="${JENKINS_HOME}/.ssh/"
840+SSH_KEY_PATH="${SSH_PATH}/bazaar.launchpad.net/system-enablement-ci-bot"
841+SSH="ssh -i $SSH_KEY_PATH/id_rsa $REMOTE_USER@$REMOTE_WORKER"
842+SCP="scp -i $SSH_KEY_PATH/id_rsa"
843+
844+RESULTS_ID=$(md5sum $(find $WORKSPACE/results/*.snap | tail -n1) | cut -d' ' -f 1)
845+REMOTE_RESULTS_BASE_DIR=/home/$REMOTE_USER/results
846+
847+$SSH mkdir -p $REMOTE_RESULTS_BASE_DIR/$RESULTS_ID
848+$SCP $WORKSPACE/results/*.snap $REMOTE_USER@$REMOTE_WORKER:$REMOTE_RESULTS_BASE_DIR/$RESULTS_ID/
849+
850+# Save the id of our results so it can be used by a subsequent build
851+# in a properties file which is then being read from jenkins and its
852+# content passed as parameters to triggered builds.
853+echo "RESULTS_ID=$RESULTS_ID" >> $WORKSPACE/build-props
854+cat $WORKSPACE/build-props
855diff --git a/jobs/snap/snap-build-worker.yaml b/jobs/snap/snap-build-worker.yaml
856new file mode 100644
857index 0000000..36626a0
858--- /dev/null
859+++ b/jobs/snap/snap-build-worker.yaml
860@@ -0,0 +1,73 @@
861+- job-template:
862+ name: '{name}-snap-build-worker'
863+ project-type: freestyle
864+ defaults: global
865+ description: "Build a snap on launchpad"
866+ disabled: false
867+ display-name: "{name}-snap-build-worker"
868+ concurrent: true
869+ node: snap && build
870+ parameters:
871+ - string:
872+ name: TARGET_GIT_REPO
873+ default:
874+ description: "Target git repository"
875+ - string:
876+ name: TARGET_GIT_REPO_BRANCH
877+ default: master
878+ description: "Branch of the target git repository to build from"
879+ - string:
880+ name: SERIES
881+ default: xenial
882+ description: "Ubuntu archive series to build for"
883+ - string:
884+ name: FORCE
885+ default: "0"
886+ description: "Set to 1 to force the build"
887+ - string:
888+ name: SOURCE_GIT_REPO
889+ default:
890+ description: "Source git repository"
891+ - string:
892+ name: SOURCE_GIT_REPO_BRANCH
893+ default:
894+ description: "Branch of the source git repository to use"
895+ - string:
896+ name: MERGE_PROPOSAL
897+ default:
898+ description: "Link to the merge proposal this build relates to"
899+ - string:
900+ name: REVISION
901+ default:
902+ description: "Cleanup the whole workspace"
903+ - string:
904+ name: CLEANUP_WORKSPACE
905+ default: "0"
906+ description: "Cleanup the whole workspace"
907+ - string:
908+ name: ARCHITECTURES
909+ default: "{obj:architectures}"
910+ description: "Specific architectures to build for. Separate multiple architectures by ,"
911+ - string:
912+ name: REMOTE_WORKER
913+ default: "{obj:remote_worker}"
914+ description: "The remote server to execute the spread jobs on. There's no need to change from the default value unless you know what you're doing."
915+ - string:
916+ name: REMOTE_USER
917+ default: "{obj:remote_user}"
918+ description: "The remote server username used to ssh to $REMOTE_WORKER."
919+ - string:
920+ name: CORE_CHANNEL
921+ default: stable
922+ description: "Channel of the core snap to use for testing the build snap"
923+ - string:
924+ name: RESULTS_ID
925+ default: ''
926+ description: "Alphanumeric identifier used to pass build artifacts through different jobs"
927+ builders:
928+ - shell:
929+ !include-raw:
930+ - common-job-prepare.sh
931+ - shell:
932+ !include-raw-escape:
933+ - snap-build-worker.sh
934diff --git a/jobs/snap/snap-build.yaml b/jobs/snap/snap-build.yaml
935new file mode 100644
936index 0000000..a8aa6dc
937--- /dev/null
938+++ b/jobs/snap/snap-build.yaml
939@@ -0,0 +1,85 @@
940+- job-template:
941+ name: '{name}-snap-build'
942+ project-type: freestyle
943+ defaults: global
944+ description: "Build a snap with subsequent test execution"
945+ disabled: false
946+ display-name: "{name}-snap-build"
947+ concurrent: true
948+ parameters:
949+ - string:
950+ name: TARGET_GIT_REPO
951+ default:
952+ description: "Target git repository"
953+ - string:
954+ name: TARGET_GIT_REPO_BRANCH
955+ default: master
956+ description: "Branch of the target git repository to build from"
957+ - string:
958+ name: SERIES
959+ default: xenial
960+ description: "Ubuntu archive series to build for"
961+ - string:
962+ name: FORCE
963+ default: "0"
964+ description: "Set to 1 to force the build"
965+ - string:
966+ name: SOURCE_GIT_REPO
967+ default:
968+ description: "Source git repository"
969+ - string:
970+ name: SOURCE_GIT_REPO_BRANCH
971+ default:
972+ description: "Branch of the source git repository to use"
973+ - string:
974+ name: MERGE_PROPOSAL
975+ default:
976+ description: "Link to the merge proposal this build relates to"
977+ - string:
978+ name: REVISION
979+ default:
980+ description: "Cleanup the whole workspace"
981+ - string:
982+ name: CLEANUP_WORKSPACE
983+ default: "0"
984+ description: "Cleanup the whole workspace"
985+ builders:
986+ - trigger-builds:
987+ - project: '{name}-snap-build-worker'
988+ current-parameters: true
989+ predefined-parameters: |
990+ ARCHITECTURES=amd64
991+ RESULTS_ID=$BUILD_TAG
992+ block: true
993+ - project: '{name}-snap-test'
994+ current-parameters: true
995+ predefined-parameters: |
996+ RESULTS_ID=$BUILD_TAG
997+ block: true
998+ - project: '{name}-snap-cleanup'
999+ current-parameters: true
1000+ predefined-parameters: |
1001+ RESULTS_ID=$BUILD_TAG
1002+ publishers:
1003+ - archive:
1004+ artifacts: '**/*.snap'
1005+ latest-only: false
1006+ allow-empty: true
1007+ fingerprint: false
1008+ - trigger-parameterized-builds:
1009+ - project: '{name}-snap-update-mp'
1010+ condition: "SUCCESS"
1011+ predefined-parameters: |
1012+ CI_RESULT=PASSED
1013+ CI_BUILD=${{BUILD_URL}}
1014+ CI_BRANCH="${{SOURCE_GIT_REPO_BRANCH}}@${{SOURCE_GIT_REPO}}"
1015+ CI_MERGE_PROPOSAL=${{MERGE_PROPOSAL}}
1016+ CI_REVISION=${{REVISION}}
1017+ - project: '{name}-snap-update-mp'
1018+ condition: "UNSTABLE_OR_WORSE"
1019+ predefined-parameters: |
1020+ CI_RESULT=FAILED
1021+ CI_BUILD=${{BUILD_URL}}
1022+ CI_BRANCH="${{SOURCE_GIT_REPO_BRANCH}}@${{SOURCE_GIT_REPO}}"
1023+ CI_MERGE_PROPOSAL=${{MERGE_PROPOSAL}}
1024+ CI_REVISION=${{REVISION}}
1025diff --git a/jobs/snap/snap-cleanup.sh b/jobs/snap/snap-cleanup.sh
1026new file mode 100644
1027index 0000000..c4b2e02
1028--- /dev/null
1029+++ b/jobs/snap/snap-cleanup.sh
1030@@ -0,0 +1,34 @@
1031+#!/bin/sh
1032+#
1033+# Copyright (C) 2017 Canonical Ltd
1034+#
1035+# This program is free software: you can redistribute it and/or modify
1036+# it under the terms of the GNU General Public License version 3 as
1037+# published by the Free Software Foundation.
1038+#
1039+# This program is distributed in the hope that it will be useful,
1040+# but WITHOUT ANY WARRANTY; without even the implied warranty of
1041+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
1042+# GNU General Public License for more details.
1043+#
1044+# You should have received a copy of the GNU General Public License
1045+# along with this program. If not, see <http://www.gnu.org/licenses/>.
1046+
1047+set -x
1048+
1049+. "$WORKSPACE/.build_env"
1050+
1051+SSH_PATH="${JENKINS_HOME}/.ssh/"
1052+SSH_KEY_PATH="${SSH_PATH}/bazaar.launchpad.net/system-enablement-ci-bot"
1053+SSH="ssh -i $SSH_KEY_PATH/id_rsa $REMOTE_USER@$REMOTE_WORKER"
1054+
1055+REMOTE_RESULTS_BASE_DIR=/home/$REMOTE_USER/results
1056+
1057+# Delete auxiliary repo used in the build
1058+$BUILD_SCRIPTS/tools/delete-ci-repo.py \
1059+ --git-repo=https://git.launchpad.net/~snappy-hwe-team/snappy-hwe-snaps/+git/$CI_REPO
1060+
1061+$SSH rm -rf $REMOTE_RESULTS_BASE_DIR/$RESULTS_ID
1062+
1063+# Now remove any container that might have been left behind...
1064+$SSH sudo docker rm \$\(sudo docker ps -q --filter=status=exited --filter=ancestor=snap-spread-tests\) || true
1065diff --git a/jobs/snap/snap-cleanup.yaml b/jobs/snap/snap-cleanup.yaml
1066new file mode 100644
1067index 0000000..8709f48
1068--- /dev/null
1069+++ b/jobs/snap/snap-cleanup.yaml
1070@@ -0,0 +1,33 @@
1071+- job-template:
1072+ name: '{name}-snap-cleanup'
1073+ project-type: freestyle
1074+ defaults: global
1075+ description: "Cleanup artifacts left over from a snap build"
1076+ disabled: false
1077+ display-name: "{name}-snap-cleanup"
1078+ concurrent: true
1079+ node: snap && build
1080+ parameters:
1081+ - string:
1082+ name: CI_REPO
1083+ default: ""
1084+ description: "Auxiliary repo for the build, that we will remove"
1085+ - string:
1086+ name: "RESULTS_ID"
1087+ default: ""
1088+ description: "Alphanumeric Id of the results being staged on the remote worker"
1089+ - string:
1090+ name: REMOTE_WORKER
1091+ default: "{obj:remote_worker}"
1092+ description: "The remote server to execute the spread jobs on. There's no need to change from the default value unless you know what you're doing."
1093+ - string:
1094+ name: REMOTE_USER
1095+ default: "{obj:remote_user}"
1096+ description: "The remote server username used to ssh to $REMOTE_WORKER."
1097+ builders:
1098+ - shell:
1099+ !include-raw:
1100+ - common-job-prepare.sh
1101+ - shell:
1102+ !include-raw-escape:
1103+ - snap-cleanup.sh
1104diff --git a/jobs/snap/snap-nightly.yaml b/jobs/snap/snap-nightly.yaml
1105new file mode 100644
1106index 0000000..ae5a71f
1107--- /dev/null
1108+++ b/jobs/snap/snap-nightly.yaml
1109@@ -0,0 +1,41 @@
1110+- job-template:
1111+ name: '{name}-snap-nightly'
1112+ project-type: matrix
1113+ concurrent: false
1114+ node: monitor
1115+ sequential: true
1116+ properties:
1117+ - build-discarder:
1118+ days-to-keep: 30
1119+ - rebuild:
1120+ rebuild-disabled: true
1121+ axes:
1122+ - axis:
1123+ type: user-defined
1124+ name: CORE_CHANNEL
1125+ values: '{obj:nightly_core_channels}'
1126+ - axis:
1127+ type: user-defined
1128+ name: SNAP
1129+ values: '{obj:nightly_snaps}'
1130+ - axis:
1131+ type: user-defined
1132+ name: ARCHITECTURE
1133+ values: '{obj:nightly_architectures}'
1134+ wrappers:
1135+ - timeout:
1136+ timeout: 30
1137+ abort: true
1138+ - timestamps
1139+ builders:
1140+ - trigger-builds:
1141+ - project: '{name}-snap-build-worker'
1142+ predefined-parameters: |
1143+ TARGET_GIT_REPO={base_snap_repo_url}/$SNAP
1144+ TARGET_GIT_REPO_BRANCH=master
1145+ SOURCE_GIT_REPO=
1146+ SOURCE_GIT_REPO_BRANCH=
1147+ REVISION=
1148+ ARCHITECTURES=$ARCHITECTURE
1149+ CORE_CHANNEL=$CORE_CHANNEL
1150+ block: true
1151diff --git a/jobs/snap/snap-project-jobs.yaml b/jobs/snap/snap-project-jobs.yaml
1152new file mode 100644
1153index 0000000..1442ff0
1154--- /dev/null
1155+++ b/jobs/snap/snap-project-jobs.yaml
1156@@ -0,0 +1,11 @@
1157+- job-group:
1158+ name: snap-project-jobs
1159+ jobs:
1160+ - '{name}-snap-nightly'
1161+ - '{name}-snap-build-worker'
1162+ - '{name}-snap-build'
1163+ - '{name}-snap-cleanup'
1164+ - '{name}-snap-release'
1165+ - '{name}-snap-test'
1166+ - '{name}-snap-trigger-ci'
1167+ - '{name}-snap-update-mp'
1168diff --git a/jobs/snap/snap-release.sh b/jobs/snap/snap-release.sh
1169new file mode 100644
1170index 0000000..ca1e50b
1171--- /dev/null
1172+++ b/jobs/snap/snap-release.sh
1173@@ -0,0 +1,90 @@
1174+#!/bin/sh
1175+#
1176+# Copyright (C) 2017 Canonical Ltd
1177+#
1178+# This program is free software: you can redistribute it and/or modify
1179+# it under the terms of the GNU General Public License version 3 as
1180+# published by the Free Software Foundation.
1181+#
1182+# This program is distributed in the hope that it will be useful,
1183+# but WITHOUT ANY WARRANTY; without even the implied warranty of
1184+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
1185+# GNU General Public License for more details.
1186+#
1187+# You should have received a copy of the GNU General Public License
1188+# along with this program. If not, see <http://www.gnu.org/licenses/>.
1189+
1190+set -ex
1191+
1192+. "$WORKSPACE/.build_env"
1193+
1194+if [ -z "$VERSION" ]; then
1195+ echo "ERROR: No version specified"
1196+ exit 1
1197+fi
1198+
1199+echo "Snap to be released: $SNAP_NAME"
1200+echo "Version to be released: $VERSION"
1201+echo "New development version: $NEXT_VERSION"
1202+
1203+GIT_USER=system-enablement-ci-bot
1204+REPOSITORY_URL="git+ssh://$GIT_USER@git.launchpad.net/~snappy-hwe-team/snappy-hwe-snaps/+git/$SNAP_NAME"
1205+
1206+if [ -e $SNAP_NAME ]; then
1207+ rm -rf $SNAP_NAME
1208+fi
1209+
1210+set_git_identity() {
1211+ git config user.name "System Enablement CI Bot"
1212+ git config user.email "ce-system-enablement@lists.canonical.com"
1213+}
1214+
1215+bump_version_and_tag() {
1216+ sed -i -e "s/^version:\ .*/version: $1/g" snapcraft.yaml
1217+ git add snapcraft.yaml
1218+ git commit -m "Bump version to $1"
1219+ git tag -a -m "$1" $1 HEAD
1220+}
1221+
1222+RELEASE_BASE_BRANCH=master
1223+if [ "$RELEASE_FROM_STABLE" -eq 1 ]; then
1224+ RELEASE_BASE_BRANCH=stable
1225+fi
1226+
1227+git clone -b $RELEASE_BASE_BRANCH $REPOSITORY_URL $SNAP_NAME
1228+cd $SNAP_NAME
1229+
1230+if [ ! -e snapcraft.yaml ]; then
1231+ echo "ERROR: No top-level snapcraft.yaml file!"
1232+ exit 1
1233+fi
1234+
1235+set_git_identity
1236+bump_version_and_tag $VERSION
1237+
1238+if [ "$RELEASE_FROM_STABLE" -eq 1 ]; then
1239+ git push origin $RELEASE_BASE_BRANCH
1240+ git push origin $VERSION
1241+
1242+ $BUILD_SCRIPTS/tools/trigger-lp-build.py -s $SNAP_NAME -p
1243+else
1244+ if ! git branch -r | grep origin/stable ; then
1245+ git checkout -b stable origin/master
1246+ else
1247+ git checkout -b stable origin/stable
1248+ fi
1249+ git merge $RELEASE_BASE_BRANCH
1250+
1251+ git push origin stable
1252+ git push origin $RELEASE_BASE_BRANCH
1253+ git push origin $VERSION
1254+
1255+ # Build before we change master branch
1256+ $BUILD_SCRIPTS/tools/trigger-lp-build.py -s $SNAP_NAME -p
1257+
1258+ git checkout $RELEASE_BASE_BRANCH
1259+ sed -i -e "s/^version:\ .*/version: ${NEXT_VERSION}-dev/g" snapcraft.yaml
1260+ git add snapcraft.yaml
1261+ git commit -m "Open development for ${NEXT_VERSION}-dev"
1262+ git push origin $RELEASE_BASE_BRANCH
1263+fi
1264diff --git a/jobs/snap/snap-release.yaml b/jobs/snap/snap-release.yaml
1265new file mode 100644
1266index 0000000..b32b8c0
1267--- /dev/null
1268+++ b/jobs/snap/snap-release.yaml
1269@@ -0,0 +1,59 @@
1270+- job-template:
1271+ name: '{name}-snap-release'
1272+ project-type: freestyle
1273+ defaults: global
1274+ description: "A job implementing the release process used for snaps"
1275+ disabled: false
1276+ display-name: "{name}-snap-release"
1277+ concurrent: true
1278+ node: snap && release
1279+ parameters:
1280+ - string:
1281+ name: SNAP_NAME
1282+ default: ""
1283+ description: |
1284+ Name of the snap which should be released
1285+
1286+ Normally the repositories we have on https://code.launchpad.net/~snappy-hwe-team/snappy-hwe-snaps/+git/
1287+ match with the snap name. In some cases this is not true, in those you have to set the repository
1288+ name here. For example for 'canonical-se-engineering-tests' it is 'engineering-tests' as this is
1289+ the repository name.
1290+ - string:
1291+ name: VERSION
1292+ default: ""
1293+ description: "New version of the snap"
1294+ - string:
1295+ name: NEXT_VERSION
1296+ default: ""
1297+ description: |
1298+ Version which will follow the version specified in the VERSION field.
1299+ For example if you specify VERSION = "1.1" next version is most likely
1300+ "1.2". The NEXT_VERSION parameter is used to write it into the component
1301+ snapcraft.yaml as "$NEXT_VERSION-dev" to clearly indicate that snaps
1302+ build from master are development versions.
1303+
1304+ Please note that NEXT_VERSION is not set in stone and can be overriden
1305+ at any time by changes merged into master.
1306+ - string:
1307+ name: CLEANUP_WORKSPACE
1308+ default: "0"
1309+ description: "Cleanup the whole workspace"
1310+ - string:
1311+ name: SERIES
1312+ default: xenial
1313+ description: "Ubuntu archive series to build for"
1314+ - string:
1315+ name: RELEASE_FROM_STABLE
1316+ default: 0
1317+ description: |
1318+ Set to '1' to force a release from stable branch without merging with
1319+ master. This can be used when single changes are picked into stable
1320+ manually and need to be released without pulling anything else from
1321+ master. PLEASE ENSURE THAT THOSE CHANGE GO INTO MASTER TOO!!
1322+ builders:
1323+ - shell:
1324+ !include-raw:
1325+ - common-job-prepare.sh
1326+ - shell:
1327+ !include-raw-escape:
1328+ - snap-release.sh
1329diff --git a/jobs/snap/snap-test.sh b/jobs/snap/snap-test.sh
1330new file mode 100644
1331index 0000000..6fe3ce7
1332--- /dev/null
1333+++ b/jobs/snap/snap-test.sh
1334@@ -0,0 +1,116 @@
1335+#!/bin/sh
1336+#
1337+# Copyright (C) 2016 Canonical Ltd
1338+#
1339+# This program is free software: you can redistribute it and/or modify
1340+# it under the terms of the GNU General Public License version 3 as
1341+# published by the Free Software Foundation.
1342+#
1343+# This program is distributed in the hope that it will be useful,
1344+# but WITHOUT ANY WARRANTY; without even the implied warranty of
1345+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
1346+# GNU General Public License for more details.
1347+#
1348+# You should have received a copy of the GNU General Public License
1349+# along with this program. If not, see <http://www.gnu.org/licenses/>.
1350+
1351+set -ex
1352+
1353+. "$WORKSPACE/.build_env"
1354+
1355+SSH_PATH="${JENKINS_HOME}/.ssh/"
1356+SSH_KEY_PATH="${SSH_PATH}/bazaar.launchpad.net/system-enablement-ci-bot"
1357+
1358+SSH="ssh -i $SSH_KEY_PATH/id_rsa $REMOTE_USER@$REMOTE_WORKER"
1359+SCP="scp -i $SSH_KEY_PATH/id_rsa"
1360+
1361+REMOTE_WORKSPACE=/home/$REMOTE_USER/$BUILD_TAG
1362+REMOTE_RESULTS_BASE_DIR=/home/$REMOTE_USER/results
1363+
1364+tmp_srcdir=`mktemp -d`
1365+git clone --depth 1 -b $SOURCE_GIT_REPO_BRANCH $SOURCE_GIT_REPO $tmp_srcdir/src
1366+cd $tmp_srcdir/src
1367+# This will fail as we have set set -e above when the revision isn't part of
1368+# of the repository we've cloned.
1369+git branch --contains $SOURCE_GIT_REPO_REVISION | grep "$SOURCE_GIT_REPO_BRANCH"
1370+git checkout -b ci-test $SOURCE_GIT_REPO_REVISION
1371+
1372+# Components have the ability to disable CI tests if they can't provide any.
1373+# This only accepted in a few cases and should be generally avoided.
1374+if [ -e $tmp_srcdir/src/.ci_tests_disabled ]; then
1375+ echo "WARNING: Component has no CI tests so not running anything here"
1376+ exit 0
1377+fi
1378+# We require either a run-tests.sh interface script for the spread tests
1379+# or the basic spread.yaml spread test definition file, otherwise we fail
1380+# the Jenkins job. Spread tests are required for all MRs.
1381+if [ ! -e "$tmp_srcdir/src/run-tests.sh" ] && [ ! -e "$tmp_srcdir/src/spread.yaml" ]; then
1382+ echo "ERROR: missing spread test: you must provide a spread test"
1383+ exit 1
1384+fi
1385+rm -rf $tmp_srcdir
1386+
1387+if [ -n "$RESULTS_ID" ]; then
1388+ $SSH mkdir -p $REMOTE_WORKSPACE/results
1389+ $SSH cp -v $REMOTE_RESULTS_BASE_DIR/$RESULTS_ID/*.snap $REMOTE_WORKSPACE/results
1390+fi
1391+
1392+$SSH sudo apt-get --yes --force-yes install docker.io
1393+
1394+cat << EOF > $WORKSPACE/run-tests.sh
1395+#!/bin/sh
1396+set -ex
1397+
1398+export TERM=linux
1399+export DEBIAN_FRONTEND=noninteractive
1400+export PATH=/build/bin:$PATH
1401+
1402+# At this time it's necessary to build spread manually because
1403+# the snapped version does not include the qemu/kvm backend.
1404+# Once the snapped version includes this backend, then we can
1405+# change the manual building of spread with making sure the snap
1406+# package is installed.
1407+export GOPATH=`mktemp -d`
1408+go get -d -v github.com/snapcore/spread/...
1409+go build github.com/snapcore/spread/cmd/spread
1410+mkdir /build/bin
1411+cp spread /build/bin
1412+
1413+git clone --depth 1 -b $SOURCE_GIT_REPO_BRANCH $SOURCE_GIT_REPO /build/src
1414+cd /build/src
1415+git checkout -b ci-tests $SOURCE_GIT_REPO_REVISION
1416+
1417+# Copy any stage results from previous generic-build-snap-worker builds
1418+cp -v /build/results/*.snap /build/src
1419+
1420+if [ -e "run-tests.sh" ] ; then
1421+ if [ ! -z "$CHANNEL" ] ; then
1422+ ./run-tests.sh --channel=$CHANNEL --test-from-channel --debug --force-new-image
1423+ else
1424+ ./run-tests.sh --debug --force-new-image
1425+ fi
1426+else
1427+ if [ ! -z "$CHANNEL" ] ; then
1428+ SNAP_CHANNEL=$CHANNEL spread -debug
1429+ else
1430+ spread -debug
1431+ fi
1432+fi
1433+EOF
1434+
1435+$SSH mkdir -p $REMOTE_WORKSPACE
1436+$SCP $WORKSPACE/run-tests.sh $REMOTE_USER@$REMOTE_WORKER:$REMOTE_WORKSPACE
1437+$SSH chmod u+x $REMOTE_WORKSPACE/run-tests.sh
1438+
1439+$SSH mkdir -p $REMOTE_WORKSPACE/docker
1440+$SCP $WORKSPACE/build-scripts/docker/spread-tests/Dockerfile \
1441+ $REMOTE_USER@$REMOTE_WORKER:$REMOTE_WORKSPACE/docker
1442+$SSH time sudo docker build -t snap-spread-tests $REMOTE_WORKSPACE/docker
1443+
1444+$SSH time sudo docker run \
1445+ -v /dev:/dev \
1446+ -v $REMOTE_WORKSPACE:/build \
1447+ --privileged \
1448+ snap-spread-tests /build/run-tests.sh
1449+
1450+$SSH sudo rm -rf $REMOTE_WORKSPACE
1451diff --git a/jobs/snap/snap-test.yaml b/jobs/snap/snap-test.yaml
1452new file mode 100644
1453index 0000000..27e506d
1454--- /dev/null
1455+++ b/jobs/snap/snap-test.yaml
1456@@ -0,0 +1,65 @@
1457+- job-template:
1458+ name: '{name}-snap-test'
1459+ project-type: freestyle
1460+ defaults: global
1461+ description: "Run tests for a single snap on a remote agent which also allows spread execution inside KVM/QEMU"
1462+ disabled: {run-tests}
1463+ display-name: "{name}-snap-test"
1464+ concurrent: true
1465+ node: snap && test
1466+ parameters:
1467+ - string:
1468+ name: SOURCE_GIT_REPO
1469+ default: ""
1470+ description: "Source git repository"
1471+ - string:
1472+ name: SOURCE_GIT_REPO_BRANCH
1473+ default: ""
1474+ description: "Branch of the source git repository to use"
1475+ - string:
1476+ name: CHANNEL
1477+ default: ""
1478+ description: "Run tests against an image build with a core snap from the specified channel"
1479+ - string:
1480+ name: REMOTE_WORKER
1481+ default: "{obj:remote_worker}"
1482+ description: "The remote server to execute the spread jobs on. There's no need to change from the default value unless you know what you're doing."
1483+ - string:
1484+ name: REMOTE_USER
1485+ default: "{obj:remote_user}"
1486+ description: "The remote server username used to ssh to $REMOTE_WORKER."
1487+ - string:
1488+ name: CLEANUP_WORKSPACE
1489+ default: "0"
1490+ description: "Cleanup the whole workspace"
1491+ - string:
1492+ name: SERIES
1493+ default: xenial
1494+ description: "Ubuntu archive series to build for"
1495+ - string:
1496+ name: REBUILD_ROOTFS
1497+ default: 0
1498+ description: "Rebuild the chroot rootfs or not. Default is to not rebuild and use a previous job's rootfs."
1499+ - string:
1500+ name: RESULTS_ID
1501+ default: ""
1502+ description: "Alphanumeric Id of the results being staged on the remote worker"
1503+ - string:
1504+ name: CORE_CHANNEL
1505+ default: "stable"
1506+ description: "Channel used for the core snap inside the test environment. Defaults to 'stable'."
1507+ - string:
1508+ name: CI_BRANCH
1509+ default: ""
1510+ description: "Branch on which the tests should be executed"
1511+ - string:
1512+ name: CI_REPO
1513+ default: ""
1514+ description: "Git repository to use for testing (MUST contain $CI_BRANCH)"
1515+ builders:
1516+ - shell:
1517+ !include-raw:
1518+ - common-job-prepare.sh
1519+ - shell:
1520+ !include-raw-escape:
1521+ - snap-test.sh
1522diff --git a/jobs/snap/snap-trigger-ci.sh b/jobs/snap/snap-trigger-ci.sh
1523new file mode 100644
1524index 0000000..a8bb798
1525--- /dev/null
1526+++ b/jobs/snap/snap-trigger-ci.sh
1527@@ -0,0 +1,25 @@
1528+#!/bin/bash
1529+#
1530+# Copyright (C) 2016 Canonical Ltd
1531+#
1532+# This program is free software: you can redistribute it and/or modify
1533+# it under the terms of the GNU General Public License version 3 as
1534+# published by the Free Software Foundation.
1535+#
1536+# This program is distributed in the hope that it will be useful,
1537+# but WITHOUT ANY WARRANTY; without even the implied warranty of
1538+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
1539+# GNU General Public License for more details.
1540+#
1541+# You should have received a copy of the GNU General Public License
1542+# along with this program. If not, see <http://www.gnu.org/licenses/>.
1543+
1544+set -ex
1545+
1546+. "$WORKSPACE/.build_env"
1547+
1548+BUILD_SCRIPTS=$WORKSPACE/jenkins-jobs
1549+
1550+exec $BUILD_SCRIPTS/tools/trigger-snap-builds.py \
1551+ -p $LAUNCHPAD_PROJECT \
1552+ -j $SNAP_BUILD_JOB
1553diff --git a/jobs/snap/snap-trigger-ci.yaml b/jobs/snap/snap-trigger-ci.yaml
1554new file mode 100644
1555index 0000000..6e5ed7c
1556--- /dev/null
1557+++ b/jobs/snap/snap-trigger-ci.yaml
1558@@ -0,0 +1,23 @@
1559+- job-template:
1560+ name: '{name}-snap-trigger-ci'
1561+ project-type: freestyle
1562+ defaults: global
1563+ description: "Monitor Launchpad for new merge proposals"
1564+ disabled: false
1565+ display-name: "{name}-snap-trigger-ci"
1566+ concurrent: true
1567+ node: snap && misc
1568+ triggers:
1569+ - timed: # every five minutes
1570+ H/5 * * * *
1571+ properties:
1572+ - build-discarder:
1573+ num-to-keep: 10
1574+ - rebuild
1575+ builders:
1576+ - shell:
1577+ !include-raw:
1578+ - common-job-prepare.sh
1579+ - shell:
1580+ !include-raw-escape:
1581+ - snap-trigger-ci.sh
1582diff --git a/jobs/snap/snap-update-mp.sh b/jobs/snap/snap-update-mp.sh
1583new file mode 100644
1584index 0000000..bbb4b94
1585--- /dev/null
1586+++ b/jobs/snap/snap-update-mp.sh
1587@@ -0,0 +1,25 @@
1588+#!/bin/sh
1589+#
1590+# Copyright (C) 2016 Canonical Ltd
1591+#
1592+# This program is free software: you can redistribute it and/or modify
1593+# it under the terms of the GNU General Public License version 3 as
1594+# published by the Free Software Foundation.
1595+#
1596+# This program is distributed in the hope that it will be useful,
1597+# but WITHOUT ANY WARRANTY; without even the implied warranty of
1598+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
1599+# GNU General Public License for more details.
1600+#
1601+# You should have received a copy of the GNU General Public License
1602+# along with this program. If not, see <http://www.gnu.org/licenses/>.
1603+
1604+set -ex
1605+
1606+BUILD_SCRIPTS=$WORKSPACE/jenkins-jobs
1607+
1608+exec $BUILD_SCRIPTS/tools/vote-on-merge-proposal.py \
1609+ -s $CI_RESULT \
1610+ -u $CI_BUILD \
1611+ -r $CI_REVISION \
1612+ -p $CI_MERGE_PROPOSAL
1613diff --git a/jobs/snap/snap-update-mp.yaml b/jobs/snap/snap-update-mp.yaml
1614new file mode 100644
1615index 0000000..89e4e76
1616--- /dev/null
1617+++ b/jobs/snap/snap-update-mp.yaml
1618@@ -0,0 +1,33 @@
1619+- job-template:
1620+ name: '{name}-snap-update-mp'
1621+ project-type: freestyle
1622+ defaults: global
1623+ description: "Run tests for a single snap on a remote agent which also allows spread execution inside KVM/QEMU"
1624+ disabled: {update-mp}
1625+ display-name: "{name}-snap-update-mp"
1626+ concurrent: true
1627+ node: snap && misc
1628+ disabled: {update_mps}
1629+ parameters:
1630+ - string:
1631+ name: CI_RESULT
1632+ description: Result of the CI build
1633+ - string:
1634+ name: CI_BUILD
1635+ description: Jenkins URL of the build
1636+ - string:
1637+ name: CI_BRANCH
1638+ description: Launchpad branch that was processed
1639+ - string:
1640+ name: CI_MERGE_PROPOSAL
1641+ description: Launchpad merge proposal that was processed
1642+ - string:
1643+ name: CI_REVISION
1644+ description: Revision of the processed branch
1645+ builders:
1646+ - shell:
1647+ !include-raw:
1648+ - common-job-prepare.sh
1649+ - shell:
1650+ !include-raw-escape:
1651+ - snap-update-mp.sh
1652diff --git a/local.conf b/local.conf
1653new file mode 100644
1654index 0000000..0170c70
1655--- /dev/null
1656+++ b/local.conf
1657@@ -0,0 +1,11 @@
1658+[job_builder]
1659+ignore_cache=True
1660+keep_descriptions=False
1661+recursive=False
1662+allow_duplicates=False
1663+
1664+[jenkins]
1665+user=system-enablement-ci-bot
1666+password=jenkins
1667+url=http://localhost:8080
1668+query_plugins_info=False
1669diff --git a/local.yaml b/local.yaml
1670new file mode 100644
1671index 0000000..8a31cc2
1672--- /dev/null
1673+++ b/local.yaml
1674@@ -0,0 +1,40 @@
1675+- project:
1676+ name: infrastructure
1677+ bot_username: system-enablement-ci-bot
1678+ credentials_path: /var/lib/jenkins/.launchpad.credentials
1679+ allowed_users: "canonical-system-enablement"
1680+ backend_url: http://localhost:8080/
1681+ blacklisted_jobs: ""
1682+ install_packages: ""
1683+ build_slaves:
1684+ - master
1685+ jobs:
1686+ - '{name}-prepare-0-install'
1687+ - '{name}-credentials-0-ssh'
1688+ - '{name}-credentials-1-launchpad'
1689+ - '{name}-credentials-2-launchpad-plugin'
1690+
1691+- project:
1692+ name: brando
1693+ run-tests: false
1694+ update-mp: false
1695+ jobs-git-repo: https://git.launchpad.net/~morphis/snappy-hwe-snaps/+git/jenkins-jobs
1696+ jobs-git-repo-branch: f/import-jobs
1697+ bot_username: system-enablement-ci-bot
1698+ allowed_users: "canonical-system-enablement"
1699+ launchpad_project: "snappy-hwe-snaps"
1700+ update_mps: false
1701+ architectures: "amd64"
1702+ build_on_launchpad: false
1703+ all_slaves:
1704+ - master
1705+ build_slaves:
1706+ - master
1707+ remote_worker: "127.0.0.1"
1708+ remote_user: "jenkins"
1709+ nightly_architectures: []
1710+ nightly_core_channels: []
1711+ nightly_snaps: []
1712+ base_snap_repo_url: 'https://git.launchpad.net/~snappy-hwe-team/snappy-hwe-snaps/+git'
1713+ jobs:
1714+ - snap-project-jobs
1715diff --git a/run-tests.sh b/run-tests.sh
1716new file mode 100755
1717index 0000000..17e3aea
1718--- /dev/null
1719+++ b/run-tests.sh
1720@@ -0,0 +1,2 @@
1721+#!/bin/sh
1722+echo "Nothing yet!"
1723diff --git a/system-enablement.conf b/system-enablement.conf
1724new file mode 100644
1725index 0000000..6b7ea31
1726--- /dev/null
1727+++ b/system-enablement.conf
1728@@ -0,0 +1,9 @@
1729+[job_builder]
1730+recursive=True
1731+exclude=.*:tests
1732+
1733+[jenkins]
1734+user=system-enablement-jb-bot
1735+password=<API-TOKEN>
1736+url=http://system-enablement-jenkins-be.internal:8080/system-enablement/
1737+query_plugins_info=False
1738diff --git a/system-enablement.yaml b/system-enablement.yaml
1739new file mode 100644
1740index 0000000..5d0d733
1741--- /dev/null
1742+++ b/system-enablement.yaml
1743@@ -0,0 +1,44 @@
1744+- project:
1745+ name: system-enablement
1746+ run-tests: false
1747+ update-mp: false
1748+ jobs-git-repo: https://git.launchpad.net/~morphis/snappy-hwe-snaps/+git/jenkins-jobs
1749+ jobs-git-repo-branch: f/import-jobs
1750+ bot_username: system-enablement-ci-bot
1751+ install_packages: ''
1752+ backend_url: http://system-enablement-jenkins-be.internal:8080/system-enablement/
1753+ credentials_path: /var/lib/jenkins/.launchpad.credentials
1754+ allowed_users: "canonical-system-enablement"
1755+ launchpad_project: "snappy-hwe-snaps"
1756+ update_mps: false
1757+ all_slaves:
1758+ - jenkins-slave-1
1759+ - jenkins-slave-2
1760+ - jenkins-slave-3
1761+ build_slaves:
1762+ - jenkins-slave-1
1763+ - jenkins-slave-2
1764+ - jenkins-slave-3
1765+ blacklisted_jobs: ""
1766+ remote_worker: "10.101.48.223"
1767+ remote_user: "ubuntu"
1768+ nightly_architectures:
1769+ - amd64
1770+ nightly_core_channels:
1771+ - stable
1772+ - candidate
1773+ - beta
1774+ - edge
1775+ nightly_snaps:
1776+ - wifi-ap
1777+ - network-manager
1778+ base_snap_repo_url: 'https://git.launchpad.net/~snappy-hwe-team/snappy-hwe-snaps/+git'
1779+ jobs:
1780+ - infra-jobs
1781+ - '{name}-snap-nightly'
1782+ - '{name}-snap-build-worker'
1783+ - '{name}-snap-build'
1784+ - '{name}-snap-release'
1785+ - '{name}-snap-test'
1786+ - '{name}-snap-trigger-ci'
1787+ - '{name}-snap-update-mp'
1788diff --git a/tools/automerge-snap-mp.py b/tools/automerge-snap-mp.py
1789new file mode 100755
1790index 0000000..fe9b595
1791--- /dev/null
1792+++ b/tools/automerge-snap-mp.py
1793@@ -0,0 +1,128 @@
1794+#!/usr/bin/env python
1795+# -*- Mode:Python; indent-tabs-mode:nil; tab-width:4 -*-
1796+#
1797+# Copyright (C) 2016 Canonical Ltd
1798+#
1799+# This program is free software: you can redistribute it and/or modify
1800+# it under the terms of the GNU General Public License version 3 as
1801+# published by the Free Software Foundation.
1802+#
1803+# This program is distributed in the hope that it will be useful,
1804+# but WITHOUT ANY WARRANTY; without even the implied warranty of
1805+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
1806+# GNU General Public License for more details.
1807+#
1808+# You should have received a copy of the GNU General Public License
1809+# along with this program. If not, see <http://www.gnu.org/licenses/>.
1810+
1811+from launchpadlib.launchpad import Launchpad
1812+import os
1813+import sys
1814+import yaml
1815+import git
1816+import shutil
1817+import se_utils
1818+
1819+class LaunchpadVote():
1820+ APPROVE = 'Approve'
1821+ DISAPPROVE = 'Disapprove'
1822+ NEEDS_FIXING = 'Needs Fixing'
1823+
1824+def load_config():
1825+ files = [os.path.expanduser('~/.jlp/jlp.config'), 'jlp.config']
1826+ for config_file in files:
1827+ try:
1828+ config = yaml.safe_load(open(config_file, 'r'))
1829+ return config
1830+ except IOError:
1831+ pass
1832+ print("ERROR: No config file found")
1833+ sys.exit(1)
1834+
1835+def get_config_option(name):
1836+ config = load_config()
1837+ return config[name]
1838+
1839+def clean_branch_name(branch_name):
1840+ if branch_name.startswith("refs/heads/"):
1841+ return branch_name[11:]
1842+ return branch_name
1843+
1844+def correct_ssh_url(url):
1845+ if not url.startswith("git+ssh://"):
1846+ return url
1847+ new_url = "git+ssh://system-enablement-ci-bot@%s" % url[10:]
1848+ return new_url
1849+
1850+
1851+def try_merge(proposal, target_repo, target_branch, source_repo, source_branch):
1852+ print("Trying to merge %s:%s into %s:%s" % (source_repo.git_https_url, source_branch, target_repo.git_ssh_url, target_branch))
1853+
1854+ repo_path = os.path.join(os.environ["WORKSPACE"], "repo")
1855+ if os.path.exists(repo_path):
1856+ shutil.rmtree(repo_path)
1857+
1858+ repo = git.Repo.clone_from(correct_ssh_url(target_repo.git_ssh_url), repo_path, branch=target_branch)
1859+ source_remote = repo.create_remote("source", source_repo.git_https_url)
1860+ source_remote.fetch()
1861+
1862+ repo.git.config("user.name", "System Enablement CI Bot")
1863+ # FIXME: What is the real email address of the bot?
1864+ repo.git.config("user.email", "ce-system-enablement@lists.canonical.com")
1865+
1866+ repo.git.merge("--no-ff",
1867+ "-m", "Merge remote tracking branch %s" % (source_branch),
1868+ "-m", "Merge-Proposal: %s" % proposal.web_link,
1869+ "source/%s" % source_branch)
1870+
1871+ repo.git.push("origin", target_branch)
1872+
1873+def get_last_mp_vote(mp):
1874+ for vote in mp.votes:
1875+ if not vote.comment:
1876+ continue
1877+ if vote.review_type == "continuous-integration" and vote.comment:
1878+ return vote.comment.vote
1879+ return None
1880+
1881+def mp_is_disapproved(mp):
1882+ for vote in mp.votes:
1883+ if vote.comment and vote.comment.vote == LaunchpadVote.DISAPPROVE:
1884+ return True
1885+ return False
1886+
1887+lp_app = get_config_option("lp_app")
1888+lp_env = get_config_option("lp_env")
1889+credential_store_path = get_config_option('credential_store_path')
1890+launchpad = se_utils.get_launchpad(None, credential_store_path, lp_app, lp_env)
1891+
1892+project = launchpad.projects['snappy-hwe-snaps']
1893+proposals = project.getMergeProposals(status=['Approved'])
1894+
1895+failed_merges = 0
1896+
1897+print("Found %d candidate merge proposals" % len(proposals))
1898+
1899+for proposal in proposals:
1900+ if get_last_mp_vote(proposal) != LaunchpadVote.APPROVE:
1901+ print("Not merging %s as not approved by CI" % proposal.web_link)
1902+ continue
1903+
1904+ if mp_is_disapproved(proposal):
1905+ print("Not merging %s as at least one reviewer has disapproved the change" % proposal.web_link)
1906+ continue
1907+
1908+ print("Found proposal which is ready for merging: %s" % proposal.web_link)
1909+
1910+ try:
1911+ try_merge(proposal,
1912+ launchpad.load(proposal.target_git_repository_link),
1913+ clean_branch_name(proposal.target_git_path),
1914+ launchpad.load(proposal.source_git_repository_link),
1915+ clean_branch_name(proposal.source_git_path))
1916+ except:
1917+ print("ERROR: Failed to merge %s" % proposal.web_link)
1918+ failed_merges += 1
1919+
1920+if failed_merges > 0:
1921+ sys.exit(1)
1922diff --git a/tools/build-rootfs-create b/tools/build-rootfs-create
1923new file mode 100755
1924index 0000000..9a9f9ce
1925--- /dev/null
1926+++ b/tools/build-rootfs-create
1927@@ -0,0 +1,26 @@
1928+#!/bin/bash
1929+#
1930+# Copyright (C) 2016 Canonical Ltd
1931+#
1932+# This program is free software: you can redistribute it and/or modify
1933+# it under the terms of the GNU General Public License version 3 as
1934+# published by the Free Software Foundation.
1935+#
1936+# This program is distributed in the hope that it will be useful,
1937+# but WITHOUT ANY WARRANTY; without even the implied warranty of
1938+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
1939+# GNU General Public License for more details.
1940+#
1941+# You should have received a copy of the GNU General Public License
1942+# along with this program. If not, see <http://www.gnu.org/licenses/>.
1943+
1944+set -x
1945+set -e
1946+
1947+SERIES=$1
1948+TARBALL=$2
1949+
1950+mkdir -p rootfs
1951+debootstrap --components=main,universe $SERIES rootfs
1952+tar cf $TARBALL rootfs
1953+rm -rf rootfs
1954diff --git a/tools/common.sh b/tools/common.sh
1955new file mode 100755
1956index 0000000..c30d2d7
1957--- /dev/null
1958+++ b/tools/common.sh
1959@@ -0,0 +1,93 @@
1960+#!/bin/sh -ex
1961+#
1962+# Copyright (C) 2017 Canonical Ltd
1963+#
1964+# This program is free software: you can redistribute it and/or modify
1965+# it under the terms of the GNU General Public License version 3 as
1966+# published by the Free Software Foundation.
1967+#
1968+# This program is distributed in the hope that it will be useful,
1969+# but WITHOUT ANY WARRANTY; without even the implied warranty of
1970+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
1971+# GNU General Public License for more details.
1972+#
1973+# You should have received a copy of the GNU General Public License
1974+# along with this program. If not, see <http://www.gnu.org/licenses/>.
1975+
1976+# Set common variables used by the jenkins jobs
1977+set_jenkins_env ()
1978+{
1979+ SSH_PATH="${JENKINS_HOME}/.ssh/"
1980+ SSH_KEY_PATH="${SSH_PATH}/bazaar.launchpad.net/system-enablement-ci-bot"
1981+
1982+ SSH="ssh -i $SSH_KEY_PATH/id_rsa $REMOTE_USER@$REMOTE_WORKER"
1983+ SCP="scp -i $SSH_KEY_PATH/id_rsa"
1984+
1985+ REPO=https://git.launchpad.net/~snappy-hwe-team/snappy-hwe-snaps/+git/$CI_REPO
1986+ BRANCH=$CI_BRANCH
1987+
1988+ # If no CI repo/branch is set fallback to the source repo/branch set
1989+ # which will be the case for those repositories which don't contain
1990+ # a snap.
1991+ if [ -z "$CI_REPO" ]; then
1992+ REPO=$SOURCE_GIT_REPO
1993+ BRANCH=$SOURCE_GIT_REPO_BRANCH
1994+ fi
1995+
1996+ REMOTE_WORKSPACE=/home/$REMOTE_USER/$BUILD_TAG
1997+ REMOTE_RESULTS_BASE_DIR=/home/$REMOTE_USER/results
1998+}
1999+
2000+# Sets variables
2001+# TEST_TYPE={script, spread}
2002+# HW_TESTS_RESULT={0, !=0} -> {has hw tests, does not have hw tests}
2003+# FIXME Maybe depending on context the call to clone could be avoided
2004+set_test_type ()
2005+{
2006+ tmp_srcdir=$(mktemp -d)
2007+
2008+ # We use FAIL to make sure we do not exit until we free tmp_srcdir
2009+ FAIL=no
2010+ git clone --depth 1 -b "$BRANCH" "$REPO" "$tmp_srcdir"/src || FAIL=yes
2011+ cd "$tmp_srcdir"/src || FAIL=yes
2012+
2013+ TEST_TYPE=none
2014+ if [ -e "$tmp_srcdir/src/spread.yaml" ]; then
2015+ TEST_TYPE=spread
2016+ fi
2017+ # run-tests.sh gets priority over spread.yaml
2018+ if [ -e "$tmp_srcdir/src/run-tests.sh" ]; then
2019+ TEST_TYPE=script
2020+ fi
2021+
2022+ # TODO: Use https://github.com/0k/shyaml in the future for this
2023+ if grep -q "type: adhoc" spread.yaml; then
2024+ HW_TESTS_RESULT=0
2025+ else
2026+ HW_TESTS_RESULT=1
2027+ fi
2028+
2029+ # Components have the ability to disable CI tests if they can't provide any.
2030+ # This is only accepted in a few cases and should be generally avoided.
2031+ CI_TESTS_DISABLED=no
2032+ if [ -e "$tmp_srcdir"/src/.ci_tests_disabled ]; then
2033+ CI_TESTS_DISABLED=yes
2034+ fi
2035+
2036+ rm -rf "$tmp_srcdir"
2037+
2038+ if [ "$FAIL" = yes ]; then
2039+ echo "ERROR: critical in set_test_type()"
2040+ exit 1
2041+ fi
2042+
2043+ if [ "$CI_TESTS_DISABLED" = yes ]; then
2044+ echo "WARNING: Component has no CI tests so not running anything here"
2045+ exit 0
2046+ fi
2047+
2048+ if [ "$TEST_TYPE" = none ]; then
2049+ echo "ERROR: missing spread or script tests: you must provide one of them"
2050+ exit 1
2051+ fi
2052+}
2053diff --git a/tools/delete-ci-repo.py b/tools/delete-ci-repo.py
2054new file mode 100755
2055index 0000000..35c762d
2056--- /dev/null
2057+++ b/tools/delete-ci-repo.py
2058@@ -0,0 +1,45 @@
2059+#!/usr/bin/env python
2060+#
2061+# Copyright (C) 2017 Canonical Ltd
2062+#
2063+# This program is free software: you can redistribute it and/or modify
2064+# it under the terms of the GNU General Public License version 3 as
2065+# published by the Free Software Foundation.
2066+#
2067+# This program is distributed in the hope that it will be useful,
2068+# but WITHOUT ANY WARRANTY; without even the implied warranty of
2069+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
2070+# GNU General Public License for more details.
2071+#
2072+# You should have received a copy of the GNU General Public License
2073+# along with this program. If not, see <http://www.gnu.org/licenses/>.
2074+
2075+from launchpadlib.launchpad import Launchpad
2076+
2077+from argparse import ArgumentParser
2078+
2079+import se_utils
2080+
2081+print("Running delete-ci-repo")
2082+
2083+parser = ArgumentParser(description="Delete a git repository stored in launchpad")
2084+parser.add_argument('--git-repo', help="Git repository to be deleted")
2085+
2086+args = vars(parser.parse_args())
2087+
2088+git_repo = args['git_repo']
2089+ind = git_repo.find('~')
2090+if ind == -1:
2091+ print("Bad git repo {}".format(git_repo))
2092+ exit(1)
2093+
2094+lp_repo = git_repo[ind:]
2095+
2096+lp_app = se_utils.get_config_option("lp_app")
2097+lp_env = se_utils.get_config_option("lp_env")
2098+credential_store_path = se_utils.get_config_option('credential_store_path')
2099+launchpad = se_utils.get_launchpad(None, credential_store_path, lp_app, lp_env)
2100+
2101+repo = launchpad.git_repositories.getByPath(path=lp_repo)
2102+print("Removing {}".format(lp_repo))
2103+repo.lp_delete()
2104diff --git a/tools/hardware-test.sh b/tools/hardware-test.sh
2105new file mode 100755
2106index 0000000..41eee60
2107--- /dev/null
2108+++ b/tools/hardware-test.sh
2109@@ -0,0 +1,112 @@
2110+#!/bin/sh -ex
2111+#
2112+# Copyright (C) 2017 Canonical Ltd
2113+#
2114+# This program is free software: you can redistribute it and/or modify
2115+# it under the terms of the GNU General Public License version 3 as
2116+# published by the Free Software Foundation.
2117+#
2118+# This program is distributed in the hope that it will be useful,
2119+# but WITHOUT ANY WARRANTY; without even the implied warranty of
2120+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
2121+# GNU General Public License for more details.
2122+#
2123+# You should have received a copy of the GNU General Public License
2124+# along with this program. If not, see <http://www.gnu.org/licenses/>.
2125+
2126+# Runs tests on real HW. Requires set_jenkins_env and set_test_type to have
2127+# already run.
2128+run_hardware_tests ()
2129+{
2130+ TEST_RESULTS=test_results
2131+
2132+ # Just dragonboard for the moment
2133+ DRAGONBOARD_TEST=testflinger-dragonboard.yaml
2134+
2135+ # We use jq to process testflinger output
2136+ if ! which jq; then
2137+ sudo apt install --yes --allow-downgrades --allow-remove-essential \
2138+ --allow-change-held-packages jq
2139+ fi
2140+
2141+ # Initially the device has a password-less ubuntu user. But spread needs a
2142+ # user with a password, so we use the DEVICE_USER/DEVICE_PASSWORD pair to
2143+ # create it. Note: {device_ip} gets substituted by testflinger.
2144+ DEVICE_USER=test
2145+ DEVICE_PASSWORD=test
2146+ DEVICE_SSH="ssh -q -o UserKnownHostsFile=/dev/null
2147+ -o StrictHostKeyChecking=no -p 22 ubuntu@{device_ip}"
2148+
2149+ if [ "$TEST_TYPE" = script ]; then
2150+ TEST_COMMAND="./run-tests.sh --spread-system=hw-ubuntu-core-16
2151+ --external-address={device_ip}:22 --external-user=$DEVICE_USER
2152+ --external-password=$DEVICE_PASSWORD --debug"
2153+ else
2154+ TEST_COMMAND="export SPREAD_EXTERNAL_ADDRESS={device_ip}:22 &&
2155+ export SPREAD_EXTERNAL_USER=$DEVICE_USER &&
2156+ export SPREAD_EXTERNAL_PASSWORD=$DEVICE_PASSWORD &&
2157+ ./spread -vv external:hw-ubuntu-core-16"
2158+ fi
2159+
2160+ cd "$WORKSPACE"
2161+
2162+ # Run testflinger from our bare metal server so we can install it
2163+
2164+ $SSH mkdir -p "$REMOTE_WORKSPACE"
2165+
2166+ # If the snap has been built, copy over
2167+ if [ -n "$RESULTS_ID" ]; then
2168+ set +x
2169+ # We need to flatten the key here to avoid issues with yaml parsing
2170+ SSH_KEY_DATA=$(tr '\n:' '?!' < "$SSH_KEY_PATH"/id_rsa)
2171+ set -x
2172+ fi
2173+
2174+ cat << EOF > "$WORKSPACE"/"$DRAGONBOARD_TEST"
2175+job_queue: dragonboard
2176+provision_data:
2177+ channel: stable
2178+test_data:
2179+ test_cmds:
2180+ - git clone --depth 1 -b $BRANCH $REPO src
2181+ - cd src && curl -s -O https://niemeyer.s3.amazonaws.com/spread-amd64.tar.gz && tar xzvf spread-amd64.tar.gz
2182+ - $DEVICE_SSH "sudo adduser --extrausers --quiet --disabled-password --gecos '' $DEVICE_USER"
2183+ - $DEVICE_SSH "echo $DEVICE_USER:$DEVICE_PASSWORD | sudo chpasswd"
2184+ - $DEVICE_SSH "echo '$DEVICE_USER ALL=(ALL) NOPASSWD:ALL' | sudo tee /etc/sudoers.d/create-user-test"
2185+ - if [ -n $RESULTS_ID ]; then set +x; echo "$SSH_KEY_DATA" | tr '?!' '\n:' > ssh_key; set -x; chmod 600 ssh_key; scp -i ssh_key $REMOTE_USER@$REMOTE_WORKER:$REMOTE_RESULTS_BASE_DIR/$RESULTS_ID/*.snap src/; fi
2186+ - cd src && export PATH=$PATH:\$(pwd) && $TEST_COMMAND
2187+EOF
2188+
2189+ $SCP "$WORKSPACE"/"$DRAGONBOARD_TEST" "$REMOTE_USER"@"$REMOTE_WORKER":"$REMOTE_WORKSPACE"/
2190+
2191+ $SSH << EOF
2192+#!/bin/sh
2193+set -ex
2194+
2195+cd $REMOTE_WORKSPACE
2196+if ! which virtualenv; then
2197+ sudo apt install --yes --allow-downgrades --allow-remove-essential --allow-change-held-packagesvirtualenv
2198+fi
2199+
2200+git clone https://git.launchpad.net/testflinger-cli
2201+cd testflinger-cli
2202+virtualenv -p python3 env
2203+. env/bin/activate
2204+./setup.py install
2205+
2206+JOB_ID=\$(testflinger-cli submit -q $REMOTE_WORKSPACE/$DRAGONBOARD_TEST)
2207+echo "JOB_ID: \${JOB_ID}"
2208+
2209+testflinger-cli poll \${JOB_ID}
2210+testflinger-cli results \${JOB_ID} > $REMOTE_WORKSPACE/$TEST_RESULTS
2211+EOF
2212+
2213+ $SCP "$REMOTE_USER"@"$REMOTE_WORKER":"$REMOTE_WORKSPACE"/"$TEST_RESULTS" "$WORKSPACE"/
2214+
2215+ $SSH sudo rm -rf "$REMOTE_WORKSPACE"
2216+
2217+ TEST_STATUS=$(jq -r .test_status "$WORKSPACE"/"$TEST_RESULTS")
2218+ echo "Test exit status: $TEST_STATUS"
2219+
2220+ return "$TEST_STATUS"
2221+}
2222diff --git a/tools/se_utils/__init__.py b/tools/se_utils/__init__.py
2223new file mode 100644
2224index 0000000..280450c
2225--- /dev/null
2226+++ b/tools/se_utils/__init__.py
2227@@ -0,0 +1,122 @@
2228+#!/usr/bin/env python
2229+# -*- Mode:Python; indent-tabs-mode:nil; tab-width:4 -*-
2230+#
2231+# Copyright (C) 2016 Canonical Ltd
2232+#
2233+# This program is free software: you can redistribute it and/or modify
2234+# it under the terms of the GNU General Public License version 3 as
2235+# published by the Free Software Foundation.
2236+#
2237+# This program is distributed in the hope that it will be useful,
2238+# but WITHOUT ANY WARRANTY; without even the implied warranty of
2239+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
2240+# GNU General Public License for more details.
2241+#
2242+# You should have received a copy of the GNU General Public License
2243+# along with this program. If not, see <http://www.gnu.org/licenses/>.
2244+
2245+import atexit
2246+import sys
2247+import time
2248+import logging
2249+import os
2250+import yaml
2251+from shutil import rmtree
2252+from launchpadlib.credentials import RequestTokenAuthorizationEngine
2253+from lazr.restfulclient.errors import HTTPError
2254+from launchpadlib.launchpad import Launchpad
2255+from launchpadlib.credentials import UnencryptedFileCredentialStore
2256+
2257+ACCESS_TOKEN_POLL_TIME = 10
2258+WAITING_FOR_USER = """Open this link:
2259+{}
2260+to authorize this program to access Launchpad on your behalf.
2261+Waiting to hear from Launchpad about your decision. . . ."""
2262+
2263+
2264+class AuthorizeRequestTokenWithConsole(RequestTokenAuthorizationEngine):
2265+ """Authorize a token in a server environment (with no browser).
2266+
2267+ Print a link for the user to copy-and-paste into his/her browser
2268+ for authentication.
2269+ """
2270+
2271+ def __init__(self, *args, **kwargs):
2272+ # as implemented in AuthorizeRequestTokenWithBrowser
2273+ kwargs['consumer_name'] = None
2274+ kwargs.pop('allow_access_levels', None)
2275+ super(AuthorizeRequestTokenWithConsole, self).__init__(*args, **kwargs)
2276+
2277+ def make_end_user_authorize_token(self, credentials, request_token):
2278+ """Ask the end-user to authorize the token in their browser.
2279+
2280+ """
2281+ authorization_url = self.authorization_url(request_token)
2282+ print WAITING_FOR_USER.format(authorization_url)
2283+ # if we don't flush we may not see the message
2284+ sys.stdout.flush()
2285+ while credentials.access_token is None:
2286+ time.sleep(ACCESS_TOKEN_POLL_TIME)
2287+ try:
2288+ credentials.exchange_request_token_for_access_token(
2289+ self.web_root)
2290+ break
2291+ except HTTPError, e:
2292+ if e.response.status == 403:
2293+ # The user decided not to authorize this
2294+ # application.
2295+ raise e
2296+ elif e.response.status == 401:
2297+ # The user has not made a decision yet.
2298+ pass
2299+ else:
2300+ # There was an error accessing the server.
2301+ raise e
2302+
2303+# launchpadlib is not thread/process safe so we are creating launchpadlib
2304+# cache in /tmp per process which gets cleaned up at the end
2305+# see also lp:459418 and lp:1025153
2306+launchpad_cachedir = os.path.join('/tmp', str(os.getpid()), '.launchpadlib')
2307+
2308+# `launchpad_cachedir` is leaked upon unexpected exits
2309+# adding this cleanup to stop directories filling up `/tmp/`
2310+atexit.register(rmtree, os.path.join('/tmp',
2311+ str(os.getpid())),
2312+ ignore_errors=True)
2313+
2314+
2315+def get_launchpad(launchpadlib_dir=None, credential_store_path=None, lp_app=None, lp_env=None):
2316+ """ return a launchpad API class. In case launchpadlib_dir is
2317+ specified used that directory to store launchpadlib cache instead of
2318+ the default """
2319+ store = UnencryptedFileCredentialStore(credential_store_path)
2320+ authorization_engine = AuthorizeRequestTokenWithConsole(lp_env, lp_app)
2321+ lib_dir=launchpad_cachedir
2322+ if launchpadlib_dir != None:
2323+ lib_dir = launchpadlib_dir
2324+ return Launchpad.login_with(lp_app, lp_env,
2325+ credential_store=store,
2326+ authorization_engine=authorization_engine,
2327+ launchpadlib_dir=lib_dir,
2328+ version='devel')
2329+
2330+# Load configuration for the current agent we're running on. All agents were
2331+# provisioned when they were setup with a proper configuration. See
2332+# https://wiki.canonical.com/InformationInfrastructure/Jenkaas/UserDocs for
2333+# more details.
2334+def load_config():
2335+ files = [os.path.expanduser('~/.jlp/jlp.config'), 'jlp.config']
2336+ for config_file in files:
2337+ try:
2338+ config = yaml.safe_load(open(config_file, 'r'))
2339+ return config
2340+ except IOError:
2341+ pass
2342+ print("ERROR: No config file found")
2343+ sys.exit(1)
2344+
2345+# Return a configuration option from the agent configuration specified by the
2346+# name argument.
2347+def get_config_option(name):
2348+ config = load_config()
2349+ return config[name]
2350diff --git a/tools/se_utils/__init__.pyc b/tools/se_utils/__init__.pyc
2351new file mode 100644
2352index 0000000..f1be9ca
2353Binary files /dev/null and b/tools/se_utils/__init__.pyc differ
2354diff --git a/tools/snapbuild.sh b/tools/snapbuild.sh
2355new file mode 100755
2356index 0000000..f84afb3
2357--- /dev/null
2358+++ b/tools/snapbuild.sh
2359@@ -0,0 +1,97 @@
2360+#!/bin/sh
2361+#
2362+# Copyright (C) 2017 Canonical Ltd
2363+#
2364+# This program is free software: you can redistribute it and/or modify
2365+# it under the terms of the GNU General Public License version 3 as
2366+# published by the Free Software Foundation.
2367+#
2368+# This program is distributed in the hope that it will be useful,
2369+# but WITHOUT ANY WARRANTY; without even the implied warranty of
2370+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
2371+# GNU General Public License for more details.
2372+#
2373+# You should have received a copy of the GNU General Public License
2374+# along with this program. If not, see <http://www.gnu.org/licenses/>.
2375+
2376+set -ex
2377+
2378+if [ "$(id -u)" -ne 0 ]; then
2379+ echo "ERROR: You have to run this script as root!"
2380+ exit 1
2381+fi
2382+
2383+SERIES=xenial
2384+SOURCE_DIR=
2385+RESULTS_DIR=
2386+
2387+while [ -n "$1" ]; do
2388+ case "$1" in
2389+ --series=*)
2390+ SERIES=${1#*=}
2391+ shift
2392+ ;;
2393+ --source-dir=*)
2394+ SOURCE_DIR=${1#*=}
2395+ shift
2396+ ;;
2397+ --results-dir=*)
2398+ RESULTS_DIR=${1#*=}
2399+ shift
2400+ ;;
2401+ *)
2402+ echo "ERROR: Unknown options $1"
2403+ exit 1
2404+ esac
2405+done
2406+
2407+if [ -z "$SERIES" ]; then
2408+ echo "ERROR: No series specified"
2409+ exit 1
2410+fi
2411+
2412+if [ -z "$SOURCE_DIR" ]; then
2413+ echo "ERROR: No source dir specified"
2414+ exit 1
2415+fi
2416+
2417+if [ -z "$RESULTS_DIR" ]; then
2418+ echo "ERROR: No results dir specified"
2419+ exit 1
2420+fi
2421+
2422+if [ ! -e /build/chroots/$SERIES-rootfs.tar ] ; then
2423+ mkdir -p /build/chroots
2424+ WORKDIR=$(mktemp -d)
2425+ mkdir -p $WORKDIR/rootfs
2426+ debootstrap --components=main,universe $SERIES $WORKDIR/rootfs
2427+ (cd $WORKDIR/rootfs; tar cf /build/chroots/$SERIES-rootfs.tar .)
2428+ rm -rf $WORKDIR
2429+fi
2430+
2431+BUILDDIR=$(mktemp -d)
2432+
2433+cleanup() {
2434+ rm -rf $BUILDDIR
2435+}
2436+
2437+trap cleanup INT EXIT
2438+
2439+tar xf /build/chroots/$SERIES-rootfs.tar -C $BUILDDIR
2440+
2441+cp -ra $SOURCE_DIR $BUILDDIR/src
2442+
2443+cat << EOF > $BUILDDIR/do-build.sh
2444+#!/bin/sh
2445+set -ex
2446+echo "deb http://archive.ubuntu.com/ubuntu/ xenial-updates main universe restricted" >> /etc/apt/sources.list
2447+apt update
2448+apt upgrade -y
2449+apt install -y snapcraft
2450+cd /src
2451+snapcraft clean
2452+snapcraft
2453+EOF
2454+chmod +x $BUILDDIR/do-build.sh
2455+
2456+sudo chroot $BUILDDIR /do-build.sh
2457diff --git a/tools/test-snap.sh b/tools/test-snap.sh
2458new file mode 100755
2459index 0000000..4851645
2460--- /dev/null
2461+++ b/tools/test-snap.sh
2462@@ -0,0 +1,100 @@
2463+#!/bin/sh -ex
2464+#
2465+# Copyright (C) 2017 Canonical Ltd
2466+#
2467+# This program is free software: you can redistribute it and/or modify
2468+# it under the terms of the GNU General Public License version 3 as
2469+# published by the Free Software Foundation.
2470+#
2471+# This program is distributed in the hope that it will be useful,
2472+# but WITHOUT ANY WARRANTY; without even the implied warranty of
2473+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
2474+# GNU General Public License for more details.
2475+#
2476+# You should have received a copy of the GNU General Public License
2477+# along with this program. If not, see <http://www.gnu.org/licenses/>.
2478+
2479+# Import used functions
2480+. "$WORKSPACE"/build-scripts/scripts/hardware-test.sh
2481+
2482+# Runs snap tests
2483+run_snap_tests ()
2484+{
2485+ if [ -n "$RESULTS_ID" ]; then
2486+ $SSH mkdir -p "$REMOTE_WORKSPACE"/results
2487+ $SSH cp -v "$REMOTE_RESULTS_BASE_DIR"/"$RESULTS_ID"/*.snap "$REMOTE_WORKSPACE"/results
2488+ fi
2489+
2490+ $SSH sudo apt-get --yes --force-yes install docker.io
2491+
2492+ cat << EOF > "$WORKSPACE"/run-tests.sh
2493+#!/bin/sh
2494+set -ex
2495+
2496+export TERM=linux
2497+export DEBIAN_FRONTEND=noninteractive
2498+export PATH=/build/bin:$PATH
2499+
2500+# At this time it's necessary to build spread manually because
2501+# the snapped version does not include the qemu/kvm backend.
2502+# Once the snapped version includes this backend, then we can
2503+# change the manual building of spread with making sure the snap
2504+# package is installed.
2505+export GOPATH=$(mktemp -d)
2506+go get -d -v github.com/snapcore/spread/...
2507+go build github.com/snapcore/spread/cmd/spread
2508+mkdir /build/bin
2509+cp spread /build/bin
2510+
2511+git clone --depth 1 -b $BRANCH $REPO /build/src
2512+cd /build/src
2513+
2514+# Copy any stage results from previous generic-build-snap-worker builds
2515+if [ "\$(find /build/results -path ./misc -prune -o -name '*.snap' -print | wc -l)" -gt 0 ]; then
2516+ cp -v /build/results/*.snap /build/src
2517+fi
2518+
2519+if [ -e "run-tests.sh" ] ; then
2520+ EXTRA_ARGS=
2521+ if [ -n "$CHANNEL" ] ; then
2522+ # If CHANNEL is specified we use that channel for image construction and
2523+ # also load the snap from that channel for testing.
2524+ EXTRA_ARGS="--test-from-channel=$CHANNEL"
2525+ fi
2526+ if [ -n "$CORE_CHANNEL" ]; then
2527+ EXTRA_ARGS="--channel=$CORE_CHANNEL"
2528+ fi
2529+
2530+ ./run-tests.sh --force-new-image \$EXTRA_ARGS
2531+else
2532+ if [ ! -z "$CHANNEL" ] ; then
2533+ SNAP_CHANNEL=$CHANNEL spread -v
2534+ else
2535+ spread -v
2536+ fi
2537+fi
2538+EOF
2539+
2540+ $SSH mkdir -p "$REMOTE_WORKSPACE"
2541+ $SCP "$WORKSPACE"/run-tests.sh "$REMOTE_USER"@"$REMOTE_WORKER":"$REMOTE_WORKSPACE"
2542+ $SSH chmod u+x "$REMOTE_WORKSPACE"/run-tests.sh
2543+
2544+ $SSH mkdir -p "$REMOTE_WORKSPACE"/docker
2545+ $SCP "$WORKSPACE"/build-scripts/docker/spread-tests/Dockerfile \
2546+ "$REMOTE_USER"@"$REMOTE_WORKER":"$REMOTE_WORKSPACE"/docker
2547+ $SSH time sudo docker build -t snap-spread-tests "$REMOTE_WORKSPACE"/docker
2548+
2549+ $SSH time sudo docker run \
2550+ --rm \
2551+ -v /dev:/dev \
2552+ -v "$REMOTE_WORKSPACE":/build \
2553+ --privileged \
2554+ snap-spread-tests /build/run-tests.sh
2555+
2556+ $SSH sudo rm -rf "$REMOTE_WORKSPACE"
2557+
2558+ # Now run tests on real hardware if defined in the backend
2559+ if [ "$HW_TESTS_RESULT" -eq 0 ] ; then
2560+ run_hardware_tests
2561+ fi
2562+}
2563diff --git a/tools/trigger-lp-build.py b/tools/trigger-lp-build.py
2564new file mode 100755
2565index 0000000..5434281
2566--- /dev/null
2567+++ b/tools/trigger-lp-build.py
2568@@ -0,0 +1,230 @@
2569+#! /usr/bin/python
2570+
2571+import os
2572+import sys
2573+import time
2574+import random
2575+import smtplib
2576+import string
2577+import urllib2
2578+import zlib
2579+
2580+from datetime import datetime
2581+from os.path import basename
2582+from launchpadlib.launchpad import Launchpad
2583+
2584+from argparse import ArgumentParser
2585+
2586+import se_utils
2587+
2588+parser = ArgumentParser(description="Build a specific snap on launchpad")
2589+parser.add_argument('-s', '--snap', required=True,
2590+ help="Name of the snap to build")
2591+parser.add_argument('-p', '--publish', action='store_true',
2592+ help="Trigger a publish build instead of a daily (default)")
2593+parser.add_argument('-n', '--new', action='store_true', help="Create a new ephemeral snap build on launchpad")
2594+parser.add_argument('--git-repo', help="Git repository to be used for new ephemeral snap build")
2595+parser.add_argument('--git-repo-branch', help="Git repository branch to be used for new ephemeral snap build")
2596+parser.add_argument('-a', '--architectures', help="Specify architectures to build for. Separate multiple architectures by ','")
2597+parser.add_argument('-r', '--results-dir', help="Specify where results should be saved")
2598+
2599+args = vars(parser.parse_args())
2600+
2601+ephemeral_build=False
2602+results_dir=os.path.join(os.getcwd(), "results")
2603+
2604+if 'results_dir' in args:
2605+ results_dir=args['results_dir']
2606+
2607+if args['new']:
2608+ ephemeral_build=True
2609+ if args['git_repo'] == None or args['git_repo_branch'] == None:
2610+ print("ERROR: No git repository or a branch supplied")
2611+ sys.exit(1)
2612+
2613+series = 'xenial'
2614+
2615+lp_app = se_utils.get_config_option("lp_app")
2616+lp_env = se_utils.get_config_option("lp_env")
2617+credential_store_path = se_utils.get_config_option('credential_store_path')
2618+launchpad = se_utils.get_launchpad(None, credential_store_path, lp_app, lp_env)
2619+
2620+team = launchpad.people['snappy-hwe-team']
2621+ubuntu = launchpad.distributions['ubuntu']
2622+release = ubuntu.getSeries(name_or_version=series)
2623+primary_archive = ubuntu.getArchive(name='primary')
2624+
2625+snap=None
2626+if ephemeral_build:
2627+ snap_arches=[]
2628+ if 'architectures' in args and args['architectures'] != None:
2629+ snap_arches = args["architectures"].split(",")
2630+
2631+ if len(snap_arches) == 0:
2632+ print("WARNING: No architectures to build specified. Will only build for amd64.")
2633+ snap_arches=["amd64"]
2634+
2635+ processors=[]
2636+ for arch in snap_arches:
2637+ try:
2638+ p = launchpad.processors.getByName(name=arch)
2639+ processors.append(p.self_link)
2640+ except:
2641+ print("ERROR: Failed to find processor for '{}' architecture".format(arch))
2642+ sys.exit(1)
2643+
2644+ build_name = 'ci-%s-%s' % (args["snap"], ''.join(random.choice(string.ascii_lowercase + string.digits) for _ in range(16)))
2645+ snap = launchpad.snaps.new(name=build_name,
2646+ processors=processors,
2647+ auto_build=False, distro_series=release,
2648+ git_repository_url=args['git_repo'],
2649+ git_path='%s' % args["git_repo_branch"],
2650+ owner=team)
2651+else:
2652+ build_name = "%s-daily" % args["snap"]
2653+ if args["publish"] == True:
2654+ build_name = "%s-publish" % args["snap"]
2655+ snap = launchpad.snaps.getByName(name=build_name, owner=team)
2656+
2657+if snap == None:
2658+ print("ERROR: Failed to create snap build on launchpad")
2659+
2660+# Not every snap is build agains all arches.
2661+arches = [processor.name for processor in snap.processors]
2662+if not ephemeral_build and args['architectures'] != None:
2663+ wanted_arches = args["architectures"].split(",")
2664+ possible_arches = []
2665+ for arch in wanted_arches:
2666+ if not arch in arches:
2667+ print("WARNING: Can't build snap for architecture {} as it is not enabled in the build job".format(args["snap"]))
2668+ continue
2669+ possible_arches.append(arch)
2670+ arches = possible_arches
2671+
2672+if len(arches) == 0:
2673+ print("ERROR: No architectures available to build for")
2674+ sys.exit(1)
2675+
2676+# Add a big fat warning that we don't really care about fixing things when
2677+# the job will be canceled after the following lines are printed out.
2678+print("!!!!!!! POINT OF NO RETURN !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!")
2679+print("DO NOT CANCEL THIS JOB AFTER THIS OR BAD THINGS WILL HAPPEN")
2680+print("!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!")
2681+
2682+stamp = datetime.now().strftime('%Y-%m-%d %H:%M:%S')
2683+print("Trying to trigger builds at: {}".format(stamp))
2684+
2685+# sometimes we see error such as "u'Unknown architecture lpia for ubuntu xenial'"
2686+# and in order to workaround let's validate the arches agains set of valid
2687+# architectures that the snap can choose from
2688+
2689+
2690+# We will now trigger a build for each whitelisted architecture, collect the
2691+# build job url and the wait for all builds to finish and collect their results
2692+# to vote for a successful or failed build.
2693+triggered_builds = []
2694+triggered_build_urls = {}
2695+valid_arches = ['armhf', 'i386', 'amd64', 'arm64', 's390x', 'powerpc', 'ppc64el']
2696+for build_arch in arches:
2697+ # sometimes we see error such as "u'Unknown architecture lpia for
2698+ # ubuntu xenial'" and in order to workaround let's validate the arches
2699+ # agains set of valid architectures that the snap can choose from
2700+ if build_arch not in valid_arches:
2701+ print("WARNING: Can't build snap for architecture {} as it is not enabled in the build job".format(args["snap"]))
2702+ continue
2703+
2704+ arch = release.getDistroArchSeries(archtag=build_arch)
2705+ request = snap.requestBuild(archive=primary_archive, distro_arch_series=arch, pocket='Proposed')
2706+ build_id = str(request).rsplit('/', 1)[-1]
2707+ triggered_builds.append(build_id)
2708+ triggered_build_urls[build_id] = request.self_link
2709+ print("Arch: {} is building under: {}".format(build_arch, request.self_link))
2710+
2711+failures = []
2712+successful = []
2713+while len(triggered_builds):
2714+ for build in triggered_builds:
2715+ try:
2716+ response = snap.getBuildSummariesForSnapBuildIds(snap_build_ids=[build])
2717+ except:
2718+ print("Could not get response for {} (was there an LP timeout?)".format(build))
2719+ continue
2720+ status = response[build]['status']
2721+ if status == "FULLYBUILT":
2722+ successful.append(build)
2723+ triggered_builds.remove(build)
2724+ continue
2725+ elif status == "FAILEDTOBUILD":
2726+ failures.append(build)
2727+ triggered_builds.remove(build)
2728+ continue
2729+ elif status == "CANCELLED":
2730+ print("INFO: {} snap build was canceled for id: {}".format(args["snap"], build))
2731+ triggered_builds.remove(build)
2732+ continue
2733+ if len(triggered_builds) > 0:
2734+ time.sleep(60)
2735+
2736+if len(failures):
2737+ for failure in failures:
2738+ try:
2739+ response = snap.getBuildSummariesForSnapBuildIds(snap_build_ids=[failure])
2740+ except:
2741+ print ("Could not get failure data for {} (was there an LP timeout?)".format(build))
2742+ continue
2743+
2744+ if not failure in response:
2745+ print("Launchpad didn't returned us the snap build summary we ask it for!?")
2746+ continue
2747+
2748+ build_summary = response[failure]
2749+ arch = 'unknown'
2750+ buildlog = None
2751+ if 'build_log_url' in build_summary:
2752+ buildlog = build_summary['build_log_url']
2753+
2754+ if buildlog != None and len(buildlog) > 0:
2755+ parts = arch = str(buildlog).split('_')
2756+ if len(parts) >= 4:
2757+ arch = parts[4]
2758+ elif buildlog == None:
2759+ buildlog = 'not available'
2760+
2761+ print("INFO: {} snap {} build at {} failed for id: {} log: {}".format(args["snap"], arch, stamp, failure, buildlog))
2762+
2763+ # For ephermal builds we need to print out the log file as it will be gone after
2764+ # the launchpad build is removed.
2765+ if ephemeral_build and buildlog != None:
2766+ response = urllib2.urlopen(buildlog)
2767+ log_data = zlib.decompress(response.read(), 16+zlib.MAX_WBITS)
2768+ print(log_data)
2769+
2770+# Fetch build results for successful builds and store those in the output
2771+# directory so that the caller can reuse them.
2772+if len(successful):
2773+ for success in successful:
2774+ try:
2775+ snap_build = launchpad.load(triggered_build_urls[success])
2776+ urls = snap_build.getFileUrls()
2777+ if len(urls):
2778+ for u in urls:
2779+ print("Downloading snap from %s ..." % u)
2780+ response = urllib2.urlopen(u)
2781+ if not os.path.exists(results_dir):
2782+ os.makedirs(results_dir)
2783+ path = os.path.join(results_dir, os.path.basename(u))
2784+ with open(path, "w") as out_file:
2785+ out_file.write(response.read())
2786+ except:
2787+ print ("Could not retrieve snap build data for {} (was there an LP timeout?)".format(build))
2788+ continue
2789+
2790+
2791+if ephemeral_build:
2792+ snap.lp_delete()
2793+
2794+if len(failures):
2795+ # Let the build fail as at least a single snap has failed to build
2796+ sys.exit(1)
2797+
2798+print("Done!")
2799diff --git a/tools/trigger-snap-builds.py b/tools/trigger-snap-builds.py
2800new file mode 100755
2801index 0000000..dc74456
2802--- /dev/null
2803+++ b/tools/trigger-snap-builds.py
2804@@ -0,0 +1,218 @@
2805+#!/usr/bin/env python
2806+# -*- Mode:Python; indent-tabs-mode:nil; tab-width:4 -*-
2807+#
2808+# Copyright (C) 2016 Canonical Ltd
2809+#
2810+# This program is free software: you can redistribute it and/or modify
2811+# it under the terms of the GNU General Public License version 3 as
2812+# published by the Free Software Foundation.
2813+#
2814+# This program is distributed in the hope that it will be useful,
2815+# but WITHOUT ANY WARRANTY; without even the implied warranty of
2816+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
2817+# GNU General Public License for more details.
2818+#
2819+# You should have received a copy of the GNU General Public License
2820+# along with this program. If not, see <http://www.gnu.org/licenses/>.
2821+
2822+from launchpadlib.launchpad import Launchpad
2823+import jenkins
2824+import os
2825+import sys
2826+import yaml
2827+import re
2828+from jlp import launchpadutils
2829+from jlp import jenkinsutils
2830+
2831+from argparse import ArgumentParser
2832+
2833+import se_utils
2834+
2835+parser = ArgumentParser(description="Trigger snap builds for pending merge proposals")
2836+parser.add_argument('-p', '--project', required=True,
2837+ help="Launchpad project to check for new merge-proposals")
2838+parser.add_argument('-j', '--job', required=True,
2839+ help="Jenkins job to trigger")
2840+
2841+args = vars(parser.parse_args())
2842+
2843+lp_app = se_utils.get_config_option("lp_app")
2844+lp_env = se_utils.get_config_option("lp_env")
2845+credential_store_path = se_utils.get_config_option('credential_store_path')
2846+launchpad = se_utils.get_launchpad(None, credential_store_path, lp_app, lp_env)
2847+
2848+project = launchpad.projects[args['project']]
2849+
2850+proposals = project.getMergeProposals()
2851+
2852+def load_config():
2853+ files = [os.path.expanduser('~/.jlp/jlp.config'), 'jlp.config']
2854+ for config_file in files:
2855+ try:
2856+ config = yaml.safe_load(open(config_file, 'r'))
2857+ return config
2858+ except IOError:
2859+ pass
2860+ print("ERROR: No config file found")
2861+ sys.exit(1)
2862+
2863+def get_config_option(name):
2864+ config = load_config()
2865+ return config[name]
2866+
2867+j = jenkins.Jenkins(get_config_option('jenkins_url'),
2868+ get_config_option('jenkins_user'),
2869+ get_config_option('jenkins_password'))
2870+
2871+jenkins_job = args['job']
2872+jenkins_build_token = get_config_option('jenkins_build_token')
2873+
2874+job_info = j.get_job_info(jenkins_job)
2875+if not job_info['buildable']:
2876+ print("ERROR: Job is not buildable (propably disabled)")
2877+ sys.exit(1)
2878+
2879+
2880+def get_latest_revision(mp):
2881+ """Return the latest revision of the given merge proposal.
2882+
2883+ :param mp: handle to merge proposal
2884+ """
2885+ if '+git' in mp.web_link:
2886+ for ref in mp.source_git_repository.refs_collection:
2887+ if ref.path == mp.source_git_path:
2888+ return ref.commit_sha1
2889+ return str(0)
2890+ else:
2891+ return mp.source_branch.revision_count
2892+
2893+def clean_branch_name(branch_name):
2894+ if branch_name.startswith("refs/heads/"):
2895+ return branch_name[11:]
2896+ return branch_name
2897+
2898+def series_from_branch_name(branch_name):
2899+ if branch_name.startswith("vivid/"):
2900+ return "vivid"
2901+ return "xenial"
2902+
2903+def get_latest_revision(mp):
2904+ """Return the latest revision of the given merge proposal.
2905+
2906+ :param mp: handle to merge proposal
2907+ """
2908+ if '+git' in mp.web_link:
2909+ for ref in mp.source_git_repository.refs_collection:
2910+ if ref.path == mp.source_git_path:
2911+ return ref.commit_sha1
2912+ return str(0)
2913+ else:
2914+ return mp.source_branch.revision_count
2915+
2916+def get_review_revision_regex(mp):
2917+ if '+git' in mp.web_link:
2918+ return '^(PASSED|FAILED): Continuous integration, rev:([0-9a-f]+)'
2919+ else:
2920+ return '^(PASSED|FAILED): Continuous integration, rev:(\d+)'
2921+
2922+def get_latest_review(launchpad_user, mp):
2923+ """Return the latest revision reviewed by the given launchpad_user.
2924+
2925+ This function expects a review comment in the following format:
2926+ '^(PASSED|FAILED): Continuous integration, rev:(\d+)'
2927+
2928+ :param launchpad_user: handle to launchpad user
2929+ :param mp: handle to merge proposal
2930+ """
2931+ revision = 0
2932+ launchpad_review_type = get_config_option('launchpad_review_type')
2933+ for comment in mp.all_comments:
2934+ if comment.author.name == launchpad_user.name:
2935+ if comment.vote_tag == launchpad_review_type:
2936+ m = re.search(
2937+ get_review_revision_regex(mp),
2938+ comment.message_body)
2939+ if m:
2940+ revision = m.group(2)
2941+ return revision
2942+
2943+def latest_candidate_validated(launchpad_user, mp):
2944+ """Return if the latest candidate revision of the merge proposal is
2945+ validated.
2946+
2947+ :param launchpad_user: handle to launchpad user used to validate the
2948+ merge proposals
2949+ :param mp: handle to merge proposal
2950+ """
2951+
2952+ latest_review = get_latest_review(launchpad_user, mp)
2953+ print 'Latest review is revision: ' + str(latest_review)
2954+ latest_revision = get_latest_revision(mp)
2955+ print 'Latest revision is: ' + str(latest_revision)
2956+ if latest_review == latest_revision:
2957+ print 'Skipping this MP. Current revision: ' + str(latest_revision)
2958+ return True
2959+ return False
2960+
2961+# NOTE: We can't use the testing_in_progress method from jlp as it checks for
2962+# the lower case parameter 'merge_proposal' of the build job and ours is using
2963+# the upper case variant 'MERGE_PROPOSAL'.
2964+def testing_in_progress(mp, jenkins_job):
2965+ if jenkinsutils.is_job_or_downstream_building(
2966+ jenkins_job, job_params={'MERGE_PROPOSAL': mp.web_link}):
2967+ print('Skipping this MP. It is currently being tested by Jenkins.')
2968+ return True
2969+ return False
2970+
2971+project_blacklist = []
2972+
2973+for proposal in proposals:
2974+ launchpad_user = launchpad.people(get_config_option('launchpad_login'))
2975+
2976+ print("Checking proposal %s .." % proposal.web_link)
2977+
2978+ # Ignore certain projects which don't build here as they are
2979+ # fetching source from somewhere else.
2980+ ignore = False
2981+ for project in project_blacklist:
2982+ if project in proposal.web_link:
2983+ ignore = True
2984+ break
2985+
2986+ if ignore:
2987+ print "Ignoring %s" % proposal.web_link
2988+ continue
2989+
2990+ if not launchpadutils.users_allowed_to_trigger_jobs([proposal.registrant]):
2991+ continue
2992+
2993+ if latest_candidate_validated(launchpad_user, proposal):
2994+ continue
2995+
2996+ if testing_in_progress(proposal, jenkins_job):
2997+ continue
2998+
2999+ target_repo = launchpad.load(proposal.target_git_repository_link)
3000+ source_repo = launchpad.load(proposal.source_git_repository_link)
3001+
3002+ target_git_url = target_repo.git_https_url
3003+ if target_git_url == None:
3004+ target_git_url = target_repo.git_ssh_url
3005+
3006+ source_git_url = source_repo.git_https_url
3007+ if source_git_url == None:
3008+ source_git_url = target_repo.git_ssh_url
3009+
3010+ target_branch = clean_branch_name(proposal.target_git_path)
3011+ jenkins_params = {
3012+ 'TARGET_GIT_REPO': target_git_url,
3013+ 'TARGET_GIT_REPO_BRANCH': target_branch,
3014+ 'SOURCE_GIT_REPO': source_git_url,
3015+ 'SOURCE_GIT_REPO_BRANCH': clean_branch_name(proposal.source_git_path),
3016+ 'MERGE_PROPOSAL': proposal.web_link,
3017+ 'REVISION': get_latest_revision(proposal),
3018+ 'SERIES': series_from_branch_name(target_branch),
3019+ }
3020+
3021+ print("Triggering build job for %s" % proposal.web_link)
3022+ j.build_job(jenkins_job, jenkins_params, jenkins_build_token)
3023diff --git a/tools/vote-on-merge-proposal.py b/tools/vote-on-merge-proposal.py
3024new file mode 100755
3025index 0000000..bec6da5
3026--- /dev/null
3027+++ b/tools/vote-on-merge-proposal.py
3028@@ -0,0 +1,271 @@
3029+#!/usr/bin/env python
3030+# -*- Mode:Python; indent-tabs-mode:nil; tab-width:4 -*-
3031+#
3032+# Copyright (C) 2016 Canonical Ltd
3033+#
3034+# This program is free software: you can redistribute it and/or modify
3035+# it under the terms of the GNU General Public License version 3 as
3036+# published by the Free Software Foundation.
3037+#
3038+# This program is distributed in the hope that it will be useful,
3039+# but WITHOUT ANY WARRANTY; without even the implied warranty of
3040+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
3041+# GNU General Public License for more details.
3042+#
3043+# You should have received a copy of the GNU General Public License
3044+# along with this program. If not, see <http://www.gnu.org/licenses/>.
3045+
3046+import atexit
3047+import sys
3048+import time
3049+import logging
3050+import os
3051+import yaml
3052+import re
3053+from shutil import rmtree
3054+from argparse import ArgumentParser
3055+from launchpadlib.credentials import RequestTokenAuthorizationEngine
3056+from lazr.restfulclient.errors import HTTPError
3057+from launchpadlib.launchpad import Launchpad
3058+from launchpadlib.credentials import UnencryptedFileCredentialStore
3059+from jlp import get_config_option
3060+from jlp import launchpadutils, jenkinsutils, logger
3061+
3062+logger = logging.getLogger('jenkins-launchpad-plugin')
3063+stdout_handler = logging.StreamHandler(stream=sys.stdout)
3064+formatter = logging.Formatter('%(levelname)s: %(message)s')
3065+stdout_handler.setFormatter(formatter)
3066+logger.addHandler(stdout_handler)
3067+
3068+parser = ArgumentParser(description="Vote on a Launchpad merge proposal.")
3069+parser.add_argument('-s', '--status')
3070+parser.add_argument('-u', '--build-url', required=True,
3071+ help="URL of the Jenkins job")
3072+parser.add_argument('-p', '--merge-proposal', required=True,
3073+ help="URL of the merge proposal to update")
3074+parser.add_argument('-r', '--revision', required=True,
3075+ help="merge proposal candidate revision")
3076+
3077+args = vars(parser.parse_args())
3078+
3079+ACCESS_TOKEN_POLL_TIME = 10
3080+WAITING_FOR_USER = """Open this link:
3081+{}
3082+to authorize this program to access Launchpad on your behalf.
3083+Waiting to hear from Launchpad about your decision. . . ."""
3084+
3085+
3086+class AuthorizeRequestTokenWithConsole(RequestTokenAuthorizationEngine):
3087+ """Authorize a token in a server environment (with no browser).
3088+
3089+ Print a link for the user to copy-and-paste into his/her browser
3090+ for authentication.
3091+ """
3092+
3093+ def __init__(self, *args, **kwargs):
3094+ # as implemented in AuthorizeRequestTokenWithBrowser
3095+ kwargs['consumer_name'] = None
3096+ kwargs.pop('allow_access_levels', None)
3097+ super(AuthorizeRequestTokenWithConsole, self).__init__(*args, **kwargs)
3098+
3099+ def make_end_user_authorize_token(self, credentials, request_token):
3100+ """Ask the end-user to authorize the token in their browser.
3101+
3102+ """
3103+ authorization_url = self.authorization_url(request_token)
3104+ print WAITING_FOR_USER.format(authorization_url)
3105+ # if we don't flush we may not see the message
3106+ sys.stdout.flush()
3107+ while credentials.access_token is None:
3108+ time.sleep(ACCESS_TOKEN_POLL_TIME)
3109+ try:
3110+ credentials.exchange_request_token_for_access_token(
3111+ self.web_root)
3112+ break
3113+ except HTTPError, e:
3114+ if e.response.status == 403:
3115+ # The user decided not to authorize this
3116+ # application.
3117+ raise e
3118+ elif e.response.status == 401:
3119+ # The user has not made a decision yet.
3120+ pass
3121+ else:
3122+ # There was an error accessing the server.
3123+ raise e
3124+
3125+
3126+def get_launchpad(launchpadlib_dir=None):
3127+ """ return a launchpad API class. In case launchpadlib_dir is
3128+ specified used that directory to store launchpadlib cache instead of
3129+ the default """
3130+ store = UnencryptedFileCredentialStore(
3131+ get_config_option('credential_store_path'))
3132+ lp_app = get_config_option('lp_app')
3133+ lp_env = get_config_option('lp_env')
3134+ authorization_engine = AuthorizeRequestTokenWithConsole(lp_env, lp_app)
3135+ return Launchpad.login_with(lp_app, lp_env,
3136+ credential_store=store,
3137+ authorization_engine=authorization_engine,
3138+ launchpadlib_dir=launchpadlib_dir,
3139+ version='devel')
3140+
3141+def get_branch_handle_from_url(lp_handle, url):
3142+ """ Return a branch/repo handle for the given url.
3143+ Returns a launchpad branch or git repository handle for the given url.
3144+ :param lp_handle: launchpad API handle/instance
3145+ :param url: url of the branch or git repository
3146+ """
3147+ if '+git' in url:
3148+ name = url.replace('https://code.launchpad.net/', '')
3149+ logger.debug('fetching repo: ' + name)
3150+ try:
3151+ return lp_handle.git_repositories.getByPath(path=name)
3152+ except AttributeError:
3153+ logger.debug('git_repositories.getByPath was not found. You may need to set lp_version=devel in the config')
3154+ return None
3155+ else:
3156+ name = url.replace('https://code.launchpad.net/', 'lp:')
3157+ name = name.replace('https://code.staging.launchpad.net/', 'lp://staging/')
3158+ logger.debug('fetching branch: ' + name)
3159+ return lp_handle.branches.getByUrl(url=name)
3160+
3161+def get_branch_from_mp(merge_proposal):
3162+ """Return a link to branch given a link to a merge proposal.
3163+
3164+ If merge_proposal is:
3165+ https://copde.launchpad.net/~user/project/name/+merge/12345
3166+ then the result will be:
3167+ https://copde.launchpad.net/~user/project/name/
3168+
3169+ :param merge_proposal: url of a launchpad merge proposal
3170+ """
3171+ m = re.search('(.*)\+merge/[0-9]+$', merge_proposal)
3172+ if m:
3173+ return m.group(1)
3174+ return None
3175+
3176+def get_mp_handle_from_url(lp_handle, merge_proposal_link):
3177+ """ Get launchpad handle for merge proposal given a merge proposal URL.
3178+
3179+ Returns None in case the merge proposal can't be found.
3180+ :param merge_proposal_link: URL of the merge proposal
3181+ """
3182+ branch_link = get_branch_from_mp(merge_proposal_link)
3183+ if not branch_link:
3184+ logger.error('Unable to get branch link from merge proposal link.')
3185+ return None
3186+
3187+ branch = get_branch_handle_from_url(lp_handle, branch_link)
3188+ if not branch:
3189+ logger.debug('Branch {} does not exist'.format(branch_link))
3190+ return None
3191+
3192+ logger.debug('mp_link: {}.'.format(merge_proposal_link))
3193+
3194+ for mp in branch.landing_targets:
3195+ logger.debug('mp.web_link: {}'.format(mp.web_link))
3196+ if mp.web_link == merge_proposal_link:
3197+ return mp
3198+
3199+ return None
3200+
3201+class LaunchpadVote():
3202+ APPROVE = 'Approve'
3203+ DISAPPROVE = 'Disapprove'
3204+ NEEDS_FIXING = 'Needs Fixing'
3205+
3206+def get_vote_subject(mp):
3207+ """Given a mp handle return a subject for the vote message
3208+
3209+ Unfortunately there is no method in the API that gives you the "standard"
3210+ subject that launchapd is using and some email clients (gmail) are
3211+ grouping conversations into threads based on subject.
3212+
3213+ This returns what seems to be the launchpad way of doing subjects.
3214+ :param mp: launchpad merge proposal handle
3215+ """
3216+
3217+ if '+git' in mp.web_link:
3218+ source = mp.source_git_repository.display_name.replace('lp:', '') + \
3219+ ':' + \
3220+ mp.source_git_path.replace('refs/heads/', '')
3221+ target = mp.target_git_repository.display_name.replace('lp:', '') + \
3222+ ':' + \
3223+ mp.target_git_path.replace('refs/heads/', '')
3224+ return 'Re: [Merge] {} into {}'.format(source, target)
3225+ else:
3226+ return 'Re: [Merge] {} into {}'.format(
3227+ mp.source_branch.display_name,
3228+ mp.target_branch.display_name)
3229+
3230+
3231+def approve_mp(mp, revision, build_url):
3232+ """Approve a given merge proposal a revision.
3233+
3234+ :params mp: launchapd handle to the respective merge proposal
3235+ :params revision: revision that should be approved
3236+ :params build_url: jenkins build url with the details. This job is used to
3237+ generate the message with all the links to test runs as
3238+ well as artifacts (coverity, deb files, etc)
3239+ """
3240+ state = 'PASSED: Continuous integration, rev:' + str(revision)
3241+ logger.debug(state)
3242+ content = jenkinsutils.format_message_for_mp_update(build_url,
3243+ state + "\n")
3244+ mp.createComment(review_type=get_config_option('launchpad_review_type'),
3245+ vote=LaunchpadVote.APPROVE, subject=get_vote_subject(mp),
3246+ content=content)
3247+
3248+
3249+def disapprove_mp(mp, revision, build_url, reason=None):
3250+ """Disapprove a given merge proposal a revision (vote Needs Fixing).
3251+
3252+ :params mp: launchapd handle to the respective merge proposal
3253+ :params revision: revision that should be fixed
3254+ :params build_url: jenkins build url with the details. This job is used to
3255+ generate the message with all the links to test runs as
3256+ well as artifacts (coverity, deb files, etc)
3257+ :params reason: optional string that is attached to the comment
3258+ """
3259+ state = "FAILED: Continuous integration, rev:{revision}".format(
3260+ revision=revision)
3261+ if reason:
3262+ state = "{state}\n{reason}".format(state=state, reason=reason)
3263+
3264+ logger.debug(state)
3265+ content = jenkinsutils.format_message_for_mp_update(
3266+ build_url, state + "\n")
3267+ mp.createComment(review_type=get_config_option('launchpad_review_type'),
3268+ vote=LaunchpadVote.NEEDS_FIXING,
3269+ subject=get_vote_subject(mp),
3270+ content=content)
3271+
3272+# launchpadlib is not thread/process safe so we are creating launchpadlib
3273+# cache in /tmp per process which gets cleaned up at the end
3274+# see also lp:459418 and lp:1025153
3275+launchpad_cachedir = os.path.join('/tmp', str(os.getpid()), '.launchpadlib')
3276+
3277+# `launchpad_cachedir` is leaked upon unexpected exits
3278+# adding this cleanup to stop directories filling up `/tmp/`
3279+atexit.register(rmtree, os.path.join('/tmp',
3280+ str(os.getpid())),
3281+ ignore_errors=True)
3282+
3283+lp_handle = get_launchpad(launchpadlib_dir=launchpad_cachedir)
3284+mp = get_mp_handle_from_url(lp_handle, args["merge_proposal"])
3285+if not mp:
3286+ parser.error('merge proposal related to this branch was not found')
3287+
3288+# this is the status from tests
3289+overal_status = args['status']
3290+# by default reason is empty as it is usually just a failed build
3291+reason = ''
3292+
3293+if overal_status == 'PASSED':
3294+ approve_mp(mp, args['revision'], args['build_url'])
3295+else: # status == False corresponds to NOT 'PASSED'
3296+ disapprove_mp(mp,
3297+ args['revision'],
3298+ args['build_url'],
3299+ reason)

Subscribers

People subscribed via source and target branches

to all changes: