Merge lp:~fginther/charms/trusty/jenkins-remote-slave/jenkins-phablet-slave into lp:~canonical-ci-engineering/charms/trusty/jenkins-remote-slave/trunk

[Francis Ginther]

Allow specification of a jenkins master and slave name to connect to instead of a master set via a relationship. Add provisioning scripts to setup an adb host and keys for operation with the ci-teams existing jobs.

Allow specification of a jenkins master and slave name to connect to instead of a master set via a relationship. Add provisioning scripts to setup an adb host and keys for operation with the ci-teams existing jobs.

[Vincent Ladeuil (vila)]

Nice job !

A few inline comments but no blockers, I think I don't quite understand the details of how it works (but I can't see anything that should break either ;).

review: Approve

[Evan (ev)]

Largely looks good, but I have a few concerns inline.

review: Needs Fixing

[Evan (ev)]

Also, we should convert this piece by piece to python with tests as we iterate on it. Not worth blocking this MP on that, but we should address it soon.

[Paul Larson (pwlars)]

Responded to a few of these for now

[Evan (ev)]

Comments inline.

[Francis Ginther (fginther)]

To address the multiple comments on "files/jenkins-slave/jenkins-slave-remote". This file is a copy of the one supplied by the jenkins-slave package. The only values we actually care about are: NAME,
JENKINS_HOSTNAME and JENKINS_URL. An improved sed command would allow us to use the one supplied by the package itself and not have to supply the copy.

files/jenkins-slave/jenkins-slave-remote.conf is also supplied by the jenkins-slave package, but its version is missing the start/stop/respawn portion.

I'll attempt to respond the comments for changes, but may have to iterate with plars on a few. Will make note if this is the case.

[Paul Larson (pwlars)]

A few responses below.

Francis, since this is your branch, I can't modify it. If it would be easier, I'm happy to have placeholders for some of these comments and merge it pretty much as is, then we can more easily collaborate on hitting the FIXMEs in parallel.

review: Approve

[Paul Larson (pwlars)]

Please pull lp:~pwlars/charms/trusty/jenkins-slave/phablet-slave-fixups rev 24 - this moves the install.d hook for jenkins slaves into the main install hook and recycles the existing template upstart files rather than adding our own copies as discussed.

[Paul Larson (pwlars)]

Also added lp:~pwlars/charms/trusty/jenkins-slave/phablet-slave-fixups rev 25-27 (see comments below for what they are related to). I'll also be adding a 28 in just a moment to fix something I realized is wrong with how I did the install conversion for rev 24. minor fix - will be in rev 28 of my branch.

[lp:~fginther/charms/trusty/jenkins-remote-slave/jenkins-phablet-slave]

24. By Francis Ginther on 2014-09-26

Update metadata.yaml as this charm is distinct from jenkins-slave.

25. By Francis Ginther on 2014-09-26

Convert 4000-lp-keys.sh to use 'install'.

26. By Francis Ginther on 2014-09-26

Drop the extra jenkins-slave-remote files and just modify those supplied by the jenkins-slave package. Add fixmes and task links for future updates to the wifi.conf setup and removing jenkins sudo accesss.

27. By Francis Ginther on 2014-09-26

Add subunit dependency to 3000-ci-tools.sh.

28. By Francis Ginther on 2014-09-26

Add FIXME comment for ignoring failures on stop, annotate other FIXME comments and explain the purpose of 0000-init.sh.

29. By Francis Ginther on 2014-09-27

Check and modify the correct sudoers file.

30. By Francis Ginther on 2014-09-27

Always install jenkins-slave from the in-charm deb package.

31. By Francis Ginther on 2014-10-02

Perform an apt-get update prior to install jenkins-slave packages.

[Paul Larson (pwlars)]

Seems to work ok here with one very minor fix. Please pull r30 from my branch

review: Approve

[lp:~fginther/charms/trusty/jenkins-remote-slave/jenkins-phablet-slave]

32. By Francis Ginther on 2014-10-02

Add jenkins to the utah group so that we can run utah tests without sudo and fix global replacement of 'jenkins-slave' in the upstart conf file.

=== modified file 'config.yaml'
--- config.yaml	2014-03-26 03:14:41 +0000
+++ config.yaml	2014-10-02 19:07:57 +0000
@@ -3,6 +3,14 @@
     type: string
     default: git gcc make bzr
     description: Tooling to deploy on jenkins slave node
+  master_url:
+    type: string
+    description: Jenkins master url
+    default: ""
+  slave_name:
+    type: string
+    description: Slave name on jenkins master to associate with node
+    default: ""
   labels:
     type: string
     description: Jenkins labels to associate with jenkins slave node

=== added file 'files/99-adb.rules'
--- files/99-adb.rules	1970-01-01 00:00:00 +0000
+++ files/99-adb.rules	2014-10-02 19:07:57 +0000
@@ -0,0 +1,1 @@
+SUBSYSTEM=="usb", ATTR{idVendor}=="*", MODE="0777"

=== added directory 'files/lp'
=== added file 'files/lp/authentication.conf'
--- files/lp/authentication.conf	1970-01-01 00:00:00 +0000
+++ files/lp/authentication.conf	2014-10-02 19:07:57 +0000
@@ -0,0 +1,4 @@
+[Launchpad]
+host = .launchpad.net
+scheme = ssh
+user = ps-jenkins

=== added file 'files/lp/bazaar.conf'
--- files/lp/bazaar.conf	1970-01-01 00:00:00 +0000
+++ files/lp/bazaar.conf	2014-10-02 19:07:57 +0000
@@ -0,0 +1,3 @@
+[DEFAULT]
+email = [e-mail]
+launchpad_username = ps-jenkins

=== added file 'files/lp/config'
--- files/lp/config	1970-01-01 00:00:00 +0000
+++ files/lp/config	2014-10-02 19:07:57 +0000
@@ -0,0 +1,2 @@
+Host bazaar.launchpad.net
+    IdentityFile /var/lib/jenkins/.ssh/%h/%r/id_rsa

=== modified file 'hooks/install'
--- hooks/install	2014-05-15 18:46:43 +0000
+++ hooks/install	2014-10-02 19:07:57 +0000
@@ -2,27 +2,38 @@
 
 set -eux
 
+slave_name=`config-get slave_name`
+master_url=`config-get master_url`
 
 # Install the slave if it is not installed already.
 install_slave () {
     juju-log "Installing jenkins-slave..."
-    if [[ ! -f /etc/init/jenkins-slave.conf ]]
+    if [[ ! -f /etc/init/jenkins-slave-remote-${slave_name}.conf ]]
     then
-        if [[ $(apt-cache madison jenkins-slave) =~ .*jenkins-slave.* ]]
-        then
-            apt-get -y install -qq jenkins-slave wget
-        else
-            # This series doesn't provide a jenkins.
-            # Install the same slave package as the precise Jenkins master.
-            apt-get -y install -qq wget adduser default-jre-headless upstart-job
-            dpkg -i files/jenkins-slave_*.deb
-        fi
+        # Install the same slave package as the precise Jenkins master.
+        # Always install from the in-charm deb so that the modifications
+        # are always applied to the same files.
+        apt-get -y install -qq wget adduser default-jre-headless upstart-job
+        dpkg -i files/jenkins-slave_*.deb
     else
         juju-log "Jenkins-slave is already installed"
     fi
 }
 
 
+modify_jenkins_settings () {
+        install /etc/init/jenkins-slave.conf /etc/init/jenkins-slave-remote-${slave_name}.conf
+        sed -i -e "/stop on/arespawn\nrespawn limit 10 5" \
+               -e "s!jenkins-slave!jenkins-slave-remote-${slave_name}!g" \
+                /etc/init/jenkins-slave-remote-${slave_name}.conf
+        install /etc/default/jenkins-slave /etc/default/jenkins-slave-remote-${slave_name}
+        sed -i -e "s!^JENKINS_HOSTNAME.*!JENKINS_HOSTNAME=${slave_name}!" \
+               -e "s!^#*JENKINS_URL.*!JENKINS_URL=${master_url}!" \
+               -e "s!jenkins-slave!jenkins-slave-remote-${slave_name}!" \
+               /etc/default/jenkins-slave-remote-${slave_name}
+}
+
+
 # Install extra packages needed by the slave.
 install_tools () {
     juju-log "Installing tools..."
@@ -48,8 +59,17 @@
     fi
 }
 
+# Always refresh the apt db before installing any packages
+# Use multiple attempts to avoid transient hash mismatch errors
+apt-get update || apt-get update || apt-get update
 
+# For config-changed, we want to ensure slave gets stopped and restarted
+# FIXME: 20140926 fginther - check if the upstart job is running first
+# https://app.asana.com/0/15652576021427/16759381549805
+hooks/stop || true
 install_slave
+modify_jenkins_settings
 install_tools
 install_extra_hooks
+hooks/start
 exit 0

=== added file 'hooks/install.d/0000-init.sh'
--- hooks/install.d/0000-init.sh	1970-01-01 00:00:00 +0000
+++ hooks/install.d/0000-init.sh	2014-10-02 19:07:57 +0000
@@ -0,0 +1,10 @@
+#!/bin/bash
+set -ex
+
+host_ip=`unit-get public-address`
+host_name=`hostname`
+
+# This resolves the problem of the unit not being able to resolve itself.
+if ! grep $host_name /etc/hosts; then
+    echo "$host_ip $host_name" >> /etc/hosts
+fi

=== added file 'hooks/install.d/3000-ci-tools.sh'
--- hooks/install.d/3000-ci-tools.sh	1970-01-01 00:00:00 +0000
+++ hooks/install.d/3000-ci-tools.sh	2014-10-02 19:07:57 +0000
@@ -0,0 +1,31 @@
+#!/bin/bash
+set -ex
+
+#Install tools needed for working with devices
+add-apt-repository ppa:phablet-team/tools < /dev/null
+add-apt-repository ppa:utah/stable < /dev/null
+
+apt-get update || apt-get update || apt-get update
+apt-get install -y --assume-yes phablet-tools ubuntu-device-flash utah \
+	autopkgtest bzr-builder subunit
+
+#Update the udev rules needed for adb devices
+cp files/99-adb.rules /etc/udev/rules.d
+udevadm control --reload-rules
+udevadm trigger
+
+#FIXME: 20140926 plars - find a better place for this
+# Existing tests assume /home/ubuntu/magners-wifi
+#https://app.asana.com/0/8312808705920/16730142900572
+cp files/secrets/wifi.conf /home/ubuntu/magners-wifi
+
+#Add jenkins to the utah group so that we can run utah tests without sudo
+addgroup jenkins utah
+
+#Setup sudo permissions for the jenkins user
+#FIXME: 20140926 plars - remove the need for this
+#https://app.asana.com/0/8312808705920/16730142900575
+if [ ! -f /etc/sudoers.d/jenkins ] ||
+	! grep -q jenkins /etc/sudoers.d/jenkins; then
+    echo "jenkins ALL=(ALL) NOPASSWD: ALL" > /etc/sudoers.d/jenkins
+fi

=== added file 'hooks/install.d/4000-lp-keys.sh'
--- hooks/install.d/4000-lp-keys.sh	1970-01-01 00:00:00 +0000
+++ hooks/install.d/4000-lp-keys.sh	2014-10-02 19:07:57 +0000
@@ -0,0 +1,19 @@
+#!/bin/bash
+set -ex
+
+install --owner=jenkins --group=jenkins -d \
+	/var/lib/jenkins/.ssh/bazaar.launchpad.net/ps-jenkins \
+	/var/lib/jenkins/.bazaar
+
+install --owner=jenkins --group=jenkins --mode=600 \
+	files/secrets/id_rsa \
+	/var/lib/jenkins/.ssh/bazaar.launchpad.net/ps-jenkins/
+install --owner=jenkins --group=jenkins \
+	files/lp/config files/secrets/known_hosts \
+	/var/lib/jenkins/.ssh/
+install --owner=jenkins --group=jenkins \
+	files/lp/authentication.conf files/lp/bazaar.conf \
+	/var/lib/jenkins/.bazaar
+install --owner=jenkins --group=jenkins \
+	files/secrets/.launchpad.credentials \
+	/var/lib/jenkins

=== modified file 'hooks/start'
--- hooks/start	2011-08-02 08:53:54 +0000
+++ hooks/start	2014-10-02 19:07:57 +0000
@@ -1,3 +1,5 @@
 #!/bin/bash
 
-start jenkins-slave || true
+slave_name=`config-get slave_name`
+
+start jenkins-slave-remote-${slave_name} || true

=== modified file 'hooks/stop'
--- hooks/stop	2011-08-02 08:53:54 +0000
+++ hooks/stop	2014-10-02 19:07:57 +0000
@@ -1,3 +1,5 @@
 #!/bin/bash
 
-stop jenkins-slave
+slave_name=`config-get slave_name`
+
+stop jenkins-slave-remote-${slave_name}

=== modified file 'metadata.yaml'
--- metadata.yaml	2014-05-15 16:22:53 +0000
+++ metadata.yaml	2014-10-02 19:07:57 +0000
@@ -1,14 +1,15 @@
-name: jenkins-slave
-maintainer: James Page <[e-mail]>
-summary: Jenkins CI Slave
+name: jenkins-remote-slave
+maintainer: Ubuntu Engineering CI Team <[e-mail]>
+summary: Jenkins CI Remote Slave
 description: |
   Jenkins is a Continous Integration server supporting
   flexible continous integration and deployment methodologies
   and more.
   .
-  This charm provides support for jenkins slaves
+  This charm provides support for remote jenkins slaves connected to an
+  existing jenkins master.
 categories:
   - applications
 provides:
   slave:
-    interface: jenkins-slave
+    interface: jenkins-remote-slave