Merge ~michihenning/jenkaas-jobs:install-extra-packages-during-build into jenkaas-jobs:master

[Michi Henning]

Make sure that install_packages is used during first build step, in case a dependency is added to a project config after the chroots were created.

While adding unity-scopes-api, I found that we need click-dev, which wasn't installed on the slaves.

Just adding click-dev to install_packages doesn't help, because that is used when the chroots are built. However, if we add a new project to our config and find that something extra needs to be installed, we can't fix it without modifying jenkaas-jobs because install_packages is expanded into prepare-0-install.sh, and we can't modify that from our config without also incrementing the job_version in prepare-0-install.yaml.

The easiest solution seems to be to also look at install_packages during build-0-fetch and call apt install. If the extra packages are installed already, it finishes very quickly and does not harm. If they are missing, they are added before the build kicks off.

I suspect that we could also remove install_packages from prepare-0-install altogether (but I didn't do that). install_packages has an effect only if a dependency for a project is known *before* the chroots are created, but that's generally not the case.

[Michał Sawicz (saviq)]

Thing is, that means running a NOOP `apt-get install` for 99% of the runs. IIUC this is needed in `debian/rules` clean?

I want to go in the direction bileto went and not run clean at all during source package builds. Would that work for you?

review: Needs Information

[Michi Henning (michihenning)]

Are you talking about something like the billet_pre_release_hook? I guess that would work. So, the plan would be for the source package build to invoke some script that we can provide as part of the project config (outside jenkaas-jobs)?

[Michał Sawicz (saviq)]

Yes, I would actually like to use that same script - no need for anything else?

[Michi Henning (michihenning)]

At the moment, the bileto hook for scopes-api doesn't install anything. I guess we add the apt-get install command for click-dev there though.
So, yes, I guess that'll work, thanks!

[Michi Henning (michihenning)]

Ah, no change needed in the script. Once the control file is generated, we should be good.

[Michał Sawicz (saviq)]

OK I'll disapprove this then.

review: Disapprove

[Michi Henning (michihenning)]

Sorry, should have done that myself, thanks!

Unmerged commits

2fded35... by Michi Henning on 2016-05-12

Moved apt command to more logical place.

79d8e4b... by Michi Henning on 2016-05-10

Install extra packages during build, otherwise they won't be there if the main project file is updated after the chroots were created.

c109369... by Michi Henning on 2016-05-10

Removed project-specific launchpad node label matching. The trigger jobs now run on any node with a "launchpad" label.

diff --git a/.gitignore b/.gitignore
index 1408662..96dd851 100644
--- a/.gitignore
+++ b/.gitignore
@@ -23,3 +23,6 @@ var/
 
 .project
 .pydevproject
+
+# Temporary files
+*.swp
diff --git a/jenkaas_jobs/yaml/build/build-0-fetch.sh b/jenkaas_jobs/yaml/build/build-0-fetch.sh
index ac989cc..768b8e1 100644
--- a/jenkaas_jobs/yaml/build/build-0-fetch.sh
+++ b/jenkaas_jobs/yaml/build/build-0-fetch.sh
@@ -3,6 +3,11 @@
 # log all commands and abort on error
 set -xe
 
+# make sure that packages that were newly added to the main project file are installed
+if [ -n "${extra_packages}" ]; then
+    sudo apt-get install --yes ${extra_packages}
+fi
+
 # sanitize input
 bzr revno ${source_branch} || (echo "ERROR: source branch incorrect"; exit 1)
 [ -z "${source_revision}" ] || bzr revno -r ${source_revision} ${source_branch} || (echo "ERROR: source revision incorrect"; exit 2)
@@ -40,4 +45,4 @@ dch -v ${VERSION} "Automatic build of ${source_branch} at revision ${source_revi
 [ -n "${target_branch}" ] && dch "Based on ${target_branch}"
 
 # build source package
-bzr builddeb --builder="debuild -uc -us" --source --result-dir=${OUTDIR}
 \ No newline at end of file
+bzr builddeb --builder="debuild -uc -us" --source --result-dir=${OUTDIR}
diff --git a/jenkaas_jobs/yaml/build/build-0-fetch.yaml b/jenkaas_jobs/yaml/build/build-0-fetch.yaml
index 15e338e..b5a5ef2 100644
--- a/jenkaas_jobs/yaml/build/build-0-fetch.yaml
+++ b/jenkaas_jobs/yaml/build/build-0-fetch.yaml
@@ -9,6 +9,10 @@
       artifactNumToKeep: '{artifact_keep}'
     parameters:
     - branch_parameters
+    - string:
+        name: extra_packages
+        default: '{install_packages}'
+        description: A space-separated list of additional packages to install on slaves prior to starting a build.
     properties:
     - rebuild
     wrappers:
@@ -27,4 +31,4 @@
         only-if-success: false
         fingerprint: false
     - workspace-cleanup:
-        fail-build: true
 \ No newline at end of file
+        fail-build: true
diff --git a/jenkaas_jobs/yaml/launchpad/lp-trigger-autolanding.yaml b/jenkaas_jobs/yaml/launchpad/lp-trigger-autolanding.yaml
index d1f8b97..2376dd9 100644
--- a/jenkaas_jobs/yaml/launchpad/lp-trigger-autolanding.yaml
+++ b/jenkaas_jobs/yaml/launchpad/lp-trigger-autolanding.yaml
@@ -2,7 +2,7 @@
     name: lp-{project}-trigger-autolanding
     project-type: freestyle
     description: 'Auto-land top-approved changes for {branch}'
-    node: launchpad-{project}
+    node: launchpad
     logrotate:
       numToKeep: 10
     triggers:
diff --git a/jenkaas_jobs/yaml/launchpad/lp-trigger.yaml b/jenkaas_jobs/yaml/launchpad/lp-trigger.yaml
index 92cde53..3faacb4 100644
--- a/jenkaas_jobs/yaml/launchpad/lp-trigger.yaml
+++ b/jenkaas_jobs/yaml/launchpad/lp-trigger.yaml
@@ -2,7 +2,7 @@
     name: lp-{project}-trigger
     project-type: freestyle
     description: 'Monitor Launchpad for changes in {branch}'
-    node: launchpad-{project}
+    node: launchpad
     logrotate:
       numToKeep: 10
     triggers:
diff --git a/tests/fixtures/launchpad/lp-one-trigger.xml b/tests/fixtures/launchpad/lp-one-trigger.xml
index f6698c8..977982c 100644
--- a/tests/fixtures/launchpad/lp-one-trigger.xml
+++ b/tests/fixtures/launchpad/lp-one-trigger.xml
@@ -16,7 +16,7 @@
     </com.sonyericsson.rebuild.RebuildSettings>
   </properties>
   <scm class="hudson.scm.NullSCM"/>
-  <assignedNode>launchpad-one</assignedNode>
+  <assignedNode>launchpad</assignedNode>
   <canRoam>false</canRoam>
   <disabled>false</disabled>
   <blockBuildWhenDownstreamBuilding>false</blockBuildWhenDownstreamBuilding>
diff --git a/tests/fixtures/launchpad/lp-three-trigger-autolanding.xml b/tests/fixtures/launchpad/lp-three-trigger-autolanding.xml
index 1694b1e..f34e0ac 100644
--- a/tests/fixtures/launchpad/lp-three-trigger-autolanding.xml
+++ b/tests/fixtures/launchpad/lp-three-trigger-autolanding.xml
@@ -6,7 +6,7 @@
   <blockBuildWhenDownstreamBuilding>false</blockBuildWhenDownstreamBuilding>
   <blockBuildWhenUpstreamBuilding>false</blockBuildWhenUpstreamBuilding>
   <concurrentBuild>false</concurrentBuild>
-  <assignedNode>launchpad-three</assignedNode>
+  <assignedNode>launchpad</assignedNode>
   <canRoam>false</canRoam>
   <logRotator>
     <daysToKeep>-1</daysToKeep>
diff --git a/tests/fixtures/launchpad/lp-three-trigger.xml b/tests/fixtures/launchpad/lp-three-trigger.xml
index 76a17f2..62ea6e7 100644
--- a/tests/fixtures/launchpad/lp-three-trigger.xml
+++ b/tests/fixtures/launchpad/lp-three-trigger.xml
@@ -16,7 +16,7 @@
     </com.sonyericsson.rebuild.RebuildSettings>
   </properties>
   <scm class="hudson.scm.NullSCM"/>
-  <assignedNode>launchpad-three</assignedNode>
+  <assignedNode>launchpad</assignedNode>
   <canRoam>false</canRoam>
   <disabled>false</disabled>
   <blockBuildWhenDownstreamBuilding>false</blockBuildWhenDownstreamBuilding>
diff --git a/tests/fixtures/launchpad/lp-two-trigger-autolanding.xml b/tests/fixtures/launchpad/lp-two-trigger-autolanding.xml
index 6af75ad..6192baf 100644
--- a/tests/fixtures/launchpad/lp-two-trigger-autolanding.xml
+++ b/tests/fixtures/launchpad/lp-two-trigger-autolanding.xml
@@ -6,7 +6,7 @@
   <blockBuildWhenDownstreamBuilding>false</blockBuildWhenDownstreamBuilding>
   <blockBuildWhenUpstreamBuilding>false</blockBuildWhenUpstreamBuilding>
   <concurrentBuild>false</concurrentBuild>
-  <assignedNode>launchpad-two</assignedNode>
+  <assignedNode>launchpad</assignedNode>
   <canRoam>false</canRoam>
   <logRotator>
     <daysToKeep>-1</daysToKeep>
diff --git a/tests/fixtures/launchpad/lp-two-trigger.xml b/tests/fixtures/launchpad/lp-two-trigger.xml
index 95407c0..092ae55 100644
--- a/tests/fixtures/launchpad/lp-two-trigger.xml
+++ b/tests/fixtures/launchpad/lp-two-trigger.xml
@@ -16,7 +16,7 @@
     </com.sonyericsson.rebuild.RebuildSettings>
   </properties>
   <scm class="hudson.scm.NullSCM"/>
-  <assignedNode>launchpad-two</assignedNode>
+  <assignedNode>launchpad</assignedNode>
   <canRoam>false</canRoam>
   <disabled>false</disabled>
   <blockBuildWhenDownstreamBuilding>false</blockBuildWhenDownstreamBuilding>