Merge lp:~josvaz/vmbuilder/jenkins_kvm+fix-eth1boot into lp:~ubuntu-on-ec2/vmbuilder/jenkins_kvm

[Jose L. VG]

Remove the ifup call that was breaking the boot

ifup should be unnecesary with allow-hotplug, but in this case it was making the first boot hung, even if invoked in the background.

I also made some improvements to the udev rule to avoid it being invoked on eth0 and use RUN instead of PROGRAM.

[Jose L. VG (josvaz)]

Commit 774 re-applies the broken eth1+ autoconfig patch.
Commit 775 is the actual fix.
Commit 776 is a enhancement.

[Jose L. VG (josvaz)]

Although it looks good so far, this is still under testing.
Feel free to review and even Approve, but code won't be merged until all testing is completed succesfully.

[Philip Roche (philroche)]

Syntax looks good to me.

review: Approve

[Jose L. VG (josvaz)]

Will need more work, testing reveals that trusty fails to boot now.
This is a surprise, as the eth1+ patch tries to apply most changes to xenial, onward, leaving previous releases like trusty almost untouched, apart from using /run... for the ephemeral net devices config instead of /etc/network/interfaces/dynamic.d

[Jose L. VG (josvaz)]

upstart code path fixed with commit 777, please review

[Jose L. VG (josvaz)]

This MP has passed manual multi-nic tests for the combinations:
{Standard_DS2_v2,A8} x {precise, trusty, xenial, yakkety, zesty}

No issues or regressions found.

[Robert C Jennings (rcj)]

This looks good. Glad to hear that testing covered the two instance types across the full set of supported suites.

review: Approve

[Jose L. VG (josvaz)]

I will be merging this once I pass all AzureTests against all suites using test images

[Jose L. VG (josvaz)]

xenial tests:

https://cloud-images-jenkins.canonical.com/view/Azure/job/Azure-Tests/787/

Ubuntu_DAILY_BUILD-xenial-16_04-LTS-amd64-server-20161104-ethx-1478604131-en-us-30GB

[Jose L. VG (josvaz)]

The xenial test image passed AzureTests:
https://cloud-images-jenkins.canonical.com/view/Azure/job/Azure-Tests/787

[Jose L. VG (josvaz)]

Trusty tests:
Ubuntu_DAILY_BUILD-trusty-14_04_5-LTS-amd64-server-20161104-ethx-1478613062-en-us-30GB
https://cloud-images-jenkins.canonical.com/view/Azure/job/Azure-Tests/788/

[Jose L. VG (josvaz)]

Trusted passed

(Passed = unstable ONLY in lxd tests, success everywhere else)

[Jose L. VG (josvaz)]

Just discovered the precise test image is NOT properly built:
- It has the udev changes IN, instead of the upstart ones.

I checked trusty (that just passed the tests) and is ALSO using the udev change sinstead of the upstart.

I think something did not work as expected when bulding the Azure tests images. I am guessing I might have missed a build parameter setting. Precise & trusty were built as if SUITE was set to xenial or yakkety.

I can let yakkety pass the tests (as the image will be buolt correctly for it) and then repeat trusty test and add precise tests ONCE I get test images properly built for them.

[Jose L. VG (josvaz)]

Just a clarification, so far NOTHING is found wrong with the MP, the test images might have been misbuilt due to improper inputs. When I did test the MP manually trusty & precise where setup with upstart and no udev, as expected.

[Jose L. VG (josvaz)]

Yakkety tests:
Ubuntu_DAILY_BUILD-yakkety-16_10-amd64-server-20161105-ethx-1478613973-en-us-30GB
https://cloud-images-jenkins.canonical.com/view/Azure/job/Azure-Tests/789/

[Jose L. VG (josvaz)]

Need to setup
https://cloud-images-jenkins.canonical.com/view/Azure/job/CloudImages_Azure-Custom_or_Test_Images/configure
So that the KVM scipt ALSO gets passed the proper SUITE variable.

Right now some scripts infer SUITE from VERSION. But the value is NOT job wide scope, so if it is not detected in the script, it is NOT set.

[Jose L. VG (josvaz)]

Yakkety tests failed in a couple of LXD test cases:
https://cloud-images-jenkins.canonical.com/view/Azure/job/Azure-Tests/789/

Rerunning those as I think they are transients.

[Jose L. VG (josvaz)]

Yakkety tests passed, the issues were transient as suspected:
https://cloud-images-jenkins.canonical.com/view/Azure/job/Azure-Tests/789/
https://cloud-images-jenkins.canonical.com/view/Azure/job/Azure-Tests/790/

Pending are the precise, trusty & zesty tests, but for that I need first to fix this job:
https://cloud-images-jenkins.canonical.com/view/Azure/job/CloudImages_Azure-Custom_or_Test_Images/configure
To make sure the test images are generated with a proper SUITE setting.

[Jose L. VG (josvaz)]

Job fixed to compute SUITE from VERSION and export the value to a properties file that is later injected to all subsequent jobs:
https://cloud-images-jenkins.canonical.com/view/Azure/job/CloudImages_Azure-Custom_or_Test_Images/configure

Re-running trusty & precise test image generation:

trusty
https://cloud-images-jenkins.canonical.com/view/Azure/job/CloudImages_Azure-Custom_or_Test_Images/304

precise
https://cloud-images-jenkins.canonical.com/view/Azure/job/CloudImages_Azure-Custom_or_Test_Images/305

[Jose L. VG (josvaz)]

For some reason, the KVM builds of precise & trusty is STILL not getting my SUITE setting right.

[Jose L. VG (josvaz)]

Turns out that inside KVM the SUITE var used is in lowercase "suite"!

So instead of fixing the lack of consistency I should probably just fix this MP instead and use ${suite} instead of ${SUITE}

[Jose L. VG (josvaz)]

Suite variable used now by this MP is lowercase, so lets try it...

[Jose L. VG (josvaz)]

precise test image created correctly now:

Ubuntu_DAILY_BUILD-precise-12_04_5-LTS-amd64-server-20161109-ethx-1479204698-en-us-30GB

https://cloud-images-jenkins.canonical.com/view/Azure/job/CloudImages_Azure-Custom_or_Test_Images/307/console

[Jose L. VG (josvaz)]

proper precise test image in test here:
https://cloud-images-jenkins.canonical.com/view/Azure/job/Azure-Tests/791/

[Jose L. VG (josvaz)]

trusty test image createc correctly:
Ubuntu_DAILY_BUILD-trusty-14_04_5-LTS-amd64-server-20161111-ethx-1479205498-en-us-30GB

https://cloud-images-jenkins.canonical.com/view/Azure/job/CloudImages_Azure-Custom_or_Test_Images/308/console

[Jose L. VG (josvaz)]

proper trusty test image in test here:
https://cloud-images-jenkins.canonical.com/view/Azure/job/Azure-Tests/792/

[Jose L. VG (josvaz)]

trusty test image passed AzureTests as expected (only unstable at lxd tests):
https://cloud-images-jenkins.canonical.com/job/Azure-Tests/792/

precise test experienced a single failure, a transient:
https://cloud-images-jenkins.canonical.com/view/Azure/job/Azure-Tests/791/

Rerun of the failed precise test passed at:
https://cloud-images-jenkins.canonical.com/view/Azure/job/Azure-Tests/793/

[Jose L. VG (josvaz)]

This MP right now passes all AzureTests for all supported suites.

Zesty is missing but I fear I will continue to have issues creating a test image for it.

Please review and approve, also let me know if we should or not wait for zesty tests to marge this MP.

[Jose L. VG (josvaz)]

Might have fixed the zesty test image building for Azure, testing it here:
https://cloud-images-jenkins.canonical.com/view/Azure/job/CloudImages_Azure-Custom_or_Test_Images/311/console

[Jose L. VG (josvaz)]

zesty test image produced:
https://cloud-images-jenkins.canonical.com/view/Azure/job/CloudImages_Azure-Custom_or_Test_Images/311/console

Ubuntu_DAILY_BUILD-zesty-17_04-amd64-server-20161111-ethx-1479217570-en-us-30GB

Now I need to validate is probably built with this MP changes applied to the image.

[Jose L. VG (josvaz)]

Zesty image is good, it has the MP changes as expected.

Testing now at:
https://cloud-images-jenkins.canonical.com/view/Azure/job/Azure-Tests/794/

[Jose L. VG (josvaz)]

Zesty passed:
https://cloud-images-jenkins.canonical.com/view/Azure/job/Azure-Tests/794/

As zesty can pass right now in Azure, which is azure-general fails due to a known issue, which bug I can't find at the moment.

[Jose L. VG (josvaz)]

Please, review, all test passed this is ready to go!

[Jose L. VG (josvaz)]

BTW, the zesty azure-general issue is:
https://bugs.launchpad.net/cloud-images/+bug/1638964

[Philip Roche (philroche)]

LGTM

review: Approve

[Robert C Jennings (rcj)]

+1 on the SuItE change.

[Robert C Jennings (rcj)]

review: Approve

[lp:~josvaz/vmbuilder/jenkins_kvm+fix-eth1boot]

775. By Jose L. VG on 2016-11-16

Re-apply eth1+ changes to be fixed

776. By Jose L. VG on 2016-11-16

Remove the ifup call that was breaking the boot

ifup should be unnecesary with allow-hotplug, but in this case it was
making the first boot hung, even if invoked in the background.

777. By Jose L. VG on 2016-11-16

Enhance udev rule: skip eth0 and do RUN

The udev expressions are not full regex, eth[1-9]* just means:
'eth' followed by a number 1-9 and 'zero or more chars'
That will NOT match eth0 (or eth0something) but will match eth1 to eth99
(and will also match things like eth2xd)

Also looking at udev man RUN is more appropriate as it is an action,
while PROGRAM is still part of the matching filters.

778. By Jose L. VG on 2016-11-16

Place mkdir at the proper time for upstart

The mkdir was was misplaced for upstart code path, as the new directory
we use under /run is already mounted by the OS we don't need to create it
at image preparation time, BUT we do need it to be there for autoconfig
eth1+ to work.

779. By Jose L. VG on 2016-11-16

Fix ${SUITE} to ${suite}

Turns out that KVM templates in general (and Azure ones specifically) use the
variable name $suite} lowercase instead of the more common uppercase ${SUITE}
in jenkins and jerff env variables.

[Jose L. VG (josvaz)]

This has been "git pull --rebase"d so that now they can be mergen cleanly

[Jose L. VG (josvaz)]

merged cleanly I meant

=== modified file 'templates/img-extra-nets.tmpl'
--- templates/img-extra-nets.tmpl	2016-11-02 12:46:14 +0000
+++ templates/img-extra-nets.tmpl	2016-11-16 17:32:59 +0000
@@ -1,29 +1,16 @@
 ######################
 ## BEGIN NICS
 ## This adds upstart jobs for the creation of hot-plug NICS
+## For xenial onward, udev is configured instead
 # vi: ts=4 noexpandtab syntax=sh
 
-int_d="/etc/network/interfaces.dynamic.d"
-xchroot mkdir -p ${int_d}
-
-## Place readme to notify people what is going on
-cat << EOF > ${mp}${int_d}/README
-WARNING: This is mounted over during boot!
-
-If you are seeing this file, it means that you have disabled the tmpfs mount
-   in /etc/fstab for ${int_d}. Therefore, this directory
-   is ephemeral.
-
-This directory is normally populated at boot for any interface that is not
-    eth0 with a dynamic configuration.
-
-${CLOUD_IMG_STR}
-EOF
-
-# Give an upstart job for defining a secondary interface
-# if needed.
-cat << EOF > ${mp}/etc/init/network-conf-ethX.conf
-# Enable DHCP configuration for additional ethX devices
+int_d="/run/network/interfaces.ephemeral.d"
+
+function config_upstart {
+    # Give an upstart job for defining a secondary interface
+    # if needed.
+    cat << EOF > ${mp}/etc/init/network-conf-ethX.conf
+# Enable DHCP configuration for eth1+ devices as they appear
 ${CLOUD_IMG_STR}
 
 description "Dynamically populate ${int_d}"
@@ -39,7 +26,10 @@
     case \$INTERFACE in
         eth0) #skip, this is static
               ;;
-        eth*) cat << EOM > \$net_int_d/\$INTERFACE.cfg
+        eth*)
+            # Ensure \${net_int_d} exists on the ephemeral mount
+            mkdir -p \${net_int_d}
+            cat << EOM > \$net_int_d/\$INTERFACE.cfg
 allow-hotplug \$INTERFACE
 iface \$INTERFACE inet dhcp
 EOM
@@ -50,15 +40,62 @@
     esac
 end script
 EOF
-
-# Add an ephemeral mount for holding additional NIC interfaces
-cat << EOF >> ${mp}/etc/fstab
-
+}
+
+function config_udev {
+    # Give a udev script to define eth1+ interfaces, as they appear
+    cat << EOF > ${mp}/usr/local/sbin/ephemeral_eth.sh
+#!/bin/bash
 ${CLOUD_IMG_STR}
-# The following is used to dynamically configured additional
-# NICs. Do not remove unless you know what you are doing.
-none ${int_d} tmpfs   nodev,noexec,nosuid,size=64K 0 0
-EOF
+# Enable DHCP configuration for ethx devices other than eth0
+# Called by udev to define eth1+ interfaces, as they appear
+
+INTERFACE=\$1
+net_int_d="${int_d}"
+
+function configure_nic {
+    # Ensure \${net_int_d} exists on the ephemeral mount
+    mkdir -p \${net_int_d}
+
+    # Setup the interface config file
+    cat << EOM > \${net_int_d}/\${INTERFACE}.cfg
+# Added by \$0
+allow-hotplug \${INTERFACE}
+iface \${INTERFACE} inet dhcp
+EOM
+}
+
+case \${INTERFACE} in
+    eth0) #skip, this is static
+        ;;
+    eth*)
+        if [ ! -e \${net_int_d}/\${INTERFACE}.cfg ]
+        then
+            configure_nic
+        fi
+        ;;
+    *)
+        ;;
+esac
+EOF
+    # Make the script executable
+    xchroot chmod +x /usr/local/sbin/ephemeral_eth.sh 
+
+    # Place an udev rule to call the script when an net device is added
+    cat << EOF > ${mp}/etc/udev/rules.d/10-net-device-added.rules
+# Run /usr/local/sbin/ephemeral_eth.sh every time a network device is added
+ACTION=="add", SUBSYSTEM=="net", KERNEL=="eth[1-9]*", RUN="/usr/local/sbin/ephemeral_eth.sh %k"
+EOF
+}
+
+case ${suite} in
+    precise|trusty)
+        config_upstart
+        ;;
+    *)
+        config_udev
+        ;;
+esac
 
 # Add the network ephemeral mount...
 cat << EOF >> ${mp}/etc/network/interfaces