Merge ~bbaude/cloud-init:azure_bounce into cloud-init:master

[Brent Baude]

Bounce network interface for Azure

When deploying on Azure and using only cloud-init, you must "bounce"
the network interface to trigger a DDNS update. This allows dhclient
to register the hostname with Azure so that DNS works correctly on their
private networks (i.e. between vm and vm).

The agent path was already doing the bounce so this creates parity between the built-in path and the agent.

This PR adds the ability to set the hostname and bounce the network interfaces when provisioning in Azure with just cloud-init. The hostname must be set and network bounced in order for DNS to work correctly in Azure when communicating between azure vms.

[Server Team CI bot (server-team-bot)]

FAILED: Continuous integration, rev:3392d841335cad4615381f274071729a9866e9a7
https://jenkins.ubuntu.com/server/job/cloud-init-ci/142/
Executed test runs:
    FAILURE: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-amd64/142/console
    FAILURE: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-arm64/142/console
    FAILURE: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-ppc64el/142/console
    FAILURE: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-s390x/142/console
    FAILURE: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=vm-i386/142/console

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/142/rebuild

review: Needs Fixing (continuous-integration)

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:4e9065e46fa8bdad3e32ea00f3ede709bbd201d5
https://jenkins.ubuntu.com/server/job/cloud-init-ci/146/
Executed test runs:
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-amd64/146
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-arm64/146
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-ppc64el/146
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-s390x/146
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=vm-i386/146

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/146/rebuild

review: Approve (continuous-integration)

[Joshua Powers (powersj)]

After adding commit 9040e78, this ran integration tests successfully:
https://paste.ubuntu.com/24217659/

[Dan Watkins (daniel-thewatkins)]

One step closer to killing the agent, I like it!

review: Approve

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:34b9adebdb26c1b3e68da5a3ad484d513c4e004d
https://jenkins.ubuntu.com/server/job/cloud-init-ci/147/
Executed test runs:
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-amd64/147
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-arm64/147
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-ppc64el/147
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-s390x/147
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=vm-i386/147

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/147/rebuild

review: Approve (continuous-integration)

diff --git a/cloudinit/sources/DataSourceAzure.py b/cloudinit/sources/DataSourceAzure.py
index c5af8b8..48a3e1d 100644
--- a/cloudinit/sources/DataSourceAzure.py
+++ b/cloudinit/sources/DataSourceAzure.py
@@ -111,50 +111,62 @@ class DataSourceAzureNet(sources.DataSource):
         root = sources.DataSource.__str__(self)
         return "%s [seed=%s]" % (root, self.seed)
 
-    def get_metadata_from_agent(self):
-        temp_hostname = self.metadata.get('local-hostname')
+    def bounce_network_with_azure_hostname(self):
+        # When using cloud-init to provision, we have to set the hostname from
+        # the metadata and "bounce" the network to force DDNS to update via
+        # dhclient
+        azure_hostname = self.metadata.get('local-hostname')
+        LOG.debug("Hostname in metadata is {}".format(azure_hostname))
         hostname_command = self.ds_cfg['hostname_bounce']['hostname_command']
-        agent_cmd = self.ds_cfg['agent_command']
-        LOG.debug("Getting metadata via agent.  hostname=%s cmd=%s",
-                  temp_hostname, agent_cmd)
-        with temporary_hostname(temp_hostname, self.ds_cfg,
+
+        with temporary_hostname(azure_hostname, self.ds_cfg,
                                 hostname_command=hostname_command) \
                 as previous_hostname:
             if (previous_hostname is not None and
-               util.is_true(self.ds_cfg.get('set_hostname'))):
+                    util.is_true(self.ds_cfg.get('set_hostname'))):
                 cfg = self.ds_cfg['hostname_bounce']
+
+                # "Bouncing" the network
                 try:
-                    perform_hostname_bounce(hostname=temp_hostname,
+                    perform_hostname_bounce(hostname=azure_hostname,
                                             cfg=cfg,
                                             prev_hostname=previous_hostname)
                 except Exception as e:
                     LOG.warn("Failed publishing hostname: %s", e)
                     util.logexc(LOG, "handling set_hostname failed")
 
-            try:
-                invoke_agent(agent_cmd)
-            except util.ProcessExecutionError:
-                # claim the datasource even if the command failed
-                util.logexc(LOG, "agent command '%s' failed.",
-                            self.ds_cfg['agent_command'])
-
-            ddir = self.ds_cfg['data_dir']
-
-            fp_files = []
-            key_value = None
-            for pk in self.cfg.get('_pubkeys', []):
-                if pk.get('value', None):
-                    key_value = pk['value']
-                    LOG.debug("ssh authentication: using value from fabric")
-                else:
-                    bname = str(pk['fingerprint'] + ".crt")
-                    fp_files += [os.path.join(ddir, bname)]
-                    LOG.debug("ssh authentication: "
-                              "using fingerprint from fabirc")
-
-            missing = util.log_time(logfunc=LOG.debug, msg="waiting for files",
-                                    func=wait_for_files,
-                                    args=(fp_files,))
+    def get_metadata_from_agent(self):
+        temp_hostname = self.metadata.get('local-hostname')
+        agent_cmd = self.ds_cfg['agent_command']
+        LOG.debug("Getting metadata via agent.  hostname=%s cmd=%s",
+                  temp_hostname, agent_cmd)
+
+        self.bounce_network_with_azure_hostname()
+
+        try:
+            invoke_agent(agent_cmd)
+        except util.ProcessExecutionError:
+            # claim the datasource even if the command failed
+            util.logexc(LOG, "agent command '%s' failed.",
+                        self.ds_cfg['agent_command'])
+
+        ddir = self.ds_cfg['data_dir']
+
+        fp_files = []
+        key_value = None
+        for pk in self.cfg.get('_pubkeys', []):
+            if pk.get('value', None):
+                key_value = pk['value']
+                LOG.debug("ssh authentication: using value from fabric")
+            else:
+                bname = str(pk['fingerprint'] + ".crt")
+                fp_files += [os.path.join(ddir, bname)]
+                LOG.debug("ssh authentication: "
+                          "using fingerprint from fabirc")
+
+        missing = util.log_time(logfunc=LOG.debug, msg="waiting for files",
+                                func=wait_for_files,
+                                args=(fp_files,))
         if len(missing):
             LOG.warn("Did not find files, but going on: %s", missing)
 
@@ -220,6 +232,8 @@ class DataSourceAzureNet(sources.DataSource):
         write_files(ddir, files, dirmode=0o700)
 
         if self.ds_cfg['agent_command'] == AGENT_START_BUILTIN:
+            self.bounce_network_with_azure_hostname()
+
             metadata_func = partial(get_metadata_from_fabric,
                                     fallback_lease_file=self.
                                     dhclient_lease_file)
diff --git a/cloudinit/sources/__init__.py b/cloudinit/sources/__init__.py
index 3d01072..1829450 100644
--- a/cloudinit/sources/__init__.py
+++ b/cloudinit/sources/__init__.py
@@ -210,7 +210,7 @@ class DataSource(object):
         else:
             hostname = toks[0]
 
-        if fqdn:
+        if fqdn and domain != defdomain:
             return "%s.%s" % (hostname, domain)
         else:
             return hostname