Merge ~paul-meyer/cloud-init:lp1717611 into cloud-init:master

[Paul Meyer]

Azure: wait longer for SSH pub keys to arrive.

Currently the Azure data source waits up to 60 seconds. This has proven
not to be sufficient to provide resiliency to unrelated transient failures
in other parts of the infrastructure. Azure already has logic outside of
the VM to abort hung provisioning. This changes lengthens the time out to
15 minutes.

LP: #1717611

Azure: wait longer for SSH pub keys to arrive.
Currently the Azure data source waits up to 60 seconds. This has proven not to be sufficient to provide resiliency to unrelated transient failures in other parts of the infrastructure. Azure already has logic outside of the VM to abort hung provisioning. This changes lengthens the time out to 15 minutes.

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:63c7c48543ae779590668f8ccc9f571fd039bed1
https://jenkins.ubuntu.com/server/job/cloud-init-ci/323/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Unit & Style Tests
    SUCCESS: Ubuntu LTS: Build
    SUCCESS: Ubuntu LTS: Integration
    SUCCESS: MAAS Compatability Testing
    IN_PROGRESS: Declarative: Post Actions

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/323/rebuild

review: Approve (continuous-integration)

[Scott Moser (smoser)]

It still seems like it makes sense to have a timeout of some sort.
cloud-init is blocking boot at this point, and you may well have another way into your instance if cloud-init would let go.

Can't we just set this to 600 seconds or something?

[Paul Meyer (paul-meyer)]

Discussing internally with the provisioning folks...

[Ryan Harper (raharper)]

Is it possible to set this value via vendor-data? Surely then it may be tuned by the cloud without a change to cloud-init (even if we decide on 600 or 900 or whatever else) as a default value.

[Paul Meyer (paul-meyer)]

From historical data, we've seen infrastructure updates (that can take the wireserver temporarily offline) take 5+ minutes. How about 15 minutes to be on the safe side?
BTW, we're aware that this is not the best way to transport SSH keys, so we're working on that separately.

[Scott Moser (smoser)]

this is down a Azure only path.
We could probably make it configurable in some mechanism through vendor-data as ryan suggested, but... if the platform tells us they need it to be this high, then i'm willing to say "they know best".

review: Approve

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:edc8fba4deff06de8cb7127ae555e67a819197d0
https://jenkins.ubuntu.com/server/job/cloud-init-ci/325/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Unit & Style Tests
    SUCCESS: Ubuntu LTS: Build
    SUCCESS: Ubuntu LTS: Integration
    SUCCESS: MAAS Compatability Testing
    IN_PROGRESS: Declarative: Post Actions

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/325/rebuild

review: Approve (continuous-integration)

[Chad Smith (chad.smith)]

Sure this works for me. We already log how long we've decided to maxwait. I see no problem with dropping the maxwait default on wait_for_files since there are only two callers, and they are in the Azure datasource.

review: Approve

diff --git a/cloudinit/sources/DataSourceAzure.py b/cloudinit/sources/DataSourceAzure.py
index b5a95a1..80c2bd1 100644
--- a/cloudinit/sources/DataSourceAzure.py
+++ b/cloudinit/sources/DataSourceAzure.py
@@ -317,9 +317,13 @@ class DataSourceAzure(sources.DataSource):
                 LOG.debug("ssh authentication: "
                           "using fingerprint from fabirc")
 
-        missing = util.log_time(logfunc=LOG.debug, msg="waiting for files",
+        # wait very long for public SSH keys to arrive
+        # https://bugs.launchpad.net/cloud-init/+bug/1717611
+        missing = util.log_time(logfunc=LOG.debug,
+                                msg="waiting for SSH public key files",
                                 func=wait_for_files,
-                                args=(fp_files,))
+                                args=(fp_files, 900))
+
         if len(missing):
             LOG.warning("Did not find files, but going on: %s", missing)
 
@@ -656,7 +660,7 @@ def pubkeys_from_crt_files(flist):
     return pubkeys
 
 
-def wait_for_files(flist, maxwait=60, naplen=.5, log_pre=""):
+def wait_for_files(flist, maxwait, naplen=.5, log_pre=""):
     need = set(flist)
     waited = 0
     while True: