Merge lp:~hloeung/charm-helpers/status-set-retry-a-few-times into lp:charm-helpers

[Haw Loeung]

We're constantly seeing update-status hook failures where a 'juju resolved --retry' would fix. The latest example:

| ksplice/2 error idle 1.25.8 10.25.9.229 hook failed: "update-status"

Juju logs shows the following:

| 2017-02-28 06:20:49 INFO juju-log Application Version: 1.2.31
| 2017-02-28 06:20:49 INFO update-status error: connection is shut down
| 2017-02-28 06:20:49 INFO juju-log status-set failed: active Effective kernel 4.4.0-64-generic

This makes status_set() retry up to 10 times before actually failing.

[Stuart Bishop (stub)]

Is there a Juju bug for the problem this is working around? It should be cited in the comments so future-us knows when the work around can be removed.

review: Needs Fixing

Unmerged revisions

696. By Haw Loeung on 2017-02-28

Updated status_set() to re-try at least 10 times with random splay.

=== modified file 'charmhelpers/core/hookenv.py'
--- charmhelpers/core/hookenv.py	2017-02-01 00:12:12 +0000
+++ charmhelpers/core/hookenv.py	2017-02-28 06:35:17 +0000
@@ -29,7 +29,9 @@
 import subprocess
 import sys
 import errno
+import random
 import tempfile
+import time
 from subprocess import CalledProcessError
 
 import six
@@ -813,13 +815,19 @@
             '{!r} is not a valid workload state'.format(workload_state)
         )
     cmd = ['status-set', workload_state, message]
-    try:
-        ret = subprocess.call(cmd)
-        if ret == 0:
-            return
-    except OSError as e:
-        if e.errno != errno.ENOENT:
-            raise
+    count = 1
+    while (count <= 10):
+        count += 1
+        try:
+            ret = subprocess.call(cmd)
+            if ret == 0:
+                return
+        except OSError as e:
+            if e.errno != errno.ENOENT:
+                raise
+        # For large environments, we could have a storm of update-status hooks
+        # firing (e.g. when jujud-machine-0 has been bounced) so let's splay.
+        time.sleep(random.randint(0, 60))
     log_message = 'status-set failed: {} {}'.format(workload_state,
                                                     message)
     log(log_message, level='INFO')