Merge lp:~hloeung/ubuntu-repository-cache/increase-juju-run-retries into lp:ubuntu-repository-cache

[Haw Loeung]

Increase retries on juju-run failures to reduce issues with incomplete metadata sync

We do this by increasing the no. of retries as well as the time to sleep before next retry.

[Paul Collins (pjdc)]

review: Approve (lgtm)

[🤖 Canonical IS Merge Bot (canonical-is-mergebot)]

This merge proposal is being monitored by mergebot. Change the status to Approved to merge.

[🤖 Canonical IS Merge Bot (canonical-is-mergebot)]

Change successfully merged at revision 366

=== modified file 'lib/ubuntu_repository_cache/metadata_sync.py'
--- lib/ubuntu_repository_cache/metadata_sync.py	2022-07-28 01:56:44 +0000
+++ lib/ubuntu_repository_cache/metadata_sync.py	2022-08-08 22:22:33 +0000
@@ -295,7 +295,7 @@
 
     # Work around Juju instability issues causing juju-run to abort
     # (LP:1977798).
-    max_retries = 5
+    max_retries = 10
     for attempt in range(1, max_retries):
         try:
             check_call(cmd, stdout=subprocess.DEVNULL)
@@ -304,7 +304,7 @@
             # reraise it after exhausting all available attempts.
             exp = e
             # Exponential backoff.
-            sleep = attempt * random.randint(1, 60)
+            sleep = attempt * random.randint(10, 60)
             log("Attempt {} failed, sleeping {} - retrying with {}".format(attempt, sleep, meta_ver))
             metric_name = 'ubuntu_repository_cache_metadata_sync_failures'
             label = 'meta_ver={}'.format(meta_ver)