Merge lp:~hloeung/ubuntu-repository-cache/metadata-peer-sync-retry into lp:ubuntu-repository-cache

[Haw Loeung]

Add retry logic around syncing metadata snapshot to peers

[🤖 Canonical IS Merge Bot (canonical-is-mergebot)]

This merge proposal is being monitored by mergebot. Change the status to Approved to merge.

[lp:~hloeung/ubuntu-repository-cache/metadata-peer-sync-retry]

376. By Haw Loeung on 2022-08-21

Break out of retry loop on success

[James Simpson (jsimpso)]

LGTM

review: Approve

[🤖 Canonical IS Merge Bot (canonical-is-mergebot)]

Change successfully merged at revision 375

=== modified file 'lib/ubuntu_repository_cache/mirror.py'
--- lib/ubuntu_repository_cache/mirror.py	2022-08-20 23:39:48 +0000
+++ lib/ubuntu_repository_cache/mirror.py	2022-08-21 23:32:03 +0000
@@ -1,6 +1,8 @@
 '''Functions relating to mirrored Ubuntu package repository content'''
 
+import datetime
 import os
+import random
 import sys
 import time
 import traceback
@@ -89,14 +91,33 @@
     for peer in peers:
         rsync_dest = '{}@{}:{}'.format(user, peer, dest_dir)
         LOG('Syncing {} to {}.'.format(source_dir, rsync_dest))
-        try:
-            _rsync(source_dir, rsync_dest, link_dest=link_dir, user=user, retries=10)
-        except Exception:
-            LOG('Sync to {} failed'.format(peer))
-            traceback.print_exc(file=sys.stdout)
+
+        errmsgs = []
+        # Work around Juju instability issues causing juju-run to abort
+        # (LP:1977798 & LP:1984060).
+        max_retries = 10
+        for attempt in range(1, max_retries):
+            try:
+                _rsync(source_dir, rsync_dest, link_dest=link_dir, user=user, retries=10)
+            except Exception as e:
+                LOG('Sync to {} failed'.format(peer))
+                LOG(e.output.decode())
+                errmsgs.append("{}: Sync to {} failed".format(datetime.datetime.now().strftime("%Y-%m-%d %H:%M:%S"), peer))
+                errmsgs.append(e.output.decode())
+                # Exponential backoff.
+                sleep = random.randint(attempt, 10 * attempt)
+                time.sleep(sleep)
+            else:
+                successful_peers.append(peer)
+                LOG('Sync to {} complete.'.format(peer))
+                break
         else:
-            successful_peers.append(peer)
-            LOG('Sync to {} complete.'.format(peer))
+            print('\n'.join(errmsgs))
+            msg = "Failed after all available attempts ({}) with peer {}".format(max_retries, peer)
+            LOG(msg)
+            print("{}: {}".format(datetime.datetime.now().strftime("%Y-%m-%d %H:%M:%S"), msg))
+            sys.stdout.flush()
+
     return successful_peers