Merge lp:~hloeung/ubuntu-repository-cache/metadata-sync-failure-output-ordering into lp:ubuntu-repository-cache

[Haw Loeung]

Bump rsync timeout to reduce metadata age failures & alerts; Fix ordering of errors in output

[🤖 Canonical IS Merge Bot (canonical-is-mergebot)]

This merge proposal is being monitored by mergebot. Change the status to Approved to merge.

[lp:~hloeung/ubuntu-repository-cache/metadata-sync-failure-output-ordering]

375. By Haw Loeung on 2022-08-21

Minor fixing

[Paul Collins (pjdc)]

review: Approve (lgtm)

[🤖 Canonical IS Merge Bot (canonical-is-mergebot)]

Change successfully merged at revision 374

=== modified file 'lib/ubuntu_repository_cache/metadata_sync.py'
--- lib/ubuntu_repository_cache/metadata_sync.py	2022-08-17 01:40:08 +0000
+++ lib/ubuntu_repository_cache/metadata_sync.py	2022-08-21 22:21:33 +0000
@@ -11,6 +11,7 @@
 import shutil
 import socket
 import subprocess
+import sys
 import time
 import traceback
 
@@ -307,6 +308,7 @@
             msg = "{}".format(output)
             log(msg)
         except subprocess.CalledProcessError as e:
+            send_to_influx(render_influx(metric_name, label, attempt))
             failures = attempt
             # Save the exact exception and reuse when we need to
             # reraise it after exhausting all available attempts.
@@ -318,18 +320,18 @@
             log(e.output.decode())
             errmsgs.append("{}: {}".format(datetime.datetime.now().strftime("%Y-%m-%d %H:%M:%S"), msg))
             errmsgs.append(e.output.decode())
-            send_to_influx(render_influx(metric_name, label, attempt))
             time.sleep(sleep)
         else:
             break
     else:
+        send_to_influx(render_influx(metric_name, label, max_retries))
+        print('\n'.join(errmsgs))
         # Exhausted all available attempts, let's re-raise the
         # exception.
         msg = "Failed after all available attempts ({}) with {}".format(max_retries, meta_ver)
         log(msg)
         print("{}: {}".format(datetime.datetime.now().strftime("%Y-%m-%d %H:%M:%S"), msg))
-        print('\n'.join(errmsgs))
-        send_to_influx(render_influx(metric_name, label, max_retries))
+        sys.stdout.flush()
         raise exp
 
     send_to_influx(render_influx(metric_name, label, failures))

=== modified file 'lib/ubuntu_repository_cache/mirror.py'
--- lib/ubuntu_repository_cache/mirror.py	2022-07-22 03:55:30 +0000
+++ lib/ubuntu_repository_cache/mirror.py	2022-08-21 22:21:33 +0000
@@ -40,7 +40,7 @@
         '--compress',
         '--delete',
         '--delete-before',
-        '--timeout=120',
+        '--timeout=300',
         '-e',
         'sudo -u {} ssh -l {}'.format(user, user),
     ]