Merge autopkgtest-cloud:instance-cleanup-client-error into autopkgtest-cloud:master

[Brian Murray]

The cleanup-instances script was not cleaning up instances because it was unable to write to the influx database as we hit the same limit as earlier but with "max-values-per-tag".

[Brian Murray (brian-murray)]

Here's an example of logging the error:

Jul 15 23:58:11 juju-4d1272-prod-proposed-migration-9 cloud-worker-maintenance[2674356]: WARNING:root:Unable to write to InfluxDB: 400: {"error":"partial write: max-values-per-tag limit exceeded (100000/100000): measurement=\"autopkgtest_delete_event\" tag=\"message\" value=\"instance adt-kinetic-arm64-libdata-phrasebook-perl-20220616-020233-juju-4d1272-prod-proposed-migration-9 (833ad1be-fa29-49ab-bcef-89b9b4b62597) is 717.9 hours old, deleting\" dropped=2"}

2b81192... by Brian Murray on 2022-07-16

Log the error we encountered.

3b507b8... by Brian Murray on 2022-07-16

Change error message to match the worker one

[Paride Legovini (paride)]

LGTM, thanks!

review: Approve

[Brian Murray (brian-murray)]

This was merged here IDK why the MP wasn't marked as merged:

https://git.launchpad.net/autopkgtest-cloud/commit/?id=e2bcdc6f2f6e856e77b7161f212e4fe0a0623b8c

diff --git a/charms/focal/autopkgtest-cloud-worker/autopkgtest-cloud/tools/cleanup-instances b/charms/focal/autopkgtest-cloud-worker/autopkgtest-cloud/tools/cleanup-instances
index f2910ad..1d0e871 100755
--- a/charms/focal/autopkgtest-cloud-worker/autopkgtest-cloud/tools/cleanup-instances
+++ b/charms/focal/autopkgtest-cloud-worker/autopkgtest-cloud/tools/cleanup-instances
@@ -10,6 +10,7 @@ from urllib.error import HTTPError
 
 import novaclient.client, novaclient.exceptions
 from influxdb import InfluxDBClient
+from influxdb.exceptions import InfluxDBClientError
 from keystoneauth1 import session
 from keystoneauth1.identity import v2, v3
 
@@ -178,4 +179,8 @@ for instance in nova.servers.list():
         logging.warning("instance %s has invalid name" % instance.name)
 
     if measurements and influx_client:
-        influx_client.write_points(measurements)
+        try:
+            influx_client.write_points(measurements)
+        except InfluxDBClientError as err:
+            logging.warning("Write to InfluxDB failed: %s" % err)
+            pass