Handle telegraf down gracefully; Add pushing metrics for last successful test run

Bug #1914475 reported by Haw Loeung
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
APT Stress Test Charm
Fix Released
Medium
Haw Loeung

Bug Description

Hi,

On an environment with issues with telegraf, apt-stresstest fails but not gracefully:

| apt-stresstest@machine-3:~$ /usr/local/bin/test_apt_mirrors.py -f /etc/apt-stresstest-mirrors.yaml -p bash,ubuntu-wallpapers-xenial --update-timeout 20 --download-timeout 45 -s 127.0.0.1:8094 -v
| apt_transaction_duration_seconds,test=update_1,mirror=us.archive.ubuntu.com,ip=20.62.184.178 value=0.918013
| DEBUG: Sending data to influxdb: 127.0.0.1:8094
| Traceback (most recent call last):
| File "/usr/local/bin/test_apt_mirrors.py", line 289, in <module>
| sys.exit(main())
| File "/usr/local/bin/test_apt_mirrors.py", line 267, in main
| run_tests(
| File "/usr/local/bin/test_apt_mirrors.py", line 166, in run_tests
| output_results(
| File "/usr/local/bin/test_apt_mirrors.py", line 156, in output_results
| send_to_influx(send_to, result)
| File "/usr/local/bin/test_apt_mirrors.py", line 145, in send_to_influx
| s.connect((host, port))
| ConnectionRefusedError: [Errno 111] Connection refused

I think the charm should push metrics for last successful run. We can then add prometheus alerting rules to catch those not run with say the last hour.

Related branches

Haw Loeung (hloeung)
summary: - Add pushing metrics for last successful test run
+ Handle telegraf down gracefully; Add pushing metrics for last successful
+ test run
Haw Loeung (hloeung)
Changed in apt-stresstest-charm:
status: New → In Progress
importance: Undecided → Medium
assignee: nobody → Haw Loeung (hloeung)
Revision history for this message
Haw Loeung (hloeung) wrote :

The charm already pushes metrics for successful runs - apt_transaction_total_duration_seconds and apt_mirror_units_count

Changed in apt-stresstest-charm:
status: In Progress → Fix Committed
Haw Loeung (hloeung)
Changed in apt-stresstest-charm:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.