MAAS

[2.3, HWTv2] Fio does not always gather metrics

Bug #1726534 reported by Lee Trager
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
MAAS
Fix Released
Critical
Unassigned
MAAS 2.3.0beta3

Bug Description

I ran the Fio test on the hardware in the CI and none of the CI hardware is showing any metrics. This may be because the disks in the CI are hard drives and not SSDs.

Related branches

~ltrager/maas:lp1726534
Newell Jensen (community): Approve
MAAS Lander: Approve
Diff: 13 lines (+1/-1)
1 file modified
src/metadataserver/builtin_scripts/fio.py (+1/-1)
Revision history for this message
Lee Trager (ltrager) wrote :
Revision history for this message
Andres Rodriguez (andreserl) wrote :

I have run this test multiple times and have never seen this issue. I’ve run it in the CI and locally. Are there any Maas logs attached to this?

Changed in maas:
status: Triaged → Incomplete
Revision history for this message
Lee Trager (ltrager) wrote :

I think the issue is a race condition. We pass through all of fio's output so its written to our log and sent back to MAAS. When fio finishes running we read that log to get the results. What I suspect is happening is during high load fio finishes and the call is sent to write the data to the disk however our processing code runs before that happens.

Adding os.fsync to the script runner may fix this. If not we will have to modify fio to capture and log fio's output.

Changed in maas:
status: Incomplete → Triaged
Lee Trager (ltrager)
Changed in maas:
status: Triaged → In Progress
MAAS Lander (maas-lander)
Changed in maas:
status: In Progress → Fix Committed
Andres Rodriguez (andreserl)
Changed in maas:
milestone: 2.3.0beta4 → 2.3.0beta3
Andres Rodriguez (andreserl)
Changed in maas:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Bug attachments

Remote bug watches

Bug watches keep track of this bug in other bug trackers.