pt-stalk doesn't run vmstat, iostat, and mpstat for --run-time

Bug #955860 reported by Miguel Angel Nieto
10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Percona Toolkit moved to https://jira.percona.com/projects/PT
Fix Released
High
Daniel Nichter
2.0
Fix Released
High
Daniel Nichter
2.1
Fix Released
High
Daniel Nichter

Bug Description

When you run pt-stalk just to collect information with --no-stalk, it ignores --sleep and --iterations:

root@debian:# /root/pt-stalk --collect --no-stalk --threshold=0 --run-time 61 --sleep=120 --iterations=10 -- -uroot -pmsandbox -h 127.0.0.1 -P 5520
2012_03_15_11_14_08 Starting /root/pt-stalk --function=status --variable=Threads_running --threshold=0 --match= --cycles=0 --interval=0 --iterations=1 --run-time=61 --sleep=0 --dest=/var/lib/pt-stalk --prefix= --notify-by-email= --log=/var/log/pt-stalk.log --pid=/var/run/pt-stalk.pid

With sleep=0, pt-stalk run so fast that you don't get some useful information, like iostat or vmstat for example. The workaround is the use of --stalk option with low values:

root@debian:# /root/pt-stalk --collect --variable=Threads_running --iterations=2 --threshold=0 --run-time 60 --sleep=120 --dest=/root/percona_low_load/
2012_03_15_05_11_33 Starting /root/pt-stalk --function=status --variable=Threads_running --threshold=0 --match= --cycles=5 --interval=1 --iterations=2 --run-time=60 --sleep=120 --dest=/root/percona_low_load/ --prefix= --notify-by-email= --log=/var/log/pt-stalk.log --pid=/var/run/pt-stalk.pid

Now, pt-stalk has time to collect all the data from the system.

Related branches

Revision history for this message
Aurimas Mikalauskas (aurimas-mikalauskas) wrote :

When I want to collect some information now() rather than letting pt-stalk trigger on some condition, I usually use:

  pt-stalk --cycles=1 --iterations=1 --threshold=1

However, there's the same problem that there's just one sample of vmstat, mpstat & iostat while I'd expect 30 in this case.

Revision history for this message
Baron Schwartz (baron-xaprb) wrote :

When running the tool with bash -x, I found that it's doing this:

vmstat 1 1
vmstat 1 2

It should be doing this:

vmstat 1 30
vmstat 30 2

It looks to me like $OPT_INTERVAL, which is supposed to be the amount of time between checks for the trigger, is instead being used for the total runtime of vmstat here. Also to iostat and mpstat. It should be replaced by $OPT_RUN_TIME.

The temporary workaround for this bug is probably this patch:

=== modified file 'bin/pt-stalk'
--- bin/pt-stalk 2012-03-07 23:41:54 +0000
+++ bin/pt-stalk 2012-03-16 14:22:46 +0000
@@ -710,16 +710,16 @@
       $CMD_SYSCTL -a >> "$d/$p-sysctl" &
    fi
    if [ "$CMD_VMSTAT" ]; then
- $CMD_VMSTAT 1 $OPT_INTERVAL >> "$d/$p-vmstat" &
- $CMD_VMSTAT $OPT_INTERVAL 2 >> "$d/$p-vmstat-overall" &
+ $CMD_VMSTAT 1 $OPT_RUN_TIME >> "$d/$p-vmstat" &
+ $CMD_VMSTAT $OPT_RUN_TIME 2 >> "$d/$p-vmstat-overall" &
    fi
    if [ "$CMD_IOSTAT" ]; then
- $CMD_IOSTAT -dx 1 $OPT_INTERVAL >> "$d/$p-iostat" &
- $CMD_IOSTAT -dx $OPT_INTERVAL 2 >> "$d/$p-iostat-overall" &
+ $CMD_IOSTAT -dx 1 $OPT_RUN_TIME >> "$d/$p-iostat" &
+ $CMD_IOSTAT -dx $OPT_RUN_TIME 2 >> "$d/$p-iostat-overall" &
    fi
    if [ "$CMD_MPSTAT" ]; then
- $CMD_MPSTAT -P ALL 1 $OPT_INTERVAL >> "$d/$p-mpstat" &
- $CMD_MPSTAT -P ALL $OPT_INTERVAL 1 >> "$d/$p-mpstat-overall" &
+ $CMD_MPSTAT -P ALL 1 $OPT_RUN_TIME >> "$d/$p-mpstat" &
+ $CMD_MPSTAT -P ALL $OPT_RUN_TIME 1 >> "$d/$p-mpstat-overall" &
    fi

Changed in percona-toolkit:
status: New → Confirmed
importance: Undecided → High
milestone: none → 2.0.5
Revision history for this message
Aurimas Mikalauskas (aurimas-mikalauskas) wrote :

Thanks Baron, that gets the job done.

Revision history for this message
Baron Schwartz (baron-xaprb) wrote :

This is a proper patch for the bug.

Revision history for this message
Baron Schwartz (baron-xaprb) wrote :

Miguel, can you confirm that we're indeed talking about the same bug, and the patch fixes the problem you've experienced too?

Revision history for this message
Miguel Angel Nieto (miguelangelnieto) wrote :

It works. Thank you.

Revision history for this message
Baron Schwartz (baron-xaprb) wrote :

We need to ensure the fix is merged to both the 2.0 and 2.1 branches.

summary: - pt-stalk ignores --sleep and --iterations when running with --no-stalk
+ pt-stalk ignores --sleep and --iterations with --no-stalk
tags: added: option-ignored pt-stalk
Revision history for this message
Daniel Nichter (daniel-nichter) wrote : Re: pt-stalk ignores --sleep and --iterations with --no-stalk

For the record, the tool is supposed to ignore --sleep and --interval because --no-stalk means "collect once and exit". The bug was, as Baron pointed out, that --interval was being used instead of --run-time. So the collection sub-process was running for --run-time, but vmstat, iostate, and mpstat were not.

summary: - pt-stalk ignores --sleep and --iterations with --no-stalk
+ pt-stalk doesn't run vmstat, iostat, and mpstat for --run-time
Revision history for this message
Shahriyar Rzayev (rzayev-sehriyar) wrote :

Percona now uses JIRA for bug reports so this bug report is migrated to: https://jira.percona.com/browse/PT-301

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.