Tuesday, November 25, 2014

10 pidstat Examples to Debug Performance Issues of Linux Process

pidstat stands for PID Statistics.
This tool can monitor an individual process that is managed by kernel and generate a report. It can monitor either a specific PID (process id), or all the process running on the system.
pidstat is a part of sysstat utility.
This tool reports various statistics including CPU used by a process, disk usage statistics of a process, statistics for threads associated with selected tasks and child processes.

You should install systat package to use this command.
yum install sysstat

1. Display Statistics for ALL Running Process (or a Specific Process)

Use -p ALL option to view performance statistics of all running process as shown below.
By default this will display the CPU usage. But, you can change this to any other performance statistics as shown in later examples.
# pidstat -p ALL | wc -l
165

# pidstat -p ALL | head
Linux 3.0.101-0.7.17-default (thegeekstuff)    07/30/14        _x86_64_

05:00:03          PID    %usr %system  %guest    %CPU   CPU  Command
05:00:03            1    0.00    0.00    0.00    0.00     0  init
05:00:03            2    0.00    0.00    0.00    0.00     0  kthreadd
05:00:03            3    0.00    0.00    0.00    0.00     0  ksoftirqd/0
05:00:03            6    0.00    0.00    0.00    0.00     0  migration/0
05:00:03            7    0.00    0.00    0.00    0.00     0  watchdog/0
05:00:03            8    0.00    0.00    0.00    0.00     0  cpuset
05:00:03            9    0.00    0.00    0.00    0.00     0  khelper
Use -p PID to monitor performance statistics for a particular process as shown below.
# pidstat -p 13203
Linux 3.0.101-0.7.17-default (thegeekstuff)    07/30/14        _x86_64_

05:04:29          PID    %usr %system  %guest    %CPU   CPU  Command
05:04:29        13203    0.00    0.00    0.00    0.00     0  vim
If you are new to systat package, you should also understand the performance statistics data provided by sar command.

2. Display Performance Statistics based on Process Name using -C

The following example will display performance statistics for all the process that matches a specific keyword (for example: mysql).
# pidstat -C "mysql"
Linux 3.0.101-0.7.17-default (thegeekstuff)    07/30/14        _x86_64_

06:33:14          PID    %usr %system  %guest    %CPU   CPU  Command
06:33:14        23132    0.00    0.00    0.00    0.00     0  mysqld_safe
06:33:14        23493    0.06    0.02    0.00    0.08     0  mysqld
Note: In the above example, the option -C stands for “Command Name”. i.e It will search the command name of the process with the given keyword.

3. Repeat the Output at Certain Interval

By default, the output will not be repeated. For example, option -u is to display the CPU usage statistics of tasks and this is the default statistics given by pidstat command. Thsi will display the output only one time.
# pidstat -p 23493
Linux 3.0.101-0.7.17-default (thegeekstuff)    07/30/14        _x86_64_

06:52:35          PID    %usr %system  %guest    %CPU   CPU  Command
06:52:36        23493    1.00    0.00    0.00    1.00     0  mysqld
To repeat the output, specific the interval in seconds as the last parameter. For example, the following example will repeat the output every 1 second (until you press Ctrl-C).
# pidstat -p 23493 1
Linux 3.0.101-0.7.17-default (thegeekstuff)    07/30/14        _x86_64_

06:52:35          PID    %usr %system  %guest    %CPU   CPU  Command
06:52:36        23493    1.00    0.00    0.00    1.00     0  mysqld
06:52:37        23493    0.00    0.00    0.00    0.00     0  mysqld
06:52:38        23493    0.00    0.00    0.00    0.00     0  mysqld
06:52:39        23493    0.00    0.00    0.00    0.00     0  mysqld
06:52:40        23493    0.00    0.00    0.00    0.00     0  mysqld
06:52:41        23493    0.00    0.00    0.00    0.00     0  mysqld
..
The following will repeat the output every 5 seconds (until you press Ctrl-C).
# pidstat -p 23493 5

4. Display I/O Statistics for a Specific Process using -d

Use option -d to report I/O statistics of process. It output displays different attributes like PID, disk read and write speed in kB/s as shown below.
The following example displays the disk usage for PID 23493 every 1 second.
# pidstat -p 23493 -d 1
Linux 3.0.101-0.7.17-default (thegeekstuff)    07/30/14        _x86_64_

06:48:33          PID   kB_rd/s   kB_wr/s kB_ccwr/s  Command
06:48:34        23493      0.00      0.00      0.00  mysqld
06:48:35        23493      0.00      0.00      0.00  mysqld
06:48:36        23493      0.00      0.00      0.00  mysqld
06:48:37        23493      0.00      0.00      0.00  mysqld
06:48:38        23493      0.00      0.00      0.00  mysqld
06:48:39        23493      0.00      0.00      0.00  mysqld
06:48:40        23493      0.00      0.00      0.00  mysqld
06:48:41        23493      0.00      0.00      0.00  mysqld

5. Display Paging Activity for a Specific Process using -r

Use option -r to display page faults and memory utilization for a given task (PID).
# pidstat -p 23493 -r 1
Linux 3.0.101-0.7.17-default (thegeekstuff)    07/30/14        _x86_64_

06:58:48          PID  minflt/s  majflt/s     VSZ    RSS   %MEM  Command
06:58:49        23493      0.00      0.00  398876  33468   3.28  mysqld
06:58:50        23493      0.00      0.00  398876  33468   3.28  mysqld
06:58:51        23493      0.00      0.00  398876  33468   3.28  mysqld
06:58:52        23493      0.00      0.00  398876  33468   3.28  mysqld
06:58:53        23493      0.00      0.00  398876  33468   3.28  mysqld
06:58:54        23493      0.00      0.00  398876  33468   3.28  mysqld
06:58:55        23493      0.00      0.00  398876  33468   3.28  mysqld

6. Display Command Name and its Arguments using option -l

By default, pidstat displays only the command name. i.e Without the full path of the command and its argument. For example, in the command column you’ll see only “java” (which is just the name of the program).
$ pidstat -C java
Linux 3.0.101-0.7.17-default (thegeekstuff)    07/30/14        _x86_64_

11:00:25       PID    %usr %system  %guest    %CPU   CPU  Command
11:00:25      3288    0.04    0.03    0.00    0.07     0  java
11:00:25     17861    0.03    0.02    0.00    0.05     6  java
However when you use option -l, it will display the full path of the command, and all its arguments as shown below.
$ pidstat -C java -l
Linux 3.0.101-0.7.17-default (thegeekstuff)    07/30/14        _x86_64_

11:00:31       PID    %usr %system  %guest    %CPU   CPU  Command
11:00:31      3288    0.04    0.03    0.00    0.07     0  /usr/bin/java -Djava.u
11:00:31     17861    0.03    0.02    0.00    0.05     6  java -jar /home/rabbit/myapp.jar /home/app/conf/myapp.conf
In order to get the statistics on regular intervals for tasks, just pass the number of seconds at which you wish to see the statistics,
# pidstat -p 23493 1
Linux 3.0.101-0.7.17-default (thegeekstuff)    07/30/14        _x86_64_

06:25:50          PID    %usr %system  %guest    %CPU   CPU  Command
06:25:51        23493    0.00    0.00    0.00    0.00     0  mysqld
06:25:52        23493    0.00    0.00    0.00    0.00     0  mysqld
As seen above, watching the mysqld process’s statistics for every 1 second.
You can also fetch few process ids from top command output and watch it here to understand its usage of system resources on regular intervals.

7. Display Output at Regular Intervals for X Number of Times

It is also possible to get reports for certain number of times at given intervals for list of process as shown below.
Add the number of times as the last parameter (after the intervals in seconds).
For example, the following will display the output 5 times (at a regular interval of 2 seconds). At the end of the report, it will also display the “Average” values.
# pidstat 2 5
Linux 3.0.101-0.7.17-default (thegeekstuff)    07/30/14        _x86_64_

07:14:11          PID    %usr %system  %guest    %CPU   CPU  Command
07:14:13         1445    0.00    0.50    0.00    0.50     0  pidstat
07:14:13        28222    0.50    0.00    0.00    0.50     0  cma

07:14:13          PID    %usr %system  %guest    %CPU   CPU  Command
07:14:15         1445    0.00    0.50    0.00    0.50     0  pidstat
07:14:15        28222    0.00    0.50    0.00    0.50     0  cma

07:14:15          PID    %usr %system  %guest    %CPU   CPU  Command
07:14:17         1445    0.50    0.00    0.00    0.50     0  pidstat
07:14:17        19614    0.00    0.50    0.00    0.50     0  nailswebd

07:14:17          PID    %usr %system  %guest    %CPU   CPU  Command
07:14:19         1445    0.50    0.50    0.00    1.00     0  pidstat

07:14:19          PID    %usr %system  %guest    %CPU   CPU  Command
07:14:21         1445    0.00    0.51    0.00    0.51     0  pidstat
07:14:21        28222    0.00    0.51    0.00    0.51     0  cma

Average:          PID    %usr %system  %guest    %CPU   CPU  Command
Average:         1445    0.20    0.40    0.00    0.60     -  pidstat
Average:        19614    0.00    0.10    0.00    0.10     -  nailswebd
Average:        28222    0.10    0.20    0.00    0.30     -  cma

8. Display Statistics of Selected Process and its Child using -T

Use option -T to specify either CHILD or TASKS. In this case the statistics will be reported for TASKS, or tasks and all their children. You can also specify ALL.
Possible values for -T: CHILD, or TASKS, or ALL.
# pidstat -T CHILD | head 
Linux 3.0.101-0.7.17-default (thegeekstuff)    07/30/14        _x86_64_

10:13:34  IST       PID    usr-ms system-ms  guest-ms  Command
10:13:34  IST         1      7950      3340         0  init
10:13:34  IST         3         0       420         0  ksoftirqd/0
10:13:34  IST         8         0        10         0  migration/0
10:13:34  IST        10         0      1190         0  rcu_sched
10:13:34  IST        11        10         0         0  watchdog/0
10:13:34  IST        12        10         0         0  watchdog/1
10:13:34  IST        13         0       450         0  ksoftirqd/1

9. Display Statistics of Dependent Process in Tree Format using -t

Using option -t, you can display the output in a tree format as shown below.
# pidstat -t -C "mysql"
Linux 3.0.101-0.7.17-default (thegeekstuff)    07/30/14        _x86_64_

18:47:54          PID       TID    %usr %system  %guest    %CPU   CPU  Command
18:47:54         1646        -     0.00    0.00    0.00    0.00     0  mysql
18:47:54           -       1646    0.00    0.00    0.00    0.00     0  |__mysql
18:47:54        23132        -     0.00    0.00    0.00    0.00     0  mysqld_safe
18:47:54           -      23132    0.00    0.00    0.00    0.00     0  |__mysqld_safe
18:47:54        23493        -     0.06    0.02    0.00    0.08     0  mysqld
18:47:54           -      23493    0.00    0.00    0.00    0.00     0  |__mysqld
18:47:54           -      23504    0.00    0.00    0.00    0.00     0  |__mysqld
18:47:54           -      23509    0.00    0.00    0.00    0.00     0  |__mysqld
18:47:54           -      23510    0.00    0.00    0.00    0.00     0  |__mysqld
18:47:54           -      23512    0.00    0.00    0.00    0.00     0  |__mysqld
18:47:54           -      23515    0.00    0.00    0.00    0.00     0  |__mysqld
18:47:54           -      23516    0.01    0.00    0.00    0.02     0  |__mysqld
18:47:54           -      23517    0.00    0.00    0.00    0.00     0  |__mysqld
18:47:54           -      23518    0.01    0.01    0.00    0.02     0  |__mysqld
18:47:54           -      23519    0.00    0.00    0.00    0.00     0  |__mysqld
..

10. Display All Statistics Horizontally on a Single Line using -h

If you ask pidstat to report more than one statistics it displays one statistics after another statistics. In the following example, it will first display the performance statistics for option “r”, then option “u”, and finally option “d”.
# pidstat -rud
However if you want all of those statistics to be displayed in a single line for the individual process, use the option -h as shown below.
# pidstat -rud -h | head
Linux 3.0.101-0.7.17-default (thegeekstuff)    07/30/14        _x86_64_

#      Time       PID    %usr %system  %guest    %CPU   CPU  minflt/s  majflt/s     VSZ    RSS   %MEM   kB_rd/s   kB_wr/s kB_ccwr/s  Command
 1406823329         1    0.04    0.19    0.00    0.24     2     21.50      0.03   26944   2768   0.07     -1.00     -1.00     -1.00  init
 1406823329         3    0.00    0.03    0.00    0.03     0      0.00      0.00       0      0   0.00     -1.00     -1.00     -1.00  ksoftirqd/0
 1406823329         8    0.00    0.00    0.00    0.00     0      0.00      0.00       0      0   0.00     -1.00     -1.00     -1.00  migration/0
 1406823329        10    0.00    0.08    0.00    0.08     2      0.00      0.00       0      0   0.00     -1.00     -1.00     -1.00  rcu_sched
 1406823329        13    0.00    0.03    0.00    0.03     1      0.00      0.00       0      0   0.00     -1.00     -1.00     -1.00  ksoftirqd/1
 1406823329        14    0.00    0.00    0.00    0.00     1      0.00      0.00       0      0   0.00     -1.00     -1.00     -1.00  migration/1
 1406823329        18    0.00    0.01    0.00    0.01     2      0.00      0.00       0      0   0.00     -1.00     -1.00     -1.00  ksoftirqd/2

No comments:

Post a Comment