Linux: how can I determine if an application is doing sequential or random disk I/O?

Question

Everyone knows that databases tend to do lots of small random I/O while big data things like Kafka tend to do large sequential I/O, but if I'm approaching this as a sysadmin without making assumptions, how do I determine what a running application is doing on my system, or in general whether my system is seeing? Without knowing how the application is written, how do I determine whether it is doing mostly sequential or random I/O, thereby making it easier for me to choose the right kind of disks, etc, to deploy?

I can use iostat to get the average request size (avgrq-rz) of each IOP and a count of the number of IOPS (r/s + w/s). How do I determine whether these are mostly sequential or random?

Answer 1

Yes, there're actually couple of tools you can use to monitor this. Such as the one you mentioned, iotop and the iostat command.

Depending on your distribution, you can use either of the following to install iotop:

$ sudo apt-get install iotop or

$ yum install iotop

Run: root@tomcat-1-vm:/# iotop (to see list of disk I/O processess that are running)

Total DISK READ :       0.00 B/s | Total DISK WRITE :       0.00 B/s
Actual DISK READ:       0.00 B/s | Actual DISK WRITE:       0.00 B/s
  TID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN     IO>    COMMAND                                                                                  
24576 be/4 www-data    0.00 B/s    0.00 B/s  0.00 %  0.00 % apache2 -k start
    1 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % init
    2 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [kthreadd]
    3 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [ksoftirqd/0]
24580 be/4 www-data    0.00 B/s    0.00 B/s  0.00 %  0.00 % apache2 -k start
    5 be/0 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [kworker/0:0H]
    6 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [kworker/u2:0]
    7 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [rcu_sched]
    8 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [rcu_bh]
    9 rt/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [migration/0]
   10 be/0 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [lru-add-drain]
   11 rt/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [watchdog/0]
   12 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [cpuhp/0]
   13 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [kdevtmpfs]
   14 be/0 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [netns]
   15 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [khungtaskd]

You may also display I/O activity just by passing the o phrase

Example: root@tomcat-1-vm:/# iotop -o

Total DISK READ :       0.00 B/s | Total DISK WRITE :      11.91 K/s
Actual DISK READ:       0.00 B/s | Actual DISK WRITE:      11.91 K/s
  TID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN     IO>    COMMAND                                                            
  835 be/4 tomcat8     0.00 B/s    7.94 K/s  0.00 %  0.00 % java -Djava.util.logging.config.fi~he.catalina.startup.Bootstrap start
 1371 be/4 tomcat8     0.00 B/s    3.97 K/s  0.00 %  0.00 % java -Djava.util.logging.config.fi~he.catalina.startup.Bootstrap start

To learn more about disk i/o ouput and information you can check the proc filesystem page

Again depending on the distribution, you may install iostat as follows:

$ sudo apt-get install sysstat or

$ yum install sysstat

Run: root@tomcat-1-vm:/# iostat -dx 5

Linux 4.9.0-6-amd64 (tomcat-1-vm)       05/29/2018      _x86_64_        (1 CPU)

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda               0.00     0.22    0.01    0.31     0.31     3.67    24.34     0.00   12.56    8.50   12.73   4.04   0.13

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda               0.00     0.40    0.00    0.40     0.00     3.21    16.00     0.01   18.00    0.00   18.00  18.00   0.72

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda               0.00     0.00    0.00   25.30     0.00  5546.18   438.41     4.38  173.17    0.00  173.17   3.14   7.95

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda               0.00     2.81    0.00    0.40     0.00    12.83    64.00     0.00    0.00    0.00    0.00   0.00   0.00

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda               0.00     1.20    0.00    4.21     0.00    21.64    10.29     0.03    7.24    0.00    7.24   0.38   0.16

You may also refer to the following iostat examples post on options you can include for more report.

Another command to use is the dstat.

Example: root@tomcat-1-vm:/# sudo apt-get install dstat

root@tomcat-1-vm:/# dstat -tdD total,sda,sdb,sdc,md1 60 
----system---- -dsk/total----dsk/sda--
     time     | read  writ: read  writ
29-05 23:58:15| 318B 3797B: 318B 3797B
29-05 23:59:15|   0   473k:   0   473k
29-05