XML Feeds
dstat and disk device names
by Karanbir Singh 
Running dstat normally gives you something like this :
# dstat --nocolor ----total-cpu-usage---- -dsk/total- -net/total- ---paging-- ---system-- usr sys idl wai hiq siq| read writ| recv send| in out | int csw 3 0 97 0 0 0| 820k 456k| 0 0 | 800B 866B|1054 255
And you can split the disk metrics up based on devices using something like this :
# dstat -D sda,sdb,total --nocolor ----total-cpu-usage---- --dsk/sda-- --dsk/sdb-- -dsk/total- -net/total- ---paging-- ---system-- usr sys idl wai hiq siq| read writ: read writ: read writ| recv send| in out | int csw 1 1 97 0 0 0| 176k 77k: 30k 162k: 411k 478k| 0 0 |3608B 4005B|1447 1458 33 2 63 0 1 2| 0 0 : 0 216k: 0 432k|2470k 1611k| 0 0 |2915 6967 31 2 65 0 1 1| 0 0 : 0 0 : 0 0 |2210k 1338k| 0 0 |2866 6051
But you can also get deeper level device names / nodes to measure - which is required in many cases if you have drivers for storage creating nodes further down the tree than /dev. Eg. I have a setup where there are four mysql instances running, each with its own dedicated storage :
# mount | grep srv /dev/cciss/c0d1 on /var/lib/mysql/node1 type ext3 (rw) /dev/cciss/c0d2 on /var/lib/mysql/node2 type ext3 (rw) /dev/cciss/c0d3 on /var/lib/mysql/node3 type ext3 (rw) /dev/cciss/c0d4 on /var/lib/mysql/node4 type ext3 (rw) /dev/cciss/c0d5 on /srv/wal type ext3 (rw)
so in this case, to get dstat reporting working you need to mention just the component level, like this :
# dstat -D cciss/c0d1,cciss/c0d2,cciss/c0d3 --nocolor ----total-cpu-usage---- dsk/cciss/c dsk/cciss/c dsk/cciss/c -net/total- ---paging-- ---system-- usr sys idl wai hiq siq| read writ: read writ: read writ| recv send| in out | int csw 3 0 97 0 0 0| 209k 89k: 201k 86k: 204k 85k| 0 0 | 799B 865B|1055 256 90 1 8 1 0 0|3196k 68k: 764k 0 : 816k 24k|1067k 1650k| 0 0 |1999 609 95 1 3 1 0 0|2548k 0 : 0 4084k:2448k 5700k| 660k 791k| 0 0 |1611 571 96 1 2 1 0 0|2628k 0 : 808k 0 :1620k 0 | 352k 798k| 0 0 |1835 1605
And you get the details, for each block device.
- KB
12/Jun/2009 11:04:37 am, 
4 comments
Excellent post, I've enjoyed following your blog.
I recently setup a Red Hat Enterprise Linux 5 (x86_64) server attached to an IBM DS4300 (1722) via device-mapper-multipath. However, I'm not able to produce statistics on the mapper/mpath0p1 device. Are you aware of any incompatibilites between dstat and device-mapper-multipath?
Thanks,
John
P.S. Here is a brief look at one of my attempts...
[johncall@fprddb ~]$ mount
/dev/sda3 on / type ext3 (rw)
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620)
/dev/sda1 on /boot type ext3 (rw)
tmpfs on /dev/shm type tmpfs (rw)
/dev/mapper/mpath0p1 on /opt type ext3 (rw)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
[johncall@fprddb ~]$ dstat -D mapper/mpath0p1 --nocolor
Module disk24old has problems. (No objects found, no stats available)
----total-cpu-usage---- -net/total- ---paging-- ---system--
usr sys idl wai hiq siq| recv send| in out | int csw
0 0 100 0 0 0| 0 0 | 0 0 |1002 29
0 0 100 0 0 0| 302B 904B| 0 0 |1006 30
I recently setup a Red Hat Enterprise Linux 5 (x86_64) server attached to an IBM DS4300 (1722) via device-mapper-multipath. However, I'm not able to produce statistics on the mapper/mpath0p1 device. Are you aware of any incompatibilites between dstat and device-mapper-multipath?
Thanks,
John
P.S. Here is a brief look at one of my attempts...
[johncall@fprddb ~]$ mount
/dev/sda3 on / type ext3 (rw)
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620)
/dev/sda1 on /boot type ext3 (rw)
tmpfs on /dev/shm type tmpfs (rw)
/dev/mapper/mpath0p1 on /opt type ext3 (rw)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
[johncall@fprddb ~]$ dstat -D mapper/mpath0p1 --nocolor
Module disk24old has problems. (No objects found, no stats available)
----total-cpu-usage---- -net/total- ---paging-- ---system--
usr sys idl wai hiq siq| recv send| in out | int csw
0 0 100 0 0 0| 0 0 | 0 0 |1002 29
0 0 100 0 0 0| 302B 904B| 0 0 |1006 30
12/Jun/2009 @ 14:52
Comment from: John McNulty [Visitor]
@John Call
dstat is nice, but I'd stick with using "iostat -mx" for disk stats if I were you. You get much more information than dstat provides. How much time the system is spending in %iowait, and %util for each disk are the most immediate indicators of a potential I/O issue. You will need to ls -l the /dev/mpath/mpath0p1 device to find out the dm device it maps to. Or another way is to look at the major/minor number pair for /dev/mapper/mpath0p1 (e.g. 253,7) and then grep for this here:
# grep 253:7 /sys/block/*/dev
/sys/block/dm-7/dev:253:7
I don't think there are any disk reporting tools that have moved with the times yet enough to do the smart thing and report IO stats using device mapper names.
dstat is nice, but I'd stick with using "iostat -mx" for disk stats if I were you. You get much more information than dstat provides. How much time the system is spending in %iowait, and %util for each disk are the most immediate indicators of a potential I/O issue. You will need to ls -l the /dev/mpath/mpath0p1 device to find out the dm device it maps to. Or another way is to look at the major/minor number pair for /dev/mapper/mpath0p1 (e.g. 253,7) and then grep for this here:
# grep 253:7 /sys/block/*/dev
/sys/block/dm-7/dev:253:7
I don't think there are any disk reporting tools that have moved with the times yet enough to do the smart thing and report IO stats using device mapper names.
13/Jun/2009 @ 10:00
Comment from: Dag Wieers [Visitor] · http://dag.wieers.com/blog/
@John McNulty: It shouldn't be hard to create a plugin that shows per disk %iowait and %util values based on the sources from iostat. I looked at it once, but the iostat code is pretty hard to read (at least compared to the average dstat plugin)
Also, smarter device name support should not be that hard either, we just need someone who thinks it is important enough to do it.
In general, if you look at the amount of functionality dstat is gaining over the past months for general troubleshooting and performance monitoring, I hope it gathers some people adding more plugins.
Try eg. --toplatency --topcputime, or --topbio --topbio, on a recent subversion checkout and you'll see what I mean.
Also, smarter device name support should not be that hard either, we just need someone who thinks it is important enough to do it.
In general, if you look at the amount of functionality dstat is gaining over the past months for general troubleshooting and performance monitoring, I hope it gathers some people adding more plugins.
Try eg. --toplatency --topcputime, or --topbio --topbio, on a recent subversion checkout and you'll see what I mean.
20/Nov/2009 @ 14:14
Comment from: Dag Wieers [Visitor] · http://dag.wieers.com/blog/
@John McNulty: You're lucky, I just wrote the disk utilization plugin on the train, commuting home. It is now committed into subversion, you can use it with the --diskutil option.
I am still not sure if that will be the final plugin name, I might change it by a plugin that shows more extended information about a disk, than just the utilization percentage.
I am still not sure if that will be the final plugin name, I might change it by a plugin that shows more extended information about a disk, than just the utilization percentage.
20/Nov/2009 @ 17:19