System monitoring and reporting using sysstat and dstat on CentOS/RHEL 7

It's often useful to measure and view various system utilisation metrics, such as for:

• cpu
• memory
• disk
• network

There are a lot of tools that can measure and collect these metrics, including:

• dstat - suited for creating realtime reports
• sysstat - suited for creating historical reports

dstat in action

Dstat shows realtime system utilisation data. First you need to install it:

$ yum install dstat

running dstat on it's own gives a steady stream of output until you exit out:

[root@target man]# dstat
You did not select any stats, using -cdngy by default.
----total-cpu-usage---- -dsk/total- -net/total- ---paging-- ---system--
usr sys idl wai hiq siq| read  writ| recv  send|  in   out | int   csw
  0   0 100   0   0   0|  23k   53k|   0     0 |   0     0 |  84    72
  0   0 100   0   0   0|   0     0 |  60B  338B|   0     0 |  88    82
  0   0 100   0   0   0|   0     0 |  60B  122B|   0     0 |  56    54
  0   0 100   0   0   0|   0     0 |  60B   98B|   0     0 |  53    52
  0   1 100   0   0   0|   0     0 |  60B   98B|   0     0 |  66    60
  0   0 100   0   0   0|   0     0 |  60B   98B|   0     0 |  53    55
.
.
.
...etc

This gives a general overview. But if you want look at just the cpu stats then you use the '-c' option:

[root@target man]# dstat -c
----total-cpu-usage----
usr sys idl wai hiq siq
  0   0 100   0   0   0
  0   0 100   0   0   0
  0   0 100   0   0   0
  0   0 100   0   0   0^C

You take a similar approach for (d)isk, (m)emory, (n)etwork, (p)rocesses, (r)ead-write requests.

Also you can output these stats into a csv file like this:

$ dstat --output /tmp/dstat.csv
You did not select any stats, using -cdngy by default.
----total-cpu-usage---- -dsk/total- -net/total- ---paging-- ---system--
usr sys idl wai hiq siq| read  writ| recv  send|  in   out | int   csw
  0   0 100   0   0   0|  22k   51k|   0     0 |   0     0 |  81    70
  0   0 100   0   0   0|   0     0 |  60B  130B|   0     0 |  39    32
  0   0 100   0   0   0|   0     0 |  60B  106B|   0     0 |  21    17
  0   0 100   0   0   0|   0     0 |  60B   98B|   0     0 |  43    35 ^C

Note it displays the output like normal as well as writing it to a file.

Where the output file looks like this:

cat /tmp/dstat.csv
"Dstat 0.7.2 CSV output"
"Author:","Dag Wieers ",,,,"URL:","http://dag.wieers.com/home-made/dstat/"
"Host:","target.cb.net",,,,"User:","root"
"Cmdline:","dstat --output /tmp/dstat",,,,"Date:","28 Feb 2018 17:39:48 UTC"

"total cpu usage",,,,,,"dsk/total",,"net/total",,"paging",,"system",
"usr","sys","idl","wai","hiq","siq","read","writ","recv","send","in","out","int","csw"
0.173,0.123,99.670,0.009,0.0,0.026,22395.765,51954.426,0.0,0.0,0.0,0.0,80.816,69.846
0.0,0.0,100.0,0.0,0.0,0.0,0.0,0.0,60.0,130.0,0.0,0.0,39.0,32.0
0.0,0.0,100.0,0.0,0.0,0.0,0.0,0.0,60.0,106.0,0.0,0.0,21.0,17.0
0.0,0.0,100.0,0.0,0.0,0.0,0.0,0.0,60.0,98.0,0.0,0.0,43.0,35.0

However this approach is quite crude, since it's hard to query this info for historical information. That's where sysstat comes to the rescue.

sysstat in action

Sysstat is actually a collection of tools for gathering and reporting system stats. You first need to install it:

$ yum install -y sysstat

sysstat has a few config files and by default runs using a cron job:

$ rpm -qc sysstat
/etc/cron.d/sysstat
/etc/sysconfig/sysstat
/etc/sysconfig/sysstat.ioconf

Cron job can be customised by editing the /etc/sysconfig/sysstat file:

$ cat /etc/sysconfig/sysstat
# sysstat-10.1.5 configuration file.

# How long to keep log files (in days).
# If value is greater than 28, then log files are kept in
# multiple directories, one for each month.
HISTORY=28

# Compress (using gzip or bzip2) sa and sar files older than (in days):
COMPRESSAFTER=31

# Parameters for the system activity data collector (see sadc manual page)
# which are used for the generation of log files.
SADC_OPTIONS="-S DISK"

# Compression program to use.
ZIP="bzip2"

The cron job itself runs 2 tools sa1 and sa2:

$ cat /etc/cron.d/sysstat
# Run system activity accounting tool every 10 minutes
*/10 * * * * root /usr/lib64/sa/sa1 1 1
# 0 * * * * root /usr/lib64/sa/sa1 600 6 &
# Generate a daily summary of process accounting at 23:53
53 23 * * * root /usr/lib64/sa/sa2 -A

Both of these cron jobs creates daily reports and store them in the /var/log/sa directory. They cron jobs creates the following files:

• sa1 generates /var/log/sa/sa{dd} files. It also appends to this file every 10 minutes. This file gradually grows every 10 minutes until midnight. Then a new file is generated.
• sa2 generates /var/log/sa/sar{dd} files, once per day

Where {dd} represents the day of the month:

$ date
Wed Feb 28 18:35:15 UTC 2018
[root@target sa]# ls -lrt /var/log/sa/sa28
-rw-r--r--. 1 root root 18372 Feb 28 18:30 /var/log/sa/sa28

For more info about sa1 and sa2, see the man pages for these. sa1 essentially stores system statistics in special non-human-readable binary files, called '/var/log/sa/sa{dd}':

Since sadd files are not human readable, you need to use the sadf tool query the content of the sadf file. E.g. to view the data between 6pm to 7pm on the 28th of the month, then we do:

$ [root@codingbee sa]# sadf -s 18:00:00 -e 19:00:00 /var/log/sa/sa28
codingbee.net	-1	2018-02-28 18:20:21 UTC	LINUX-RESTART
codingbee.net	600	2018-02-28 18:40:01 UTC	all	%user	0.01
codingbee.net	600	2018-02-28 18:40:01 UTC	all	%nice	0.00
codingbee.net	600	2018-02-28 18:40:01 UTC	all	%system	0.02
codingbee.net	600	2018-02-28 18:40:01 UTC	all	%iowait	0.00
codingbee.net	600	2018-02-28 18:40:01 UTC	all	%steal	0.00
codingbee.net	600	2018-02-28 18:40:01 UTC	all	%idle	99.97
codingbee.net	600	2018-02-28 18:50:01 UTC	all	%user	0.01
codingbee.net	600	2018-02-28 18:50:01 UTC	all	%nice	0.00
codingbee.net	600	2018-02-28 18:50:01 UTC	all	%system	0.02
codingbee.net	600	2018-02-28 18:50:01 UTC	all	%iowait	0.00
codingbee.net	600	2018-02-28 18:50:01 UTC	all	%steal	0.00
codingbee.net	600	2018-02-28 18:50:01 UTC	all	%idle	99.98

sadf is actually a wrapper around another cli utility called sar (System Activity Report). Sar also comes with the sysstat package. See:

https://www.google.co.uk/search?q=sar+command&oq=sar+command&aqs=chrome..69i57j69i60j0l4.3023j0j4&sourceid=chrome&ie=UTF-8

Here are some other useful commands:

uptime (prints load averages)
du
df
free
top
ps
pstree
pgrep

other commands

htop
nmon
sysstat
dstat
netdata
sysdig

http://www.brendangregg.com/linuxperf.html

https://medium.com/netflix-techblog/netflix-at-velocity-2015-linux-performance-tools-51964ddb81cf

[post-content post_name=rhsca-quiz]