It’s often useful to measure and view various system utilisation metrics, such as for:
Announcement
You can find all my latest posts on medium.- cpu
- memory
- disk
- network
There are a lot of tools that can measure and collect these metrics, including:
- dstat – suited for creating realtime reports
- sysstat – suited for creating historical reports
dstat in action
Dstat shows realtime system utilisation data. First you need to install it:
$ yum install dstat
running dstat on it’s own gives a steady stream of output until you exit out:
[root@target man]# dstat You did not select any stats, using -cdngy by default. ----total-cpu-usage---- -dsk/total- -net/total- ---paging-- ---system-- usr sys idl wai hiq siq| read writ| recv send| in out | int csw 0 0 100 0 0 0| 23k 53k| 0 0 | 0 0 | 84 72 0 0 100 0 0 0| 0 0 | 60B 338B| 0 0 | 88 82 0 0 100 0 0 0| 0 0 | 60B 122B| 0 0 | 56 54 0 0 100 0 0 0| 0 0 | 60B 98B| 0 0 | 53 52 0 1 100 0 0 0| 0 0 | 60B 98B| 0 0 | 66 60 0 0 100 0 0 0| 0 0 | 60B 98B| 0 0 | 53 55 . . . ...etc
This gives a general overview. But if you want look at just the cpu stats then you use the ‘-c’ option:
[root@target man]# dstat -c ----total-cpu-usage---- usr sys idl wai hiq siq 0 0 100 0 0 0 0 0 100 0 0 0 0 0 100 0 0 0 0 0 100 0 0 0^C
You take a similar approach for (d)isk, (m)emory, (n)etwork, (p)rocesses, (r)ead-write requests.
Also you can output these stats into a csv file like this:
$ dstat --output /tmp/dstat.csv You did not select any stats, using -cdngy by default. ----total-cpu-usage---- -dsk/total- -net/total- ---paging-- ---system-- usr sys idl wai hiq siq| read writ| recv send| in out | int csw 0 0 100 0 0 0| 22k 51k| 0 0 | 0 0 | 81 70 0 0 100 0 0 0| 0 0 | 60B 130B| 0 0 | 39 32 0 0 100 0 0 0| 0 0 | 60B 106B| 0 0 | 21 17 0 0 100 0 0 0| 0 0 | 60B 98B| 0 0 | 43 35 ^C
Note it displays the output like normal as well as writing it to a file.
Where the output file looks like this:
cat /tmp/dstat.csv "Dstat 0.7.2 CSV output" "Author:","Dag Wieers",,,,"URL:","http://dag.wieers.com/home-made/dstat/" "Host:","target.cb.net",,,,"User:","root" "Cmdline:","dstat --output /tmp/dstat",,,,"Date:","28 Feb 2018 17:39:48 UTC" "total cpu usage",,,,,,"dsk/total",,"net/total",,"paging",,"system", "usr","sys","idl","wai","hiq","siq","read","writ","recv","send","in","out","int","csw" 0.173,0.123,99.670,0.009,0.0,0.026,22395.765,51954.426,0.0,0.0,0.0,0.0,80.816,69.846 0.0,0.0,100.0,0.0,0.0,0.0,0.0,0.0,60.0,130.0,0.0,0.0,39.0,32.0 0.0,0.0,100.0,0.0,0.0,0.0,0.0,0.0,60.0,106.0,0.0,0.0,21.0,17.0 0.0,0.0,100.0,0.0,0.0,0.0,0.0,0.0,60.0,98.0,0.0,0.0,43.0,35.0
However this approach is quite crude, since it’s hard to query this info for historical information. That’s where sysstat comes to the rescue.
sysstat in action
Sysstat is actually a collection of tools for gathering and reporting system stats. You first need to install it:
$ yum install -y sysstat
sysstat has a few config files and by default runs using a cron job:
$ rpm -qc sysstat /etc/cron.d/sysstat /etc/sysconfig/sysstat /etc/sysconfig/sysstat.ioconf
Cron job can be customised by editing the /etc/sysconfig/sysstat
file:
$ cat /etc/sysconfig/sysstat # sysstat-10.1.5 configuration file. # How long to keep log files (in days). # If value is greater than 28, then log files are kept in # multiple directories, one for each month. HISTORY=28 # Compress (using gzip or bzip2) sa and sar files older than (in days): COMPRESSAFTER=31 # Parameters for the system activity data collector (see sadc manual page) # which are used for the generation of log files. SADC_OPTIONS="-S DISK" # Compression program to use. ZIP="bzip2"
The cron job itself runs 2 tools sa1
and sa2
:
$ cat /etc/cron.d/sysstat # Run system activity accounting tool every 10 minutes */10 * * * * root /usr/lib64/sa/sa1 1 1 # 0 * * * * root /usr/lib64/sa/sa1 600 6 & # Generate a daily summary of process accounting at 23:53 53 23 * * * root /usr/lib64/sa/sa2 -A
Both of these cron jobs creates daily reports and store them in the /var/log/sa
directory. They cron jobs creates the following files:
- sa1 generates
/var/log/sa/sa{dd}
files. It also appends to this file every 10 minutes. This file gradually grows every 10 minutes until midnight. Then a new file is generated. - sa2 generates
/var/log/sa/sar{dd}
files, once per day
Where {dd} represents the day of the month:
$ date Wed Feb 28 18:35:15 UTC 2018 [root@target sa]# ls -lrt /var/log/sa/sa28 -rw-r--r--. 1 root root 18372 Feb 28 18:30 /var/log/sa/sa28
For more info about sa1 and sa2, see the man pages for these. sa1 essentially stores system statistics in special non-human-readable binary files, called ‘/var/log/sa/sa{dd}’:
Since sadd files are not human readable, you need to use the sadf tool query the content of the sadf file. E.g. to view the data between 6pm to 7pm on the 28th of the month, then we do:
$ [root@codingbee sa]# sadf -s 18:00:00 -e 19:00:00 /var/log/sa/sa28 sherc.sg-host.com -1 2018-02-28 18:20:21 UTC LINUX-RESTART sherc.sg-host.com 600 2018-02-28 18:40:01 UTC all %user 0.01 sherc.sg-host.com 600 2018-02-28 18:40:01 UTC all %nice 0.00 sherc.sg-host.com 600 2018-02-28 18:40:01 UTC all %system 0.02 sherc.sg-host.com 600 2018-02-28 18:40:01 UTC all %iowait 0.00 sherc.sg-host.com 600 2018-02-28 18:40:01 UTC all %steal 0.00 sherc.sg-host.com 600 2018-02-28 18:40:01 UTC all %idle 99.97 sherc.sg-host.com 600 2018-02-28 18:50:01 UTC all %user 0.01 sherc.sg-host.com 600 2018-02-28 18:50:01 UTC all %nice 0.00 sherc.sg-host.com 600 2018-02-28 18:50:01 UTC all %system 0.02 sherc.sg-host.com 600 2018-02-28 18:50:01 UTC all %iowait 0.00 sherc.sg-host.com 600 2018-02-28 18:50:01 UTC all %steal 0.00 sherc.sg-host.com 600 2018-02-28 18:50:01 UTC all %idle 99.98
sadf is actually a wrapper around another cli utility called sar (System Activity Report). Sar also comes with the sysstat package. See:
https://www.google.co.uk/search?q=sar+command&oq=sar+command&aqs=chrome..69i57j69i60j0l4.3023j0j4&sourceid=chrome&ie=UTF-8
Here are some other useful commands:
uptime (prints load averages)
du
df
free
top
ps
pstree
pgrep
other commands
htop
nmon
sysstat
dstat
netdata
sysdig
http://www.brendangregg.com/linuxperf.html
https://medium.com/netflix-techblog/netflix-at-velocity-2015-linux-performance-tools-51964ddb81cf
[post-content post_name=rhsca-quiz]
$ yum install dstat
$ yum install sysstat
$ dstat -c
$ dstat –output /tmp/dstat.csv
cronjob
– /var/log/sa/sa{dd}
– /var/log/sa/sar{dd}
binary files.
sadf
$ sadf -d -s 18:00:00 -e 19:00:00 /var/log/sa/sa28
# here we have specified a (d)elimited file with (s)tart and (e)nd date.
The sar command. you can also pass in sar command related options and switches by using ‘–‘, e.g.:
$ $ sadf -d -s 18:00:00 -e 19:00:00 /var/log/sa/sa28 -urd -n DEV