Overview
By the end of this article you should be able to answer the following questions:
Announcement
I have released my new course on Udemy, Kubernetes By Example. Sign up now to get free lifetime access!$ systemctl status rsyslog
– debug
– info
– notice
– warn (or warning)
– err
– crit
– alert
– emerg
/etc/rsyslog.conf
$ tail -F /var/log/messages
# or
$ tailf /var/log/messages
$ logger “hello world”
Intro
Nearly all services (e.g. httpd, vsftpd, and yum) records messages in their various log files. However they don’t write these logs entries directly to the associated log files (i.e. they don’t do “echo “xxx” >> log-file.txt”). Instead, these services send their log entries to an intermediary service called rsyslog (short for “remote system logger”). rsyslog then writes the log entries to the appropriate log file.
rsyslog actually runs as a service:
$ systemctl status rsyslog
rsyslog.service - System Logging Service
Loaded: loaded (/usr/lib/systemd/system/rsyslog.service; enabled)
Active: active (running) since Sun 2015-05-10 09:40:34 BST; 2h 6min ago
Main PID: 638 (rsyslogd)
CGroup: /system.slice/rsyslog.service
└─638 /usr/sbin/rsyslogd -n
May 10 09:40:34 localhost.localdomain systemd[1]: Started System Logging Service.
For rsyslog to work, each message is sent to rsyslog with a priority label. Here are the priority levels:
- debug
- info
- notice
- warn (or warning)
- err
- crit
- alert
- emerg
This comes in useful when we come to look at the rsyslog’s main config file, which is:
$ cat /etc/rsyslog.conf # rsyslog configuration file # For more information see /usr/share/doc/rsyslog-*/rsyslog_conf.html # If you experience problems, see http://www.rsyslog.com/doc/troubleshoot.html #### MODULES #### # The imjournal module bellow is now used as a message source instead of imuxsock. $ModLoad imuxsock # provides support for local system logging (e.g. via logger command) $ModLoad imjournal # provides access to the systemd journal #$ModLoad imklog # reads kernel messages (the same are read from journald) #$ModLoad immark # provides --MARK-- message capability # Provides UDP syslog reception #$ModLoad imudp #$UDPServerRun 514 # Provides TCP syslog reception #$ModLoad imtcp #$InputTCPServerRun 514 #### GLOBAL DIRECTIVES #### # Where to place auxiliary files $WorkDirectory /var/lib/rsyslog # Use default timestamp format $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat # File syncing capability is disabled by default. This feature is usually not required, # not useful and an extreme performance hit #$ActionFileEnableSync on # Include all config files in /etc/rsyslog.d/ $IncludeConfig /etc/rsyslog.d/*.conf # this is important. # Turn off message reception via local log socket; # local messages are retrieved through imjournal now. $OmitLocalLogging on # File to store the position in the journal $IMJournalStateFile imjournal.state #### RULES #### # Log all kernel messages to the console. # Logging much else clutters up the screen. #kern.* /dev/console # Log anything (except mail) of level info or higher. # Don't log private authentication messages! *.info;mail.none;authpriv.none;cron.none /var/log/messages # The authpriv file has restricted access. authpriv.* /var/log/secure # Log all the mail messages in one place. mail.* -/var/log/maillog # Log cron stuff cron.* /var/log/cron # Everybody gets emergency messages *.emerg :omusrmsg:* # Save news errors of level crit and higher in a special file. uucp,news.crit /var/log/spooler # Save boot messages also to boot.log local7.* /var/log/boot.log # ### begin forwarding rule ### # The statement between the begin ... end define a SINGLE forwarding # rule. They belong together, do NOT split them. If you create multiple # forwarding rules, duplicate the whole block! # Remote Logging (we use TCP for reliable delivery) # # An on-disk queue is created for this action. If the remote host is # down, messages are spooled to disk and sent when it is up again. #$ActionQueueFileName fwdRule1 # unique name prefix for spool files #$ActionQueueMaxDiskSpace 1g # 1gb space limit (use as much as possible) #$ActionQueueSaveOnShutdown on # save messages to disk on shutdown #$ActionQueueType LinkedList # run asynchronously #$ActionResumeRetryCount -1 # infinite retries if host is down # remote host is: name/ip:port, e.g. 192.168.0.1:514, port optional #*.* @@remote-host:514 # ### end of the forwarding rule ###
Let’s take a look at parts of this file, first we have:
. . # Include all config files in /etc/rsyslog.d/ $IncludeConfig /etc/rsyslog.d/*.conf . .
This rsyslog.d directory that can hold other rsyslog related config files:
$ pwd /etc/rsyslog.d $ ls -l total 8 -rw-r--r--. 1 root root 2564 Sep 19 2014 gluster.conf.example -rw-r--r--. 1 root root 49 Mar 26 13:03 listen.conf
Next we have the following really important section:
. . # Log anything (except mail) of level info or higher. # Don't log private authentication messages! *.info;mail.none;authpriv.none;cron.none /var/log/messages . .
In plain English, the above line says: log ALL messages of priority “info” and above (*.info), but filter out all “mail” related messages (mail.none), and all “news” related messages (news.none), and all authentication messages (authpriv.none), and all “cron” messages (cron.none). As you can see, nearly all messages gets sent to /var/log/messages.
The messages that have been filtered out, are actually logged elsewhere, as indicated by the following extract:
. . # The authpriv file has restricted access. authpriv.* /var/log/secure # Log all the mail messages in one place. mail.* -/var/log/maillog # Log cron stuff cron.* /var/log/cron . .
The “-” in -/var/log/maillog means that the log entries can temporarily reside in a buffer if necessary.
Finally we have:
# Everybody gets emergency messages *.emerg :omusrmsg:*
Here a special “omusrmsg” plugin is being used so that any emergency message is appended to ALL log files.
One final thing to remember, if you do make any changes to the above config files, then you need to restart the rsyslog service for the changes to take into effect:
systemctl status rsyslog
rsyslog.service - System Logging Service
Loaded: loaded (/usr/lib/systemd/system/rsyslog.service; enabled)
Active: active (running) since Sun 2015-05-10 09:40:34 BST; 4h 43min ago
Main PID: 638 (rsyslogd)
CGroup: /system.slice/rsyslog.service
└─638 /usr/sbin/rsyslogd -n
May 10 09:40:34 localhost.localdomain systemd[1]: Started System Logging Service.
You can monitor log entries in realtime using the tail command (with the follow flag):
$ tail -F /var/log/messages
Or you could do:
$ tailf /var/log/messages
Another option you can do is:
$ vim /var/log/messages
The benefit of this approach is that vim will intelligently use syntax highlighting to highlight error messages.
If you have multiple log files, then you can view them all in real time like this:
$ tail -F /var/log/httpd/*.log ==> /var/log/httpd/default_error.log <== ==> /var/log/httpd/foreman_access.log <== 127.0.0.1 - - [19/Sep/2015:21:05:02 +0100] "HEAD / HTTP/1.1" 301 - "-" "-" 127.0.0.1 - - [19/Sep/2015:21:05:01 +0100] "HEAD / HTTP/1.1" 301 - "-" "-" 127.0.0.1 - - [25/Sep/2015:22:03:40 +0100] "HEAD / HTTP/1.1" 500 - "-" "-" 127.0.0.1 - - [25/Sep/2015:22:07:12 +0100] "HEAD / HTTP/1.1" 301 - "-" "-" 127.0.0.1 - - [26/Sep/2015:08:17:29 +0100] "HEAD / HTTP/1.1" 301 - "-" "-" 127.0.0.1 - - [26/Sep/2015:08:17:30 +0100] "HEAD / HTTP/1.1" 301 - "-" "-" 127.0.0.1 - - [26/Sep/2015:08:22:10 +0100] "HEAD / HTTP/1.1" 301 - "-" "-" 127.0.0.1 - - [26/Sep/2015:08:22:10 +0100] "HEAD / HTTP/1.1" 301 - "-" "-" 127.0.0.1 - - [26/Sep/2015:16:35:43 +0100] "HEAD / HTTP/1.1" 301 - "-" "-" 127.0.0.1 - - [26/Sep/2015:16:35:43 +0100] "HEAD / HTTP/1.1" 301 - "-" "-" ==> /var/log/httpd/foreman_error.log <== ==> /var/log/httpd/foreman-ssl_access_ssl.log <== 127.0.0.1 - - [26/Sep/2015:16:36:18 +0100] "GET /node/puppetmaster.local?format=yml HTTP/1.1" 200 321 "-" "Ruby" 127.0.0.1 - - [26/Sep/2015:16:36:20 +0100] "POST /api/reports HTTP/1.1" 201 511 "-" "Ruby" 127.0.0.1 - - [26/Sep/2015:17:05:49 +0100] "GET /node/puppetmaster.local?format=yml HTTP/1.1" 200 321 "-" "Ruby" 127.0.0.1 - - [26/Sep/2015:17:05:51 +0100] "POST /api/hosts/facts HTTP/1.1" 201 773 "-" "Ruby" 127.0.0.1 - - [26/Sep/2015:17:05:51 +0100] "GET /node/puppetmaster.local?format=yml HTTP/1.1" 200 321 "-" "Ruby" 127.0.0.1 - - [26/Sep/2015:17:05:52 +0100] "POST /api/reports HTTP/1.1" 201 518 "-" "Ruby" 127.0.0.1 - - [26/Sep/2015:17:35:48 +0100] "GET /node/puppetmaster.local?format=yml HTTP/1.1" 200 321 "-" "Ruby" 127.0.0.1 - - [26/Sep/2015:17:35:50 +0100] "POST /api/hosts/facts HTTP/1.1" 201 773 "-" "Ruby" 127.0.0.1 - - [26/Sep/2015:17:35:50 +0100] "GET /node/puppetmaster.local?format=yml HTTP/1.1" 200 321 "-" "Ruby" 127.0.0.1 - - [26/Sep/2015:17:35:51 +0100] "POST /api/reports HTTP/1.1" 201 510 "-" "Ruby" ==> /var/log/httpd/foreman-ssl_error_ssl.log <== ==> /var/log/httpd/puppet_access_ssl.log <== 127.0.0.1 - - [26/Sep/2015:17:05:48 +0100] "GET /production/node/puppetmaster.local?transaction_uuid=ce413f09-b8be-4585-91f1-2395f72fa880&fail_on_404=true HTTP/1.1" 200 5010 "-" "Ruby" 127.0.0.1 - - [26/Sep/2015:17:05:50 +0100] "GET /production/file_metadatas/pluginfacts?links=manage&recurse=true&ignore=.svn&ignore=CVS&ignore=.git&checksum_type=md5 HTTP/1.1" 200 303 "-" "Ruby" 127.0.0.1 - - [26/Sep/2015:17:05:50 +0100] "GET /production/file_metadatas/plugins?links=manage&recurse=true&ignore=.svn&ignore=CVS&ignore=.git&checksum_type=md5 HTTP/1.1" 200 303 "-" "Ruby" 127.0.0.1 - - [26/Sep/2015:17:05:51 +0100] "POST /production/catalog/puppetmaster.local HTTP/1.1" 200 579 "-" "Ruby"
Note: tailf can't handle multiple log files.
Finally you can add your own custom log entries to /var/log/messages using the logger command. For example to add an entry with the message "hello world":
$ logger "hello world"
Now let's confirm that this has worked:
$ tail -3 /var/log/messages Nov 11 20:10:01 puppetmaster systemd: Starting Session 17 of user root. Nov 11 20:10:01 puppetmaster systemd: Started Session 17 of user root. Nov 11 20:13:36 puppetmaster root: hello world