Chapter 7
7. Logging
Learn about system logs and the /var/log directory.
Subsections of 7. Logging
1. System Logging
Lesson Content
The services, kernel, daemons, etc on your system are constantly doing something, this data is actually sent to be saved on your system in the form of logs. This allows us to have a human readable journal of the events that are happening on our system. This data is usually kept in the /var directory, the /var directory is where we keep our variable data, such as logs!
How are these messages even getting received on your system? There is a service called syslog that sends this information to the system logger.
Syslog actually contains many components, one of the important ones is a daemon running called syslogd (newer Linux distributions use rsyslogd), that waits for event messages to occur and filter the ones it wants to know about, and depending on what it’s supposed to do with that message, it will send it to a file, your console or do nothing with it.
You would think that this system logger is the centralized place to manage logs, but unfortunately it’s not. You’ll see many applications that write their own logging rules and generate different log files, however in general the format of logs should include a timestamp and the event details.
Here is an example of a line from syslog:
pete@icebox:~$ less /var/log/syslog
Jan 27 07:41:32 icebox anacron[4650]: Job `cron.weekly' startedHere we can see that at Jan 27 07:41:32 our cron service ran the cron.weekly job. You can view all the event messages that syslog collects with in the /var/log/syslog file.
Exercise
Look at your /var/log/syslog file and see what else is happening on your machine.
Quiz Question
# What is the daemon that manages log on newer Linux systems?
> The syslog daemon is a server process that provides a message logging facility for application and system processes.
1. [ ] lslog
2. [ ] initlog
3. [ ] syslog
4. [x] rsyslogd
2. syslog
Lesson Content
The syslog service manages and sends logs to the system logger. Rsyslog is an advanced version of syslog, most Linux distributions should be using this new version. The output of all the logs the syslog service collects can be found at /var/log/syslog (every message except auth messages).
To find out what files are maintained by our system logger, look at the configuration files in /etc/rsyslog.d:
pete@icebox:~$ less /etc/rsyslog.d/50-default.conf
# First some standard log files. Log by facility.
#
auth,authpriv.* /var/log/auth.log
*.*;auth,authpriv.none -/var/log/syslog
#cron.* /var/log/cron.log
#daemon.* -/var/log/daemon.log
kern.* -/var/log/kern.log
#lpr.* -/var/log/lpr.log
mail.* -/var/log/mail.log
#user.* -/var/log/user.logThese rules to log files are denoted by the selector on the left column and the action on the right column. The action tells us where to send the log information, in a file, console, etc. Remember not every application and service uses rsyslog to manage their logs, so if you want to know specifically what is logged you’ll have to look inside this directory.
Let’s actually see logging in action, you can manually send a log with the logger command:
logger -s HelloNow look inside your /var/log/syslog and you should see this entry in your logs!
Exercise
Look at your /etc/rsyslog.d configuration file and see what else is being logged via the system logger.
Quiz Question
# What command can you use to manually log a message?
> Just type logger `````` on the command line and your message will be added to the end of the /var/log/syslog file.
1. [ ] syslog
2. [ ] log
3. [ ] fuser
4. [x] logger
3. General Logging
Lesson Content
There are many log files you can view on your system, many important ones can be found under /var/log. We won’t go through them all, but we’ll discuss a couple of the major ones.
There are two general log files you can view to get a glimpse of what your system is doing:
/var/log/messages
This log contains all non-critical and non-debug messages, includes messages logged during bootup (dmesg), auth, cron, daemon, etc. Very useful to get a glimpse of how your machine is acting.
/var/log/syslog
This logs everything except auth messages, it’s extremely useful for debugging errors on your machine.
These two logs should be more than enough when troubleshooting issues with your system, However, if you just want to view a specific log component, there are also separate logs for those as well.
Exercise
Look at your /var/log/messages and /var/log/syslog files and see what the differences are.
Quiz Question
# What log file logs everything except auth messages?
> /var/log. This is such a crucial folder on your Linux systems. Open up a terminal window and issue the command cd /var/log. Now issue the command ls and you will see the logs housed within this directory.
1. [ ] log
2. [ ] init
3. [ ] logger
4. [x] syslog
4. Kernel Logging
Lesson Content
/var/log/dmesg On boot-time your system logs information about the kernel ring buffer. This shows us information about hardware drivers, kernel information and status during bootup and more. This log file can be found at /var/log/dmesg and gets reset on every boot, you may not actually see any use in it now, but if you were to ever have issues with something during bootup or a hardware issue, dmesg is the best place to look. You can also view this log using the dmesg command.
/var/log/kern.log Another log you can use to view kernel information is the /var/log/kern.log file, this logs the kernel information and events on your system, it also logs dmesg output.
Exercise
Look at your dmesg and kern logs, what differences do you notice?
Quiz Question
# What command can be used to view kernel bootup messages?
> While the dmesg command can display the entire contents of the Linux kernel message buffer there are ways to have it choose just what you want to see. The dmesg command displays the content of the kernel's message buffer since the system's most recent boot.
1. [ ] auth.log
2. [ ] sys.log
3. [ ] kern.log
4. [x] dmesg
5. Authentication Logging
Lesson Content
Authentication logging can be very useful to look at if you are having issues logging in.
/var/log/auth.log
This contains system authorization logs, such as user login and the authentication method used.
Sample snippet:
Jan 31 10:37:50 icebox pkexec: pam_unix(polkit-1:session): session opened for user root by (uid=1000)Exercise
Do some failed logins and then a successful one, look at your /var/log/auth.log and see what happened.
Quiz Question
# What log is used for user authentication?
> /var/log/auth. log – Contains system authorization information, including user logins and authentication machinsm that were used.
1. [ ] boot.log
2. [ ] sys.log
3. [ ] kern.log
4. [x] auth.log
6. Managing Log Files
Lesson Content
Log files generate lots of data and they store this data on your hard disks, however there are lots of issues with this, for the most part we just want to be able to see newer logs, we also want to manage our disk space efficiently, so how do we do all of this? The answer is with logrotate.
The logrotate utility does log management for us. It has a configuration file that allows us to specify how many and what logs to keep, how to compress our logs to save space and more. The logrotate tool is usually run out of cron once a day and the configuration files can be found in /etc/logrotate.d.
There are other logrotating tools you can use to manage your logs, but logrotate is the most common one.
Exercise
Look at your logrotate configuration file and see how it manages some of your logs.
Quiz Question
# What utility is used to manage logs?
> A key best practice for logging is to centralize or aggregate your logs in a single location, especially if you have multiple servers or architecture tiers. Modern applications often have several tiers of infrastructure that can include a mix of on-premises servers and cloud services.
1. [ ] man logger
2. [ ] logger
3. [ ] log
4. [x] logrotate