Logging

Core Tools

Pithy

1. journald - collect and index local logs (binary format)
2. syslog - process, filter, route logs (local or remote)
3. logrotate - manage retention (compress, delete old files)

Introduction

Daemons, the kernel and applications all emit log data that accumulates on finite disks. Logs have limited useful life and may need to be queried, filtered, analyzed, compressed, archived and eventually discarded.

Tip

Logs are the first place you should look when a daemon crashes or refuses to start or when an error plagues a system that is trying to boot.

Log management breaks down into three subtasks:

1. Collecting logs from varied sources
2. Querying, filtering and monitoring messages
3. Managing retention and expiration, keeping data as long as it’s useful or legally required, but no longer

Historically, UNIX handled this through syslog, which gives applications a standard interface for submitting messages but only addresses collection. This is why many programs bypass syslog and write their own ad hoc log files.

systemd journal is a second attempt at sanity (ps it’s way better): it collects messages, stores them in an indexed compressed binary format and provides a CLI for viewing and filtering. It can run standalone or alongside syslog.

Log file locations

• Most log files are found in /var/log, but some applications write their log files elsewhere.
• Log files are generally owned by root, although conventions for the ownership and mode of log files vary.

Key	Values
Where	F = Configuration file, H = Hardwired, S = Syslog
Frequency of logrotate	D = Daily, M = Monthly, NNm = Size-based (in MB, e.g., 1m), W = Weekly
Systems	D = Debian and Ubuntu (D* = Debian only), R = Red Hat and CentOS, F = FreeBSD

a. passwd, sshd login and shutdown also write to the authorization log.
b. Must be read with the faillog utility
c. lastlog2 on Debian, last login is interactive logins only (not daemons)

Some helpful commands

lastlog # reports the most recent login of all users or of a given user
last

Systemd journald

Pithy

journald sits in an event loop, when a message arrives it parses and enriches it based on its source before writing to the journal.

Blog on journald

Journald

Unlike syslog, which typically saves log messages to plain text files, the systemd journal stores messages in a binary format. All message attributes are indexed automatically, which makes the log easier and faster to search.

The journal collects and indexes messages from several sources:

• /dev/log syslog socket
• /dev/kmsg kernel messages (replaces klogd)
• /run/systemd/journal/stdout stdout
• /run/systemd/journal/socket native journal API
• kernel audit subsystem

To forward these logs to another system use one of the following:

• systemd-journal-remote
• 3rd party alloy/fluent bit
• old school syslog forwarding

Journal config

Default/base config is stored /etc/systemd/journald.conf, custom configs should go in /etc/systemd/journald.conf.d/*.conf. The base file contains commented out version of every possible option along with their default values.

A few settings

[Journal]
Storage=auto         # writes to /var/log/journal for persistance ONLY if it exists
Storage=persistent   # EXPLICITLY ensure logs survive reboots

SystemMaxUse=2G      # Cap total journal size
MaxRentionSec=3month # Time based pruning regardless of size
MaxFileSec=1week     # How often the active journal file is rotated
ForwardToSylog=no    # If running alongside syslog forwarding remotely

Journalctl

For Linux distributions running systemd, the quickest and easiest way to view logs is to use the journalctl.

Commands

journalctl -u ssh # view specific unit
journalctl -f  # live
journalctl --disk-usage # may need to run with sudo for full
journalctl -b 0 -u ssh # ssh logs in current boot
journalctl --since=yesterday --until=now
journalctl -n 100 /usr/sbin/sshd # last 100 from specific binary

Syslog

Note

syslog is a protocol and concept for how to structure log messages and transport them.
rsyslog (rocket-fast syslog) is a specific implementation and is default syslog daemon on most Linux distros.

Before syslog was defined every program was free to make up its own logging policy.

Think about log messages as a stream of events and rsyslog as an event-stream processing engine. Log message “events” are submitted as inputs, processed by filters and forwarded to output destinations (files, terminals, other machines).

• The rsyslogd process typically starts at boot and runs continuously.
• Programs that are syslog aware write log entries to the special file /dev/log, a UNIX domain socket.

rsyslog configuration test

1. My example system ubuntu 24 has supplant the rsyslogd socket with a symlink to systemd’s journald, to centralize logging, ingestion and storage.

   $ ls -l /dev/log
   lrwxrwxrwx 1 root root 28 May 26 08:23 /dev/log -> /run/systemd/journal/dev-log
   $ ls -l /run/systemd/journal/dev-log 
   srw-rw-rw- 1 root root 0 May 26 08:23 /run/systemd/journal/dev-log

   Checking the systemd status of the service it is running - rsyslogd continues to run as a daemon for additional filtering and processing.

2. Let’s check what rsyslogd is doing, no freeloaders!

   #/etc/rsyslog.conf
   
   # global  configs are here..... 
   # .....[snip].....
   # Include all config files in /etc/rsyslog.d/
   #
   $IncludeConfig /etc/rsyslog.d/*.conf

   As expected rsyslog is pointing to a folder of configs /etc/rsyslog.d/*.conf

   #/etc/rsyslog.d/50-default.conf
   
   # Looking at this first rule, filting the facility (source) field that matches auth
   # or authpriv* and it's forwarding to the file /var/log/auth.log
   auth,authpriv.*                 /var/log/auth.log

   So it is doing something!

3. Let’s try making a change

   sudo vim /etc/rsyslog.d/50-default.conf
   
   # Let's forward the logs syslog filters to a different path
   auth,authpriv.*                 /var/log/spongebob.log
   
   # Reload rsyslog's config
   sudo systemctl restart rsyslog
   
   # And boom!
   $ cat /var/log/spongebob.log 
   2026-06-04T21:15:59.529208-04:00 k3s sudo: pam_unix(sudo:session): session closed for user root
   2026-06-04T21:16:03.960824-04:00 k3s sshd[1118358]: Received disconnect from 192.168.2.11 port 60762:11: disconnected by user
   2026-06-04T21:16:03.960992-04:00 k3s sshd[1118358]: Disconnected from user mat 192.168.2.11 port 60762

rsyslog configurations

Formats

Rsyslog has 3 syntaxes

# sysklogd format (original/simplest), implicit:
auth,authpriv.*                 /var/log/auth.log

# RainerScript (modern), explicit
auth,authpriv.* action(type="omfile" file="/var/log/auth.log")

# Legacy rsyslog directives ($ syntax)
# Primarily for global config and module loading not filter/action rules

Modules

• modules are written C, they parse and can mutate messages
• modules follow the naming convention:
  • im* input modules
  • om* output modules
  • mm* message modifiers

Examples of modules are:

• imjournal systemd journal
• imfile converts a plain text file to syslog format
• imtcp & imudp accept messages over tcp and udp, useful
• immark module produces timestamp messages at regular intervals, this can help you figure out that your machine crashed between random hours not just “some time last night”
• omfile write messages to a file, most commonly used
• omfwd forwards message to a remote syslog server (tcp/udp) used for centralized logging
• ommysql send messages to a MySQL database

Config Examples

Simple

This reads from a log file and stores to another log file, contrived by a good sanity test before sending elsewhere.

# /etc/rsyslog.d/55-kubeview.conf  
module(load="imfile" mode="inotify")
input(
    type="imfile"
    Tag="kubeview"
    File="/var/log/kubeview.log"
    Severity="info"
)
if $programname startswith "kubeview" then {
  action(type="omfile" file="/var/log/kubeview-out.log")
}

TODO: send to a central server

https://docs.rsyslog.com/doc/configuration/modules/idx_output.html

Rsyslog TLS

Rsyslog can send and receive log messages over TLS on port 6514.

Kernel and boot-time logging

For kernel logging at boot time, kernel log entries are stored in an internal buffer of limited size. The buffer is large enough to accommodate messages about all the kernel’s boot-time activities. When the system is up and running, a user process accesses the kernel’s log buffer and disposes of its contents.

On Linux systems, systemd-journald reads kernel messages from the kernel buffer by reading the device file /dev/kmsg.

# View the kernel and boot logs
journalctl -k or # --dmesg.
sudo dmesg

Log management and rotation

Logrotate

Pithy

Logrotate is a program that has builtin functions to Target and handle logs.
The core goal prevent a single log file from growing infinitely, instead compress and later delete (configurable)

• logroute is normally run using cron once a day.

• /etc/logrotate.conf standard cfg is here,

• /etc/logrotate.d the canonical place for logrotate config files.

• logrotate-aware software packages (there are many) can drop in log management instructions as part of their installation procedure, thus greatly simplifying administration.

Tip

You could write custom logic to handle your logs using a systemd timer or a cron job… but why would you? logrotate implements all the typical things you’d want to do with a log and it’s easier for others to understand how they’re being managed.

Logrotate commands

Real Example

1. I want to stop my kubeview log from getting so big, 1KB is too big!

   # /etc/logrotate.d/kubeview
   
   /var/log/kubeview.log {
       size 1K   # rotate once log hits 1KB - note: must be UPPERCASE
       rotate 3  # keep logs from last 3 rotates
       compress  # compress all noncurrent version of the log file
   }
   
   # Note for some reason the comments broke the logrotate parser, so if you're having
   # issues try remove the comments

   Note

   We’re not specifying a date like daily,weekly,monthly instead we’re doing a size. logrotate is run on a cronjob /etc/cron.daily/logrotate so it will check the size on that schedule, no need for a time and a size be specified.

2. Let’s test if my logrotate config is working

   # -d = "debug", it won't actually run rather it will print what would have done
   sudo logrotate -d /etc/logrotate.d/kubeview
   
   # unimportant details are omitted
   reading config file /etc/logrotate.d/kubeview
   Handling 1 logs                                                   # 1 log file match my config
   rotating pattern: /var/log/kubeview.log  1024 bytes (3 rotations) # Rotate at 1024 bytes (1KB), keep 3 old compressed versions
   considering log /var/log/kubeview.log                             # about to check if rotation is needed
   Last rotated at 2026-06-06 00:00                                  # logrotate last ran at midnight
   log needs rotating                                                # the logic triggered to rotate the log (size > 1KB)
   
   renaming /var/log/kubeview.log.1.gz to /var/log/kubeview.log.2.gz # casecade, everything shifts down one
   renaming /var/log/kubeview.log.2.gz to /var/log/kubeview.log.3.gz # rotate only keeps 3 so 4 gets deleted
   renaming /var/log/kubeview.log.3.gz to /var/log/kubeview.log.4.gz
   log /var/log/kubeview.log.4.gz doesn't exist -- won't try to dispose of it 
   
   renaming /var/log/kubeview.log to /var/log/kubeview.log.1         # current log becomes .1
   compressing log with: /bin/gzip                                   # kubeview.log.1 get gzipped to kubeview.log.1.gz

3. Looks good to me, now let’s manually run it to see what it will actually do when cron takes over

   # -f forces the logrotate config to trigger
   $ sudo logrotate -f /etc/logrotate.d/kubeview
   
   $ ls -lh /var/log/kubeview.log*              
   -rw-r--r-- 1 root root  591 Jun  6 11:08 /var/log/kubeview.log.1.gz
   -rw-r--r-- 1 root root 1.1K Jun  5 11:13 /var/log/kubeview.log.2.gz
   $ sudo systemctl restart kubeview # triggering some logs
   $ ls -lh /var/log/kubeview.log*   # the new log file is created and the 2 compressed logs exist
   -rw-r--r-- 1 root root  755 Jun  6 11:10 /var/log/kubeview.log
   -rw-r--r-- 1 root root  591 Jun  6 11:08 /var/log/kubeview.log.1.gz
   -rw-r--r-- 1 root root 1.1K Jun  5 11:13 /var/log/kubeview.log.2.gz

Logs at Scale!

ELK

Elasticsearch is a scalable database and search engine with a RESTful API for querying data
Logstash accepts data from many sources. It can also read data directly from TCP or UDP sockets and from the traditional syslog. Once messages have been ingested, Logstash can write them to a wide variety of destinations, including, of course, Elasticsearch.
Filebeat also can ship logs to logstash or Elasticsearch in a similar way to syslog. Kibana is the frontend

LaaS - logging as a service

Splunk is the most mature and trusted, on-prem and hosted.
AWS CloudWatch can collect logs from AWS services and from your apps.

Log Policies / Centralization

Keep these questions in mind when designing your logging strategy:

• How many systems and applications will be included?
• What type of storage infrastructure is available?
• How long must logs be retained?
• What types of events are important?

For most applications, consider capturing at least the following information:

• Username or user ID
• Event success or failure
• Source address for network events
• Date and time
• Sensitive data added, altered, or removed
• Event details

These log warehouse systems have no real role in the organization’s daily business beyond satisfying audibility requirements

Centralization takes work and at smaller sites it may not represent a net benefit.

• Twenty servers as a reasonable threshold for considering centralization.
• Below that size, just ensure that logs are rotated properly and are archived frequently enough to avoid filling up a disk.