Leveraging `systemd-journald`, `journalctl`, and `logrotate` for Robust System Monitoring
On a CentOS 8.5 server, logs are not merely verbose output; they are the definitive record of every kernel event, application action, security incident, and system operation. For the modern system administrator, mastering log analysis is paramount for proactive system health monitoring, rapid troubleshooting, security auditing, and compliance. This module explores the sophisticated logging architecture of CentOS 8.5, focusing on its `systemd` integration and powerful tools.
Important Note on CentOS 8.5 EOL
While this guide focuses on CentOS 8.5 for educational purposes, please remember that CentOS Linux 8 reached End-of-Life (EOL) on December 31, 2021. For production environments, consider migrating to CentOS Stream 8 (which is an upstream for RHEL), Rocky Linux, AlmaLinux, or RHEL itself for continued support and security updates.
System Log Architecture: The `systemd-journald` Core
CentOS 8.5, like its RHEL counterparts, fully embraces `systemd` and its integral logging component, systemd-journald. This daemon provides a centralized, structured logging system that supplants the older, more fragmented `syslog` model for many services.
Advantages of `systemd-journald` in CentOS 8.5
- Universal Collection: Captures messages from the kernel, initrd, early boot, services, applications (via stdout/stderr), and traditional syslog messages (forwarded by `rsyslog`). This eliminates the need for services to manage their own log files, simplifying application development and deployment.
- Binary & Structured Storage: Logs are stored in a binary format, not plain text. This allows for rich metadata (timestamps, PIDs, UIDs, `systemd` unit names, cgroup information, SELinux context) to be associated with each log entry. This structure enables highly efficient indexing and searching, making `journalctl` significantly faster than `grep` on plain text logs.
- Reliability & Security: Journal files can be made immutable (`Seal=yes` in `journald.conf`) and cryptographically signed to prevent tampering, crucial for forensic analysis and compliance. They can also persist across reboots (default for CentOS 8.5 if `/var/log/journal` exists) or be volatile (stored in `/run/log/journal`, which is cleared on reboot).
- Resource Efficiency: `journald` efficiently manages disk space usage (`SystemMaxUse`, `SystemKeepFree`), can rate-limit messages to prevent log floods (`RateLimitInterval`, `RateLimitBurst`), and is designed to be performant, even under heavy logging loads.
`rsyslog` Coexistence and Remote Logging
While `journald` is the primary logging system, `rsyslog` (the traditional `syslog` daemon) still plays a vital role in CentOS 8.5. `journald` forwards all messages it collects to `rsyslog` by default, which then writes them to the conventional text files in /var/log (e.g., /var/log/messages, /var/log/secure, /var/log/boot.log). This ensures backward compatibility for applications expecting traditional log files and, more importantly, facilitates **network-based logging** (sending logs to a central syslog server for aggregation and analysis).
You can configure `rsyslog` behavior in /etc/rsyslog.conf to set up a centralized log server or client. For `journald` persistence, check /etc/systemd/journald.conf; the default `Storage=auto` typically means persistent if `/var/log/journal` exists, volatile otherwise.
Logging Events to the System (Implicitly and Explicitly)
Most system events are logged automatically. Services managed by `systemd` (which is almost all of them in CentOS 8.5) have their standard output (stdout) and standard error (stderr) captured directly by `journald`. Kernel messages are also automatically collected from the `printk` buffer.
Explicitly Logging from the Command Line or Scripts
System administrators and developers can explicitly log messages to the journal (and thus to `rsyslog`) using the `systemd-cat` command or the traditional `logger` utility. This is crucial for custom scripts, application debugging, or recording ad-hoc events.
# Example 1: Log a custom informational message with a specific tag
$ echo "Application starting up. Initializing modules." | systemd-cat -p info -t my_web_app
# Example 2: Log a warning message using the traditional 'logger' command
$ logger -p user.warn "Unauthorized access attempt detected from unknown source."
# Example 3: Log a critical message from a shell script, specifying the systemd unit as identifier
$ echo "CRITICAL ERROR: Database connection failed during nightly backup!" | systemd-cat -p crit -u backup_service.service
After running these, you can verify their presence in the journal:
$ journalctl -t my_web_app # Verify Example 1
$ journalctl -p warn _COMM=logger # Verify Example 2 (logger command itself is the comm)
$ journalctl -u backup_service.service -p crit # Verify Example 3
Understanding Log Priorities (Levels)
Logs have severity levels, often referred to as priorities. Knowing these helps in filtering and understanding urgency:
- 0: emerg (system unusable - e.g., kernel panic)
- 1: alert (action must be taken immediately - e.g., critical hardware error)
- 2: crit (critical conditions - e.g., disk full, core dump)
- 3: err (error conditions - e.g., application cannot perform a function)
- 4: warning (warning conditions - e.g., resource nearly exhausted)
- 5: notice (normal but significant condition - e.g., service restart, user login)
- 6: info (informational messages - e.g., routine operation status)
- 7: debug (debug-level messages - extensive information, usually for development)
You can specify these with `-p` or `--priority` in `journalctl`. Ranges like `-p err..crit` are also supported.
Collecting Information: Mastering `journalctl`
journalctl is your primary interface for querying the `systemd` journal. Its power lies in its ability to filter on a vast array of metadata fields, offering a level of precision and speed unmatched by traditional `grep` on plain text logs. Let's explore essential commands and advanced filtering for effective log investigation.
Basic Viewing and Real-time Monitoring
These commands are your entry points into the journal.
# Example 4: View all journal entries
$ journalctl
# Example 5: Follow the log in real-time (like 'tail -f')
$ journalctl -f
# Example 6: Follow logs specifically for the Nginx web server unit
$ journalctl -f -u nginx.service
Boot-Specific Analysis
`journalctl` retains logs from multiple boots, allowing you to easily investigate past system starts/crashes.
# Example 7: List all available boots with their IDs, timestamps, and uptime
$ journalctl --list-boots
# Example 8: Show logs from the current boot (most common for recent issues)
$ journalctl -b
# Example 9: Show logs from the previous boot
$ journalctl -b -1
# Example 10: Show logs from a specific boot ID (e.g., from `journalctl --list-boots`)
$ journalctl -b <boot_id>
Time-Based Filtering: Precision Investigation
Crucial for incident response and historical analysis. Supports various flexible formats for `--since` and `--until`.
# Example 11: Show logs from a specific date and time range
$ journalctl --since "2023-01-20 10:00:00" --until "2023-01-20 10:30:00"
# Example 12: Show logs from the last two hours
$ journalctl --since "2 hours ago"
# Example 13: Show all errors since yesterday
$ journalctl --since "yesterday" -p err
# Example 14: Show logs between two specific dates
$ journalctl --since "2023-08-01" --until "2023-08-07"
Filtering by `systemd` Unit, Process, and User
The structured nature of `journald` makes filtering by metadata extremely efficient.
# Example 15: What has the SSH service been doing?
$ journalctl -u sshd.service
Aug 07 10:15:23 server1 sshd[1024]: Server listening on 0.0.0.0 port 22.
Aug 07 11:30:05 server1 sshd[2048]: Accepted publickey for Mubasher from 192.168.1.100 port 54321
Aug 07 11:32:19 server1 sshd[2055]: Failed password for invalid user attacker from 1.2.3.4 port 12345
# Example 16: Show all messages from the 'firewalld' executable
$ journalctl _COMM=firewalld
# Example 17: Track all logs related to a specific Process ID (PID)
$ journalctl _PID=12345
# Example 18: Show all messages originating from the root user (UID 0)
$ journalctl _UID=0
# Example 19: Filter by a specific kernel message (e.g., related to network interface 'eth0')
$ journalctl -k | grep -i "eth0"
Output Formats for Analysis and Scripting
`journalctl` offers various output formats, useful for different analytical needs, including programmatic parsing.
-o short: The default, `syslog`-like output.-o verbose: Show all available fields for each entry.-o json: Output in JSON format, excellent for programmatic parsing.- `json-pretty`: Pretty-printed JSON.
-o cat: Show just the message field, without metadata.-o export: Binary format suitable for transfer to another system (can be imported via `journalctl --file=...`).
# Example 20: Output Nginx service logs in pretty JSON format
$ journalctl -u nginx.service -o json-pretty | head -n 20
# Example 21: Export logs from the last boot for forensic analysis on another machine
$ journalctl -b -o export > /tmp/last_boot_journal.export
The true power of `journalctl` comes from combining multiple filters.
# Show all critical errors from the sshd service in the last 24 hours
$ journalctl -u sshd.service -p crit..err --since "24 hours ago"
# Show all logs from user 'Mubasher' that happened after the last reboot
$ journalctl _UID=$(id -u Mubasher) -b
Controlling Journal Size and Retention Policies
By default, `journald` usually persists logs in `/var/log/journal`. You can configure its behavior in `/etc/systemd/journald.conf` to manage disk space and retention.
[Journal]
#Storage=persistent # Ensures logs are stored in /var/log/journal. 'auto' is default.
#SystemMaxUse=1G # Maximum disk space for system journal files, e.g., 500M, 1G
#SystemKeepFree=200M # Minimum free disk space to leave on the filesystem
#SystemMaxFileSize=100M # Maximum size of individual journal files
#ForwardToSyslog=yes # Ensures logs are sent to rsyslog (default)
#Compress=yes # Compress data in journal files (default)
To view current disk usage of the journal and explicitly clean old logs:
# Example 22: View current journal disk usage
$ journalctl --disk-usage
# Example 23: Reduce journal size to 1GB by deleting oldest entries
$ sudo journalctl --vacuum-size=1G
# Example 24: Remove logs older than 2 weeks
$ sudo journalctl --vacuum-time=2weeks
Log File Rotation: Managing Traditional Text Logs with `logrotate`
As mentioned, `rsyslog` continues to write to traditional text files in /var/log. To prevent these files from consuming all disk space, CentOS 8.5 utilizes the logrotate utility. This tool is configured to run daily (typically via `/etc/cron.daily/logrotate` or a `systemd` timer) and manages the archiving, compression, and deletion of old text log files.
How `logrotate` Operates
- Configuration: `logrotate` reads its main configuration from
/etc/logrotate.confand additional application-specific configurations from/etc/logrotate.d/. Each file in `logrotate.d` defines rules for specific applications or services (e.g., `httpd`, `sshd`, `mariadb`). - Rotation Criteria: Rules define when a log file should be rotated (e.g., `daily`, `weekly`, `monthly`, or `size 100M` when it reaches a certain size).
- Archiving: When rotated, the current log file (e.g.,
/var/log/secure) is renamed (e.g.,/var/log/secure.1or `secure-20230807`). - New Log Creation: A new, empty log file with the original name is created (e.g.,
/var/log/secure). - Service Notification (`postrotate` script): For services that cache their log file handles (like `rsyslog`, `httpd`), `logrotate` needs to signal the service to reopen its log files so it starts writing to the new empty file. This is typically done with a `postrotate` script block that reloads or restarts the service (e.g., `systemctl reload rsyslog.service`). The `sharedscripts` option prevents the script from running multiple times if multiple log files are rotated by the same block.
- Compression: Older rotated logs are often compressed (e.g., `secure.2.gz`) using `gzip` (default for `compress`) to save disk space.
- Deletion: After a configured number of rotations (`rotate 4` for 4 weeks) or age (`maxage 365`), the oldest log files are deleted.
Example `logrotate` Configuration (`/etc/logrotate.d/syslog`)
This snippet shows how common `syslog` files are managed, critical for system health and security auditing:
/var/log/messages
/var/log/secure
/var/log/maillog
/var/log/spooler
/var/log/boot.log
/var/log/cron
{
daily # Rotate daily
compress # Compress old logs
rotate 4 # Keep 4 rotated logs (plus the current one). Typically means 4 days if daily.
create 0600 root root # Create new log file with these permissions (owner root, group root, read/write for owner only)
# The 'sharedscripts' option is important here: if multiple logs in this block are rotated,
# the postrotate script will only run ONCE after all of them are processed.
sharedscripts
postrotate
# Sends a HUP signal to the old syslogd PID to tell it to reopen its log files.
# The '|| true' prevents the script from failing if syslogd.pid doesn't exist.
/bin/kill -HUP `cat /var/run/syslogd.pid 2> /dev/null` 2> /dev/null || true
# For modern systems, reloading the rsyslog service is the preferred way
/usr/bin/systemctl reload rsyslog.service >/dev/null 2>&1 || true
endscript
}
Manually Testing and Debugging `logrotate`
When creating or modifying `logrotate` configurations, always test them before relying on them in production.
# Example 25: Force `logrotate` to run all configured rotations (useful for testing)
$ sudo logrotate -f /etc/logrotate.conf
# Example 26: Run `logrotate` in debug mode. It shows what it *would* do without making any changes.
$ sudo logrotate -d /etc/logrotate.conf
# Example 27: Test a specific configuration file in debug mode
$ sudo logrotate -d /etc/logrotate.d/nginx
Scenario: You've noticed unusual SSH login attempts on your server. You need to investigate the failed attempts, identify the source IPs, and check for any suspicious user activity.
- Initial Triage: Identify Failed SSH Logins
Filter `journalctl` for `sshd.service` and look for messages indicating authentication failures.
You notice a high volume from a particular IP: `192.0.2.1`.$ journalctl -u sshd.service --since "1 day ago" | grep "Failed password" -i - Deep Dive: Analyze All Activity from a Suspicious IP
Now, query the journal for *all* entries originating from that suspicious IP address, across all services.
$ journalctl _SOURCE_ADDRESS=192.0.2.1 --since "2 hours ago"This reveals a pattern of failed logins for various users, and perhaps even some successful attempts if the attacker was lucky.
- Identify Attempts for Non-existent Users:
Attackers often try common usernames. Filter for failed attempts targeting non-existent users.
This helps distinguish brute-force attacks from legitimate users making typos.$ journalctl -u sshd.service --since "4 hours ago" | grep "invalid user" - Check for Successful Logins from Unexpected IPs/Users:
Cross-reference your findings with successful logins to ensure no compromise.
Look for logins from IPs that are not whitelisted or for users who shouldn't be logging in.$ journalctl -u sshd.service --since "1 day ago" | grep "Accepted publickey" $ journalctl -u sshd.service --since "1 day ago" | grep "Accepted password" - Contextual Analysis with Journal Fields:
If you find a suspicious `_PID`, you can trace its full activity. If a `_UID` logs in from a suspicious IP, check their subsequent actions.
# Example: Find all journal entries related to PID 12345 $ journalctl _PID=12345 # Example: See all activity for a specific user ID for the current boot $ journalctl _UID=1000 -b - Conclusion & Mitigation:
Based on the definitive evidence from `journalctl`, you can now formulate a mitigation strategy:
- Add the suspicious IP `192.0.2.1` to `firewalld`'s blacklist (or use `fail2ban`).
- Ensure `sshd` is configured to disallow password authentication (favoring SSH keys).
- Review `AllowUsers`, `AllowGroups`, `DenyUsers`, `DenyGroups` in `/etc/ssh/sshd_config`.
- If any successful unauthorized logins are found, immediately isolate the server, disable the compromised user, change all credentials, and perform a full forensic investigation.
This lab demonstrates how `journalctl` provides the granular detail necessary for robust security auditing and incident response on CentOS 8.5.