Investigating System Anomalies using journalctl and /var/log/auth.log
When a Hovixa VPS behaves unpredictably—whether it's a sudden service crash, an unauthorized login attempt, or a mysterious performance dip—the solution is always encoded in the system logs. Modern Linux systems use a dual-logging approach: systemd journal (binary logs managed by journalctl) and traditional text logs (located in /var/log/). Analyzing these in tandem is critical for accurate incident response.
1. Parsing traditional logs: /var/log/auth.log
On Debian and Ubuntu systems, /var/log/auth.log is the authoritative record for all security-related events, including SSH logins, sudo executions, and user creation. This is your first stop when investigating a potential security anomaly.
Key Investigation Commands:
- Search for failed logins:
grep "Failed password" /var/log/auth.log - Monitor logins in real-time:
tail -f /var/log/auth.log - Identify suspicious sudo usage:
grep "COMMAND=" /var/log/auth.log
2. Mastering journalctl for Service Anomalies
journalctl queries the binary logs collected by systemd-journald. Unlike text logs, these include metadata that allows you to filter by specific service, priority level, or time range without complex sed or awk scripts.
Advanced Filtering Examples:
# 1. View logs for a specific service since a specific time
journalctl -u nginx --since "2026-02-27 12:00:00"
# 2. Filter for critical errors only (Priority 3 and below)
journalctl -p err..emerg
# 3. View logs for the current boot only
journalctl -b
3. Correlating Time and Events
Anomalies are often multi-faceted. If Nginx crashed at 14:05, you should check journalctl for service errors and auth.log for concurrent SSH sessions to see if a configuration change caused the failure.
| Anomalous Event | Primary Log Source | What to Look For |
|---|---|---|
| Service Core Dump | `journalctl -u [service]` | Segmentation faults or out-of-memory (OOM) kills. |
| Unauthorized Access | `/var/log/auth.log` | Unexpected IPs or logins during non-work hours. |
| Kernel Panics / Hardware | `dmesg` or `journalctl -k` | I/O errors, NVMe timeouts, or driver failures. |
| General System Instability | `/var/log/syslog` | Cron job failures or disk space warnings. |
4. Technical Implementation Details
- Binary vs. Text: The journal is stored in a structured binary format, making it much faster to search than flat text files. However,
/var/log/auth.logis still used because many legacy security tools (like Fail2Ban) are designed to parse plain text. - Persistent Journals: By default, some distributions store the journal in RAM (
/run/log/journal/), meaning logs are lost on reboot. EnsureStorage=persistentis set in/etc/systemd/journald.confon your Hovixa VPS to preserve history. - Log Rotation: Check
/etc/logrotate.d/to ensure your text logs are being compressed and archived. This prevents/var/log/from consuming your NVMe storage capacity.
Sysadmin Advice: Use journalctl -f while you are performing a complex task like a WordPress migration or a LEMP upgrade. Watching the logs in real-time allows you to catch and fix minor "Warning" messages before they escalate into "Critical" system failures.
