Reading and Analyzing Linux Logs

Why Logs Matter

#

When something breaks on a Linux system — a service won’t start, a request fails, the machine reboots unexpectedly — logs are where the answer lives. Learning to read them quickly is the difference between fixing a problem in minutes and guessing for hours.

Modern Linux systems keep logs in two places:

• The systemd journal — a structured, binary log managed by journald, queried with journalctl. This is where most service and system logs go on current distributions.
• Traditional text files in /var/log — plain-text logs written by applications and, on some systems, by a syslog daemon like rsyslog.

You need to be comfortable with both. This post covers where logs live, the tools to read them, and the patterns you’ll actually use when diagnosing a problem.

Where Logs Live

#

The /var/log directory is the traditional home for log files:

What’s present varies by distribution and installed software. Have a look at what’s on your system:

ls -lh /var/log

The -h gives human-readable file sizes, which helps you spot logs that are growing out of control.

Reading the Systemd Journal

#

On systemd-based distributions (nearly all modern ones), journalctl is the primary tool. It reads the structured journal that captures output from services, the kernel, and the boot process.

The basics

#

# All journal entries, oldest first (opens in a pager)
journalctl

# Jump to the end (most recent entries)
journalctl -e

# Follow new entries in real time — like tail -f
journalctl -f

By default journalctl pipes through less, so you can search with /, scroll, and quit with q.

Filter by service

#

This is the most common use — you want the logs for one specific service:

# All logs for the nginx service
journalctl -u nginx

# Follow a service's logs live
journalctl -u nginx -f

# Multiple services at once
journalctl -u nginx -u php-fpm

Filter by time

#

# Since a relative time
journalctl --since "1 hour ago"
journalctl --since "10 min ago"

# Since an absolute time
journalctl --since "2026-09-06 14:00:00"

# A specific window
journalctl --since "2026-09-06 14:00:00" --until "2026-09-06 15:00:00"

# Today only
journalctl --since today

Filter by priority

#

Log entries carry a severity level. Filter to just the important ones:

# Errors and worse
journalctl -p err

# A specific service, errors only
journalctl -u nginx -p err

Priority levels, from most to least severe:

Specifying -p err includes everything at that level and above (more severe).

Filter by boot

#

# Logs from the current boot
journalctl -b

# List all recorded boots
journalctl --list-boots

# Logs from the previous boot (useful after a crash/reboot)
journalctl -b -1

journalctl -b -1 is invaluable when a machine rebooted unexpectedly — it shows you what was happening right before it went down.

Kernel messages

#

# Kernel messages from this boot
journalctl -k

# Kernel messages, errors only
journalctl -k -p err

Combining filters

#

The real power comes from stacking filters:

# nginx errors in the last hour
journalctl -u nginx -p err --since "1 hour ago"

# Everything since the last boot, errors and above, following live
journalctl -b -p err -f

Working with Traditional Log Files

#

Plenty of applications still write plain-text logs to /var/log, and not everything goes through the journal. For these, you use standard text tools.

tail — see the end of a log

#

# Last 20 lines
tail -n 20 /var/log/syslog

# Follow the file as it grows (the workhorse of live debugging)
tail -f /var/log/nginx/access.log

# Follow multiple files at once
tail -f /var/log/nginx/access.log /var/log/nginx/error.log

tail -f is one of the most-used commands in log analysis. Run it, then trigger the action you’re debugging in another terminal, and watch what appears.

less — read and search a whole log

#

less /var/log/syslog

Inside less:

grep — find specific entries

#

grep is how you find the needle in the haystack:

# Find all lines mentioning an error
grep -i error /var/log/syslog

# Case-insensitive, with line numbers
grep -in "connection refused" /var/log/nginx/error.log

# Show 3 lines of context around each match
grep -i -C 3 "connection refused" /var/log/syslog

# Count how many times something occurs
grep -c "404" /var/log/nginx/access.log

# Search across all logs in a directory
grep -ri "out of memory" /var/log/

For compressed, rotated logs (see below), use zgrep:

zgrep -i error /var/log/syslog.2.gz

Practical Patterns

#

Watch a service while you restart it

#

The single most useful debugging move — follow the logs in one terminal while you act in another:

# Terminal 1
journalctl -u nginx -f

# Terminal 2
sudo systemctl restart nginx

You’ll see exactly why it fails to start, if it does.

Find what happened at a specific time

#

If a problem occurred at 2:15 PM, narrow to that window:

journalctl --since "2026-09-06 14:10:00" --until "2026-09-06 14:20:00"

Investigate an unexpected reboot

#

# What was logged right before the last shutdown?
journalctl -b -1 -e

# Kernel messages from the previous boot
journalctl -k -b -1

Look for Out of memory, hardware errors, or a clean shutdown sequence versus an abrupt cut.

Find failed login attempts

#

# Debian/Ubuntu
grep "Failed password" /var/log/auth.log

# RHEL/Fedora
grep "Failed password" /var/log/secure

# Or from the journal (unit is "ssh" on Debian/Ubuntu, "sshd" on RHEL/Fedora)
journalctl -u ssh --since today | grep -i "failed"

Count and rank the most common errors

#

Combine tools to see which errors dominate:

grep -i error /var/log/syslog | awk '{$1=$2=$3=""; print}' | sort | uniq -c | sort -rn | head

This strips the timestamp, groups identical messages, counts them, and shows the most frequent first. It turns a wall of log lines into a ranked summary. The $1=$2=$3="" assumes the standard syslog format where the first three fields are the month, day, and time — adjust the field count for other log formats.

Analyze web server traffic

#

# Top 10 requested URLs
awk '{print $7}' /var/log/nginx/access.log | sort | uniq -c | sort -rn | head

# Top 10 client IPs
awk '{print $1}' /var/log/nginx/access.log | sort | uniq -c | sort -rn | head

# Count responses by status code
awk '{print $9}' /var/log/nginx/access.log | sort | uniq -c | sort -rn

Log Rotation: Why Logs Disappear

#

Logs don’t grow forever. A tool called logrotate periodically renames, compresses, and eventually deletes old logs to keep disks from filling up. That’s why you’ll see files like:

/var/log/syslog
/var/log/syslog.1
/var/log/syslog.2.gz
/var/log/syslog.3.gz

The unnumbered file is current. .1 is the previous period. Numbered .gz files are older and compressed. If you’re looking for an event from last week, it may be in a rotated, compressed file — use zgrep and zless to read those without decompressing them first:

zless /var/log/syslog.2.gz
zgrep -i error /var/log/syslog.3.gz

The systemd journal has its own retention, controlled by size and time limits in /etc/systemd/journald.conf. Check how much space it’s using:

journalctl --disk-usage

And vacuum old entries if needed:

# Keep only the last 500MB
sudo journalctl --vacuum-size=500M

# Keep only the last 2 weeks
sudo journalctl --vacuum-time=2weeks

Quick Reference

#

Best Practices

#

• Follow logs live while reproducing an issue — journalctl -f or tail -f in one terminal, trigger the action in another. Seeing the log appear in real time removes all guesswork about which entry corresponds to your action.
• Filter by time first, then narrow — when you know roughly when something happened, --since/--until cuts a huge log down to a readable window before you start grepping.
• Use priority filters to cut noise — -p err on a busy service instantly hides the routine chatter and surfaces the problems.
• Know your distro’s file names — syslog/auth.log on Debian-based systems, messages/secure on RHEL-based ones. Looking in the wrong file wastes time.
• Remember rotated logs exist — if the event isn’t in the current log, check the numbered and .gz files with zgrep and zless. Don’t assume the data is gone.
• Watch disk usage — runaway logs fill disks and take services down with them. du -sh /var/log/* and journalctl --disk-usage tell you what’s growing.
• Never store secrets in logs — if you’re writing an application, keep passwords, tokens, and personal data out of log output. Logs are frequently read, shipped to other systems, and retained longer than you expect.