Mastering journalctl: The systemd Journal in Depth

Beyond the Basics

#

If you’ve read Reading and Analyzing Linux Logs, you already know the essential journalctl moves — filtering by service, time, priority, and boot. This post goes deeper into the systemd journal itself: how it stores data, how to filter on structured fields, how to run full-text searches, and how to control disk usage.

The journal isn’t just a log file with a query tool bolted on. Every entry is a structured record with dozens of fields — the service name, process ID, user ID, the executable path, priority, and more. Once you understand the fields, you can slice the journal with precision that plain-text logs can’t match.

Output Formats

#

By default journalctl prints a syslog-like format. The -o option changes that, and some formats reveal information the default hides.

# Default (syslog-style)
journalctl -o short

# With full ISO timestamps including timezone
journalctl -o short-iso

# With microsecond precision
journalctl -o short-precise

# Every field of every entry — shows the full structure
journalctl -o verbose

# JSON, one object per line (for scripting/piping to jq)
journalctl -o json

# Pretty-printed JSON
journalctl -o json-pretty

# Just the message text, nothing else
journalctl -o cat

-o verbose is the one to reach for when you want to see what fields are available on an entry:

journalctl -u nginx -o verbose -n 1

This dumps a single nginx entry with every field — _PID, _UID, _SYSTEMD_UNIT, _EXE, PRIORITY, and more. Those underscore-prefixed fields are the ones you filter on next.

-o cat is the opposite — strip everything except the message. Useful when piping to other tools:

journalctl -u nginx -o cat | grep -c "GET /api"

Filtering by Field

#

This is the feature that sets the journal apart from text logs. You filter with FIELD=value arguments, and multiple values for the same field are ORed while different fields are ANDed.

# Everything from process ID 1234
journalctl _PID=1234

# Everything run by a specific user (by UID)
journalctl _UID=1000

# Everything from a specific executable
journalctl _EXE=/usr/sbin/sshd

# Combine fields (AND) — sshd entries from a specific PID
journalctl _EXE=/usr/sbin/sshd _PID=980

# Multiple values for one field (OR) — two units
journalctl _SYSTEMD_UNIT=nginx.service _SYSTEMD_UNIT=php-fpm.service

Discovering available fields and values

#

You don’t have to guess field names. List them:

# List all field names present in the journal
journalctl -N

# List all values a given field has taken
journalctl -F _SYSTEMD_UNIT

-F _SYSTEMD_UNIT gives you every unit that has ever logged — a quick way to find the exact service name to filter on.

Common fields worth knowing

#

The distinction between _SYSTEMD_UNIT and the -u flag matters: -u nginx is a convenience wrapper that also pulls in related coredumps and messages about the unit, while _SYSTEMD_UNIT=nginx.service is a strict match on that exact field. For most work, -u is what you want; the raw field is there when you need precision.

Full-Text Search

#

journalctl has a built-in search that beats piping to grep because it operates on the structured message field and works alongside every other filter.

# Entries whose message matches a pattern (Perl-compatible regex)
journalctl --grep "connection refused"

# --grep uses smart-case: an all-lowercase pattern matches case-insensitively,
# while a pattern with any uppercase is matched case-sensitively.
# Force one behavior explicitly:
journalctl --grep "timeout" --case-sensitive=yes
journalctl --grep "Timeout" --case-sensitive=no

# Combine with other filters — search within one service, since a time
journalctl -u nginx --since "1 hour ago" --grep "upstream"

--grep uses PCRE, so you get full regex power:

# Match either "error" or "failed"
journalctl --grep "error|failed"

# Match a specific HTTP 5xx status in access logs
journalctl -u nginx --grep " 5[0-9]{2} "

Because --grep composes with -u, --since, and -p, you can search a narrow slice instead of the whole journal — much faster than grepping everything.

Persistent vs. Volatile Storage

#

A common surprise: on some systems, journalctl -b -1 returns nothing, because the journal isn’t being saved across reboots. Whether the journal persists depends on storage configuration.

# Check current disk usage of the journal
journalctl --disk-usage

Storage is controlled by Storage= in /etc/systemd/journald.conf:

The gotcha is auto: if /var/log/journal doesn’t exist, logs live only in RAM and vanish on reboot. To make the journal persistent:

# Create the directory and set ownership/permissions correctly
sudo mkdir -p /var/log/journal
sudo systemd-tmpfiles --create --prefix /var/log/journal

# Or set Storage=persistent explicitly, then restart journald
sudo sed -i 's/^#\?Storage=.*/Storage=persistent/' /etc/systemd/journald.conf
sudo systemctl restart systemd-journald

After this, journalctl -b -1 and older boots will be available — critical for diagnosing crashes that trigger a reboot.

Managing Disk Usage

#

The journal is capped by size and time limits, but you can trim it manually or tune the limits.

# How much space is the journal using?
journalctl --disk-usage

# Vacuum: keep only the most recent 500MB
sudo journalctl --vacuum-size=500M

# Vacuum: drop anything older than 2 weeks
sudo journalctl --vacuum-time=2weeks

# Vacuum: keep at most 10 journal files
sudo journalctl --vacuum-files=10

To set persistent limits, edit /etc/systemd/journald.conf:

[Journal]
# Cap total journal size
SystemMaxUse=1G
# Keep at least this much free on the filesystem
SystemKeepFree=2G
# Cap the size of any single journal file
SystemMaxFileSize=128M
# Drop entries older than this
MaxRetentionSec=1month

Apply changes with:

sudo systemctl restart systemd-journald

User Services

#

Services you run under your own user (via systemctl --user) log to a separate user journal. The system-wide journalctl won’t show them by default.

# Your own user's journal
journalctl --user

# A specific user service
journalctl --user -u my-app.service

# Follow a user service live
journalctl --user -u my-app.service -f

If you covered running services as a user in SSH Tunnels and Port Forwarding (persistent tunnels via systemctl --user), this is how you read their logs.

Exporting and Scripting

#

For analysis outside the journal — feeding a dashboard, archiving, or processing with other tools — export in a machine-readable format.

# JSON per line, ideal for jq
journalctl -u nginx -o json --since today > nginx-today.json

# Extract the message text of error-level entries (PRIORITY is a string in JSON,
# so convert it to a number before comparing)
journalctl -u nginx -o json --since today \
  | jq -r 'select((.PRIORITY | tonumber) <= 3) | .MESSAGE'

# Export in the journal export format for import on another machine
journalctl -o export > journal-export.bin

The JSON output pairs naturally with the command-line text tools covered in The Linux sed Command and standard shell pipelines — extract the field you care about, then sort, count, and rank as usual.

Quick Reference

#

Best Practices

#

• Use -o verbose to discover fields, then filter on them — when a -u filter is too broad or too narrow, dump one entry with -o verbose, find the field you actually want (_EXE, _PID, _COMM), and filter on it directly.
• Prefer --grep over piping to grep — it composes with -u, --since, and -p, so you search a narrow slice instead of the entire journal. It’s faster and keeps the structured context.
• Make the journal persistent on any machine you’ll debug — without /var/log/journal, you lose the previous boot’s logs, which is exactly what you need after an unexpected reboot. Set Storage=persistent.
• Cap journal size on constrained systems — set SystemMaxUse and MaxRetentionSec so logs can’t fill the disk. A full disk takes services down with it.
• Remember user services log separately — journalctl --user for anything started with systemctl --user. The system journal won’t show it.
• Export to JSON for real analysis — when you need more than ad-hoc filtering, -o json piped to jq turns the journal into structured data you can query, count, and aggregate.
• Filter before you follow — on a busy system, journalctl -f alone is a firehose. Add -u, -p err, or --grep so you only watch what matters.