Rsyslog

Rsyslog is a high-performance, open-source system logging daemon that processes syslog messages with flexible filtering, custom templates, and remote forwarding capabilities. It’s the default logger on most modern Linux distributions.

Installation

Ubuntu/Debian

# Install rsyslog
sudo apt update
sudo apt install rsyslog

# Install optional modules
sudo apt install rsyslog-mysql rsyslog-pgsql rsyslog-elasticsearch

RHEL/CentOS/Fedora

# Install rsyslog
sudo dnf install rsyslog

# Optional database modules
sudo dnf install rsyslog-mysql rsyslog-pgsql rsyslog-elasticsearch

Build from Source

# Clone repository
git clone https://github.com/rsyslog/rsyslog.git
cd rsyslog

# Install dependencies
sudo apt-get install build-essential autoconf automake libnet-dev zlib1g-dev

# Build and install
./configure --enable-mysql
make
sudo make install
sudo useradd -r -s /bin/false syslog

Service Management

Control rsyslog Daemon

# Start rsyslog service
sudo systemctl start rsyslog

# Stop rsyslog service
sudo systemctl stop rsyslog

# Restart rsyslog (apply configuration changes)
sudo systemctl restart rsyslog

# Reload configuration without dropping messages
sudo systemctl reload rsyslog

# Check service status
sudo systemctl status rsyslog

# Enable on boot
sudo systemctl enable rsyslog

# Disable on boot
sudo systemctl disable rsyslog

# Check if running
sudo systemctl is-active rsyslog

Verify Rsyslog Installation

# Check version
rsyslogd -version

# List compiled modules
rsyslogd -v 2>&1 | grep "modules"

# Test configuration syntax
sudo rsyslogd -N1

# Run in foreground (debug mode)
sudo rsyslogd -d -x -A

Configuration Basics

Main Configuration File

# Primary configuration file
/etc/rsyslog.conf

# Configuration directory (drop-in snippets)
/etc/rsyslog.d/

# Example: custom rules
sudo nano /etc/rsyslog.d/50-default.conf

# Test configuration for syntax errors
sudo rsyslogd -N1

Basic Configuration Structure

# Load modules
$ModLoad imuxsock       # Unix sockets
$ModLoad imklog         # Kernel logs
$ModLoad imtcp          # TCP input

# Log destinations
# File-based logging
*.*  /var/log/messages

# Remote server
*.*  @@remote-host:514

# Rules format: selector action
# selector: facility.severity
# action: destination (file, remote host, pipe, etc.)

Rule Syntax and Filters

Basic Logging Rules

# Log all messages to file
*.*  /var/log/all.log

# Log kernel messages
kern.*  /var/log/kern.log

# Log auth/authpriv messages
auth,authpriv.*  /var/log/auth.log

# Log cron messages
cron.*  /var/log/cron.log

# Log mail facility
mail.*  /var/log/mail.log

# Log by severity level
*.err  /var/log/errors.log
*.warning  /var/log/warnings.log
*.info  /var/log/info.log

Property-Based Filtering

# Filter by hostname
:hostname, isequal, "webserver" /var/log/webserver.log
action(type="omfile" File="/var/log/webserver.log")

# Filter by program name
:programname, isequal, "apache2" /var/log/apache2-custom.log
:programname, contains, "mysql" /var/log/mysql-custom.log

# Filter by message content
:msg, contains, "error" /var/log/errors.log
:msg, startswith, "WARN" /var/log/warnings.log
:msg, regex, "kernel.*panic" /var/log/panics.log

Expressions and Conditions

# Negation (NOT)
!auth.*  /var/log/non-auth.log

# Multiple facilities
auth,authpriv,kern.*  /var/log/combined.log

# Stop processing (discard)
debug,info  ~

# Multiple actions (repeat rule)
kern.*  /var/log/kern.log
kern.*  @@remote-server:514

# Modern format with module
if $msg contains "error" then {
    action(type="omfile" file="/var/log/errors.log")
}

Template Configuration

Custom Log Format

# Simple template
$Template MyFormat,"%TIMESTAMP% %HOSTNAME% %syslogtag%%msg:::drop-last-lf%\n"

# Detailed template
$Template DetailedFormat,"%TIMESTAMP:1:10:date-rfc3339% %TIMESTAMP:12:19:date-rfc3339% %HOSTNAME% %programname% [%procid%]: %msg%\n"

# JSON template (for Elasticsearch, etc.)
$Template JSONFormat,"{\"@timestamp\":\"%TIMESTAMP:1:23:date-rfc3339%\",\"host\":\"%HOSTNAME%\",\"program\":\"%programname%\",\"pid\":%procid%,\"message\":\"%msg%\"}\n"

# Use template
auth.*  action(type="omfile" file="/var/log/auth.log" template="DetailedFormat")

Constant Values in Templates

# Static values in logs
$Template VerboseFormat,"%TIMESTAMP:1:10:date-rfc3339% [%syslogfacility-text%][%syslogseverity-text%] %msg%\n"

# Hostname always set
$Template FixedHostFormat,"LocalServer - %msg%\n"

# Application-specific
$Template AppFormat,"[APP:%programname%] %msg%"

Log File Rotation

# Rotate logs daily
$ActionFileDefaultTemplate RSYSLOG_FileFormat
$FileOwner syslog
$FileGroup adm
$FileCreateMode 0640
$DirCreateMode 0755
$Umask 0022

# Daily rotation via date format
:programname, isequal, "myapp" /var/log/myapp-%$now%-%$year%-%$month%-%$day%.log

Remote Logging

Send Logs to Remote Server

# TCP remote logging (reliable)
*.* @@remote-server.example.com:514

# UDP remote logging (fast, lossy)
*.* @remote-server.example.com:514

# Multiple remote servers (failover)
*.* @@primary.example.com:514
*.* @@backup.example.com:514

# Specific facility to remote
auth,authpriv.* @@seclog-server.example.com:601

# With custom template
*.* action(type="omfwd" target="logserver.example.com" port="514" \
    protocol="tcp" template="DetailedFormat")

Receive Remote Logs

# Listen for TCP syslog
$ModLoad imtcp
$InputTCPServerRun 514

# Listen for UDP syslog
$ModLoad imudp
$UDPServerRun 514

# Listen on specific interface
$InputTCPServerBindRuleset remote_rules
input(type="imtcp" port="514" ruleset="remote_rules")

# Parse remote hostname
$RulesetCreateMainQueue off
ruleset(name="remote_rules") {
    action(type="omfile" file="/var/log/remote/%HOSTNAME%/messages.log")
}

TLS/SSL Encryption

# Load GnuTLS module
$ModLoad imtcp

# Enable TLS for TCP
$InputTCPServerRun 6514
$InputTCPServerStreamDriverMode 1
$InputTCPServerStreamDriverAuthMode x509/name
$InputTCPServerStreamDriverPermittedPeers "*.example.com"

# Client-side TLS
action(type="omfwd" target="logserver.example.com" port="6514" \
    protocol="tcp" \
    streamDriver="gtls" \
    streamDriverMode="1" \
    streamDriverAuthMode="x509/name" \
    streamDriverPermittedPeers="logserver.example.com")

Database Logging

MySQL Backend

# Load MySQL module
$ModLoad ommysql

# Create database
mysql -u root -p << EOF
CREATE DATABASE syslog;
CREATE USER 'syslog'@'localhost' IDENTIFIED BY 'password';
GRANT ALL ON syslog.* TO 'syslog'@'localhost';
FLUSH PRIVILEGES;

# Create table
USE syslog;
CREATE TABLE SystemEvents (
    ID int unsigned not null auto_increment primary key,
    CustomerID bigint,
    ReceivedAt datetime not null,
    DeviceReportedTime datetime,
    Facility smallint,
    Priority smallint,
    FromHost varchar(60),
    Message text,
    NTSeverity int,
    Importance int,
    EventSource varchar(60),
    EventUser varchar(60),
    EventCategory int,
    EventID int,
    EventBinaryUserData longblob,
    LastOccurrence datetime,
    COUNT int,
    PROCESSED tinyint default 0,
    zbReceiveTime datetime,
    INDEX idx_ReceivedAt (ReceivedAt),
    INDEX idx_FromHost (FromHost)
);
EOF

# Configure rsyslog for MySQL
$ModLoad ommysql
*.* :ommysql:localhost,syslog,syslog,password

# Or modern format
action(type="ommysql" server="localhost" db="syslog" \
    uid="syslog" pwd="password")

PostgreSQL Backend

# Create database
sudo -u postgres psql << EOF
CREATE DATABASE syslog;
CREATE USER syslog WITH PASSWORD 'password';
GRANT ALL PRIVILEGES ON DATABASE syslog TO syslog;

\c syslog
CREATE TABLE SystemEvents (
    id SERIAL PRIMARY KEY,
    receivedat TIMESTAMP,
    fromhost VARCHAR(60),
    priority SMALLINT,
    facility SMALLINT,
    message TEXT
);
EOF

# Load module and configure
$ModLoad ompgsql
action(type="ompgsql" server="localhost" db="syslog" \
    uid="syslog" pwd="password")

Troubleshooting

Common Issues

Issue: Logs not appearing in files

# Check rsyslog is running
sudo systemctl status rsyslog

# Test configuration syntax
sudo rsyslogd -N1

# Check file permissions
ls -la /var/log/

# Verify rsyslog has write access
sudo touch /var/log/test.log
sudo chown syslog:adm /var/log/test.log
sudo chmod 640 /var/log/test.log

# Test with logger command
logger "Test message"
tail /var/log/syslog

Issue: Remote logging not working

# Check if rsyslog is listening on ports
sudo netstat -tlnp | grep rsyslogd

# Verify network connectivity
telnet remote-server 514

# Check firewall rules
sudo ufw status
sudo firewall-cmd --list-ports

# Enable port if blocked
sudo ufw allow 514/tcp
sudo ufw allow 514/udp

# Test with tcpdump
sudo tcpdump -i any port 514

Issue: High CPU or memory usage

# Check rsyslog process
ps aux | grep rsyslogd

# Reduce queue size in config
$ActionQueueSize 1000
$ActionQueueDiscardMark 100

# Disable unnecessary inputs
# Comment out unused $ModLoad lines

# Check for message storms
grep -c "" /var/log/syslog

Issue: Permission denied writing to log file

# Fix log directory permissions
sudo chmod 755 /var/log
sudo chown root:root /var/log

# Fix file ownership
sudo chown syslog:adm /var/log/auth.log
sudo chmod 640 /var/log/auth.log

# Verify syslog user exists
id syslog

# Create if missing
sudo useradd -r -s /bin/false syslog

Debug Mode

# Run rsyslog in foreground with debug output
sudo rsyslogd -d -x -A

# Enable debug in configuration
$DebugFile /var/log/rsyslog-debug.log
$DebugLevel 2

# Check debug output
tail -f /var/log/rsyslog-debug.log

# Query statistics (rsyslog 8.x+)
$ModLoad impstats
action(type="omfile" file="/var/log/rsyslog-stats.log" \
    template="stat" interval="300")

Performance Tuning

Queue Configuration

# Set main queue size (newer format)
global(
    action.processingQueue.size="10000"
    action.processingQueue.discardMark="9000"
    action.processingQueue.discardSeverity="5"
)

# Legacy format
$ActionQueueSize 10000
$ActionQueueDiscardMark 9000
$ActionQueueDiscardSeverity 5
$ActionQueueType LinkedList

# Persist queue to disk
$ActionQueueFileName fwdRule1
$ActionQueueMaxDiskSpace 2g
$ActionQueueSaveOnShutdown on
$ActionQueueType LinkedList

Module Tuning

# Increase max file descriptor limit
$MaxOpenFiles 2048

# Set number of listener threads
$InputTCPServerInputName imtcp
input(type="imtcp" port="514" threads="8")

# Batch processing
$ActionBatchProcessing on
$ActionBatchSize 1000
$ActionBatchTimeout 3000

# Enable high precision timestamps
$ActionFileDefaultTemplate RSYSLOG_FileFormat_PrecisionUTC

Connection and Network Tuning

# TCP keepalive for remote forwarding
action(type="omfwd" target="logserver" port="514" \
    protocol="tcp" \
    socketBufSize="1m" \
    keepAlive="on")

# Enable compression for remote logs
action(type="omfwd" target="logserver" port="514" \
    protocol="tcp" \
    zipLevel="9")

# Timeout and retry settings
action(type="omfwd" target="logserver" port="514" \
    protocol="tcp" \
    actionTimeout="10" \
    actionRetryCount="3")

Practical Configuration Examples

Complete rsyslog.conf

# Default settings
$FileOwner syslog
$FileGroup adm
$FileCreateMode 0640
$DirCreateMode 0755
$Umask 0022

# Load modules
$ModLoad imuxsock
$ModLoad imklog
$ModLoad imtcp
$InputTCPServerRun 514

# Define templates
$Template AuthLog,"%TIMESTAMP:1:10:date-rfc3339% %HOSTNAME% %syslogtag%%msg%\n"
$Template RemoteLog,"From %HOSTNAME%: %msg%\n"

# Standard facility rules
auth,authpriv.*  /var/log/auth.log;AuthLog
*.*;auth,authpriv.none  -/var/log/syslog
cron.*  /var/log/cron.log
kern.*  /var/log/kern.log
mail.*  /var/log/mail.log
mail.err  /var/log/mail.err

# Emergency messages
*.emerg  :omusrmsg:*

# Remote forwarding
*.* @@logserver.example.com:514

# Stop processing
& stop

Multi-Server Configuration

# /etc/rsyslog.d/30-servers.conf

# Define a ruleset for each server
ruleset(name="web_servers") {
    action(type="omfile" file="/var/log/servers/web/%HOSTNAME%/messages.log")
}

ruleset(name="db_servers") {
    action(type="omfile" file="/var/log/servers/db/%HOSTNAME%/messages.log")
}

# Match by hostname
if $hostname startswith "web" then {
    call web_servers
} else if $hostname startswith "db" then {
    call db_servers
} else {
    action(type="omfile" file="/var/log/other.log")
}

Application-Specific Logging

# /etc/rsyslog.d/50-app-logging.conf

# Apache/Nginx access logs to separate file
:programname, isequal, "nginx"  /var/log/nginx-processed.log

# Docker container logs
:programname, regex, "docker.*"  /var/log/docker.log

# Application errors to high-priority queue
:severity, gte, "err" -/var/log/app-errors.log
:msg, contains, "CRITICAL"  /var/log/app-critical.log

# JSON format for applications
$Template JSONFormat,"{\"timestamp\":\"%TIMESTAMP:1:23:date-rfc3339%\",\"host\":\"%HOSTNAME%\",\"app\":\"%programname%\",\"level\":\"%syslogseverity-text%\",\"message\":\"%msg%\"}\n"

:programname, isequal, "myapp" action(type="omfile" \
    file="/var/log/myapp.json" template="JSONFormat")

Log Parsing and Processing

Using liblognorm for Parsing

# Enable normalization module
$ModLoad mmnormalize

# Parse with pattern
action(type="mmnormalize" rulebase="/etc/rsyslog.d/rules/apache.rulebase")

# Elasticsearch-friendly output
action(type="omfile" file="/var/log/normalized.json")

Property Extraction

# Extract severity and facility names
$Template WithSeverity,"%TIMESTAMP% [%syslogseverity-text%] %msg%\n"
$Template WithFacility,"%TIMESTAMP% <%syslogfacility-text%.%syslogseverity-text%> %msg%\n"

# Extract timestamp variations
$Template ISO8601,"%TIMESTAMP:::date-iso8601% %msg%\n"
$Template RFC3339,"%TIMESTAMP:1:23:date-rfc3339% %msg%\n"

# Message length limiting
$Template ShortMsg,"%msg:1:80%...\n"

Conditional Actions

# Drop duplicate messages
action(type="mmdblookup")

# If facility is local0, send to app logs
if $syslogfacility-text == "local0" then {
    action(type="omfile" file="/var/log/app-messages.log")
}

# High-severity messages get multiple destinations
if $syslogseverity <= 3 then {
    action(type="omfile" file="/var/log/critical.log")
    action(type="omfwd" target="alert-server" port="514")
    action(type="omusrmsg" users="*")
}

Security and Access Control

File Permissions

# Set default permissions for new log files
$FileOwner syslog
$FileGroup adm
$FileCreateMode 0640
$DirCreateMode 0755

# Per-file permissions
action(type="omfile" file="/var/log/auth.log" \
    fileOwner="root" fileGroup="adm" fileCreateMode="0640")

# Verify permissions
ls -la /var/log/auth.log
ls -la /var/log/syslog

Log Rotation

# Install logrotate configuration
cat > /etc/logrotate.d/rsyslog << EOF
/var/log/auth.log
/var/log/syslog
/var/log/cron.log
{
    daily
    rotate 7
    compress
    delaycompress
    missingok
    notifempty
    create 0640 syslog adm
    postrotate
        systemctl reload rsyslog > /dev/null 2>&1 || true
    endscript
}
EOF

# Manual log rotation trigger
sudo logrotate -f /etc/logrotate.d/rsyslog

Best Practices

Configuration Management

• Store rsyslog.d/*.conf files in version control
• Test configuration changes before restarting
• Use rsyslogd -N1 to validate syntax
• Document all custom rules and templates
• Back up original /etc/rsyslog.conf
• Use drop-in files in /etc/rsyslog.d/ for customizations
• Comment all custom rules with purpose
• Create separate files for different log sources

Performance and Tuning

• Monitor disk I/O for high-volume logging
• Use disk queues for remote forwarding to prevent message loss
• Adjust queue sizes based on message volume
• Enable compression for remote syslog
• Use UDP for high-volume, lossy acceptable scenarios
• Use TCP or TLS for critical logs
• Monitor memory usage with ps aux | grep rsyslogd
• Implement log rotation to prevent disk saturation

Security

• Restrict log file permissions (640 or 600)
• Run rsyslog as unprivileged syslog user
• Use TLS for remote logging over untrusted networks
• Implement authentication for remote syslog collection
• Encrypt sensitive logs at rest (e.g., auth.log)
• Audit log access with auditctl
• Separate logs by sensitivity level
• Protect database credentials in configs

High Availability

• Configure failover remote servers
• Use disk queues for message persistence
• Implement load balancing for log collection
• Monitor rsyslog daemon health
• Set up log aggregation architecture
• Test failover scenarios regularly
• Document recovery procedures
• Implement log replication strategies

Related Tools

• syslog-ng - Alternative syslog daemon
• journalctl - systemd journal queries
• logrotate - Log file rotation
• Elasticsearch - Log storage/analysis
• Splunk - Enterprise log management

---

Last updated: 2026-03-30