Logs in Linux: Centralize and Analyze with rsyslog

Logging is the backbone of system administration, security auditing and performance monitoring. On Linux, syslog-compatible daemons collect, store and forward logs generated by applications, the kernel and system services. rsyslog is the modern, high-performance syslog implementation that supports advanced features such as reliable transport, filtering, message rewriting, and database output.

1. Why Centralize Logs

Unified Visibility: One console for viewing logs from multiple machines.
Security Compliance: Retain logs in a tamper-evident, centralized store.
Resource Optimization: Reduce disk usage on endpoints archive older logs centrally.
Simplified Analysis: Centralization enables correlation, alerting and trend analysis.

2. rsyslog Architecture Overview

rsyslog extends the traditional syslog model with modular inputs, filters and outputs. Key components include:

Immodules (Inputs):
- imuxsock for local socket (#47dev#47log).
- imudp, imtcp for network reception.
- imjournal for systemd journal integration.
Queueing: Disk- or memory-based queues for reliable delivery.
Filters Rulesets: Programmatic control over routing, rewriting and discarding messages.
Output Modules:
- omfile for local files.
- omfwd for forwarding to remote syslog servers.
- ommysql, ompgsql for database inserts.

3. Secure Transport: VPN and TLS

When transmitting logs across untrusted networks, you should encrypt the channel. Options include:

Transport Layer Security (TLS) via imtcp StreamDriver.
Virtual Private Networks (VPNs) such as
OpenVPN or
WireGuard
to isolate syslog traffic.

Example: TLS setup in rsyslog.conf

# Enable TLS on TCP listener
module(load=imtcp)
module(load=gtls)
input(type=imtcp port=6514 StreamDriver=gtls StreamDriverMode=1 StreamDriverAuthMode=x509/name)
# Certificates
global(
  DefaultNetstreamDriverCAFile=/etc/rsyslog/ca-cert.pem
  DefaultNetstreamDriverCertFile=/etc/rsyslog/server-cert.pem
  DefaultNetstreamDriverKeyFile=/etc/rsyslog/server-key.pem
)

4. Configuring Central Server and Clients

4.1 Central rsyslog Server

Install: apt-get install rsyslog or yum install rsyslog.

Edit /etc/rsyslog.conf:

Enable TCP/UDP listeners:

module(load=imudp)
input(type=imudp port=514)
module(load=imtcp)
input(type=imtcp port=514)

Define ruleset for remote messages:

template(name=RemoteLogs type=string 
  string=/var/log/remote/%HOSTNAME%/%PROGRAMNAME%.log)
if (fromhost-ip != 127.0.0.1) then {
  action(type=omfile dynaFile=RemoteLogs)
  stop
}

Create directory structure: mkdir -p /var/log/remote.
Restart systemctl restart rsyslog.

4.2 Client Configuration

Edit /etc/rsyslog.conf or /etc/rsyslog.d/50-remote.conf:

.  @@central.example.com:514    # TCP, double @ for TCP TLS, @ for UDP

Optionally configure TLS:

module(load=imtcp)
module(load=gtls)
global(DefaultNetstreamDriverCAFile=/etc/rsyslog/ca-cert.pem)
. action(
  type=omfwd
  target=central.example.com port=6514
  protocol=tcp
  StreamDriver=gtls StreamDriverMode=1
)

Restart: systemctl restart rsyslog.

5. Log Processing and Filtering

rsyslog’s powerful filtering syntax lets you select messages by:

Facility and severity: authpriv., .warn.
Program name or process ID: programname == sshd.
Message properties: msg contains error.
GeoIP or DNS lookup: For incoming remote host IPs.

Example: Route kernel messages to a separate file:

if (syslogfacility-text == kern) then {
  action(type=omfile file=/var/log/kernel.log)
  stop
}

6. Storage, Rotation and Archiving

While rsyslog writes logs, tools like logrotate handle archival:

Directive	Description
rotate 7	Keep seven rotated logs before deletion.
daily	Rotate logs every day.
compress	Gzip old logs to save space.

Sample /etc/logrotate.d/remote-logs:

/var/log/remote//.log {
  daily
  rotate 14
  compress
  missingok
  notifempty
  sharedscripts
  postrotate
    systemctl kill -s HUP rsyslog
  endscript
}

7. Log Analysis and Visualization

Once centralized, logs can feed into analysis tools:

ELK Stack (Elasticsearch, Logstash, Kibana): Powerful search and dashboards.
Graylog: Web UI, alerting and streams.
Grafana Loki: Efficient, label-based log indexing.
Splunk: Enterprise-grade analytics.

Integrating rsyslog with Elasticsearch:

module(load=omelasticsearch)
action(
  type=omelasticsearch
  server=localhost serverport=9200
  template=RSYSLOG_TraditionalFileFormat
  searchIndex=syslog
  dynSearchIndex=on
)

8. Best Practices

Use TLS or VPN tunnels to protect sensitive data in transit.
Implement rate limiting to guard against log floods and DoS.
Centralize time synchronization with NTP or Chrony for accurate timestamps.
Enforce access controls and audit rsyslog configuration changes.
Document retention policies to balance storage and compliance needs.

9. Further Resources

By centralizing and analyzing logs with rsyslog, administrators gain powerful control over system visibility, compliance and forensic capabilities. Whether you manage a handful of servers or an entire data center, the techniques outlined here provide a robust foundation for building a secure, efficient log management infrastructure.

LINUXMIND.DEV

Logs in Linux: Centralize and Analyze with rsyslog