Fluentd Cheatsheet¶

Installation¶

Platform	Command
Ubuntu/Debian (td-agent)	`curl -fsSL https://toolbelt.treasuredata.com/sh/install-ubuntu-jammy-td-agent4.sh \\| sh`
RHEL/CentOS	`curl -L https://toolbelt.treasuredata.com/sh/install-redhat-td-agent4.sh \\| sh`
macOS	`brew install fluentd`
Ruby Gem	`gem install fluentd`
Docker	`docker pull fluent/fluentd:latest`
Kubernetes	Deploy as DaemonSet (see Configuration section)

Basic Commands¶

Command	Description
`fluentd -c fluent.conf`	Start Fluentd with specified configuration file
`fluentd -c fluent.conf -vv`	Run with verbose debug output
`fluentd -c fluent.conf --dry-run`	Validate configuration without starting
`fluentd --setup ./fluent`	Create default configuration directory structure
`fluentd --version`	Display Fluentd version information
`sudo systemctl start td-agent`	Start td-agent service (Linux)
`sudo systemctl stop td-agent`	Stop td-agent service
`sudo systemctl restart td-agent`	Restart td-agent service
`sudo systemctl status td-agent`	Check td-agent service status
`sudo systemctl reload td-agent`	Reload configuration without restarting
`sudo systemctl enable td-agent`	Enable td-agent to start on boot
`sudo journalctl -u td-agent -f`	Follow td-agent service logs in real-time
`echo '{"msg":"test"}' \\| fluent-cat debug.test`	Send test log message to Fluentd
`curl -X POST -d 'json={"event":"test"}' http://localhost:8888/test.cycle`	Send HTTP test log
`td-agent-gem list \\| grep fluent-plugin`	List installed Fluentd plugins

Advanced Usage¶

Command	Description
`fluentd -c fluent.conf -d /var/run/fluentd.pid`	Run Fluentd in daemon mode with PID file
`fluentd -c fluent.conf -o /var/log/fluentd.log`	Run with output to specific log file
`fluentd -c fluent.conf --workers 4`	Run with multiple worker processes
`fluentd -c fluent.conf -vvv`	Run with trace-level logging for debugging
`fluentd --show-plugin-config=input:tail`	Display configuration options for specific plugin
`td-agent-gem install fluent-plugin-elasticsearch`	Install Elasticsearch output plugin
`td-agent-gem install fluent-plugin-kafka -v 0.17.5`	Install specific version of Kafka plugin
`td-agent-gem update fluent-plugin-s3`	Update S3 plugin to latest version
`td-agent-gem uninstall fluent-plugin-mongo`	Remove MongoDB plugin
`td-agent-gem search -r fluent-plugin`	Search for available plugins in repository
`fluent-cat --host 192.168.1.100 --port 24224 app.logs`	Send logs to remote Fluentd instance
`fluent-cat app.logs < /path/to/logfile.json`	Send log file contents to Fluentd
`docker run -d -p 24224:24224 -v /data/fluentd:/fluentd/etc fluent/fluentd`	Run Fluentd in Docker with mounted config
`sudo kill -USR1 $(cat /var/run/td-agent/td-agent.pid)`	Gracefully reload Fluentd (reopen log files)
`sudo kill -USR2 $(cat /var/run/td-agent/td-agent.pid)`	Reopen Fluentd log files without reload

Configuration¶

Main Configuration File Locations¶

td-agent (Linux): /etc/td-agent/td-agent.conf
Gem installation: ./fluent/fluent.conf
Docker: /fluentd/etc/fluent.conf

Basic Configuration Structure¶

# Source: Input plugins
<source>
  @type forward
  port 24224
  bind 0.0.0.0
</source>

# Filter: Process/transform logs
<filter app.**>
  @type record_transformer
  <record>
    hostname "#{Socket.gethostname}"
    tag ${tag}
  </record>
</filter>

# Match: Output plugins
<match app.**>
  @type elasticsearch
  host elasticsearch.local
  port 9200
  index_name fluentd
  type_name fluentd
</match>

Source Plugins (Input)¶

# Forward input (receive from other Fluentd instances)
<source>
  @type forward
  port 24224
  bind 0.0.0.0
</source>

# Tail log files
<source>
  @type tail
  path /var/log/nginx/access.log
  pos_file /var/log/td-agent/nginx-access.pos
  tag nginx.access
  <parse>
    @type nginx
  </parse>
</source>

# HTTP input
<source>
  @type http
  port 8888
  bind 0.0.0.0
  body_size_limit 32m
  keepalive_timeout 10s
</source>

# Syslog input
<source>
  @type syslog
  port 5140
  bind 0.0.0.0
  tag system.syslog
</source>

Filter Plugins (Processing)¶

# Add/modify record fields
<filter app.**>
  @type record_transformer
  <record>
    hostname "#{Socket.gethostname}"
    environment production
    timestamp ${time}
  </record>
</filter>

# Parse unstructured logs
<filter app.logs>
  @type parser
  key_name message
  <parse>
    @type json
  </parse>
</filter>

# Grep filter (include/exclude)
<filter app.**>
  @type grep
  <regexp>
    key level
    pattern /^(ERROR|FATAL)$/
  </regexp>
</filter>

# Modify tag
<match app.raw.**>
  @type rewrite_tag_filter
  <rule>
    key level
    pattern /^ERROR$/
    tag app.error.${tag}
  </rule>
</match>

Match Plugins (Output)¶

# Elasticsearch output
<match app.**>
  @type elasticsearch
  host elasticsearch.local
  port 9200
  logstash_format true
  logstash_prefix fluentd
  <buffer>
    @type file
    path /var/log/fluentd/buffer/elasticsearch
    flush_interval 10s
    retry_max_interval 300s
  </buffer>
</match>

# S3 output
<match logs.**>
  @type s3
  aws_key_id YOUR_AWS_KEY_ID
  aws_sec_key YOUR_AWS_SECRET_KEY
  s3_bucket your-bucket-name
  s3_region us-east-1
  path logs/
  time_slice_format %Y%m%d%H
  <buffer time>
    timekey 3600
    timekey_wait 10m
  </buffer>
</match>

# File output
<match debug.**>
  @type file
  path /var/log/fluentd/output
  <buffer>
    timekey 1d
    timekey_use_utc true
  </buffer>
</match>

# Forward to another Fluentd
<match forward.**>
  @type forward
  <server>
    host 192.168.1.100
    port 24224
  </server>
  <buffer>
    @type file
    path /var/log/fluentd/buffer/forward
  </buffer>
</match>

# Stdout (debugging)
<match debug.**>
  @type stdout
</match>

Buffer Configuration¶

<match pattern.**>
  @type elasticsearch

  # File buffer with advanced settings
  <buffer>
    @type file
    path /var/log/fluentd/buffer

    # Flush settings
    flush_mode interval
    flush_interval 10s
    flush_at_shutdown true

    # Retry settings
    retry_type exponential_backoff
    retry_wait 10s
    retry_max_interval 300s
    retry_timeout 72h
    retry_max_times 17

    # Chunk settings
    chunk_limit_size 5M
    queue_limit_length 32
    overflow_action drop_oldest_chunk

    # Compression
    compress gzip
  </buffer>
</match>

# Memory buffer for high-performance
<match fast.**>
  @type forward
  <buffer>
    @type memory
    flush_interval 5s
    chunk_limit_size 1M
    queue_limit_length 64
  </buffer>
</match>

Multi-Worker Configuration¶

<system>
  workers 4
  root_dir /var/log/fluentd
</system>

# Worker-specific sources
<worker 0>
  <source>
    @type forward
    port 24224
  </source>
</worker>

<worker 1-3>
  <source>
    @type tail
    path /var/log/app/*.log
    tag app.logs
  </source>
</worker>

Label-Based Routing¶

# Route to different pipelines using labels
<source>
  @type forward
  @label @mainstream
</source>

<source>
  @type tail
  path /var/log/secure.log
  @label @security
</source>

<label @mainstream>
  <filter **>
    @type record_transformer
    <record>
      pipeline mainstream
    </record>
  </filter>

  <match **>
    @type elasticsearch
    host es-main
  </match>
</label>

<label @security>
  <filter **>
    @type grep
    <regexp>
      key message
      pattern /authentication failure/
    </regexp>
  </filter>

  <match **>
    @type s3
    s3_bucket security-logs
  </match>
</label>

Common Use Cases¶

Use Case 1: Collect Nginx Logs to Elasticsearch¶

# Install Elasticsearch plugin
sudo td-agent-gem install fluent-plugin-elasticsearch

# Configure Fluentd
sudo tee /etc/td-agent/td-agent.conf > /dev/null <<'EOF'
<source>
  @type tail
  path /var/log/nginx/access.log
  pos_file /var/log/td-agent/nginx-access.pos
  tag nginx.access
  <parse>
    @type nginx
  </parse>
</source>

<match nginx.access>
  @type elasticsearch
  host localhost
  port 9200
  logstash_format true
  logstash_prefix nginx
  <buffer>
    flush_interval 10s
  </buffer>
</match>
EOF

# Restart td-agent
sudo systemctl restart td-agent

# Verify logs are flowing
sudo journalctl -u td-agent -f

Use Case 2: Kubernetes Log Collection¶

# Deploy Fluentd DaemonSet
kubectl apply -f - <<'EOF'
apiVersion: v1
kind: ServiceAccount
metadata:
  name: fluentd
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: fluentd
rules:
- apiGroups: [""]
  resources: ["pods", "namespaces"]
  verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: fluentd
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: fluentd
subjects:
- kind: ServiceAccount
  name: fluentd
  namespace: kube-system
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: fluentd
  namespace: kube-system
spec:
  selector:
    matchLabels:
      k8s-app: fluentd-logging
  template:
    metadata:
      labels:
        k8s-app: fluentd-logging
    spec:
      serviceAccountName: fluentd
      containers:
      - name: fluentd
        image: fluent/fluentd-kubernetes-daemonset:v1-debian-elasticsearch
        env:
        - name: FLUENT_ELASTICSEARCH_HOST
          value: "elasticsearch.logging.svc.cluster.local"
        - name: FLUENT_ELASTICSEARCH_PORT
          value: "9200"
        volumeMounts:
        - name: varlog
          mountPath: /var/log
        - name: varlibdockercontainers
          mountPath: /var/lib/docker/containers
          readOnly: true
      volumes:
      - name: varlog
        hostPath:
          path: /var/log
      - name: varlibdockercontainers
        hostPath:
          path: /var/lib/docker/containers
EOF

# Check DaemonSet status
kubectl get daemonset -n kube-system fluentd
kubectl logs -n kube-system -l k8s-app=fluentd-logging --tail=50

Use Case 3: Forward Logs to S3 with Rotation¶

# Install S3 plugin
sudo td-agent-gem install fluent-plugin-s3

# Configure S3 output
sudo tee /etc/td-agent/td-agent.conf > /dev/null <<'EOF'
<source>
  @type tail
  path /var/log/app/*.log
  pos_file /var/log/td-agent/app.pos
  tag app.logs
  <parse>
    @type json
  </parse>
</source>

<match app.logs>
  @type s3

  aws_key_id YOUR_AWS_ACCESS_KEY
  aws_sec_key YOUR_AWS_SECRET_KEY
  s3_bucket my-application-logs
  s3_region us-east-1

  path logs/%Y/%m/%d/
  s3_object_key_format %{path}%{time_slice}_%{index}.%{file_extension}

  <buffer time>
    @type file
    path /var/log/td-agent/s3
    timekey 3600
    timekey_wait 10m
    chunk_limit_size 256m
  </buffer>

  <format>
    @type json
  </format>
</match>
EOF

# Restart and verify
sudo systemctl restart td-agent
sudo systemctl status td-agent

Use Case 4: Multi-Destination Log Routing¶

# Configure routing to multiple destinations
sudo tee /etc/td-agent/td-agent.conf > /dev/null <<'EOF'
<source>
  @type tail
  path /var/log/app/application.log
  pos_file /var/log/td-agent/app.pos
  tag app.logs
  <parse>
    @type json
  </parse>
</source>

# Copy logs to multiple destinations
<match app.logs>
  @type copy

  # Send to Elasticsearch
  <store>
    @type elasticsearch
    host elasticsearch.local
    port 9200
    logstash_format true
  </store>

  # Send to S3 for archival
  <store>
    @type s3
    s3_bucket app-logs-archive
    path logs/
    <buffer time>
      timekey 86400
    </buffer>
  </store>

  # Send errors to Slack
  <store>
    @type grep
    <regexp>
      key level
      pattern /^ERROR$/
    </regexp>
    @type slack
    webhook_url https://hooks.slack.com/services/YOUR/WEBHOOK/URL
    channel alerts
    username fluentd
  </store>
</match>
EOF

sudo systemctl restart td-agent

Use Case 5: Application Performance Monitoring Integration¶

# Configure APM log forwarding
sudo tee /etc/td-agent/td-agent.conf > /dev/null <<'EOF'
<source>
  @type tail
  path /var/log/app/*.log
  pos_file /var/log/td-agent/app.pos
  tag app.logs
  <parse>
    @type json
    time_key timestamp
    time_format %Y-%m-%dT%H:%M:%S.%NZ
  </parse>
</source>

# Enrich logs with metadata
<filter app.logs>
  @type record_transformer
  <record>
    hostname "#{Socket.gethostname}"
    environment ${ENV['ENVIRONMENT'] || 'production'}
    service_name myapp
    trace_id ${record['trace_id']}
  </record>
</filter>

# Calculate response time metrics
<filter app.logs>
  @type prometheus
  <metric>
    name http_request_duration_seconds
    type histogram
    desc HTTP request duration
    key response_time
  </metric>
</filter>

# Forward to APM system
<match app.logs>
  @type http
  endpoint http://apm-server:8200/intake/v2/events
  <buffer>
    flush_interval 5s
  </buffer>
</match>
EOF

sudo systemctl restart td-agent

Best Practices¶

Use file-based buffers for production: Memory buffers are faster but file buffers prevent data loss during restarts or crashes. Always use file buffers with appropriate retry settings for critical logs.
Implement proper log rotation and retention: Configure pos_file for tail inputs and set appropriate timekey values in buffers to prevent disk space issues. Use rotate_age and rotate_size for file outputs.
Tag logs hierarchically: Use dot-notation tags (e.g., app.production.web) to enable flexible routing and filtering. This allows you to match patterns like app.** or app.production.*.
Monitor Fluentd performance: Track buffer queue length, retry counts, and emit rates. Use Prometheus plugin or built-in monitoring to detect bottlenecks before they cause data loss.
Secure sensitive data: Use @type secure_forward for encrypted log transmission, filter out sensitive fields with record_modifier, and restrict file permissions on configuration files containing credentials.
Test configuration changes: Always use --dry-run to validate configuration syntax before deploying. Test routing logic with small log volumes before applying to production.
Use multi-worker mode judiciously: Enable workers for CPU-intensive operations (parsing, filtering) but be aware that some plugins don't support multi-worker mode. Start with 2-4 workers and monitor CPU usage.
Implement graceful degradation: Configure overflow_action in buffers to handle backpressure (use drop_oldest_chunk or block based on your requirements). Set reasonable retry_timeout values to prevent infinite retries.
Separate concerns with labels: Use @label directives to create isolated processing pipelines for different log types. This improves maintainability and prevents unintended routing.
Keep plugins updated: Regularly update Fluentd and plugins to get security fixes and performance improvements. Pin plugin versions in production to ensure consistency.

Troubleshooting¶

Issue	Solution
Fluentd won't start	Check syntax: `fluentd -c fluent.conf --dry-run`. Review logs: `sudo journalctl -u td-agent -n 100`. Verify file permissions on config and buffer directories.
Logs not being collected	Verify `pos_file` exists and is writable. Check file path patterns match actual log locations. Ensure log files have read permissions. Test with `tail -f` on the log file.
High memory usage	Switch from memory buffers to file buffers. Reduce `chunk_limit_size` and `queue_limit_length`. Enable multi-worker mode to distribute load. Check for memory leaks in custom plugins.
Buffer queue growing	Increase `flush_interval` or reduce log volume. Check downstream system capacity (Elasticsearch, S3). Verify network connectivity. Review `retry_max_interval` settings.
Logs being dropped	Check buffer `overflow_action` setting. Increase `queue_limit_length` and `chunk_limit_size`. Monitor disk space for file buffers. Review `retry_timeout` configuration.
Plugin installation fails	Ensure Ruby development headers installed: `sudo apt-get install ruby-dev build-essential`. Use correct gem command: `td-agent-gem` not `gem`. Check plugin compatibility with Fluentd version.
Parse errors in logs	Validate parser configuration with sample logs. Use `@type regexp` with proper regex patterns. Add error handling: `emit_invalid_record_to_error true`. Check time format strings.
Cannot connect to Elasticsearch	Verify Elasticsearch is running: `curl http://elasticsearch:9200`. Check firewall rules. Validate credentials if using authentication. Review Elasticsearch logs for rejection reasons.
Duplicate logs appearing	Check `pos_file` location is persistent across restarts. Verify only one Fluentd instance is running. Review `read_from_head` setting (should be `false` in production).
Slow log processing	Enable multi-worker mode. Optimize regex patterns in filters. Use `@type grep` before expensive parsers. Profile with `--trace` flag to identify bottlenecks.
SSL/TLS connection errors	Verify certificate paths and permissions. Check certificate expiration dates. Ensure CA bundle is up to date. Use `verify_ssl false` for testing only (not production).
Time zone issues	Set `utc` or `localtime` in time parser. Use `time_format` with timezone: `%Y-%m-%dT%H:%M:%S%z`. Configure system timezone correctly.