Deploy Fluentd & Fluent Bit on a VPS

Fluentd vs. Fluent Bit: Which One Do You Need?

Fluent Bit is written in C and designed to be lightweight (450KB RAM). Ideal as a log forwarder. Fluentd is written in Ruby with 700+ plugins, designed as a log aggregator (40-60MB baseline).

Scenario	Use
Forwarding logs from a 1-2GB RAM VPS	Fluent Bit
Dedicated log aggregation node (4GB+ RAM)	Fluentd
Kubernetes DaemonSet on lightweight nodes	Fluent Bit
Complex multi-source routing with many plugins	Fluentd
Both in a tiered pipeline	Fluent Bit (edge) → Fluentd (aggregator)

Part 1: Fluent Bit

Install Fluent Bit

Fluent Bit provides an official APT repository. Do not use the version in Ubuntu's default package index.

Quick install

curl https://raw.githubusercontent.com/fluent/fluent-bit/master/install.sh | sh

Or add the repository manually:

Manual repository setup

sudo apt-get update
sudo apt-get install -y ca-certificates curl gnupg

curl -fsSL https://packages.fluentbit.io/fluentbit.key | sudo gpg --dearmor -o /usr/share/keyrings/fluentbit-keyring.gpg

echo "deb [signed-by=/usr/share/keyrings/fluentbit-keyring.gpg] https://packages.fluentbit.io/ubuntu/jammy jammy main" \
  | sudo tee /etc/apt/sources.list.d/fluent-bit.list

sudo apt-get update
sudo apt-get install -y fluent-bit

Verify and start

/opt/fluent-bit/bin/fluent-bit --version
sudo systemctl enable fluent-bit
sudo systemctl start fluent-bit
sudo systemctl status fluent-bit

Configure Fluent Bit Basics

The main configuration file is /etc/fluent-bit/fluent-bit.conf. It uses four primary sections: [SERVICE], [INPUT], [FILTER], [OUTPUT].

Minimal test config

[SERVICE]
    Flush        5
    Daemon       Off
    Log_Level    info
    Parsers_File /etc/fluent-bit/parsers.conf

[INPUT]
    Name          systemd
    Tag           host.systemd
    Systemd_Filter _SYSTEMD_UNIT=nginx.service
    Read_From_Tail On

[OUTPUT]
    Name  stdout
    Match *

Restart after config changes

sudo systemctl restart fluent-bit
sudo journalctl -u fluent-bit -f

Practical Pipeline: Nginx + Systemd

This config collects nginx access logs and systemd journal entries, adds a hostname field, and forwards everything to a remote Fluentd aggregator.

/etc/fluent-bit/fluent-bit.conf

[SERVICE]
    Flush        5
    Daemon       Off
    Log_Level    warn
    Parsers_File /etc/fluent-bit/parsers.conf
    HTTP_Server  On
    HTTP_Listen  0.0.0.0
    HTTP_Port    2020

[INPUT]
    Name              tail
    Tag               nginx.access
    Path              /var/log/nginx/access.log
    Parser            nginx
    DB                /var/lib/fluent-bit/nginx.db
    Mem_Buf_Limit     5MB
    Skip_Long_Lines   On
    Refresh_Interval  10

[INPUT]
    Name              tail
    Tag               nginx.error
    Path              /var/log/nginx/error.log
    DB                /var/lib/fluent-bit/nginx-error.db
    Mem_Buf_Limit     5MB

[INPUT]
    Name           systemd
    Tag            host.*
    Read_From_Tail On
    Strip_Underscores On

[FILTER]
    Name   record_modifier
    Match  *
    Record hostname ${HOSTNAME}
    Record environment production

[FILTER]
    Name  grep
    Match host.*
    Regex PRIORITY [0-3]

[OUTPUT]
    Name          forward
    Match         *
    Host          10.0.0.5
    Port          24224
    Shared_Key    your_shared_key_here
    Self_Hostname ${HOSTNAME}
    tls           off

The DB parameter stores read position offsets. The grep filter passes only PRIORITY 0-3 (emergency through error).

Create state directory

sudo mkdir -p /var/lib/fluent-bit
sudo chown fluent-bit:fluent-bit /var/lib/fluent-bit

Fluent Bit Output Options

Output to Local File

File output (JSON)

[OUTPUT]
    Name   file
    Match  nginx.*
    Path   /var/log/fluent-bit/
    File   nginx.json
    Format json_lines

Output to Elasticsearch / OpenSearch

Elasticsearch output

[OUTPUT]
    Name              es
    Match             *
    Host              your-es-host
    Port              9200
    Index             logs
    Type              _doc
    HTTP_User         elastic
    HTTP_Passwd       yourpassword
    tls               On
    tls.verify        On
    Suppress_Type_Name On
    Logstash_Format   On
    Logstash_Prefix   ramnode-vps
    Retry_Limit       5

Monitor Fluent Bit

The built-in HTTP server exposes useful endpoints:

Check metrics and health

curl http://localhost:2020/api/v1/metrics
curl http://localhost:2020/api/v1/health
curl -s http://localhost:2020/api/v1/metrics | jq '.output'

Part 2: Fluentd

Install Fluentd

Fluentd makes the most sense on a dedicated aggregation node with at least 1-2GB RAM.

Install td-agent (Fluentd)

curl -fsSL https://toolbelt.treasuredata.com/sh/install-ubuntu-jammy-td-agent4.sh | sh

Enable and start

sudo systemctl enable td-agent
sudo systemctl start td-agent
sudo systemctl status td-agent

Path	Purpose
/etc/td-agent/td-agent.conf	Main config
/var/log/td-agent/td-agent.log	Fluentd log
/usr/sbin/td-agent-gem	Plugin installer

Configure Fluentd

Fluentd uses an XML-style DSL. A minimal config receiving from Fluent Bit forwarders:

/etc/td-agent/td-agent.conf

<system>
  log_level warn
  workers 2
</system>

<source>
  @type forward
  port 24224
  bind 0.0.0.0
  <security>
    shared_key your_shared_key_here
    self_hostname aggregator.yourdomain.com
  </security>
</source>

<match nginx.**>
  @type file
  path /var/log/td-agent/nginx/%Y/%m/%d/access
  append true
  <format>
    @type json
  </format>
  <buffer time>
    @type file
    path /var/log/td-agent/buffer/nginx
    timekey 1h
    timekey_wait 10m
    flush_interval 30s
    chunk_limit_size 8m
  </buffer>
</match>

<match **>
  @type file
  path /var/log/td-agent/all/%Y/%m/%d/log
  append true
  <format>
    @type json
  </format>
  <buffer time>
    @type file
    path /var/log/td-agent/buffer/all
    timekey 1h
    timekey_wait 10m
    flush_interval 60s
  </buffer>
</match>

Fluentd Plugins & Outputs

Installing Plugins

Install common plugins

sudo td-agent-gem install fluent-plugin-elasticsearch
sudo td-agent-gem install fluent-plugin-s3
sudo td-agent-gem install fluent-plugin-prometheus
sudo systemctl restart td-agent

Elasticsearch Output

Elasticsearch match block

<match **>
  @type elasticsearch
  host your-es-host
  port 9200
  user elastic
  password yourpassword
  scheme https
  ssl_verify true
  logstash_format true
  logstash_prefix ramnode
  <buffer>
    @type file
    path /var/log/td-agent/buffer/es
    flush_interval 10s
    retry_max_interval 30
    retry_forever false
    retry_max_times 17
  </buffer>
</match>

S3 / B2 Output (Long-Term Archival)

S3-compatible output

<match **>
  @type s3
  aws_key_id      YOUR_ACCESS_KEY
  aws_sec_key     YOUR_SECRET_KEY
  s3_bucket       your-log-bucket
  s3_region       us-east-1
  s3_endpoint     https://s3.us-east-004.backblazeb2.com
  path            logs/${tag}/%Y/%m/%d/
  s3_object_key_format %{path}%{time_slice}_%{index}.%{file_extension}
  store_as        gzip
  <format>
    @type json
  </format>
  <buffer tag,time>
    @type file
    path /var/log/td-agent/buffer/s3
    timekey 3600
    timekey_wait 10m
    chunk_limit_size 256m
  </buffer>
</match>

Firewall Configuration

If receiving forwarded logs from other servers, open the forward port:

UFW rules

sudo ufw allow 24224/tcp comment "Fluentd forward input"
sudo ufw allow 24224/udp comment "Fluentd forward input UDP"

Restrict to specific source IPs where possible:

Restrict by source IP

sudo ufw allow from 203.0.113.10 to any port 24224

Building the Tiered Pipeline

The most resilient setup: Fluent Bit on every application node forwarding to a single Fluentd aggregator.

output

[App Server 1]         [App Server 2]         [App Server 3]
 Fluent Bit             Fluent Bit             Fluent Bit
    |                      |                      |
    +----------+-----------+                      |
               |                                  |
               +------------------+---------------+
                                  |
                          [Aggregator VPS]
                            Fluentd
                               |
                    +----------+----------+
                    |                     |
             Elasticsearch           S3 / B2

On each application node

/etc/fluent-bit/fluent-bit.conf

[SERVICE]
    Flush     5
    Log_Level warn

[INPUT]
    Name              tail
    Tag               app.${HOSTNAME}.nginx
    Path              /var/log/nginx/access.log
    Parser            nginx
    DB                /var/lib/fluent-bit/nginx.db

[FILTER]
    Name   record_modifier
    Match  *
    Record server ${HOSTNAME}

[OUTPUT]
    Name        forward
    Match       *
    Host        10.0.0.5
    Port        24224
    Shared_Key  change_this_to_a_strong_key
    Self_Hostname ${HOSTNAME}

On the aggregator node

/etc/td-agent/td-agent.conf

<source>
  @type forward
  port 24224
  bind 0.0.0.0
  <security>
    shared_key change_this_to_a_strong_key
    self_hostname aggregator
  </security>
</source>

<filter app.**>
  @type record_transformer
  <record>
    environment production
    datacenter ramnode-nyc
  </record>
</filter>

<match app.**>
  @type elasticsearch
  host localhost
  port 9200
  logstash_format true
  logstash_prefix ramnode-apps
  <buffer>
    @type file
    path /var/log/td-agent/buffer/es
    flush_interval 5s
  </buffer>
</match>

Troubleshooting

Fluent Bit not picking up new log lines

Reset DB state

sudo rm /var/lib/fluent-bit/nginx.db
sudo systemctl restart fluent-bit

Permission errors reading log files

Fix permissions

sudo usermod -aG adm fluent-bit
sudo systemctl restart fluent-bit

High memory usage on Fluentd

Check buffer accumulation

du -sh /var/log/td-agent/buffer/
sudo tail -100 /var/log/td-agent/td-agent.log | grep -i error

Testing configs without restarting

Dry run

sudo /opt/fluent-bit/bin/fluent-bit -c /etc/fluent-bit/fluent-bit.conf --dry-run
sudo td-agent --dry-run -c /etc/td-agent/td-agent.conf

Verifying forward connectivity

Test forwarding

echo '{"test":"message"}' | /opt/fluent-bit/bin/fluent-bit \
  -i stdin -o forward://10.0.0.5:24224 -p shared_key=change_this_to_a_strong_key

Resource Usage on Budget VPS Plans

Component	RAM	CPU (idle)	CPU (active)
Fluent Bit (2-3 inputs)	30-60MB	~0%	1-3%
Fluentd (td-agent, 2 workers)	80-200MB	~1%	5-15%
Fluentd + Elasticsearch plugin	150-300MB	~1%	5-20%

On a 512MB VPS, Fluent Bit as a forwarder leaves plenty of headroom. Fluentd as an aggregator needs at least 1GB RAM dedicated to it.

Deploy Fluentd & Fluent Bit on a RamNode VPS