Monitoring Guide

    Grafana Loki Log Aggregation

    Deploy a centralized log aggregation stack with Loki, Promtail, and Grafana on your RamNode VPS. All running via Docker Compose for easy management and portability.

    Ubuntu 22.04
    Docker Compose
    Loki + Promtail + Grafana
    ⏱️ 30–45 minutes

    Prerequisites

    RamNode VPS Requirements

    Plan$4/month Starter or higher recommended
    OSUbuntu 22.04 LTS (recommended)
    RAM1 GB minimum, 2 GB recommended for production
    Disk20 GB SSD minimum (logs grow fast; add more as needed)
    NetworkAny RamNode location; OpenVZ or KVM both supported
    AccessRoot SSH access or a sudo user

    What You Will Need

    • • A running RamNode VPS with Ubuntu 22.04
    • • SSH access to the server
    • • A domain or IP address you can reach (for Grafana UI)
    • • Basic familiarity with the Linux command line
    2

    Initial Server Setup

    Update the System

    SSH into your RamNode VPS and run a full package update before installing anything:

    System Update
    ssh root@YOUR_VPS_IP
    apt update && apt upgrade -y
    apt install -y curl wget vim ufw

    Create a Non-Root User (Recommended)

    Create User
    adduser lokiuser
    usermod -aG sudo lokiuser
    su - lokiuser

    Configure the Firewall

    Open only the ports you need. Grafana listens on port 3000 and Loki on 3100. Keep Loki internal-only if possible.

    Firewall Rules
    ufw allow OpenSSH
    ufw allow 3000/tcp    # Grafana
    # Only open 3100 if you are shipping logs from remote servers
    # ufw allow 3100/tcp  # Loki (optional)
    ufw enable
    ufw status

    Security Note: Do not expose port 3100 publicly without authentication in front of Loki. On a single-node stack, Promtail and Loki communicate over the internal Docker network and 3100 does not need to be open at all.

    3

    Install Docker and Docker Compose

    Install Docker Engine

    Docker Installation
    curl -fsSL https://get.docker.com -o get-docker.sh
    sh get-docker.sh
    usermod -aG docker $USER
    newgrp docker
    docker --version

    Install Docker Compose Plugin

    Docker Compose
    apt install -y docker-compose-plugin
    docker compose version
    4

    Configure Loki

    Create the Project Directory

    Project Directory
    mkdir -p ~/loki-stack && cd ~/loki-stack
    mkdir -p config data/loki data/grafana

    Loki Configuration File

    Create the Loki config at ~/loki-stack/config/loki-config.yaml:

    config/loki-config.yaml
    auth_enabled: false
    
    server:
      http_listen_port: 3100
      grpc_listen_port: 9096
    
    common:
      instance_addr: 127.0.0.1
      path_prefix: /tmp/loki
      storage:
        filesystem:
          chunks_directory: /tmp/loki/chunks
          rules_directory: /tmp/loki/rules
      replication_factor: 1
      ring:
        kvstore:
          store: inmemory
    
    schema_config:
      configs:
        - from: 2020-10-24
          store: tsdb
          object_store: filesystem
          schema: v13
          index:
            prefix: index_
            period: 24h
    
    ruler:
      alertmanager_url: http://localhost:9093
    
    analytics:
      reporting_enabled: false

    💡 Config Note: This is a minimal single-node filesystem-backed config. For production with high log volume, consider switching the object_store to S3-compatible storage such as Wasabi or Cloudflare R2.

    5

    Configure Promtail

    Promtail is the log shipper that reads local log files and pushes them to Loki. Create ~/loki-stack/config/promtail-config.yaml:

    config/promtail-config.yaml
    server:
      http_listen_port: 9080
      grpc_listen_port: 0
    
    positions:
      filename: /tmp/positions.yaml
    
    clients:
      - url: http://loki:3100/loki/api/v1/push
    
    scrape_configs:
      - job_name: system
        static_configs:
          - targets:
              - localhost
            labels:
              job: varlogs
              host: ramnode-vps
              __path__: /var/log/*log
    
      - job_name: docker
        docker_sd_configs:
          - host: unix:///var/run/docker.sock
            refresh_interval: 5s
        relabel_configs:
          - source_labels: ['__meta_docker_container_name']
            regex: '/(.*)'
            target_label: container
    6

    Docker Compose Stack

    Create ~/loki-stack/docker-compose.yml with all three services:

    docker-compose.yml
    version: '3.8'
    
    networks:
      loki:
        driver: bridge
    
    volumes:
      loki_data:
      grafana_data:
    
    services:
      loki:
        image: grafana/loki:latest
        container_name: loki
        restart: unless-stopped
        ports:
          - '127.0.0.1:3100:3100'
        volumes:
          - ./config/loki-config.yaml:/etc/loki/local-config.yaml
          - loki_data:/tmp/loki
        command: -config.file=/etc/loki/local-config.yaml
        networks:
          - loki
    
      promtail:
        image: grafana/promtail:latest
        container_name: promtail
        restart: unless-stopped
        volumes:
          - ./config/promtail-config.yaml:/etc/promtail/config.yml
          - /var/log:/var/log:ro
          - /var/run/docker.sock:/var/run/docker.sock:ro
        command: -config.file=/etc/promtail/config.yml
        networks:
          - loki
        depends_on:
          - loki
    
      grafana:
        image: grafana/grafana:latest
        container_name: grafana
        restart: unless-stopped
        ports:
          - '0.0.0.0:3000:3000'
        environment:
          - GF_SECURITY_ADMIN_USER=admin
          - GF_SECURITY_ADMIN_PASSWORD=changeme123
          - GF_USERS_ALLOW_SIGN_UP=false
        volumes:
          - grafana_data:/var/lib/grafana
        networks:
          - loki
        depends_on:
          - loki

    Password Warning: Change GF_SECURITY_ADMIN_PASSWORD to a strong unique password before starting the stack. Never leave default credentials on a publicly accessible server.

    7

    Start and Verify the Stack

    Start All Services

    Launch Stack
    cd ~/loki-stack
    docker compose up -d
    docker compose ps

    You should see loki, promtail, and grafana all in a Running state.

    Check Loki is Healthy

    Verify Loki
    curl -s http://localhost:3100/ready
    # Expected output: ready
    
    curl -s http://localhost:3100/metrics | grep loki_build

    Check Promtail Targets

    Verify Promtail
    curl -s http://localhost:9080/targets
    # Returns JSON showing which log paths are being scraped

    Check Container Logs

    View Logs
    docker compose logs loki
    docker compose logs promtail
    docker compose logs grafana
    8

    Connect Grafana to Loki

    Open Grafana in your browser at http://YOUR_VPS_IP:3000 and log in with the admin credentials you set in docker-compose.yml.

    Add Loki as a Data Source

    1. In the left sidebar, go to Connections → Data Sources.
    2. Click Add data source.
    3. Search for and select Loki.
    4. Set the URL field to http://loki:3100 (Docker service name resolves internally).
    5. Click Save & Test. You should see a green success banner.

    Explore Logs

    1. Go to Explore (compass icon) in the left sidebar.
    2. Select Loki as the data source from the dropdown at the top.
    3. In the log browser, enter a label filter like {job="varlogs"} to view system logs.
    4. Click the Run Query button or press Shift+Enter.
    9

    Useful LogQL Queries

    Below are some ready-to-use queries for common monitoring tasks on a RamNode VPS.

    💡 LogQL Tip: Loki uses LogQL for querying. A basic query looks like {container="promtail"} |= "error" to show lines containing the word "error" from the promtail container.

    System Log Queries

    System Logs
    # All auth log entries (SSH logins, sudo, failed logins)
    {job="varlogs", filename="/var/log/auth.log"}
    
    # Failed SSH login attempts
    {job="varlogs"} |= "Failed password"
    
    # UFW firewall blocks
    {job="varlogs"} |= "UFW BLOCK"
    
    # Kernel out-of-memory events
    {job="varlogs"} |= "Out of memory"

    Container Log Queries

    Container Logs
    # All logs from a specific container
    {container="nginx"}
    
    # Errors across all containers
    {job="docker"} |= "error" | json | line_format "{{.container}}: {{.msg}}"
    
    # HTTP 5xx errors in nginx
    {container="nginx"} |~ " 5[0-9]{2} "

    Rate Queries

    Rate Queries
    # Log ingestion rate per job (lines per second)
    sum by (job) (rate({job=~".+"}[5m]))
    
    # Error rate across all containers
    sum(rate({job="docker"} |= "error" [5m]))
    10

    Retention and Disk Management

    On a RamNode VPS with limited disk space, log retention is important. Add the following block to your loki-config.yaml to enforce a retention period:

    Add to loki-config.yaml
    # Add under the root level
    limits_config:
      retention_period: 168h  # 7 days
    
    compactor:
      working_directory: /tmp/loki/compactor
      retention_enabled: true
      retention_delete_delay: 2h
      retention_delete_worker_count: 150
      compaction_interval: 10m

    After editing the config, restart Loki:

    Restart Loki
    docker compose restart loki

    Monitor Disk Usage

    Disk Usage
    # Check overall disk
    df -h
    
    # Check how much space Loki data is using
    du -sh ~/loki-stack/
    
    # Check Docker volume sizes
    docker system df -v
    11

    Shipping Logs from Other Servers

    If you have multiple RamNode VPS instances, you can ship logs from all of them to a single central Loki instance. On each additional server, install Promtail and point it at your central Loki IP.

    Install Standalone Promtail

    Install Promtail on Remote Server
    # On the remote server
    PROMTAIL_VERSION=$(curl -s https://api.github.com/repos/grafana/loki/releases/latest \
      | grep tag_name | cut -d '"' -f 4 | sed 's/v//')
    wget https://github.com/grafana/loki/releases/download/v${PROMTAIL_VERSION}/\
      promtail-linux-amd64.zip
    unzip promtail-linux-amd64.zip
    chmod +x promtail-linux-amd64
    mv promtail-linux-amd64 /usr/local/bin/promtail

    Remote Promtail Config

    Update the clients URL to point at your central Loki server:

    Remote Promtail Config Snippet
    clients:
      - url: http://CENTRAL_LOKI_IP:3100/loki/api/v1/push

    Make sure UFW allows port 3100 on the central server from the remote server's IP only:

    Allow Remote Server
    # On the central server, allow only your remote VPS
    ufw allow from REMOTE_VPS_IP to any port 3100

    Troubleshooting

    IssueSolution
    Loki not readyRun docker compose logs loki and check for config parse errors. Verify loki-config.yaml indentation is correct YAML.
    No logs in GrafanaCheck Promtail targets at http://localhost:9080/targets. Confirm the __path__ glob matches files that actually exist.
    Grafana can't reach LokiEnsure both containers are on the same Docker network (loki). Use http://loki:3100 not localhost.
    Disk fullReduce retention_period in loki-config.yaml. Run docker system prune to clear unused images.
    Permission denied on /var/logPromtail container needs read-only access. Confirm the volume mount /var/log:/var/log:ro is present.
    Port 3000 unreachableCheck ufw status. Confirm Grafana is bound to 0.0.0.0:3000 not 127.0.0.1:3000.

    Updating the Stack

    To update all images to their latest versions:

    Update Stack
    cd ~/loki-stack
    docker compose pull
    docker compose up -d
    docker compose ps