Deploy Dagster on a VPS

Dagster is a data orchestrator built around assets rather than tasks. Instead of describing a DAG of steps, you declare the tables, files, and models your pipeline produces, and Dagster works out the execution order, tracks freshness, and gives you a lineage graph across the whole platform. That asset-first model makes it a strong fit for analytics engineering work where the question is usually "is this table current" rather than "did job 47 run".

This guide covers a production deployment on a RamNode VPS: PostgreSQL for run and event storage, a gRPC code server holding your pipeline code, the Dagster daemon for schedules and sensors, the webserver for the UI, all under systemd with Nginx and TLS in front.

Architecture Overview

A working Dagster OSS deployment is three long running processes plus a database:

• Code location server. A gRPC process that loads your Python definitions. The webserver and daemon both talk to it. Keeping code in its own process means you can redeploy pipeline code without restarting the UI, and a broken import in your code does not take down Dagster itself.
• Dagster daemon. Runs schedules, evaluates sensors, and pulls queued runs off the run queue. Without the daemon, nothing runs on a schedule.
• Dagster webserver. Serves the UI and the GraphQL API. Formerly called Dagit.
• PostgreSQL. Holds run history, event logs, schedule state, and asset materialization records.

Runs execute as subprocesses launched by the daemon on the same box. That is fine for a single VPS. If you outgrow it, the same configuration moves to a DockerRunLauncher or Kubernetes without changing your pipeline code.

What You Will Need

• A RamNode VPS running Ubuntu 24.04 LTS with at least 4 GB RAM and 2 vCPU. Dagster itself is light, but your pipeline runs execute on the same machine and will dominate memory use. Size for the heaviest job you expect to run, not for Dagster.
• Root or sudo access.
• A domain or subdomain with an A record pointing at the VPS.
• Python 3.11 or 3.12.

Step 1: Prepare the System

apt update && apt upgrade -y
apt install -y python3.12 python3.12-venv python3.12-dev build-essential \
  libpq-dev git curl pkg-config

Create a dedicated user and the directory layout.

useradd --system --create-home --home-dir /opt/dagster --shell /bin/bash dagster
mkdir -p /opt/dagster/dagster_home/{logs,artifacts,storage}
mkdir -p /opt/dagster/app
chown -R dagster:dagster /opt/dagster

Add swap on plans of 4 GB or less.

fallocate -l 2G /swapfile
chmod 600 /swapfile
mkswap /swapfile
swapon /swapfile
echo '/swapfile none swap sw 0 0' >> /etc/fstab

Step 2: Install and Configure PostgreSQL

Dagster defaults to SQLite. SQLite cannot handle the daemon, the webserver, and concurrent run processes all writing event logs at once. You will hit database locked errors within a day. Use PostgreSQL from the start.

apt install -y postgresql postgresql-contrib
systemctl enable --now postgresql

sudo -u postgres psql <<'SQL'
CREATE USER dagster WITH PASSWORD 'ReplaceWithAStrongPassword';
CREATE DATABASE dagster OWNER dagster;
GRANT ALL PRIVILEGES ON DATABASE dagster TO dagster;
SQL

Dagster writes one row per step event, which adds up fast on chatty pipelines. Tune /etc/postgresql/16/main/postgresql.conf:

shared_buffers = 512MB
work_mem = 16MB
maintenance_work_mem = 128MB
effective_cache_size = 2GB
max_connections = 100

systemctl restart postgresql

Step 3: Create the Virtual Environment

sudo -u dagster -i
python3.12 -m venv /opt/dagster/venv
source /opt/dagster/venv/bin/activate
pip install --upgrade pip setuptools wheel

Install Dagster and the pieces you need.

pip install dagster dagster-webserver dagster-postgres dagster-graphql

Add integration libraries for whatever your pipelines touch:

pip install dagster-dbt dbt-postgres    # dbt projects
pip install dagster-aws                 # S3 IO managers, ECS
pip install dagster-pandas pandas       # dataframe assets
pip install dagster-duckdb duckdb       # local analytics

Keep everything in this one virtual environment. The gRPC code server, the daemon, and the webserver must all be able to import your pipeline dependencies.

Step 4: Scaffold a Project

Dagster ships a scaffolding command that produces a package layout matching its conventions.

cd /opt/dagster/app
dagster project scaffold --name analytics

That produces:

analytics/
├── analytics/
│   ├── __init__.py
│   ├── assets.py
│   └── definitions.py
├── analytics_tests/
├── pyproject.toml
└── setup.py

Install the project in editable mode so the code server can import it by module name.

cd /opt/dagster/app/analytics
pip install -e .

Write a trivial asset in analytics/assets.py so you have something to run:

import dagster as dg
import pandas as pd


@dg.asset(group_name="demo")
def raw_events(context: dg.AssetExecutionContext) -> pd.DataFrame:
    df = pd.DataFrame({"id": range(100), "value": range(100)})
    context.log.info(f"Generated {len(df)} rows")
    return df


@dg.asset(group_name="demo")
def event_summary(
    context: dg.AssetExecutionContext, raw_events: pd.DataFrame
) -> pd.DataFrame:
    summary = raw_events.groupby(raw_events["id"] % 10)["value"].sum().reset_index()
    context.log.info(f"Summarized to {len(summary)} rows")
    return summary

And wire it up in analytics/definitions.py:

import dagster as dg
from analytics import assets

daily_refresh = dg.ScheduleDefinition(
    name="daily_refresh",
    target=dg.AssetSelection.groups("demo"),
    cron_schedule="0 6 * * *",
    default_status=dg.DefaultScheduleStatus.RUNNING,
)

defs = dg.Definitions(
    assets=dg.load_assets_from_modules([assets]),
    schedules=[daily_refresh],
)

Set default_status to RUNNING so the schedule turns itself on when the code location loads. Otherwise you have to flip it on in the UI after every fresh deploy, which is a common way to discover that nothing has run for a week.

Step 5: Configure DAGSTER_HOME

DAGSTER_HOME is the directory holding dagster.yaml, which controls storage, run launching, and retention. It must be an absolute path, and every Dagster process needs it set.

Create /opt/dagster/dagster_home/dagster.yaml:

storage:
  postgres:
    postgres_db:
      username: dagster
      password:
        env: DAGSTER_PG_PASSWORD
      hostname: localhost
      db_name: dagster
      port: 5432

run_coordinator:
  module: dagster.core.run_coordinator
  class: QueuedRunCoordinator
  config:
    max_concurrent_runs: 4
    tag_concurrency_limits:
      - key: "heavy"
        limit: 1

run_launcher:
  module: dagster.core.launcher
  class: DefaultRunLauncher

compute_logs:
  module: dagster._core.storage.local_compute_log_manager
  class: LocalComputeLogManager
  config:
    base_dir: /opt/dagster/dagster_home/logs

local_artifact_storage:
  module: dagster.core.storage.root
  class: LocalArtifactStorage
  config:
    base_dir: /opt/dagster/dagster_home/artifacts

retention:
  schedule:
    purge_after_days: 30
  sensor:
    purge_after_days:
      skipped: 7
      failure: 30
      success: 7

run_monitoring:
  enabled: true
  start_timeout_seconds: 300
  max_resume_run_attempts: 2
  poll_interval_seconds: 60

telemetry:
  enabled: false

Two settings deserve attention. max_concurrent_runs is your main guard against a single VPS getting flattened by a backfill. Set it low, around one or two per vCPU. The tag_concurrency_limits block lets you mark memory hungry jobs with a heavy tag and cap them at one at a time regardless of the global limit.

run_monitoring catches runs whose process died without reporting failure, which is what happens when the kernel OOM killer picks off a job. Without it, those runs sit in STARTED forever.

Create /opt/dagster/dagster_home/workspace.yaml to point at the code server:

load_from:
  - grpc_server:
      host: 127.0.0.1
      port: 4000
      location_name: "analytics"

Store the database password in an environment file rather than the YAML.

cat > /opt/dagster/dagster.env <<'EOF'
DAGSTER_HOME=/opt/dagster/dagster_home
DAGSTER_PG_PASSWORD=ReplaceWithAStrongPassword
PYTHONUNBUFFERED=1
EOF
chown dagster:dagster /opt/dagster/dagster.env
chmod 600 /opt/dagster/dagster.env

Step 6: Create the systemd Services

Exit to root. Three units, started in order.

Code location server at /etc/systemd/system/dagster-code.service:

[Unit]
Description=Dagster Code Location Server
After=network.target postgresql.service
Requires=postgresql.service

[Service]
Type=simple
User=dagster
Group=dagster
WorkingDirectory=/opt/dagster/app/analytics
EnvironmentFile=/opt/dagster/dagster.env
ExecStart=/opt/dagster/venv/bin/dagster api grpc \
  --host 127.0.0.1 \
  --port 4000 \
  --module-name analytics.definitions \
  --attribute defs
Restart=on-failure
RestartSec=10

[Install]
WantedBy=multi-user.target

Daemon at /etc/systemd/system/dagster-daemon.service:

[Unit]
Description=Dagster Daemon
After=network.target postgresql.service dagster-code.service
Requires=postgresql.service

[Service]
Type=simple
User=dagster
Group=dagster
WorkingDirectory=/opt/dagster/dagster_home
EnvironmentFile=/opt/dagster/dagster.env
ExecStart=/opt/dagster/venv/bin/dagster-daemon run \
  --workspace /opt/dagster/dagster_home/workspace.yaml
Restart=on-failure
RestartSec=10

[Install]
WantedBy=multi-user.target

Webserver at /etc/systemd/system/dagster-webserver.service:

[Unit]
Description=Dagster Webserver
After=network.target postgresql.service dagster-code.service
Requires=postgresql.service

[Service]
Type=simple
User=dagster
Group=dagster
WorkingDirectory=/opt/dagster/dagster_home
EnvironmentFile=/opt/dagster/dagster.env
ExecStart=/opt/dagster/venv/bin/dagster-webserver \
  --host 127.0.0.1 \
  --port 3000 \
  --workspace /opt/dagster/dagster_home/workspace.yaml \
  --path-prefix ""
Restart=on-failure
RestartSec=10

[Install]
WantedBy=multi-user.target

Run exactly one daemon. Two daemons against the same database will double-fire every schedule and double-launch every sensor tick.

Enable and start.

chown -R dagster:dagster /opt/dagster
systemctl daemon-reload
systemctl enable --now dagster-code dagster-daemon dagster-webserver
systemctl status dagster-daemon --no-pager

Step 7: Configure Nginx, TLS, and Authentication

Dagster OSS has no built in authentication. Anyone who reaches the UI can launch runs, edit configuration, and read your logs. Put HTTP basic auth in front of it at minimum.

apt install -y nginx certbot python3-certbot-nginx apache2-utils
htpasswd -c /etc/nginx/.dagster-htpasswd youruser
chmod 640 /etc/nginx/.dagster-htpasswd
chown root:www-data /etc/nginx/.dagster-htpasswd

Create /etc/nginx/sites-available/dagster:

server {
    listen 80;
    server_name dagster.example.com;

    client_max_body_size 50M;

    location / {
        auth_basic "Dagster";
        auth_basic_user_file /etc/nginx/.dagster-htpasswd;

        proxy_pass http://127.0.0.1:3000;
        proxy_http_version 1.1;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;

        # Dagster streams live logs over websockets
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_read_timeout 3600;
        proxy_send_timeout 3600;
        proxy_buffering off;
    }
}

The websocket headers are not optional. Without Upgrade and Connection passed through, the run log viewer will show nothing while a run is executing and only populate after the run finishes.

ln -s /etc/nginx/sites-available/dagster /etc/nginx/sites-enabled/
rm -f /etc/nginx/sites-enabled/default
nginx -t
systemctl reload nginx
certbot --nginx -d dagster.example.com

Firewall:

ufw allow OpenSSH
ufw allow 'Nginx Full'
ufw --force enable

Confirm nothing is exposed directly.

ss -tlnp | grep -E '3000|4000'

Both should show 127.0.0.1 only.

Step 8: Verify the Deployment

Load the UI over HTTPS. You should land on the Assets page with raw_events and event_summary visible.

Check the daemon health from the CLI:

sudo -u dagster DAGSTER_HOME=/opt/dagster/dagster_home \
  /opt/dagster/venv/bin/dagster instance info

The UI has a Deployment tab with a Daemons section. All daemons should read as running with a recent heartbeat. A daemon showing "not running" while systemd reports the service as active almost always means DAGSTER_HOME differs between the daemon and the webserver, so they are reading two different instances.

Materialize the demo assets from the UI, then confirm the run appears in the Runs tab with logs.

Deploying Code Changes

The point of running code in a separate gRPC server is that you can ship pipeline changes without touching the daemon or the UI.

cd /opt/dagster/app/analytics
sudo -u dagster git pull
sudo -u dagster /opt/dagster/venv/bin/pip install -e .
systemctl restart dagster-code

Then click Reload in the code location list in the UI, or let the webserver pick it up on its next poll. Runs already in flight keep executing against the old code, since they run as their own subprocess.

If you want zero downtime on reload, run two code servers on different ports and swap the workspace entry. On a single VPS the restart is fast enough that this is rarely worth the complexity.

Troubleshooting

Code location shows a load error in the UI. The gRPC server started but your module failed to import. Check journalctl -u dagster-code -n 50. Missing dependencies in the venv are the usual cause.

Schedules exist but never fire. Confirm the daemon is running and that the schedule status is RUNNING, not STOPPED, in the Automation tab. Also confirm the schedule's timezone. Dagster cron schedules default to UTC.

Runs stay in QUEUED forever. The daemon is not running, or max_concurrent_runs is already saturated by stuck runs. Terminate the stuck runs from the UI and enable run_monitoring if you have not.

"database is locked" errors. You are still on SQLite. The storage block in dagster.yaml is not being read, usually because DAGSTER_HOME is unset or wrong for that process.

Run process killed with no failure event. The OOM killer took it. Check dmesg -T | grep -i oom. Lower max_concurrent_runs, add swap, or move the heavy job to a tag with a concurrency limit of one.

Logs are empty in the UI but present on disk. The websocket proxy is misconfigured. Recheck the Nginx Upgrade and Connection headers.

Maintenance

Dagster's event log grows continuously. On a busy instance the event_logs table will become the largest object in the database. The retention block in dagster.yaml prunes schedule and sensor ticks, but run records need explicit cleanup:

sudo -u dagster DAGSTER_HOME=/opt/dagster/dagster_home \
  /opt/dagster/venv/bin/dagster run wipe

That wipes everything, so for selective pruning write a small script using DagsterInstance.delete_run() filtered by age. Schedule it as a Dagster job so it shows up in your own run history.

Back up the metadata database on the same cadence as anything else you care about:

sudo -u postgres pg_dump dagster | gzip > /root/dagster-$(date +%F).sql.gz

Upgrade with the services stopped:

systemctl stop dagster-webserver dagster-daemon dagster-code
sudo -u dagster /opt/dagster/venv/bin/pip install --upgrade \
  dagster dagster-webserver dagster-postgres
sudo -u dagster DAGSTER_HOME=/opt/dagster/dagster_home \
  /opt/dagster/venv/bin/dagster instance migrate
systemctl start dagster-code dagster-daemon dagster-webserver

dagster instance migrate applies schema changes to the metadata database. Skipping it after a minor version bump will produce column not found errors on startup.

Hardening Notes

• Basic auth is a floor, not a ceiling. For a team, put an identity aware proxy such as oauth2-proxy or Authelia in front instead.
• The Dagster user can execute arbitrary Python through launched runs. Do not give it sudo, and do not run code locations as root.
• Store secrets in the systemd EnvironmentFile and read them with dg.EnvVar in your definitions. Never commit them to the code location repository.
• Set --read-only on a second webserver instance if you want to give stakeholders visibility without letting them launch runs.
• Restrict Postgres to loopback in pg_hba.conf unless you have a reason to expose it.

Where to Go Next

A single VPS Dagster deployment scales further than most people expect, especially if the heavy compute happens in a warehouse rather than on the box. When you do outgrow it, the migration path is swapping DefaultRunLauncher for DockerRunLauncher or K8sRunLauncher in dagster.yaml, which leaves your asset code untouched. Pair this install with a BI layer such as Apache Superset pointed at the tables your assets produce.