CI/CD Platform

Concourse CI

Deploy a modern, scalable CI/CD platform with pipeline-as-code on RamNode VPS. Containerized builds, reproducible pipelines, and distributed workers.

Ubuntu 22.04 / Debian 12

Docker Compose

Pipeline as Code

What is Concourse CI?

Concourse CI is an open-source continuous integration and deployment system designed around the principles of pipeline-as-code, containerized builds, reproducibility, and scalability. Unlike traditional CI/CD tools, Concourse uses a declarative approach where every configuration is versioned and reproducible.

Pipeline as Code

All configuration defined in versioned YAML files

Containerized Builds

Every build runs in isolated containers

Scalable

Distributed worker architecture for horizontal scaling

Prerequisites

A RamNode VPS with at least 2 CPU cores and 4GB RAM (recommended: 4 cores, 8GB RAM)
Ubuntu 22.04 LTS or Debian 12
Root or sudo access
A domain name pointed to your VPS (optional but recommended)
Basic familiarity with Docker and YAML

Initial Server Setup

Update system and configure firewall

# Update system packages
sudo apt update && sudo apt upgrade -y

# Install required dependencies
sudo apt install -y curl wget git ufw ca-certificates gnupg lsb-release

# Configure UFW firewall
sudo ufw allow 22/tcp
sudo ufw allow 80/tcp
sudo ufw allow 443/tcp
sudo ufw allow 8080/tcp   # Concourse web interface
sudo ufw --force enable

Install Docker and Docker Compose

Install Docker

# Add Docker's official GPG key
sudo install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
sudo chmod a+r /etc/apt/keyrings/docker.gpg

# Add Docker repository
echo \
  "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \
  $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

# Install Docker Engine
sudo apt update
sudo apt install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

# Start and enable Docker
sudo systemctl start docker
sudo systemctl enable docker

# Add your user to docker group (logout/login required)
sudo usermod -aG docker $USER

Verify installation:

Verify Docker

docker --version
docker compose version

Create Directory Structure

Create Concourse directories

# Create Concourse directory
sudo mkdir -p /opt/concourse
cd /opt/concourse

# Create subdirectories
sudo mkdir -p keys/web keys/worker data/db data/worker

Generate SSH Keys

Concourse uses SSH keys for secure communication between components:

Generate SSH keys

# Generate web keys
sudo ssh-keygen -t rsa -f keys/web/tsa_host_key -N '' -C "concourse-tsa"
sudo ssh-keygen -t rsa -f keys/web/session_signing_key -N '' -C "concourse-session"

# Generate worker keys
sudo ssh-keygen -t rsa -f keys/worker/worker_key -N '' -C "concourse-worker"

# Copy public keys for authorization
sudo cp keys/worker/worker_key.pub keys/web/authorized_worker_keys
sudo cp keys/web/tsa_host_key.pub keys/worker/

# Set proper permissions
sudo chmod 600 keys/web/* keys/worker/*
sudo chmod 644 keys/web/*.pub keys/worker/*.pub keys/web/authorized_worker_keys

Create Docker Compose Configuration

Create the Docker Compose file for Concourse:

/opt/concourse/docker-compose.yml

version: '3.8'

services:
  db:
    image: postgres:15-alpine
    container_name: concourse-db
    restart: unless-stopped
    environment:
      POSTGRES_DB: concourse
      POSTGRES_USER: concourse
      POSTGRES_PASSWORD: changeme_secure_password
      PGDATA: /var/lib/postgresql/data/pgdata
    volumes:
      - ./data/db:/var/lib/postgresql/data
    networks:
      - concourse-net
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U concourse"]
      interval: 10s
      timeout: 5s
      retries: 5

  web:
    image: concourse/concourse:latest
    container_name: concourse-web
    restart: unless-stopped
    command: web
    depends_on:
      db:
        condition: service_healthy
    ports:
      - "8080:8080"
    volumes:
      - ./keys/web:/concourse-keys
    environment:
      CONCOURSE_POSTGRES_HOST: db
      CONCOURSE_POSTGRES_USER: concourse
      CONCOURSE_POSTGRES_PASSWORD: changeme_secure_password
      CONCOURSE_POSTGRES_DATABASE: concourse
      CONCOURSE_EXTERNAL_URL: http://your-domain.com:8080
      CONCOURSE_ADD_LOCAL_USER: admin:changeme_admin_password
      CONCOURSE_MAIN_TEAM_LOCAL_USER: admin
      CONCOURSE_TSA_HOST_KEY: /concourse-keys/tsa_host_key
      CONCOURSE_TSA_AUTHORIZED_KEYS: /concourse-keys/authorized_worker_keys
      CONCOURSE_SESSION_SIGNING_KEY: /concourse-keys/session_signing_key
      CONCOURSE_BUILD_LOG_RETENTION_DEFAULT: "100"
      CONCOURSE_GC_INTERVAL: 30s
    networks:
      - concourse-net
    healthcheck:
      test: ["CMD", "wget", "-q", "-O", "-", "http://localhost:8080/api/v1/info"]
      interval: 30s
      timeout: 10s
      retries: 3

  worker:
    image: concourse/concourse:latest
    container_name: concourse-worker
    restart: unless-stopped
    command: worker
    privileged: true
    depends_on:
      web:
        condition: service_healthy
    volumes:
      - ./keys/worker:/concourse-keys
      - ./data/worker:/concourse-work-dir
    environment:
      CONCOURSE_TSA_HOST: web:2222
      CONCOURSE_TSA_PUBLIC_KEY: /concourse-keys/tsa_host_key.pub
      CONCOURSE_TSA_WORKER_PRIVATE_KEY: /concourse-keys/worker_key
      CONCOURSE_WORK_DIR: /concourse-work-dir
      CONCOURSE_BIND_IP: 0.0.0.0
      CONCOURSE_BAGGAGECLAIM_DRIVER: overlay
      CONCOURSE_GARDEN_MAX_CONTAINERS: "250"
    networks:
      - concourse-net
    stop_grace_period: 5m

networks:
  concourse-net:
    driver: bridge

Important: Replace the following values:

• changeme_secure_password - Strong PostgreSQL password
• changeme_admin_password - Strong admin password
• your-domain.com - Your actual domain or VPS IP address

Start Concourse CI

Launch Concourse

cd /opt/concourse
sudo docker compose up -d

# Check the status of your containers
sudo docker compose ps
sudo docker compose logs -f

Wait for all services to be healthy (this may take 1-2 minutes on first startup).

Install Fly CLI

The Fly CLI is the command-line tool for interacting with Concourse:

Install Fly CLI

# Download Fly CLI from your Concourse instance
wget "http://your-domain.com:8080/api/v1/cli?arch=amd64&platform=linux" -O fly

# Make it executable
chmod +x fly

# Move to system path
sudo mv fly /usr/local/bin/

# Verify installation
fly --version

Authenticate with Fly

# Login to Concourse (target name: 'main')
fly -t main login -c http://your-domain.com:8080 -u admin -p changeme_admin_password

# Verify you're logged in
fly -t main teams
fly -t main workers

Create Your First Pipeline

Create pipeline directory

mkdir ~/concourse-pipelines
cd ~/concourse-pipelines

Create a hello-world pipeline:

hello-world.yml

jobs:
  - name: hello-world
    plan:
      - task: say-hello
        config:
          platform: linux
          image_resource:
            type: registry-image
            source:
              repository: busybox
          run:
            path: echo
            args: ["Hello from Concourse CI on RamNode!"]

Deploy and run the pipeline:

Deploy pipeline

# Set the pipeline
fly -t main set-pipeline -p hello-world -c hello-world.yml

# Unpause the pipeline
fly -t main unpause-pipeline -p hello-world

# Trigger the job
fly -t main trigger-job -j hello-world/hello-world --watch

Setup Nginx Reverse Proxy with SSL

Install Nginx and Certbot

sudo apt install -y nginx certbot python3-certbot-nginx

Create Nginx configuration:

/etc/nginx/sites-available/concourse

server {
    listen 80;
    server_name your-domain.com;

    location / {
        return 301 https://$server_name$request_uri;
    }
}

server {
    listen 443 ssl http2;
    server_name your-domain.com;

    client_max_body_size 100M;

    location / {
        proxy_pass http://localhost:8080;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;

        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";

        proxy_read_timeout 3600;
        proxy_connect_timeout 3600;
        proxy_send_timeout 3600;
    }
}

Enable site and get SSL

# Enable the site
sudo ln -s /etc/nginx/sites-available/concourse /etc/nginx/sites-enabled/

# Test Nginx configuration
sudo nginx -t

# Obtain SSL certificate
sudo certbot --nginx -d your-domain.com

# Reload Nginx
sudo systemctl reload nginx

Update CONCOURSE_EXTERNAL_URL in docker-compose.yml to use HTTPS, then restart Concourse.

Security Best Practices

1. Change Default Credentials

Always change the default admin password immediately after deployment. Update the docker-compose.yml and restart Concourse.

2. Enable RBAC with Teams

Create teams with different access

# Create a development team
fly -t main set-team -n dev-team \
  --local-user dev-user:dev-password

# Login as the new team
fly -t dev login -c https://your-domain.com -n dev-team -u dev-user

3. Secrets Management

Use credential parameters for sensitive data:

Pipeline with secrets

resources:
  - name: private-repo
    type: git
    source:
      uri: https://github.com/org/private-repo.git
      branch: master
      username: ((github.username))
      password: ((github.password))

Monitoring and Maintenance

View Logs

View service logs

# View all logs
sudo docker compose logs -f

# View specific service logs
sudo docker compose logs -f web
sudo docker compose logs -f worker
sudo docker compose logs -f db

Database Backup

/opt/concourse/backup.sh

#!/bin/bash
BACKUP_DIR="/opt/concourse/backups"
DATE=$(date +%Y%m%d_%H%M%S)

mkdir -p $BACKUP_DIR

# Backup PostgreSQL database
docker exec concourse-db pg_dump -U concourse concourse | gzip > $BACKUP_DIR/concourse_db_$DATE.sql.gz

# Keep only last 7 days of backups
find $BACKUP_DIR -name "concourse_db_*.sql.gz" -mtime +7 -delete

echo "Backup completed: concourse_db_$DATE.sql.gz"

Schedule backup

sudo chmod +x /opt/concourse/backup.sh
sudo crontab -e

# Add daily backup at 2 AM:
0 2 * * * /opt/concourse/backup.sh

Update Concourse

Update to latest version

cd /opt/concourse
sudo docker compose pull
sudo docker compose down
sudo docker compose up -d

Troubleshooting

Workers Not Connecting

Debug worker issues

# Check worker logs
sudo docker compose logs worker

# Verify SSH keys are correctly generated
ls -la /opt/concourse/keys/web/
ls -la /opt/concourse/keys/worker/

Database Connection Issues

Check database

# Check PostgreSQL status
sudo docker compose logs db

# Verify database credentials match in docker-compose.yml

Pipeline Not Triggering

Debug pipeline

# Check resource status
fly -t main check-resource -r pipeline-name/resource-name

# View pipeline configuration
fly -t main get-pipeline -p pipeline-name

Additional Resources

Concourse CI Documentation Concourse GitHub Repository