Ollama — Run LLMs on CPU

Introduction

ChatGPT Plus costs $20/month per user. OpenAI API costs scale unpredictably with usage. And every prompt you send leaves your infrastructure, exposing sensitive data to third-party servers.

There's a better way. Modern quantized language models run surprisingly well on CPU-only VPS instances — no expensive GPU required. By the end of this 8-part series, you'll have a fully private AI platform running on a single RamNode VPS for a fraction of the cost.

Why Ollama on CPU?

GPU instances cost $50–300+/month. But modern quantized models in GGUF format run well on CPU-only hardware thanks to efficient inference engines.

Understanding Quantization

Quantization reduces model precision from 16-bit floats to lower bit widths, dramatically shrinking memory requirements with minimal quality loss:

Q4_K_M is the sweet spot — it preserves most of the model's capability while fitting comfortably in modest RAM allocations. RamNode's VPS plans offer the best price-performance ratio for this workload.

Installing Ollama

Ollama provides a one-line installer that handles everything:

curl -fsSL https://ollama.com/install.sh | sh

Verify the installation:

ollama --version

Configure for Network Access

By default, Ollama listens only on localhost. For later parts of this series (Open WebUI, n8n, etc.), configure it to accept connections from Docker containers:

sudo systemctl edit ollama.service

Add the following override:

[Service]
Environment="OLLAMA_HOST=0.0.0.0"
Environment="OLLAMA_ORIGINS=*"

Reload and restart:

sudo systemctl daemon-reload
sudo systemctl restart ollama

sudo ufw allow from 172.16.0.0/12 to any port 11434
sudo ufw deny 11434

Model Selection by RAM Tier

This is the centerpiece of your Ollama deployment. Choose models based on your VPS plan's available RAM:

ollama pull tinyllama
ollama pull phi

ollama pull mistral
ollama pull llama3.1:8b

ollama pull llama3.1:8b
ollama pull deepseek-coder:6.7b

Running Your First Inference

Interactive Chat

Start an interactive session with your model:

ollama run mistral

Type your prompt and press Enter. Use /bye to exit.

REST API

Ollama exposes a REST API for programmatic access — this is how Open WebUI, n8n, and CrewAI will connect in later parts:

curl http://localhost:11434/api/generate -d '{
  "model": "mistral",
  "prompt": "Explain containers in one paragraph.",
  "stream": false
}'

curl http://localhost:11434/api/generate -d '{
  "model": "mistral",
  "prompt": "Write a bash script to monitor disk usage.",
  "stream": true
}'

curl http://localhost:11434/api/chat -d '{
  "model": "mistral",
  "messages": [
    {"role": "system", "content": "You are a helpful DevOps assistant."},
    {"role": "user", "content": "How do I set up a cron job?"}
  ],
  "stream": false
}'

Performance Tuning

Parallel Requests

Control how many requests Ollama handles simultaneously. For CPU inference, keep this conservative:

[Service]
Environment="OLLAMA_NUM_PARALLEL=2"

Context Window vs RAM Tradeoff

Larger context windows use more RAM. The default is typically 2048 tokens. Adjust per-model:

FROM mistral
PARAMETER num_ctx 4096

ollama create mistral-4k -f Modelfile

Swap Space as Safety Net

Add swap to prevent OOM kills if a model temporarily exceeds available RAM:

sudo fallocate -l 2G /swapfile
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile
echo '/swapfile none swap sw 0 0' | sudo tee -a /etc/fstab

Monitor Resource Usage

# Watch memory and CPU in real-time
htop

# Check Ollama-specific resource usage
ollama ps

Persistence & Auto-Start

Ollama's installer configures systemd automatically. Verify it survives reboots:

sudo systemctl enable ollama
sudo systemctl status ollama

Model Pre-Pull Script

Create a script that ensures your preferred models are available after a fresh deployment:

#!/bin/bash
# Pre-pull models after restart
MODELS="mistral llama3.1:8b"

for model in $MODELS; do
  echo "Pulling $model..."
  ollama pull "$model"
done
echo "All models ready."

sudo chmod +x /usr/local/bin/ollama-pull-models.sh

Health Check

Verify Ollama is responding:

curl http://localhost:11434/api/tags

This returns a JSON list of all available models — useful for monitoring and integration testing.