Managing systemd Services

    Create, manage, and troubleshoot background services on your Linux VPS

    systemd is the init system and service manager used by all major Linux distributions — Ubuntu, Debian, AlmaLinux, Rocky Linux, and more. It manages every background process (daemon) on your server, from web servers and databases to your own custom applications. This guide covers everything you need to manage, create, and troubleshoot systemd services on your VPS.

    What You'll Learn

    Start, stop, restart, and enable services
    Write custom unit files from scratch
    View and filter logs with journalctl
    Replace cron jobs with systemd timers
    Set resource limits and security sandboxing
    Diagnose and fix common service failures

    Prerequisites

    1. Service Management Basics

    The primary tool for managing services is systemctl. Here are the commands you'll use most often:

    Essential systemctl commands
    # Start a service
    sudo systemctl start nginx
    
    # Stop a service
    sudo systemctl stop nginx
    
    # Restart a service (stop + start)
    sudo systemctl restart nginx
    
    # Reload configuration without stopping (if supported)
    sudo systemctl reload nginx
    
    # Reload or restart — tries reload first, falls back to restart
    sudo systemctl reload-or-restart nginx
    
    # Enable a service to start on boot
    sudo systemctl enable nginx
    
    # Disable a service from starting on boot
    sudo systemctl disable nginx
    
    # Enable AND start in one command
    sudo systemctl enable --now nginx
    
    # Disable AND stop in one command
    sudo systemctl disable --now nginx

    💡 Tip: enable doesn't start the service — it only sets it to start on boot. Use enable --now to do both in one step.

    2. Inspecting Service Status

    Check service status
    # Detailed status with recent log output
    sudo systemctl status nginx

    Example output:

    Example status output
    ● nginx.service - A high performance web server
         Loaded: loaded (/usr/lib/systemd/system/nginx.service; enabled; preset: enabled)
         Active: active (running) since Mon 2026-02-24 10:15:32 UTC; 1 day ago
        Process: 1234 ExecStartPre=/usr/sbin/nginx -t (code=exited, status=0/SUCCESS)
       Main PID: 1235 (nginx)
          Tasks: 3 (limit: 4614)
         Memory: 8.5M
            CPU: 1.234s
         CGroup: /system.slice/nginx.service
                 ├─1235 "nginx: master process /usr/sbin/nginx"
                 ├─1236 "nginx: worker process"
                 └─1237 "nginx: worker process"

    Key fields to check:

    FieldMeaning
    LoadedWhether the unit file was found and parsed; shows enabled or disabled for boot
    Activeactive (running), inactive (dead), or failed
    Main PIDThe process ID of the main service process
    CGroupAll child processes managed by this service
    More inspection commands
    # Check if a service is active (good for scripts)
    systemctl is-active nginx
    
    # Check if a service is enabled on boot
    systemctl is-enabled nginx
    
    # List all running services
    systemctl list-units --type=service --state=running
    
    # List all failed services
    systemctl list-units --type=service --state=failed
    
    # List all installed services (including inactive)
    systemctl list-unit-files --type=service
    
    # Show the unit file contents
    systemctl cat nginx.service
    
    # Show all properties of a service
    systemctl show nginx.service

    3. Creating a Custom Service

    Custom services let you run your own applications as background daemons — Node.js apps, Python scripts, Go binaries, or anything else. Unit files live in /etc/systemd/system/.

    Example: Node.js Application

    /etc/systemd/system/myapp.service
    [Unit]
    Description=My Node.js Application
    Documentation=https://example.com/docs
    After=network.target
    
    [Service]
    Type=simple
    User=deploy
    Group=deploy
    WorkingDirectory=/home/deploy/myapp
    ExecStart=/usr/bin/node server.js
    Restart=on-failure
    RestartSec=5
    StandardOutput=journal
    StandardError=journal
    SyslogIdentifier=myapp
    Environment="NODE_ENV=production"
    Environment="PORT=3000"
    
    [Install]
    WantedBy=multi-user.target
    Install and start the service
    # Reload systemd to pick up the new unit file
    sudo systemctl daemon-reload
    
    # Enable and start in one step
    sudo systemctl enable --now myapp
    
    # Verify it's running
    sudo systemctl status myapp

    Example: Python Application

    /etc/systemd/system/myapi.service
    [Unit]
    Description=My Python API
    After=network.target
    
    [Service]
    Type=simple
    User=deploy
    Group=deploy
    WorkingDirectory=/home/deploy/myapi
    ExecStart=/home/deploy/myapi/venv/bin/gunicorn -w 4 -b 0.0.0.0:8000 app:app
    Restart=always
    RestartSec=3
    Environment="PYTHONUNBUFFERED=1"
    
    [Install]
    WantedBy=multi-user.target

    Example: Simple Script

    /etc/systemd/system/myscript.service
    [Unit]
    Description=Run my backup script
    After=network.target
    
    [Service]
    Type=oneshot
    ExecStart=/usr/local/bin/backup.sh
    StandardOutput=journal

    ⚠️ Important: Always run sudo systemctl daemon-reload after creating or modifying a unit file. systemd caches unit files and won't see your changes until you reload.

    4. Unit File Reference

    A unit file has three sections: [Unit], [Service], and [Install]. Here are the most commonly used directives:

    [Unit] Section

    DirectiveDescription
    Description=Human-readable name shown in status output
    After=Start this service after the listed units
    Requires=Hard dependency — if the dependency fails, this service fails too
    Wants=Soft dependency — start alongside, but don't fail if it doesn't start

    [Service] Section

    DirectiveDescription
    Type=simple (default), forking, oneshot, notify
    ExecStart=Command to start the service (must be an absolute path)
    ExecStartPre=Commands to run before ExecStart (e.g., config tests)
    ExecReload=Command to run on systemctl reload
    ExecStop=Command to stop the service (default: SIGTERM)
    User= / Group=Run the service as this user/group
    WorkingDirectory=Set the working directory before starting
    Environment=Set environment variables
    EnvironmentFile=Load environment variables from a file
    Restart=no, on-failure, on-abnormal, always
    RestartSec=Seconds to wait before restarting

    Service Types Explained

    TypeUse When
    simpleThe process stays in the foreground (Node.js, Python, Go)
    forkingThe process forks and the parent exits (traditional daemons like Apache)
    oneshotThe process exits after completing (scripts, migrations)
    notifyThe process signals systemd when it's ready (advanced)

    Using Environment Files

    Instead of putting secrets directly in the unit file, use an environment file:

    /etc/myapp/env
    DATABASE_URL=postgresql://user:pass@localhost/mydb
    SECRET_KEY=your-secret-key-here
    NODE_ENV=production
    Reference in unit file
    [Service]
    EnvironmentFile=/etc/myapp/env
    Protect the env file
    sudo chmod 600 /etc/myapp/env
    sudo chown root:root /etc/myapp/env

    5. Dependencies & Ordering

    You can control the order services start and express dependencies between them:

    Example: App that needs a database
    [Unit]
    Description=My Web Application
    After=network.target mariadb.service
    Requires=mariadb.service
    
    [Service]
    Type=simple
    ExecStart=/usr/local/bin/myapp
    
    [Install]
    WantedBy=multi-user.target
    DirectiveEffect
    After=Start after the listed unit (ordering only, no dependency)
    Before=Start before the listed unit
    Requires=If the dependency stops or fails, stop this service too
    Wants=Try to start the dependency, but don't fail if it doesn't

    💡 Tip: After= controls order, while Requires= controls dependency. You usually want both together. Using Requires= without After= means they can start simultaneously.

    6. Logging with journalctl

    systemd captures all stdout/stderr from services into a structured journal. journalctl is the tool for reading it.

    Viewing service logs
    # Show all logs for a specific service
    sudo journalctl -u nginx
    
    # Follow logs in real time (like tail -f)
    sudo journalctl -u nginx -f
    
    # Show only the last 50 lines
    sudo journalctl -u nginx -n 50
    
    # Show logs since a specific time
    sudo journalctl -u nginx --since "2026-02-24 10:00"
    
    # Show logs from the last hour
    sudo journalctl -u nginx --since "1 hour ago"
    
    # Show logs from the current boot only
    sudo journalctl -u nginx -b
    
    # Show logs from the previous boot
    sudo journalctl -u nginx -b -1
    Filtering and formatting
    # Show only errors and above
    sudo journalctl -u nginx -p err
    
    # Priority levels: emerg, alert, crit, err, warning, notice, info, debug
    
    # Show in reverse order (newest first)
    sudo journalctl -u nginx -r
    
    # Output as JSON (useful for parsing)
    sudo journalctl -u nginx -o json-pretty
    
    # Show logs for multiple services
    sudo journalctl -u nginx -u php-fpm
    
    # Show kernel messages
    sudo journalctl -k
    
    # Show all logs from a specific PID
    sudo journalctl _PID=1234

    Managing Journal Size

    The journal can grow large over time. Control its size:

    Journal disk usage
    # Check how much disk space the journal uses
    sudo journalctl --disk-usage
    
    # Retain only the last 7 days of logs
    sudo journalctl --vacuum-time=7d
    
    # Limit journal size to 500MB
    sudo journalctl --vacuum-size=500M
    
    # Remove everything except the 5 most recent log files
    sudo journalctl --vacuum-files=5

    To set a permanent size limit, edit /etc/systemd/journald.conf:

    /etc/systemd/journald.conf
    [Journal]
    SystemMaxUse=500M
    SystemKeepFree=1G
    MaxRetentionSec=30day
    Apply changes
    sudo systemctl restart systemd-journald

    7. Timers (cron Alternative)

    systemd timers are a modern alternative to cron jobs. They provide better logging (journalctl), dependency management, and don't require a separate daemon.

    Step 1: Create the service to run

    /etc/systemd/system/backup.service
    [Unit]
    Description=Daily database backup
    
    [Service]
    Type=oneshot
    User=deploy
    ExecStart=/usr/local/bin/backup.sh
    StandardOutput=journal

    Step 2: Create the timer

    /etc/systemd/system/backup.timer
    [Unit]
    Description=Run backup daily at 3:00 AM
    
    [Timer]
    OnCalendar=*-*-* 03:00:00
    Persistent=true
    
    [Install]
    WantedBy=timers.target
    Enable the timer
    sudo systemctl daemon-reload
    sudo systemctl enable --now backup.timer
    
    # Verify the timer is active
    sudo systemctl list-timers
    
    # Check when it will fire next
    sudo systemctl status backup.timer

    Common Timer Schedules

    OnCalendar ValueMeaning
    *-*-* 03:00:00Daily at 3 AM
    Mon *-*-* 03:00:00Every Monday at 3 AM
    *-*-01 00:00:00First day of every month
    hourlyEvery hour
    dailyEvery day at midnight
    weeklyEvery Monday at midnight
    Test your calendar expression
    # Validate a timer schedule — shows next trigger times
    systemd-analyze calendar "*-*-* 03:00:00"
    
    # Run the service manually to test
    sudo systemctl start backup.service

    💡 Tip: Setting Persistent=true means if the server was off when the timer should have fired, it will run immediately on the next boot — similar to anacron.

    8. Resource Limits

    Prevent a service from consuming all server resources by setting limits in the [Service] section:

    Resource limit directives
    [Service]
    # Limit memory usage (kills service if exceeded)
    MemoryMax=512M
    
    # Soft memory limit (triggers reclaim)
    MemoryHigh=400M
    
    # Limit CPU usage (100% = 1 core)
    CPUQuota=50%
    
    # Limit number of tasks/threads
    TasksMax=100
    
    # Limit number of open file descriptors
    LimitNOFILE=65536
    
    # Limit number of processes
    LimitNPROC=4096

    You can also apply limits without editing the unit file using systemctl edit:

    Override limits with drop-in
    # Opens an editor to create a drop-in override
    sudo systemctl edit myapp
    
    # Add your overrides between the comments:
    # [Service]
    # MemoryMax=256M
    
    # The override is saved to:
    # /etc/systemd/system/myapp.service.d/override.conf
    
    # Reload after editing
    sudo systemctl daemon-reload
    sudo systemctl restart myapp

    9. Security Hardening

    systemd provides powerful sandboxing options that restrict what a service can do, limiting the blast radius if it's compromised:

    Security directives
    [Service]
    # Run as a non-root user
    User=deploy
    Group=deploy
    
    # Prevent gaining new privileges (e.g., via setuid binaries)
    NoNewPrivileges=true
    
    # Make the filesystem read-only except for specific paths
    ProtectSystem=strict
    ReadWritePaths=/var/www/myapp/data /var/log/myapp
    
    # Hide /home, /root, and /run/user from the service
    ProtectHome=true
    
    # Prevent writing to /dev, /proc, /sys
    ProtectKernelTunables=true
    ProtectKernelModules=true
    ProtectControlGroups=true
    
    # Restrict network access to specific address families
    RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX
    
    # Make /tmp private to this service
    PrivateTmp=true
    
    # Prevent the service from accessing /dev devices
    PrivateDevices=true
    Analyze security of a service
    # Score how secure a service configuration is (0 = best, 10 = worst)
    systemd-analyze security myapp.service

    Related Documentation

    10. Troubleshooting

    Service Won't Start

    Diagnostic steps
    # 1. Check the status and error messages
    sudo systemctl status myapp -l
    
    # 2. View the full log output
    sudo journalctl -u myapp -n 100 --no-pager
    
    # 3. Verify the unit file syntax
    sudo systemd-analyze verify /etc/systemd/system/myapp.service
    
    # 4. Check if the binary exists and is executable
    ls -la /usr/local/bin/myapp
    file /usr/local/bin/myapp
    
    # 5. Try running the command manually as the service user
    sudo -u deploy /usr/local/bin/myapp

    Service Keeps Restarting

    If a service keeps crashing and restarting, systemd will eventually stop trying and mark it as failed. Check the exit code:

    Check exit codes
    # Show recent failures with exit codes
    sudo systemctl status myapp
    
    # Reset the failure counter to try again
    sudo systemctl reset-failed myapp
    sudo systemctl start myapp

    Common Error Messages

    ErrorLikely Cause
    code=exited, status=203Binary not found or not executable — check ExecStart path
    code=exited, status=217User specified in User= does not exist
    code=exited, status=200Unit file has configuration errors
    code=killed, signal=KILLService was OOM-killed — increase memory or MemoryMax
    code=exited, status=1Application error — check journalctl for details

    Slow Boot Diagnostics

    Analyze boot time
    # Show total boot time
    systemd-analyze
    
    # Show services sorted by startup time
    systemd-analyze blame
    
    # Show the critical chain (what delayed boot)
    systemd-analyze critical-chain