Alerting & Incident Response

receivers:
  - name: "pagerduty-critical"
    pagerduty_configs:
      - routing_key: "YOUR_PAGERDUTY_INTEGRATION_KEY"
        severity: "critical"
        details:
          instance: "{{ .GroupLabels.instance }}"

route:
  receiver: "slack-warnings"
  group_by: ["alertname", "instance"]
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h

  routes:
    - match:
        severity: critical
      receiver: "pagerduty-and-slack-critical"
      repeat_interval: 1h

    - match:
        severity: warning
      receiver: "slack-warnings"
      repeat_interval: 4h

receivers:
  - name: "pagerduty-and-slack-critical"
    pagerduty_configs:
      - routing_key: "${PAGERDUTY_INTEGRATION_KEY}"
    slack_configs:
      - api_url: "https://hooks.slack.com/services/CRITICAL_WEBHOOK_URL"
        channel: "#infra-critical"
        title: "CRITICAL: {{ .GroupLabels.alertname }}"
        send_resolved: true

  - name: "slack-warnings"
    slack_configs:
      - channel: "#infra-warnings"
        title: "{{ .GroupLabels.alertname }}"
        send_resolved: true

receivers:
  - name: "discord-warnings"
    discord_configs:
      - webhook_url: "${DISCORD_WEBHOOK_WARNING}"
        title: "{{ .GroupLabels.alertname }}"

  - name: "discord-critical"
    discord_configs:
      - webhook_url: "${DISCORD_WEBHOOK_CRITICAL}"
        title: "CRITICAL: {{ .GroupLabels.alertname }}"
        message: |
          @here
          {{ range .Alerts }}
          **{{ .Annotations.summary }}**
          {{ end }}

global:
  smtp_smarthost: 'smtp.mailgun.org:587'
  smtp_from: 'alerts@yourdomain.com'
  smtp_auth_username: 'alerts@yourdomain.com'
  smtp_auth_password: "${SMTP_PASSWORD}"
  smtp_require_tls: true

receivers:
  - name: "email-weekly-digest"
    email_configs:
      - to: "ops@yourdomain.com"
        send_resolved: true

route:
  group_by: ["instance", "alertname"]
  group_wait: 30s       # Wait 30s to collect related alerts before sending
  group_interval: 5m    # Send updates for the same group every 5 minutes
  repeat_interval: 4h   # Resend a still-firing alert every 4 hours

inhibit_rules:
  - source_match:
      alertname: NodeDown
    target_match_re:
      alertname: "HighCPU|HighMemory|DiskSpaceLow"
    equal: ["instance"]

- alert: HighCPUUsage
  expr: cpu_usage > 85
  for: 5m   # Must be true for 5 minutes before alerting

# Alert: HighDiskUsage

## What this means
Disk usage on the affected instance has exceeded 80%.

## Immediate triage (< 5 minutes)
1. SSH into the affected server: ssh deploy@INSTANCE_IP
2. Check which filesystem is affected: df -h
3. Find the largest directories: du -sh /* 2>/dev/null | sort -rh | head -20

## Common causes and fixes

### Docker logs consuming space
docker system prune -f --volumes
journalctl --vacuum-size=500M

### Application logs
find /var/log -name "*.log" -size +100M
truncate -s 0 /var/log/problematic.log

## Escalate if
- Disk is over 95% and you cannot free space within 15 minutes
- The filling directory is an application data directory you do not recognize

[INVESTIGATING] We are investigating reports of [service] being unavailable.
We will update this page every 30 minutes. Started: 14:32 UTC.

[UPDATE 15:00 UTC] Root cause identified as [brief description].
Fix in progress, estimated resolution: 15:45 UTC.

[RESOLVED 15:42 UTC] Service has been restored. Root cause: [brief description].
A full postmortem will be posted within 48 hours.

Alert fires
  │
  ├─► severity: critical
  │     ├─► PagerDuty (guaranteed delivery, on-call escalation)
  │     └─► #infra-critical (Slack/Discord - team visibility)
  │
  ├─► severity: warning
  │     └─► #infra-warnings (Slack/Discord - monitored during business hours)
  │
  └─► severity: info
        └─► Email digest (weekly, low priority) or suppressed entirely

Alert resolves
  └─► #infra-resolved (separate channel, all severities)