Most servers get monitored after the first outage. By then you're reading logs to find out what happened, when an alert could have told you an hour earlier that the disk was filling up.

This guide sets up a small monitoring stack in one Docker Compose file: node_exporter collects host metrics, Prometheus stores them and evaluates alert rules, Alertmanager sends the alerts (Telegram or email), and Grafana draws the dashboards. Then it adds the five alert rules worth starting with, and leaves out the rest.

You need a Linux server with Docker Engine and the Compose plugin (docker compose version should work).

The layout

monitoring/ ├── docker-compose.yml ├── prometheus/ │ ├── prometheus.yml │ └── alerts.yml └── alertmanager/ └── alertmanager.yml 

Step 1: docker-compose.yml

services: prometheus: image: prom/prometheus:v2.53.0 volumes: - ./prometheus:/etc/prometheus:ro - prometheus-data:/prometheus command: - --config.file=/etc/prometheus/prometheus.yml - --storage.tsdb.retention.time=30d ports: - "127.0.0.1:9090:9090" restart: unless-stopped alertmanager: image: prom/alertmanager:v0.27.0 volumes: - ./alertmanager:/etc/alertmanager:ro ports: - "127.0.0.1:9093:9093" restart: unless-stopped node-exporter: image: prom/node-exporter:v1.8.1 command: - --path.rootfs=/host pid: host volumes: - /:/host:ro,rslave restart: unless-stopped grafana: image: grafana/grafana:11.1.0 environment: - GF_SECURITY_ADMIN_PASSWORD__FILE=/run/secrets/grafana_admin secrets: - grafana_admin volumes: - grafana-data:/var/lib/grafana ports: - "127.0.0.1:3000:3000" restart: unless-stopped secrets: grafana_admin: file: ./grafana_admin.txt volumes: prometheus-data: grafana-data: 

Two choices here matter more than they look:

  • Every port is bound to 127.0.0.1. Prometheus, Alertmanager and node_exporter have no login of their own. Published on 0.0.0.0, they show your whole infrastructure to anyone who finds the IP. Docker also writes its own iptables rules, so a published port can be reachable even when UFW says it's closed. Reach the UIs through an SSH tunnel (ssh -L 3000:127.0.0.1:3000 user@server) or a reverse proxy with authentication.
  • The image versions are pinned. latest changes under you on the next docker compose pull. Upgrade on purpose, after reading the release notes.

Put the Grafana admin password in grafana_admin.txt (openssl rand -base64 24 > grafana_admin.txt && chmod 600 grafana_admin.txt).

Step 2: prometheus.yml

global: scrape_interval: 30s evaluation_interval: 30s rule_files: - /etc/prometheus/alerts.yml alerting: alertmanagers: - static_configs: - targets: ["alertmanager:9093"] scrape_configs: - job_name: prometheus static_configs: - targets: ["localhost:9090"] - job_name: node static_configs: - targets: ["node-exporter:9100"] labels: server: "web-01" 

To watch more servers, run node_exporter on each one and add them as targets. Only let your Prometheus server reach their port 9100, at the firewall.

Step 3: The five alerts worth starting with

Most teams start with too many alerts, get flooded and start ignoring all of them. These five catch the outages that actually happen on most Linux servers. prometheus/alerts.yml:

groups: - name: node rules: - alert: InstanceDown expr: up == 0 for: 2m labels: { severity: critical } annotations: summary: "{{ $labels.instance }} is not responding" - alert: DiskWillFillIn24h expr: predict_linear(node_filesystem_avail_bytes{fstype!~"tmpfs|overlay|squashfs"}[6h], 24 * 3600) < 0 for: 30m labels: { severity: warning } annotations: summary: "{{ $labels.mountpoint }} on {{ $labels.instance }} will be full within 24h" - alert: DiskAlmostFull expr: (1 - node_filesystem_avail_bytes{fstype!~"tmpfs|overlay|squashfs"} / node_filesystem_size_bytes{fstype!~"tmpfs|overlay|squashfs"}) * 100 > 90 for: 10m labels: { severity: critical } annotations: summary: "{{ $labels.mountpoint }} on {{ $labels.instance }} is over 90% full" - alert: MemoryLow expr: (1 - node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes) * 100 > 90 for: 10m labels: { severity: warning } annotations: summary: "Memory on {{ $labels.instance }} is over 90% used" - alert: CpuSaturated expr: 100 - (avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 90 for: 15m labels: { severity: warning } annotations: summary: "CPU on {{ $labels.instance }} above 90% for 15 minutes" 

Why these:

  • DiskWillFillIn24h is the most useful alert most people don't have. predict_linear looks at the last 6 hours and warns you while there's still time to act, not when the database has already crashed.
  • for: on every rule keeps a short spike (a backup, a deploy) from paging anyone.
  • MemAvailable, not MemFree. Linux uses free memory as cache, so MemFree is always low and means nothing.

Step 4: alertmanager.yml (Telegram or email)

Alertmanager has a native Telegram receiver:

route: receiver: telegram group_by: ["alertname", "instance"] group_wait: 30s group_interval: 5m repeat_interval: 4h receivers: - name: telegram telegram_configs: - bot_token: "123456:ABC-your-bot-token" chat_id: -1001234567890 parse_mode: HTML send_resolved: true 

Create a bot with @botfather, add it to a group, and get the group's chat_id from https://api.telegram.org/bot<TOKEN>/getUpdates after someone posts in the group. For email, use email_configs with your SMTP server instead.

repeat_interval: 4h matters: an unfixed problem reminds you every 4 hours instead of every 5 minutes.

Step 5: Start it and test an alert

docker compose up -d docker compose ps 
  • Prometheus targets: open http://127.0.0.1:9090/targets through your tunnel. Both jobs should show UP.
  • Grafana: http://127.0.0.1:3000, log in as admin, add Prometheus as a data source (http://prometheus:9090), then import dashboard 1860 ("Node Exporter Full").
  • Test the alert path before you need it: run docker compose stop node-exporter, wait about 3 minutes, and InstanceDown should arrive in Telegram. Start it again and you should get the "resolved" message.

An alert you have never seen fire is an alert you don't know works.

Common mistakes

Mistake What happens Fix
Ports published on 0.0.0.0 Metrics and admin UIs open to the internet, even with UFW on Bind to 127.0.0.1, use a tunnel or proxy with auth
No for: on rules Alerts on every short spike, then everyone ignores them 2 to 15 minutes depending on the rule
Alerting on MemFree Constant false alarms Use MemAvailable
Never testing the alert path First real outage, no message Stop node-exporter once and watch the alert arrive
Monitoring server is the server you monitor It goes down together with everything else Monitor from a second machine, or add an external uptime check

Where to go next

Once host metrics are covered, the next layer is application metrics (Nginx, databases, queues) and logs in one place. If you'd rather have someone else run this, PikoSystem offers observability setup with Grafana and Prometheus and proactive server monitoring, and a free server monitoring review if you want a second opinion on an existing setup.

References: Prometheus: alerting rules, Alertmanager configuration, node_exporter

Parsa Shahmohammadi works on PikoSystem, an IT services and server management team.