Self-hosted Sentry: where the 16 GB actually goes

The official install guide says 4 CPU, 16 GB RAM, 20 GB disk. What it never says is why — so I installed the whole thing and measured every container.

Version 26.10.1 (getsentry/self-hosted), on a 16-core / 30 GB machine. Event load was generated synthetically and verified in ClickHouse, not assumed from HTTP 200s.

The short version

Two findings matter more than the rest: 60% of the memory is 28 copies of the same consumer, and the machine accepts events 8× faster than it can process them.

1. Where the 13.6 GiB goes

Grouped by role:

The 28 consumers are not doing 28 different jobs at scale — they are near-identical Python processes, each loading the interpreter, the Snuba libraries and a connection pool:

snuba-errors-consumer snuba-transactions-consumer snuba-replays-consumer snuba-metrics-consumer snuba-outcomes-consumer snuba-outcomes-billing-consumer snuba-outcomes-accepted snuba-profiling-profiles-consumer snuba-profiling-functions snuba-profiling-profile-chunks snuba-subscription-consumer (events / transactions / metrics / eap-items) snuba-issue-occurrence snuba-group-attributes snuba-eap-items-consumer snuba-replacer snuba-api ... 

So the 16 GB is not for one heavy service. It is for 28 copies of a small one. That distinction matters when you size a machine: you are paying for a fixed population of processes, not for throughput.

Idle CPU across the whole stack stayed under 5% (kafka was the highest at 4.06%).

2. What happens below 16 GB

I rebuilt the stack with per-service mem_limit capped to 60% and 90% of measured usage (≈8 GB and ≈12 GB budgets).

Both runs failed identically: 20 containers OOMKilled, exit 137, restart counts climbing to 11–14.

snuba-api 18 × snuba consumers (errors, transactions, replays, metrics, outcomes ×3, profiling ×3, subscription ×4, issue-occurrence, group-attributes, replacer) symbolicator 

Everything with a flat footprint survived: kafka, clickhouse, postgres, redis, seaweedfs, taskworker, web, nginx.

A caveat I have to state about my own method: scaling every limit by the same factor preserves each service's relative headroom, so the same services fail at any factor below 1.0. This experiment tells you which services are sensitive to memory pressure — it does not establish the exact total threshold. To find the real number you would need a real machine at 12, 14, 16 GB. I did not do that, so this article claims "16 GB is the measured comfortable value", not "14 GB will fail".

The failure mode is worse than a crash

nginx returns 200, the UI loads web reports healthy, dashboards are green ─────────── but ─────────── events stop being processed (consumers killed in a loop) stack traces stop resolving (symbolicator killed in a loop) 

This is the dangerous part. A stack that refuses to start gets noticed in five minutes. A stack that serves a green dashboard while silently dropping ingestion can go unnoticed for weeks.

The check that catches it

docker ps -a --format '{{.Names}}' | while read c; do n=$(docker inspect -f '{{.RestartCount}}' "$c") [ "$n" -gt 0 ] && echo "$n $c" done | sort -rn 

Any non-zero restart count on a snuba consumer or symbolicator means memory pressure — not "a flaky container".

3. Ingest is 8× faster than processing

I pushed synthetic events through /api/2/store/ with keep-alive connections, then verified what actually landed in ClickHouse.

Ingest accepted everything, with 1 ms p99:

Processing, measured by draining the backlog with no load applied:

≈235 events/s sustained. At 2,000/s ingest that means lag grows by ~1,765/s and Kafka becomes an unbounded buffer. Relay saying 200 does not mean the event was computed — it means it was queued.

4. Memory does not scale with events (and that changes what you buy)

Over 120,000+ events, total memory moved from 13.7 to 15.0 GiB — about +1.3 GiB, with no trend that suggests a per-event cost.

13.7 → 13.9 → 14.6 → 14.8 → 15.0 GiB (122,010 events processed) 

Buffers are bounded by configuration, not by volume. Buying more RAM does not increase throughput. RAM decides whether the 54 processes survive; CPU and ClickHouse write path decide how many events per second you can absorb.

Disk is the only resource that scales linearly: ≈19 KB per event (451 MB for 24,513 rows, including ClickHouse compression and Kafka retention).

5. So what should you actually run

≈235 events/s × 86,400 ≈ 20 million events/day on default settings, single node, no tuning.

6. What this does not cover

• Default configuration only — no consumer concurrency tuning, no sharding, no ClickHouse tuning
• Synthetic events are simple; real events carry stack traces and breadcrumbs and are heavier, so real throughput will be lower
• Only error events were pushed — transactions, replays and metrics were not tested
• Cloud Sentry shards and will scale far past this number
• The memory-threshold experiment is a sensitivity test, not a total-budget measurement

7. Reproducing it

# unattended install (both flags are required — either prompt aborts the run) REPORT_SELF_HOSTED_ISSUES=0 ./install.sh \ --no-report-self-hosted-issues --skip-user-creation docker compose up -d --wait # 58s to all-healthy on 16 cores docker stats --no-stream --format '{{.Name}}|{{.MemUsage}}' 

Two traps worth knowing:

1. Background installs die silently on any prompt. The telemetry question and the "create a user account" question both abort when stdin is /dev/null, and the second one reports a misleading database-migration error (web upgrade --create-kafka-topics exited with status 1) even though every migration succeeded.
2. docker update --memory may be unsupported. On this kernel it fails outright (swap limit not available), so capping memory requires recreating containers with mem_limit. Always read back docker inspect -f '{{.HostConfig.Memory}}' — a stack that "didn't crash" may simply have no limits applied.

Also: eight of the core images come from ghcr.io, which daemon.json registry mirrors do not rewrite. On a throttled route that was 164 KB/s versus 850 KB/s for Docker Hub, and a single 615 MB image stalled for half an hour. Rewrite the host before pulling.