Most backend engineers never touch the TCP congestion control setting on a server. It defaults to something reasonable, and the connection works. Then someone transfers a large file across regions, sees 15 Mbit/s on a link that should do 500, and the question of what the sender is actually doing becomes interesting fast.

Here is the mental model I use.

One number controls everything

A TCP sender keeps a number called the congestion window, cwnd: how many bytes it may have sent but not yet had acknowledged. It can't see the network, so it guesses. Start small, grow while ACKs come back, shrink when something goes missing.

Growth has two modes. In slow start the window doubles every round trip, starting around 10 segments on modern stacks. Once it crosses a threshold, ssthresh, growth turns linear, about one segment per round trip. When loss shows up, the window gets cut. Classic Reno halves it. Plot that over time and you get the familiar sawtooth.

Throughput is roughly cwnd / RTT. That one division explains most of the confusing behavior.

Why a fast link can feel slow

On a 100 Mbit/s path with a 100 ms round trip, filling the pipe takes about 830 segments in flight. Reno halves after a loss and then adds one segment per round trip, so it needs around 400 round trips, roughly 40 seconds, to climb back. If the link loses a packet every few seconds, the connection spends its whole life below capacity.

Loss rate matters more than people expect. The Mathis approximation puts steady-state throughput at about MSS / (RTT * sqrt(p)). A hundredfold drop in loss gives roughly a tenfold gain in speed. A 0.1% loss rate on a transatlantic path can cripple a transfer while the link itself sits mostly idle.

I find it easier to believe this after stepping through it by hand. I put together a small TCP congestion window simulator where you choose the starting window and which round trips lose a packet. Switching a loss from "three duplicate ACKs" to "timeout" is the instructive bit: the threshold ends up identical, but the timeout resets the window to one segment and you watch it crawl back through slow start.

Loss-based versus model-based

Reno and CUBIC both treat a dropped packet as the signal to back off. CUBIC regrows along a cubic curve instead of a line, which recovers much faster on long paths, and it cuts to 70% instead of 50%. It has been the Linux default for years.

Both share a flaw: they only learn the buffer is full when it overflows. On a link with a deep buffer, that means a standing queue and added latency for everyone sharing it.

BBR takes a different route. It estimates the bottleneck bandwidth and the minimum round-trip time, then paces its sending to match instead of waiting for loss. On lossy links it can hold throughput where CUBIC collapses, and it tends to keep queues shorter. The catch is fairness. BBR v1 could take more than its share against CUBIC flows and caused heavy retransmission in shallow buffers; later versions use loss and ECN signals to behave better.

The side-by-side details, including which signals each uses and how they recover, are in this Reno vs CUBIC vs BBR comparison.

What I would actually do

Leave CUBIC alone unless you have a measured problem. If you do, and you control the sending side of a lossy or high-latency path, try BBR:

sysctl net.ipv4.tcp_congestion_control sudo sysctl -w net.core.default_qdisc=fq sudo sysctl -w net.ipv4.tcp_congestion_control=bbr

Then measure. ss -i shows live cwnd, RTT, and retransmits per socket, and iperf3 with -P runs parallel streams, which is a quick way to check whether a single connection is the limit. Remember the setting only affects data this host sends. Changing a web server to BBR speeds up downloads to clients and does nothing for uploads from them.

Parallel streams working around a single-flow ceiling is the same reason download accelerators work. Each connection runs its own sawtooth, and their sum gets closer to the link's capacity than one stream does.