Here's a client method that looks like it was written by someone careful:
func (c *InventoryClient) Reserve(ctx context.Context, sku string, qty int) error { ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second) defer cancel() return c.post(ctx, "/reserve", reserveReq{SKU: sku, Qty: qty}) } It takes a ctx. It sets a timeout. It calls cancel. It compiles, and go vet has nothing to say about it. And it throws away every deadline the caller ever had. That first line overwrites the incoming ctx with a brand-new one rooted in context.Background(), so the 800ms budget your edge gateway gave this request is gone. From here on, this call has 5 seconds, no matter what.
That's the whole topic in one line. Go's context makes deadline propagation almost free, and "almost" is where the outages come from. This piece is about how a single timeout at the edge is supposed to cascade through every call it triggers, what the machinery actually does at each layer (in-process, over gRPC, over plain HTTP, through a proxy), and the specific places the chain snaps without anyone noticing.
If you want the broader resilience picture (retries, circuit breakers, bulkheads), that's covered in Designing for Partial Failure. This one goes deep on the deadline itself.
One clock for the whole call tree
The idea isn't Go-specific. Google's SRE book describes it in the chapter on cascading failures, and the wording is worth reading closely: "Rather than inventing a deadline when sending RPCs to backends, servers should employ deadline propagation." The tree of calls that fans out from one request should share one absolute deadline.
Their worked example: server A picks a 30-second deadline and spends 7 seconds before calling B, so the RPC to B carries 23 seconds. B spends 4 seconds before calling C, so C gets 19. Every hop inherits what's left, not a fresh number.
Then they show what happens without it. A calls B with a 10-second deadline. B takes 8 seconds to get going, then calls C with a hardcoded 20-second deadline instead of the 2 seconds it actually has. C pulls the request off its queue 5 seconds later and, in the book's words, "processes the request thinking it has 15 seconds to spare." Nobody's waiting for that answer. A gave up long ago. C is burning CPU, holding a connection, maybe taking locks, for a response that will be thrown on the floor.
That sounds like a waste problem. It's actually a capacity problem, and the same chapter has the numbers to prove it. Picture a frontend with 1,000 worker threads handling 1,000 QPS at 100ms each, so about 100 threads busy at any moment. Now 5% of requests start hanging until their deadline, and that deadline is 100 seconds. Those 50 QPS want 5,000 threads' worth of waiting. The frontend has 1,000. The book's arithmetic lands on the frontend serving 19.6% of requests, "resulting in an 80.4% error rate." A 5% backend problem became an 80% outage because the deadline was long and nobody short-circuited the doomed work.
So the goal is simple to state: one budget, set once at the edge, visible and shrinking at every hop. Let's see how much of that Go gives you for free.
What context actually does with a deadline
Some quick history, because it explains the API's shape. Sameer Ajmani introduced the context package in the July 2014 Go blog post "Go Concurrency Patterns: Context", and it lived at golang.org/x/net/context until Go 1.7 moved it into the standard library, with support wired into net, net/http, and os/exec. Go 1.8 added QueryContext, ExecContext, and friends to database/sql. Go 1.20 added cancellation causes (WithCancelCause, Cause), and Go 1.21 added WithoutCancel, AfterFunc, WithDeadlineCause, and WithTimeoutCause. We'll use most of those below.
Under the hood a deadline is just two things: a time you can read with ctx.Deadline(), and a Done() channel that closes when that time passes, when someone calls cancel, or when the parent's Done() closes, whichever comes first. Cancellation flows strictly downward. A child can't cancel its parent, and the parent's cancellation reaches every descendant.
Here's the part that surprises people. The docs for WithDeadline say the child's deadline is "adjusted to be no later than d", and more bluntly: "If the parent's deadline is already earlier than d, WithDeadline(parent, d) is semantically equivalent to parent." A child can shorten the budget. It can never extend it.
parent, cancel := context.WithTimeout(context.Background(), 100*time.Millisecond) defer cancel() child, cancelChild := context.WithTimeout(parent, 5*time.Second) defer cancelChild() pd, _ := parent.Deadline() cd, _ := child.Deadline() fmt.Println(cd.Equal(pd)) // true: the 5s asks for more than the parent has, so it's ignored <-child.Done() fmt.Println(child.Err()) // context deadline exceeded, after ~100ms This changes how you should read per-call timeouts. A WithTimeout(ctx, 5*time.Second) around a downstream call isn't a promise of 5 seconds. It's a cap: "at most 5 seconds, or whatever the caller has left, whichever is smaller." That's exactly the SRE book's advice to put "an upper bound for outgoing deadlines" on calls to noncritical backends. So hardcoded per-hop timeouts are fine in Go, as long as they're derived from the incoming ctx.
Which is why the opener is so nasty. The bug isn't the 5 seconds. It's context.Background(). The moment you root a new context there, the min-of-both rule has nothing to take the min of.
When the deadline hits, ctx.Err() returns context.DeadlineExceeded. When someone calls cancel (or the client hangs up), it returns context.Canceled. Keep those two apart in your logs and metrics. They mean different things: "we ran out of time" versus "nobody wants this anymore."
Across the wire, part 1: gRPC carries it for you
Inside a process, the deadline rides on a pointer. Between processes it has to become bytes on the wire, and this is where gRPC earns its keep.
The gRPC HTTP/2 protocol spec defines a grpc-timeout request header: a positive integer "of at most 8 digits" followed by a unit, H, M, S, m, u, or n (hours down to nanoseconds). If the header is missing, "a server should assume an infinite timeout."
Notice it's a timeout, not a deadline. The client sends "you have 1.5 seconds", not "finish by 14:03:07.250". The gRPC deadlines guide explains why: gRPC "converts the deadline to a timeout from which the already elapsed time is already deducted. This shields your system from any clock skew issues." Two machines don't have to agree on what time it is. They only have to agree on how long a second is.
You can see the mechanics in grpc-go's source (internal/transport/http2_client.go). When it builds the request headers, it does roughly this:
if dl, ok := ctx.Deadline(); ok { timeout := time.Until(dl) if timeout <= 0 { return nil, status.Error(codes.DeadlineExceeded, context.DeadlineExceeded.Error()) } headerFields = append(headerFields, hpack.HeaderField{Name: "grpc-timeout", Value: grpcutil.EncodeDuration(timeout)}) } Two details worth knowing. First, if your budget is already spent, grpc-go fails the call locally with DeadlineExceeded and never sends it, which is precisely the "don't start doomed work" behavior you want. Second, EncodeDuration picks the finest unit that fits in 8 digits. One and a half seconds is 1,500,000,000 nanoseconds, too many digits, so it goes out as 1500000u.
On the receiving side, grpc-go's server transport parses the header and wraps the stream's context with context.WithTimeout(ctx, timeout). So your handler's ctx already has the caller's remaining budget in it. The gRPC guide lists Go (and Java) among the languages where propagation is "enabled by default", but in Go, "by default" really means "if you pass the handler's ctx to the outgoing call":
func (s *checkoutServer) reserve(ctx context.Context, sku string, qty int32) error { if dl, ok := ctx.Deadline(); ok { slog.DebugContext(ctx, "reserving stock", "budget_left", time.Until(dl)) } _, err := s.inventory.Reserve(ctx, &inventorypb.ReserveRequest{Sku: sku, Qty: qty}) switch status.Code(err) { case codes.OK: return nil case codes.DeadlineExceeded: return fmt.Errorf("inventory: out of time budget: %w", err) default: return fmt.Errorf("inventory: %w", err) } } Pass ctx, and the inventory service receives a grpc-timeout equal to whatever the checkout service had left minus the time it already spent. Pass context.Background(), and inventory gets no header at all, which per the spec means infinite.
One thing gRPC can't do for you: stop your handler. The cancellation guide is explicit that "the gRPC library in general does not have a mechanism to interrupt the application-provided server handler." When the deadline passes, the server cancels the call, your ctx.Done() closes, and your code keeps running unless it looks. More on that in a minute.
Across the wire, part 2: plain HTTP carries nothing
If your services talk JSON over HTTP, here's the uncomfortable truth: there's no standard header for "this request has N milliseconds left". An IETF draft proposing a Request-Timeout header was written back in March 2011 and expired without becoming a standard. So http.NewRequestWithContext(ctx, ...) bounds how long your client waits, and the transport abandons the request when ctx fires. But the server on the other end has no idea what your budget was. It will happily process the request for as long as its own settings allow.
Other ecosystems solved this by picking their own header. The Connect protocol sends Connect-Timeout-Ms (a positive integer of at most 10 digits). Envoy sets x-envoy-expected-rq-timeout-ms on internal requests, documented as "the time in milliseconds the router expects the request to be completed" so the upstream "can make decisions based on the request timeout, e.g., early exit."
For your own HTTP services, you can do the same thing in about 30 lines. The header name below is our own convention, not a standard. Pick one and use it everywhere.
const timeoutHeader = "X-Request-Timeout-Ms" // our own convention, not a standard // Server side: turn the incoming header back into a context deadline. func DeadlineMiddleware(next http.Handler) http.Handler { return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { if v := r.Header.Get(timeoutHeader); v != "" { if ms, err := strconv.ParseInt(v, 10, 64); err == nil && ms > 0 { ctx, cancel := context.WithTimeout(r.Context(), time.Duration(ms)*time.Millisecond) defer cancel() r = r.WithContext(ctx) } } next.ServeHTTP(w, r) }) } // Client side: stamp the remaining budget on every outgoing request. type deadlineTransport struct{ base http.RoundTripper } func (t deadlineTransport) RoundTrip(req *http.Request) (*http.Response, error) { if dl, ok := req.Context().Deadline(); ok { remaining := time.Until(dl) if remaining <= 0 { return nil, context.DeadlineExceeded // don't even dial } req = req.Clone(req.Context()) // a RoundTripper must not mutate the caller's request ms := max(remaining.Milliseconds(), 1) req.Header.Set(timeoutHeader, strconv.FormatInt(ms, 10)) } return t.base.RoundTrip(req) } var client = &http.Client{Transport: deadlineTransport{base: http.DefaultTransport}} A few choices in there are deliberate:
- Send a relative duration, not a timestamp. Same reason gRPC does: it sidesteps clock skew between hosts. An absolute
X-Deadline: 2026-09-23T14:03:07.250Zonly works if every box agrees on the time to the millisecond, and they don't. - Never let a tiny budget round down to "no header". connect-go used to skip
Connect-Timeout-Msentirely when less than a millisecond remained, and a missing header means infinite. The fix truncates to 1ms and sets the header "to avoid an unbounded call." Same trap here:Milliseconds()of 0.4ms is 0, and the middleware above ignores non-positive values, hence themax(..., 1). - Clone before setting the header. The
http.RoundTrippercontract says implementations shouldn't modify the request. - Only ever shrink. The middleware wraps
r.Context(), so if the server has its own tighter limit,WithTimeout's min rule keeps it. A caller can't use the header to buy more time than the server is willing to give.
One more net/http detail that works in your favor: for incoming server requests, the docs say the context "is canceled when the client's connection closes, the request is canceled (with HTTP/2), or when the ServeHTTP method returns." So if the edge gateway gives up and hangs up, your handler's ctx fires too, and anything you passed it to (outgoing HTTP, gRPC, database/sql) gets cancelled with it. That's cancellation propagation, the SRE book's companion to deadline propagation, and you get it for free as long as you don't drop the ctx.
The proxy in the middle has its own opinion
Now put a service mesh or an API gateway between two gRPC services. The client sends grpc-timeout: 1500000u. What does the proxy do with it?
For Envoy, the answer is "probably not what you assume." The route configuration docs say that unless max_grpc_timeout is configured (it's deprecated now in favor of grpc_timeout_header_max under max_stream_duration), "the grpc-timeout header is not used and gRPC requests time out like any other requests using timeout." And that route timeout defaults to 15 seconds.
Think about what that means for the chain:
- Your edge sets 800ms, the caller's context expires on time, and the caller gets
DeadlineExceeded. Fine. - But the proxy's own timer is running at 15 seconds, not 800ms. The proxy itself enforces nothing tighter.
- And if some intermediate service dropped its
ctx, so nogrpc-timeoutwent out at all, the proxy will still cut the call at 15 seconds. That's a deadline, just not yours. It'll look like propagation is "sort of working" in load tests.
Envoy also supports grpc_timeout_header_offset (formerly grpc_timeout_offset), which subtracts a fixed amount from the incoming grpc-timeout, so the proxy times out slightly before the client and can return a clean error instead of racing it. Whether you want that depends on your mesh; the point is that it's configuration you have to go and set, not something that happens by itself. If you run Istio, Linkerd, or a cloud load balancer, go read what its defaults are before assuming the header survives with meaning.
The five places the chain breaks
Put it all together and the failure modes are pretty specific. In rough order of how often I'd expect to find them in a codebase:
1. A fresh root context somewhere in the middle
The opener. It shows up in client libraries, in "helper" functions that don't take a ctx parameter, in code written before the caller had one, and in background goroutines that got a context.Background() because it was the only thing in scope. The fix is mechanical: every function that does I/O takes ctx context.Context as its first parameter (the context docs say exactly this) and derives from it. context.Background() belongs in main, in tests, and at the top of genuinely independent background jobs. Grepping for context.Background() and context.TODO() outside those places is one of the cheapest audits you can run.
2. Fan-out that doesn't share a context
A handler that calls pricing and stock in parallel should give both goroutines a context that dies when the request does, and ideally when either sibling fails. golang.org/x/sync/errgroup does both:
func productPage(ctx context.Context, sku string) (Quote, Stock, error) { var q Quote var s Stock g, gctx := errgroup.WithContext(ctx) g.Go(func() (err error) { q, err = pricing.Quote(gctx, sku); return }) g.Go(func() (err error) { s, err = stock.Level(gctx, sku); return }) return q, s, g.Wait() } If pricing fails, gctx is cancelled and the stock call stops early. If the edge deadline hits, both stop. Hand-rolled go func() plus a sync.WaitGroup usually loses one of those two properties.
3. A handler that never looks at ctx
Cancellation in Go is cooperative. ctx.Done() closing doesn't stop anything by itself; it's a signal for code that checks. Anything that blocks on a context-aware call (HTTP, gRPC, QueryContext) checks for you. A CPU-bound loop doesn't:
func rank(ctx context.Context, rows []Row) ([]float64, error) { out := make([]float64, 0, len(rows)) for i, row := range rows { if i%1000 == 0 { // checking every iteration is fine too; this just keeps it cheap if err := ctx.Err(); err != nil { return nil, err } } out = append(out, score(row)) } return out, nil } The gRPC cancellation guide says it plainly: a long-lived handler "must periodically check if the RPC it is servicing has been cancelled and if it has, cease processing." And the SRE book has a smart exception: if the work is an expensive catch-up that checkpoints its progress, check the deadline after writing the checkpoint, so the work you already paid for isn't thrown away.
For blocking calls that don't take a context at all (a raw net.Conn, some older SDK), context.AfterFunc from Go 1.21 lets you wire one in. This is the pattern from the package's own examples, adapted:
func readWithContext(ctx context.Context, conn net.Conn, buf []byte) (int, error) { stopc := make(chan struct{}) stop := context.AfterFunc(ctx, func() { conn.SetReadDeadline(time.Now()) // unblocks the Read below close(stopc) }) n, err := conn.Read(buf) if !stop() { // The AfterFunc already ran: wait for it, then clear the deadline so the conn stays usable. <-stopc conn.SetReadDeadline(time.Time{}) return n, ctx.Err() } return n, err } 4. Retries that ignore the budget
A retry loop with exponential backoff and no awareness of the deadline is a slow way to fail. If you have 300ms left and your next backoff is 400ms, sleeping is pointless: you'll wake up to an expired context. Check the budget before you sleep, and sleep with a select so cancellation cuts it short:
func retry(ctx context.Context, attempts int, base time.Duration, op func(context.Context) error) error { var err error for i := range attempts { if err = op(ctx); err == nil { return nil } backoff := base << i if dl, ok := ctx.Deadline(); ok && time.Until(dl) < backoff { return fmt.Errorf("not enough budget left to retry: %w", err) } t := time.NewTimer(backoff) select { case <-ctx.Done(): t.Stop() return context.Cause(ctx) case <-t.C: } } return err } This is simplified (no jitter, no retryable-error check; the partial-failure article covers those), but the budget check is the part people skip.
5. A hop that doesn't speak the protocol
Plain HTTP with no timeout header, a proxy that ignores grpc-timeout, a message queue in the middle. Every one of those is a place where the in-process deadline becomes a local timeout and the downstream side starts a fresh clock. Queues deserve a special mention: a message has no caller waiting on the other end in the same sense, so "propagating the deadline" there usually means putting an expiry timestamp in the message and having the consumer drop anything already stale. That's a different design, and a good reason not to pretend an async hop is part of the synchronous budget.
Leave yourself some room
Propagating the exact remaining budget has a subtle flaw: if the downstream service uses all of it, the response arrives at the same instant your own deadline fires. The gRPC deadlines post (by Google SRE Grainne Sheerin) notes that "both the client and server make their own independent and local determination about whether the remote procedure call (RPC) was successful", so a call can succeed on the server and still fail on the client. Then you have nothing left to log, map the error, or return a fallback.
The SRE book suggests reducing "the outgoing deadline a bit (e.g., a few hundred milliseconds) to account for network transit times and post-processing in the client." For an 800ms edge budget, a few hundred milliseconds is most of it, so scale that to your own numbers. The helper is tiny:
func withReserve(ctx context.Context, reserve time.Duration) (context.Context, context.CancelFunc) { dl, ok := ctx.Deadline() if !ok { return context.WithCancel(ctx) } return context.WithDeadline(ctx, dl.Add(-reserve)) } Use it on outgoing calls where you have a meaningful fallback (a cached price, a degraded page section). If you don't have a fallback, the reserve buys you a clean error instead of a racing one, which is still worth something.
When work should outlive the request
Sometimes you want work to keep going after the caller leaves: an audit record, a cache fill, a webhook ack. The wrong fix is context.Background(), because it also throws away the request's values, like the trace ID your logs depend on. Go 1.21's context.WithoutCancel is the right tool. Per the docs, it returns a context that "is not canceled when parent is canceled" and "returns no Deadline or Err, and its Done channel is nil." Values survive, cancellation and deadline don't.
func handleRefund(w http.ResponseWriter, Source：https://dev.to/andriiboyko/context-cancellation-in-go-propagating-deadlines-across-service-calls-53ibRelated posts
Building a Private AI Control Plane for PS5 Porting: What We Learned After SQLite Locked Us Out at 2 AM
Monolith vs Microservices: A Founder's Guide
Sentinel Dev Diary: Read First, Run Second
A website for the café down the street, prepared before they ask
Where Temporary Files Come From and Why They Can Leak Office Data
