Our order service puts the order number into every log line through the logging context, the MDC, so that one search shows everything that happened to an order. A servlet filter puts the number in when a request arrives and clears it when the response goes out. On the request thread that works perfectly.
In September we moved label printing and customer emails off the request path into a Spring thread pool, so checkout would not wait for them. Two weeks later a customer complained that her confirmation email had never arrived. Support searched for her order number and found the checkout and nothing after it. Searching for email errors in that minute found a failure logged against a different order, belonging to a customer who had received his email without any trouble.
Logback keeps the MDC in a ThreadLocal. It belongs to the thread that set it, and a pool thread does not inherit it from the thread that submitted the work. So the async tasks started with an empty context and their lines carried no order number at all. That alone would have been merely unhelpful. The worse part came from the label task, which put the order number into the MDC itself and never removed it. Pool threads are reused, so the next email task on that thread logged under whatever order the last label had left behind. Some lines had no order number, some had the wrong one, and nothing in the line told you which.
For two weeks every search by order number had silently missed the async half of an order's life, and now and then shown someone else's.
The executor now has a TaskDecorator that copies the submitting thread's MDC into the task and clears it in a finally block when the task ends, whatever happened inside. Tasks may no longer call MDC.put themselves, and an architecture test in the build enforces that. Every log line also carries the thread name, and a dashboard counts log lines without an order number by logger, a number that should be zero outside startup.
A context that lives on a thread is only as accurate as the last piece of code that ran there.
– Sergey Shinder