What the interviewer is testing
Whether you can explain the query-key-value computation, why self-attention improves long-range token interactions, the quadratic cost, and the reason for scaling before softmax.
Answer framework
- Represent each token with three learned projections: a query, a key and a value.
- Compute compatibility scores with the query-key dot product, scale them by 1/sqrt(d_k), and apply softmax to obtain attention weights.
- Produce each token's output as a weighted sum of the value vectors using those attention weights.
- Allow each token to retrieve information from any position permitted by the attention mask in one layer, avoiding a recurrent bottleneck while supporting parallel sequence processing during training.
- Recognize the trade-off: full self-attention has O(n^2) compute and an attention matrix that also scales quadratically with sequence length.
Sample answer
I explain self-attention as a way for the model to decide which other tokens in the sequence matter for the token it is processing. Each token is projected into a query, a key and a value. The query represents what the token is looking for, the key represents what it offers for matching, and the value carries information that can be passed along. We compute scores by taking the query-key dot product, divide by the square root of the key dimension, and apply softmax to get normalized attention weights.
The output for each position is a weighted sum of the value vectors using those weights. The important benefit is direct information flow: each token can pull information from positions permitted by the attention mask through one layer instead of passing it through recurrent steps. Unmasked attention can access the entire sequence. causal attention can access only current and earlier positions. During training, positions can still be processed in parallel.
The trade-off is quadratic computation. For sequence length n and fixed head dimensions, full attention uses O(n^2) compute. Explicitly storing its attention matrix also takes O(n^2) memory, but implementations such as FlashAttention avoid materializing that matrix. Dividing scores by the square root of the key dimension also matters: under standard assumptions, dot-product variance grows with dimension, which can saturate softmax and produce small gradients. Scaling helps keep scores in a useful range.
Likely follow-ups
Why do we divide the attention scores by sqrt(d_k)? Without the scaling, dot products grow as the key dimension grows. The softmax can then saturate, producing very small gradients and making optimization harder.
What is the main trade-off in self-attention? It gives tokens direct access to positions permitted by the attention mask, but full attention requires O(n^2) computation at fixed head dimensions. Materializing the attention matrix also requires O(n^2) memory. memory-efficient implementations avoid storing it.
Common mistakes
- Assuming a token can only use nearby context, as in a fixed local window.
- Forgetting to scale the QK^T scores by sqrt(d_k) before applying softmax.
- Claiming the attention cost is linear in sequence length. the full attention matrix is quadratic.
- Describing the output as a plain average of values and ignoring the learned, content-dependent weights.
This article was prepared with AI assistance and reviewed before publishing. The original lives on the free question bank:
