Does Context Length Affect Inference Cost Linearly? We Measured Why It Doesn't

Cost calculators price context length linearly: ten times the input tokens, ten times the input cost. That is how the rate card is built, and the market’s own pricing behavior admits it is not what long context actually costs to serve. On DigitalOcean’s own catalogue, OpenAI’s GPT-5.5 and GPT-5.6 family step their rate at 272K tokens: GPT-5.5 goes from $5.00/$30.00 to $10.00/$45.00 per 1M input/output tokens, GPT-5.6 Sol from $4.00/$20.00 to $8.00/$30.00, GPT-5.6 Terra from $2.00/$12.00 to $4.00/$18.00, and GPT-5.6 Luna from $0.20/$1.20 to $0.40/$1.80, an input step that is exactly 2x and an output step that is exactly 1.5x across the whole family (source: DigitalOcean Inference pricing, last verified 24 August 2026).

Google’s Gemini 2.5 Pro shows the same pattern: $1.25/$10.00 per 1M input/output tokens for prompts at or below 200K, stepping to $2.50/$15.00 above it, input doubling and output rising 1.5x, matching the shape of OpenAI’s step (source: Gemini Developer API pricing, Standard tier, fetched 27 August 2026). Anthropic disagrees with itself on DigitalOcean’s catalogue: Claude Sonnet 4.5 still carries a legacy tier, $3.00/$15.00 at or below 200K tokens and $6.00/$22.50 above it, but every newer Anthropic model listed is flat with no long-context step at all, Sonnet 4.6 at $3.00/$15.00, Sonnet 5 at $2.00/$10.00, Opus 4.5 through 4.8 and Opus 5 at $5.00/$25.00, and Fable 5 at $10.00/$50.00, several of which support context windows up to 1M tokens with no premium. OpenAI and Google still price the non-linearity into their current generations. Anthropic’s current generation dropped it. That disagreement, not a unanimous industry practice, is the reason to settle the question with hardware measurements instead of reading rate cards.

This piece is the measured-economics sequel to Long-Context Inference at Scale: The Hidden Infrastructure Cost, which laid out the qualitative shape of this problem: KV cache growth colliding with fixed hardware capacity. This article puts a number on that shape for one model on one card.

The pricing structure above has a structural consequence most discussions of long-context pricing stop one step short of. A flat per-token rate charges the same amount for a token regardless of how much serving capacity the request carrying it consumes. If the cost of serving a token genuinely varies with context length, which is what this article sets out to measure, then a single flat rate cannot track that variation by construction. What that means structurally is that request shapes are priced relative to each other rather than relative to what each one costs to serve: under one flat rate, short-context and long-context requests are cross-subsidized against each other, and the ratio between what they cost to serve and what they are charged moves apart as context grows. Long-context premium tiers are the mechanism a provider uses to reintroduce that distinction, which is exactly why their existence is evidence about serving cost rather than a pricing preference. This article does not attempt to say where any provider’s rate sits relative to its own cost, which is not observable from outside. It measures the serving-cost curve on hardware you can rent, and leaves the pricing inference to the reader.

This article measures the curve behind that structural claim on one model and one card: Ministral 3 14B Instruct (DigitalOcean model ID mistral-3-14B, Hugging Face repo mistralai/Ministral-3-14B-Instruct-2512), served with vLLM v0.27.1 on a single NVIDIA H200 bare GPU Droplet (141 GB VRAM), swept across 2K, 4K, 8K, 16K, 32K, 64K, 128K, and 256K tokens of input context. It does not use the Dedicated Inference product, AMD MI325X, or a second GPU, and it does not claim the measured curve is a universal law for every model and every accelerator. It claims the curve is real for this one, and that the mechanism behind it, a fixed KV cache pool colliding with linear per-token growth, generalizes even where the specific numbers do not.

• KV cache per token for Ministral 3 14B Instruct is fixed and derived from the model’s own configuration file: 163,840 bytes (160 KiB) per token in BF16, 81,920 bytes (80 KiB) per token in FP8. This holds regardless of load, and it is linear in context length, exactly as cost calculators assume.
• What is not linear is batch capacity: with a fixed KV cache pool, measured concurrent requests on a single H200 fell from 311 at 2K to 2 at 256K. That hyperbolic collapse is why effective cost per token rises even though bytes per token stay constant.
• sliding_window: null is confirmed for this model’s config.json, meaning all 40 layers use full attention and the linear KV-cache-per-token relationship holds across the entire 262,144-token context window this model supports, unlike the interleaved 1:3 sliding-window design in the older Ministral-8B-Instruct-2410.
• A serverless rate that is flat and symmetric at $0.20 per 1M tokens for mistral-3-14B does not move with context length, while the effective cost of serving those tokens on a dedicated H200 does. A flat rate cannot track a cost curve that bends, which is what long-context premium tiers on other models exist to correct for.
• Measured total throughput fell from 19089.8 tok/s at 2K to 4967.2 tok/s at 256K, driving effective cost per million total tokens from $0.0650 to $0.2500 at 100% utilization. That is a 3.84x measured increase across the sweep range, taken from the throughput ratio 19089.8 / 4967.2 rather than from the two rounded dollar figures.
• Effective cost per token on this dedicated H200 exceeds the $0.20 per 1M symmetric DigitalOcean Serverless rate for mistral-3-14B at 256K of input context at 100% utilization (break-even utilization 124.99%, above any achievable sustained util).
• The market disagrees with itself on whether this non-linearity is real: OpenAI’s GPT-5.5 and GPT-5.6 family and Google’s Gemini 2.5 Pro still price a long-context step, while Anthropic’s current generation (Sonnet 4.6, Sonnet 5, Opus 4.5 through 5, Fable 5) has dropped the step Claude Sonnet 4.5 still carries.

To reproduce the measurements in this article, you need:

• A DigitalOcean H200 bare GPU Droplet (141 GB VRAM) with root access. See How to Choose the Right GPU for vLLM Inference for how to size a Droplet against a model’s memory footprint before you provision one.
• vLLM v0.27.1 installed on the Droplet (pip install vllm==0.27.1), which is current stable as of 11 August 2026 (source: vLLM release history and docs.vllm.ai).
• Access to the mistralai/Ministral-3-14B-Instruct-2512 repository on Hugging Face, and a Python environment able to run huggingface-cli or set HF_TOKEN.
• FlashInfer installed separately if you intend to run the FP8 KV cache sensitivity arm (see Methodology); this article does not verify a specific FlashInfer version, follow FlashInfer’s own installation instructions.
• A way to scrape vLLM’s Prometheus metrics endpoint (curl is enough) and to read its startup log, since this methodology reads capacity numbers from the log rather than computing them.

DigitalOcean lists mistral-3-14B for Serverless Inference at $0.20 per 1M input tokens and $0.20 per 1M output tokens (source: DigitalOcean Inference pricing, last verified 24 August 2026). Whether mistral-3-14B is individually line-itemed for Dedicated Inference is not stated on that page; dedicated availability for a given model is inferred from the product’s general model support, not confirmed per model, so this article runs its own GPU Droplet rather than assuming Dedicated Inference support.

Model architecture, read from the config

The relevant architecture values come from config.json in mistralai/Ministral-3-14B-Instruct-2512, main branch, commit 1861cbb11d2a33d8107d82941d5662dedc5b04d8, read 20 August 2026.

The config does not state a total parameter count; “14B” is the model name, not a verified figure from config.json, and DigitalOcean does not publish one either.

The model is multimodal: a 24-layer Pixtral vision tower is present in the checkpoint. Serving text-only does not release that tower’s weight footprint from VRAM. That affects how much memory is left over for the KV cache pool, which is exactly why this article reads the pool size from the vLLM startup log instead of computing it from an assumed weight size (see Methodology). For deeper background on how KV caching mechanics affect inference cost generally, see How KV Caching Slashes LLM Inference Costs at Scale.

KV cache per token: derived, not measured

KV cache size per token follows directly from the config: 2 × layers × kv_heads × head_dim × bytes_per_element. The two figures below are derived rather than measured: they follow from arithmetic on the verified config values above, not from a benchmark, and they are stated here as architecture facts, not as results.

layers = 40
kv_heads = 8
head_dim = 128

bf16_bytes_per_token = 2 * layers * kv_heads * head_dim * 2
fp8_bytes_per_token = 2 * layers * kv_heads * head_dim * 1

print(f"BF16: {bf16_bytes_per_token:,} bytes/token ({bf16_bytes_per_token / 1024:.0f} KiB/token)")
print(f"FP8:  {fp8_bytes_per_token:,} bytes/token ({fp8_bytes_per_token / 1024:.0f} KiB/token)")

Output
BF16: 163,840 bytes/token (160 KiB/token)
FP8:  81,920 bytes/token (80 KiB/token)

This article does not extend that derivation to total pool size, batch capacity, or concurrency at any specific context length, and it does not state a specific concurrency ratio between two context lengths as a fact or a prediction. Those numbers come from the vLLM startup log, per request, not from arithmetic on the config. The reasoning below explains why the log is the source rather than the arithmetic.

Why capacity shrinks hyperbolically, not linearly

A KV cache pool reserved by vLLM is a fixed number of bytes, call it POOL_BYTES. Each request at context length L reserves L × bytes_per_token of that pool. The number of requests that fit concurrently is approximately:

max_concurrent_requests(L) ≈ POOL_BYTES / (L × bytes_per_token)

That is a hyperbola in L, of the form k / L. As context length grows, however many requests fit concurrently at a shorter context, proportionally fewer fit at a longer one, in the same fixed pool. That is a hyperbolic decline in batch capacity, not a linear one, and it is the mechanism behind this article’s thesis: aggregate throughput is bounded by how many requests can run concurrently once the KV pool, not compute, becomes the binding constraint, and that ceiling falls off a curve, not a line, as context grows.

This formula describes the memory-bound regime, and in this sweep it held with no slack from 2K through 256K. At every point, including 2K, measured peak concurrency matched floor of the boot-log ceiling exactly: 311 at 2K against a reported 311.02x, then 155, 77, 38, 19, 9, 4, and 2 at the longer points, with no trial falling short of that floor. No compute-bound or scheduler-bound ceiling capped concurrency below the pool-implied number anywhere in the tested range. The --max-num-seqs pin sat at 512 while 2K actually reached 311 concurrent requests, so the real headroom to that pin was never approached. That does not rule out a compute or scheduler floor existing below 2K, where the formula would keep predicting higher concurrency; this sweep’s shortest point simply did not reach it.

The formula above also implies, as a pure consequence of its own arithmetic, a theoretical maximum-concurrency ratio between any two context lengths in the sweep, for example between the 2K and 256K points. That ratio is a construction-derived expectation from the formula itself, not a measured result, and it is stated here, once, for exactly that reason. What this sweep did measure is per-point concurrency, and those values landed exactly on the integer floor of each boot-log ceiling rather than on a continuous ratio between two points. Do not promote the theoretical ratio into a finding: integer flooring does not preserve a clean ratio across the curve (311 at 2K and 2 at 256K is not a published 155.5x result), and the Results section reports the measured per-point numbers instead. That ratio does not reappear in Key Takeaways or anywhere else in this article framed as a finding.

Measured sustained concurrency at each context length (solid, log₂ vertical axis), against the continuous pool ÷ context ratio before flooring to whole requests (dashed). Both axes are log₂, so a straight line of slope -1 is exactly the hyperbolic relationship derived above, and that is what the measured points trace. The --max-num-seqs pin at 512 is marked and sits above every measured point, confirming the KV pool rather than the scheduler set batch size throughout. The widening gap between the measured points and the continuous curve at the long-context end is the flooring loss: 2.43 requests fit at 256K and only 2 can run.

The second non-linearity: prefill compute, ranked against the memory effect

The hyperbolic batch-capacity collapse above is a memory effect: a fixed pool of bytes divided among requests whose footprint grows with context. There is a second, independent cost driver that is easy to conflate with the first: attention computation during the prefill pass scales worse than linearly with context length. Self-attention computes a score between every pair of tokens in the input, so the compute cost of a single prefill pass grows with the square of context length, the O(n²) term, not linearly with it. This is a property of how self-attention is defined, not a measured result or a citation-requiring claim; it follows from the mechanism itself.

These two non-linearities are separate, and this article does not size the second one with any number, invented or otherwise. The memory effect is the one this sweep is built around, on mechanism grounds rather than by assertion. Prefill compute is paid once per request, then amortized across every decode step that request goes on to produce, so its contribution to a request’s total cost falls as output length grows. It is also the specific cost that chunked prefill exists to spread across scheduler steps rather than pay in one blocking chunk, and chunked prefill is enabled by default in the engine version used here. The batch-capacity ceiling has neither property: it binds continuously for a request’s entire lifetime, and no scheduler feature relaxes it, because it is a limit on bytes rather than on time. That asymmetry, one cost amortized and actively mitigated by the serving stack, the other structural and continuous, is why the memory effect is the one that shows up in a cost-per-token curve at serving scale. It is not a claim that the compute term is negligible in absolute size, and this article does not measure it. Do not read the Results section as evidence about prefill compute in either direction.

For a sense of where the KV cache stops being a rounding error against model weights, a separate question from either non-linearity above, Brenndoerfer’s KV cache memory calculation (published 7 January 2026) works a crossover where KV cache size equals model weight size at 26,702 tokens, rounded in its own prose to approximately 27K. That figure assumes LLaMA 7B, 32 layers, 32 KV heads with standard multi-head attention and no GQA, head_dim 128, FP16, batch size 1, and roughly 13.0 GB of weights, and it does not transfer to this article’s model: Ministral 3 14B has 8 KV heads rather than 32 and 40 layers rather than 32, so its own crossover point would have to be derived independently rather than reused. The same source shows the direction that matters here: LLaMA 2 70B, with 8 KV heads instead of 32, pushes that crossover out to roughly 427,000 tokens. Fewer KV heads per layer means KV cache grows more slowly relative to weights, which is the GQA effect this model also relies on. The 27K figure is never reused in this article for that reason.

The cost identity

The formula this article applies at every context length is:

effective_cpm = (hourly_rate / (total_tps * 3600 * utilization)) * 1,000,000

total_tps is total billable throughput, input tokens plus output tokens per second, combined, not output-only throughput. This distinction matters because DigitalOcean’s serverless rate for mistral-3-14B is $0.20 per 1M input tokens and $0.20 per 1M output tokens, symmetric. Because input and output bill at the identical rate, total billable tokens is the correct basis for comparison and no weighting between the two is needed. That symmetry is stated once, here, and used without re-deriving it at every sweep point.

This article defines its own denominator from first principles rather than inheriting a crossover number from elsewhere in this series. The first article in the series (Token Economics Across Traffic Profiles on Dedicated GPUs) used decode-only throughput as the denominator for a different model and reported a different crossover as a result. This piece does not re-cite that crossover, because the two articles use different denominators on different models. If you need that earlier framework’s method, read the linked article directly rather than relying on a quoted percentage here.

Prefix caching disabled: a stated scope limit

vLLM’s V1 engine enables automatic prefix caching (APC) by default. vLLM’s own team stated this when V1 shipped, citing near-zero overhead even at a 0% cache hit rate as the reason to enable it unconditionally (vLLM V1: A Major Upgrade to vLLM’s Core Architecture, vLLM Blog, 27 January 2025), and vLLM’s current automatic prefix caching documentation confirms the default still holds for the v0.27.x line used here rather than only for the alpha that introduced it. V1 has been the default engine since v0.8.0, released March 2025. Left enabled, APC would inflate measured throughput at exactly the long-context points where this article’s thesis predicts a collapse, which would plant an artifact in the worst possible place in the curve. Headline runs in this sweep therefore use --no-enable-prefix-caching, combined with vLLM’s random dataset generator for the load test, which produces unique token sequences with no shared prefixes, so there is no ambiguity about prefix reuse contaminating a result.

This is a scope boundary, not a silent flag choice: this piece measures the no-reuse floor for context length alone. Caching’s rescue effect on that floor is covered separately in the prompt-caching break-even companion piece, linked here at the exact point the scope boundary is drawn.

Scheduler pin: verified against the boot log, not assumed

--max-num-seqs is pinned to 512 for every sweep point. At the time this methodology was written, that pin was an expectation to verify against the startup-log concurrency readout, not a claim that it sat above the pool ceiling. The verification is now complete: at 2K the boot log reports 311.02x and measured peak concurrency is 311 in all three trials, against the 512 pin. The pin did not bind at 2K or at any longer point in the sweep. The remaining headroom, 311 reached versus 512 available, is the concrete margin this sweep observed; it was never approached.

FP8 arm requires an explicitly confirmed backend

FlashAttention-2, vLLM’s default attention backend, does not support FP8 KV cache. For the FP8 sensitivity arm, VLLM_ATTENTION_BACKEND=FLASHINFER is set explicitly, and the startup log is checked to confirm FlashInfer is actually active before any FP8 result from that run is trusted.

VLLM_ATTENTION_BACKEND=FLASHINFER vllm serve mistralai/Ministral-3-14B-Instruct-2512 \
  --served-model-name mistral-3-14B \
  --max-model-len 131072 \
  --max-num-seqs 512 \
  --no-enable-prefix-caching \
  --kv-cache-dtype fp8 \
  --gpu-memory-utilization 0.90 \
  --port 8000

grep -i "flashinfer" vllm_startup_fp8_131072.log

Output
Yes

The 256K FP8 run repeats the same check independently: Yes. The two contexts are confirmed separately because a backend that loads correctly at one context length is not guaranteed to remain active at another.

For background on quantization tradeoffs beyond KV cache dtype specifically, see the quantization section of How KV Caching Slashes LLM Inference Costs at Scale.

KV pool size is read from the log, never derived

The reserved KV cache pool at each context level is read from the vLLM startup log, specifically the GPU KV cache size: N tokens and Maximum concurrency for M tokens per request: X.XXx lines, never computed as 141 GB minus an assumed weight size. Two things make that computation unreliable here: the Pixtral vision tower occupies VRAM even when serving text-only, and --gpu-memory-utilization reserves a fraction of total VRAM rather than an absolute number, so the actual bytes available for KV cache depend on values that are themselves better read from the log than assumed.

CONTEXTS=(2048 4096 8192 16384 32768 65536 131072 262144)

for ctx in "${CONTEXTS[@]}"; do
  echo "=== context length: ${ctx} tokens ==="
  vllm serve mistralai/Ministral-3-14B-Instruct-2512 \
    --served-model-name mistral-3-14B \
    --max-model-len "${ctx}" \
    --max-num-seqs 512 \
    --no-enable-prefix-caching \
    --gpu-memory-utilization 0.90 \
    --port 8000 > "vllm_startup_${ctx}.log" 2>&1 &
  SERVER_PID=$!
  sleep 30
  grep -E "GPU KV cache size|Maximum concurrency" "vllm_startup_${ctx}.log"
  kill "${SERVER_PID}"
  wait "${SERVER_PID}" 2>/dev/null
done

Output
GPU KV cache size: 636,976 tokens
Maximum concurrency for {context} tokens per request: 311.02x

That startup readout also replaces most of the manual concurrency-sweep guesswork: boot the server at each context level, read the concurrency line, then load-test around that number to confirm it rather than starting the load test blind.

Request shape holds output length constant

Input length varies across the sweep; output length is held fixed at 256 tokens for every point. Varying both simultaneously would let the output-token cost effect, which is the subject of the output token pricing companion piece, leak into a curve this article attributes to input context length alone.

Warm-up, trial count, and variance are reported, not assumed

Every sweep point discards a warm-up period before measurement starts, and reports variance rather than a single point estimate:

• Warm-up. 10 requests are discarded at each sweep point before measurement begins. This matters specifically because the first requests served at a newly booted context length pay CUDA graph capture and memory allocation costs that later requests at the same context length do not; including them in the measured window would bias throughput downward for reasons that have nothing to do with this article’s thesis.
• Trials. 3 trials are run at each sweep point. Reported throughput and latency figures are the mean across those three trials, with the trial-to-trial standard deviation shown in each table’s variance column.
• Variance. Every throughput and latency figure in the Results tables is reported with a variance measure, standard deviation or a min/max range, alongside the point estimate, not as a bare single number. Both Results tables carry a dedicated variance column for this reason.

Harness and raw data for this run

The server launch commands, vllm bench serve invocations, sizing rules, warm-up and trial protocol, and metric scrapes used for this sweep are fully specified in this Methodology section. Raw per-request result files (--save-detailed JSON with start_times, ttfts, and itls) were retained for every accepted trial and used for the concurrency and latency figures reported in Results. Those files, along with the sweep orchestration code and the vLLM startup logs each KV pool size was read from, are published alongside this article at context-length-inference-cost. Reproduction is possible either from the commands and protocol above or directly from the retained per-request data.

Request volume scales with expected duration, not a fixed count

A request at 256K context takes substantially longer to complete than one at 2K, since both prefill and decode scale with context length. Using the same --num-prompts and --request-rate at every sweep point would give sharply uneven statistical power per point, more effective sampling at short context, less at long context, without anyone deciding that on purpose. This sweep instead targets a fixed minimum of sustained-load wall-clock time per sweep point rather than a fixed request count, so that short-context points, which complete requests far faster, are not oversampled relative to long-context points, which take substantially longer per request. --num-prompts and --request-rate are chosen per context length to satisfy that duration floor, and are not held constant across the sweep. The actual values used at each point are: --request-rate inf everywhere; --num-prompts 2K 2488, 4K 1240, 8K 616, 16K 304, 32K 152, 64K 72, 128K 50, 256K 50.

vllm bench serve \
  --backend vllm \
  --model mistral-3-14B \
  --host localhost \
  --port 8000 \
  --dataset-name random \
  --random-input-len "${ctx}" \
  --random-output-len 256 \
  --num-prompts "${NUM_PROMPTS}" \
  --request-rate inf \
  --max-concurrency "${MAX_CONCURRENCY}"

NUM_PROMPTS and MAX_CONCURRENCY are set per context length from the sizing rule and boot-log ceiling above (for example at 2K: 2488 prompts, concurrency 311).

Preemption captured, not inferred

vllm:num_preemptions_total and vllm:kv_cache_usage_perc are scraped from the Prometheus metrics endpoint at each sweep point. This build’s metric names are confirmed rather than assumed, since older vLLM builds emit vllm:gpu_cache_usage_perc instead of the V1 name.

curl -s http://localhost:8000/metrics | grep -E "vllm:num_preemptions_total|vllm:kv_cache_usage_perc|vllm:gpu_cache_usage_perc"

Output
vllm:kv_cache_usage_perc{engine="0",model_name="mistral-3-14B"} 0.0
vllm:num_preemptions_total{engine="0",model_name="mistral-3-14B"} 0.0

model_name="mistral-3-14B" in that output is the --served-model-name value passed at boot, not the Hugging Face checkpoint path (mistralai/Ministral-3-14B-Instruct-2512-BF16) listed in the run metadata table below; vLLM labels its own metrics with the served name, not the source repo.

vllm:num_preemptions_total is scraped at every sweep point rather than left as a footnote, and it is a counter, so an end-of-trial scrape captures every preemption that occurred during the trial. It stayed at zero throughout (see Results), which is a real result about this load pattern.

vllm:kv_cache_usage_perc was scraped alongside it, but it is a gauge reporting instantaneous occupancy, and these scrapes were taken at end of trial when the server was already idle. It therefore read 0.0 everywhere and carries no information about the sweep. It is not reported as a result. Pool occupancy under load is instead derived in the Results section from the reserved pool size and the measured concurrency, which is a more direct measurement of the same quantity and does not depend on scrape timing. Anyone reproducing this sweep who wants the gauge itself must poll it during the run, not after it.

Prefill contention with in-flight decode, as mechanism evidence

Separately from the prefill compute non-linearity discussed in the previous section, long-context prefill can stall in-flight decode work when both share a batch. Agrawal et al., “Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve,” USENIX OSDI 2024 (arXiv:2403.02310), Figure 9, measured naive hybrid batching increasing time-between-tokens by up to 28.3x versus a decode-only batch, on LLaMA2-70B across four A100s with a 512-token budget. That result is cited here only as mechanism evidence for why TTFT and inter-token latency can degrade sharply at long context when prefill and decode compete for the same batch; it is a different model and a different accelerator, and it is not a comparable measurement to any TTFT figure reported later in this article.

Environment and run metadata

The tables below report every metric captured at each of the eight sweep points, on the BF16 headline arm, with the methodology from the previous section applied identically at every point. Every throughput and latency figure carries its own variance column per the warm-up and trial protocol above.

Capacity and preemption

Across all eight headline-arm points (2K through 256K, three trials each) and both FP8 sensitivity points (128K and 256K), end-of-trial scrapes of vllm:num_preemptions_total returned 0 every time. That counter does not reset between scrapes, so a zero at end of trial is a real result: this load pattern never triggered vLLM’s preemption path at any context length on either arm.

The kv_cache_usage_perc gauge is not reported in the table above. It is a gauge, and these scrapes were taken at end of trial when the server was already idle, so it read 0.0 at every point and says nothing about what the pool was doing under load. The Methodology section explains this in full.

The pool occupancy column replaces it, derived from two figures already measured: the reserved pool size from the startup log, and the concurrency confirmed by millisecond-precision interval sweep over each trial’s raw per-request timing. At 2K, 311 concurrent requests each holding 2,048 tokens occupy 636,928 of 636,976 available tokens, or 99.99 percent of the pool. The pool was not merely binding batch size at the short-context end, it was full to within 48 tokens.

Occupancy also falls as context grows, from 99.99 percent at 2K to 82.31 percent at 256K, and that decline is a second mechanism this sweep exposes. Concurrency is the floor of a continuous pool-to-context ratio, so whatever fraction of a request does not fit is stranded: paid for, reserved, and unusable. The quantum being discarded is one whole request’s worth of cache, which grows with context length, so the stranded share grows too. At 256K, 112,688 tokens of pool sit idle because 2.43 requests fit and only 2 can run. Effective capacity at long context is therefore worse than the smooth hyperbola in the architecture section predicts, and the extra loss comes from integer quantization rather than from KV growth.

The --max-num-seqs pin did not bind at 2K: measured peak concurrency was 311 against the 512 pin, and no point in the sweep reached 512. No row is flagged as scheduler-limited.

Throughput and latency

For latency methodology and how to read p50 against p99 separately rather than collapsing them into one number, see Why your vLLM p99 latency blows up in production and how chunked prefill and scheduling fix it and P50 vs. P99 Latency in LLM Inference. Rising TTFT p99 relative to p50 at the longer context points, once measured, is consistent with the prefill-decode contention mechanism cited in the Methodology section, though this article’s own TTFT figures are the measurement that matters here, not the Sarathi-Serve figure, which remains a different model and accelerator.

The headline chart

Measured effective cost per 1M total tokens at 100% utilization (solid), held flat at the 2K value ($0.0650/1M) as the linear-pricing reference (dashed), with the $0.20/1M serverless rate marked for comparison. X-axis is context length on a log₂ scale from 2K through 256K. The gap between the measured curve and that flat line is the argument this article exists to make: where the two lines diverge is where the linear assumption stops matching what the hardware actually does.

Reading latency against the cost curve

Preemption counts stayed at 0 at every point on both arms, and because that metric is a counter rather than a gauge, the zero is a genuine result rather than an artifact of when it was read. There is therefore no eviction cliff to read off the capacity table: the pool filled to 99.99 percent at 2K and 82.31 percent at 256K without vLLM ever preempting a request. The kv_cache_usage_perc gauge is excluded here for the reason given in the Methodology section, so it contributes nothing either way. The comparison this section was written to make is therefore between TTFT p99 and effective cost-per-token, both taken from the filled tables in this article.

Define “rises sharply” as the largest consecutive-step percentage jump along each series. Under that definition, both signals peak at the same transition:

Both series climb through the mid-range and then take their single largest step from 128K to 256K. Through 64K, p99 has realized 27.6% of its total 2K-to-256K rise and cost has realized 25.6%; through 128K those shares are 42.2% and 48.9%. Neither signal pulls clearly ahead of the other under this definition. Reading p99 against p50 at the same point does not rescue a lead/lag story either: the p99/p50 ratio falls from about 20x at 2K to 1.5x at 256K, because p50 itself rises faster than p99 at the long-context end.

What this sweep shows is co-movement, not an early latency warning that precedes the cost bend. An operator watching only a cost dashboard would not miss a separate earlier cliff in this dataset, because the sharpest latency move and the sharpest cost move land on the same step. The useful operational read is narrower: preemption stayed silent while both TTFT and cost climbed, and the pool was between 82 and 100 percent full the entire time, so a zero preemption counter is not evidence that long-context load is cheap, latency-stable, or operating with cache headroom.

The 256K point

256K stays part of the continuous curve above. It stays on that curve because the measured concurrency at this point still describes the same batch-capacity mechanism as the rest of the sweep.

Measured, log-verified concurrency at 256K on the BF16 headline arm is 2, identical across all three trials, read directly from the vLLM startup log’s concurrency line and independently confirmed by a millisecond-precision sweep over each trial’s raw per-request timing data rather than taken from vLLM’s own aggregate concurrency field. A concurrency of 2 is thin enough to raise the question of whether the measurement is still about batch capacity at all. The answer is that it is. A ceiling of a single request would be a different regime: at concurrency 1, there is no batch left to measure, no preemption between concurrent flows, and no scheduling contention, so nothing this sweep’s batch-capacity mechanism describes would still apply. Two concurrent requests still contend for the same fixed KV pool and still exhibit the batch-capacity behavior this entire sweep exists to characterize, at the point where the hyperbolic collapse the architecture section derives has run furthest. The measurement itself is also stable: 50 completed requests per trial at this point, zero failed requests, and all three trials landing on the identical concurrency value, the same stability this sweep’s other seven points show. Pulling 256K into a separate section would remove the endpoint that makes the curve’s argument. Concurrency falling from 311 at 2K to 2 at 256K is the mechanism this article measures, visible at its furthest point. The FP8 KV sensitivity arm’s own 256K concurrency figure is higher, reported separately in its own section below, and does not change this call, since that arm was never part of “the continuous curve” this section is about.

FP8 KV cache sensitivity arm

The FP8 arm is reported only for 128K and 256K, the two context lengths where FP8’s halved per-token footprint (80 KiB versus 160 KiB) is most likely to change the concurrency ceiling meaningfully. Each context length carries its own independent set of slots.

If the startup log does not confirm FlashInfer active for a given context length, that row is reported as not run rather than populated with numbers, per the methodology rule above.

Applying the cost identity from the Methodology section, effective_cpm = (hourly_rate / (total_tps * 3600 * utilization)) * 1,000,000, at the DigitalOcean H200 GPU Droplet on-demand rate of $4.47 per hour (source: GPU Droplet pricing, last verified 25 August 2026), gives an effective cost per million total tokens at each context length. Comparing that figure against the $0.20 per 1M symmetric serverless rate at each context length gives the utilization a dedicated GPU must sustain to break even.

Derivation at $4.47/hr, utilization = 1.0 (100%), using the filled total-throughput means from the Results table:

• 2K: (4.47 / (19089.8 × 3600 × 1.0)) × 1,000,000 = 0.0650434613 → $0.0650/1M; break-even utilization = 0.0650434613 / 0.20 = 0.325217 (32.52%)
• 4K: (4.47 / (18812.3 × 3600 × 1.0)) × 1,000,000 = 0.0660029165 → $0.0660/1M; break-even utilization = 0.0660029165 / 0.20 = 0.330015 (33.00%)
• 8K: (4.47 / (17933.2 × 3600 × 1.0)) × 1,000,000 = 0.0692384330 → $0.0692/1M; break-even utilization = 0.0692384330 / 0.20 = 0.346192 (34.62%)
• 16K: (4.47 / (16214.8 × 3600 × 1.0)) × 1,000,000 = 0.0765761321 → $0.0766/1M; break-even utilization = 0.0765761321 / 0.20 = 0.382881 (38.29%)
• 32K: (4.47 / (13929.6 × 3600 × 1.0)) × 1,000,000 = 0.0891387166 → $0.0891/1M; break-even utilization = 0.0891387166 / 0.20 = 0.445694 (44.57%)
• 64K: (4.47 / (11048.2 × 3600 × 1.0)) × 1,000,000 = 0.1123863314 → $0.1124/1M; break-even utilization = 0.1123863314 / 0.20 = 0.561932 (56.19%)
• 128K: (4.47 / (7988.2 × 3600 × 1.0)) × 1,000,000 = 0.1554376038 → $0.1554/1M; break-even utilization = 0.1554376038 / 0.20 = 0.777188 (77.72%)
• 256K: (4.47 / (4967.2 × 3600 × 1.0)) × 1,000,000 = 0.2499731572 → $0.2500/1M; break-even utilization = 0.2499731572 / 0.20 = 1.249866 (124.99%)

Break-even utilization is effective_cpm(100%) / 0.20. At 100% utilization, effective cost exceeds $0.20/1M first at 256K ($0.2500 > $0.20); that is also the only point whose break-even utilization exceeds 100% (124.99%). The measured 256K/2K cost ratio is 3.84x, taking 2K as the baseline to match the flat reference line in the chart above. It is computed from the unrounded figures in the derivation list above, 0.2499731572 / 0.0650434613 = 3.8432, which is identical to the total-throughput ratio 19089.8 / 4967.2 = 3.8432. Dividing the rounded display figures ($0.2500 / $0.0650) gives 3.85 instead, so the unrounded values are the ones this figure comes from.

Two figures follow from the same formula applied to the filled rows above:

• 256K: first context length at which effective cost at 100% utilization exceeds $0.20/1M ($0.2500 > $0.20; break-even utilization 124.99% > 100%)
• 3.84x: measured ratio of effective cost-per-token at 256K vs 2K, computed from the unrounded figures (0.2499731572 / 0.0650434613) and equal to the total-throughput ratio 19089.8 / 4967.2; not the formula-derived construction from the architecture section

The operator question this article exists to answer is at what specific context length a dedicated H200 stops being the cheaper way to serve Ministral 3 14B traffic, and what to do once you cross that line. That answer is 256K, read directly from the break-even column in the previous section, not estimated from the shape of the curve alone. Four responses follow, in the order an operator should actually apply them.

1. Engineer the context down first, always

Before routing decisions or hardware changes, the first lever is reducing how much context a request actually needs. Compression and retrieval instead of context-stuffing an entire document into the prompt is the first thing to try, because every token shaved off a request’s context recovers batch slots directly: the hyperbolic relationship in the architecture section means a shorter context costs less per token to prefill and lets more requests fit in the same fixed KV pool at once, which is where the measured curve in this article’s Results section actually gets its shape. Retrieval-augmented generation, retrieving only the passages a request needs rather than stuffing an entire source document into context, is the correct example of this discipline in practice. This article does not attach a specific “every N thousand tokens saved” figure to that recovery, since no such figure was ever verified; the mechanism is the claim, and the measured curve is what lets you price exactly how much a given amount of context reduction is worth at your own operating point.

2. Route by context length, not by traffic volume alone

If a meaningful share of traffic routinely exceeds the break-even context length above, route that traffic to serverless rather than dedicated hardware. The reason is the structural point from the Introduction applied to two published numbers. DigitalOcean bills mistral-3-14B on Serverless Inference at a flat $0.20 per 1M tokens that does not change with context length, while this article’s measured curve shows your effective cost per token on a dedicated H200 rising with it. Those two facts do not meet: one instrument is flat in context length and the other is not. Above the break-even point, the flat instrument is simply the cheaper one for you to be billed under, and that remains true whatever the provider’s own cost of serving those requests turns out to be. The decision does not require knowing anything about provider economics, only about which of two rates you are exposed to.

3. Re-check the FP8 KV cache arm before assuming dedicated is dead at long context

The FP8 sensitivity results at 128K and 256K in the Results section either move the break-even point out or they don’t. That is an empirical answer this article’s own sweep is built to give, not a general claim that quantized KV cache always rescues long-context economics on every model and accelerator.

4. Treat prefix and prompt caching as a separate lever, not a substitute for this measurement

This article’s headline numbers deliberately measure the no-reuse floor. If production traffic has high prefix reuse, the prompt-caching companion piece covers how much of this article’s break-even line that reuse can move, and by how much, separately.

Two callbacks this measurement quantifies

The Mixture-of-Experts inference cost piece warns that hardware should be budgeted around the cache term alongside model weights. This article’s own measured numbers are exactly that sizing rule quantified for one dense model: the KV cache pool, not the weight footprint, is what determines how batch capacity collapses with context, which is why this article reads pool size from the startup log rather than assuming it follows from weights alone.

DigitalOcean’s own inference engine update notes document long-context-efficient model architectures, compressed attention variants and reduced KV footprints, as active development. Those are the model-side response to exactly the economics this article measures on the serving side: if a future architecture shrinks bytes-per-token the way FP8 KV cache does here, the same hyperbolic-capacity mechanism applies, just with a larger POOL_BYTES-to-footprint ratio and a break-even context length that moves out accordingly.

None of this extends to a claim that every model on every accelerator hits its break-even point at the same context length as Ministral 3 14B on an H200. The mechanism, a fixed KV pool and a linear per-token cost, is architectural and generalizes; the specific number in 256K does not, and re-running the sweep in the Methodology section against your own model and card is the only way to get your own number.

Does context length affect inference cost linearly?

Not on dedicated hardware. Serverless per-token rates are usually flat or stepped, which looks linear or piecewise-linear, but on a dedicated GPU billed by the hour, KV cache per token is linear while the batch capacity that fits in a fixed KV cache pool falls hyperbolically as context grows. That means effective cost per token rises with context length even when the underlying per-token cache cost does not change, because fewer requests fit in the same GPU-hour at longer context.

Why does batch capacity fall as context length grows, if KV cache per token is constant?

Because the KV cache pool itself is a fixed number of bytes on the GPU, not something that grows with demand. Each request’s KV footprint is context_length × bytes_per_token, so at longer context, each request claims a larger share of that fixed pool, and fewer requests fit concurrently. The relationship between context length and how many requests fit is a hyperbola, not a straight line, even though the per-token cost inside that formula is itself linear.

If serverless pricing is flat, doesn’t that mean long context is fairly priced?

Not in the sense of the price tracking the cost. A flat per-token rate charges identically for a token in a 2K-context request and a token in a 256K-context request, while the measurements in this article show that the serving cost behind those two tokens is not the same on dedicated hardware. A single flat rate cannot express that difference, so what it does instead is price request shapes relative to each other. Long-context premium tiers are how providers reintroduce the distinction, and the fact that some providers charge them and others do not is the disagreement this article’s introduction starts from. For your own decision the useful consequence is narrower and does not require guessing at anyone’s costs: if you are billed a rate that is flat in context length, your bill does not rise with context the way a dedicated GPU’s effective cost per token does.

Does Ministral 3 14B use sliding-window attention?

No. sliding_window is null in the model’s config.json, and all 40 layers use full attention. This is a meaningful difference from the older Ministral-8B-Instruct-2410, which interleaves full and sliding-window layers at a 1:3 ratio. Sliding-window attention would cap KV cache growth at the window size; because this model does not use it, KV cache grows linearly with context across its full 262,144-token supported window.

How much KV cache memory does Ministral 3 14B need per token?

163,840 bytes (160 KiB) per token in BF16, and 81,920 bytes (80 KiB) per token in FP8, derived from 2 × 40 layers × 8 KV heads × 128 head_dim × bytes_per_element. These are architecture facts derived from the model’s own configuration file, not from a benchmark. They are fixed properties of the model that do not depend on the serving hardware, though how many tokens’ worth of cache actually fits on a given GPU does.

Is the KV cache memory effect the only reason long context costs more to serve?

No. There is a second, independent cost driver: attention computation during the prefill pass scales with the square of context length, not linearly, because self-attention scores every pair of tokens in the input. This article does not size that effect with a measured number. Operationally, this article’s sweep is built around the memory effect rather than the compute effect because of an asymmetry between the two: prefill compute is amortized across a request’s decode steps and is specifically what chunked prefill, on by default in the engine used here, exists to spread across scheduler steps, while the batch-capacity ceiling binds continuously for a request’s entire lifetime with no equivalent mitigation. That is not a claim the compute term is negligible; this article does not measure it either way.

What should I do first if my long-context costs are too high?

Reduce the context before changing hardware or routing. Retrieval and compression, retrieving only the passages a request needs rather than stuffing a full document into the prompt, recovers batch capacity directly, because a shorter context claims a smaller share of the fixed KV cache pool and lets more requests run concurrently. Routing by context length and re-checking quantized KV cache are the next levers, in that order, covered in the “Where the Economics Flip Your Architecture Decision” section.

What hardware and model does this article’s measurement apply to?

A single NVIDIA H200 bare GPU Droplet (141 GB VRAM) running Ministral 3 14B Instruct in BF16 with vLLM v0.27.1. The measured curve does not automatically transfer to a different model, a different GPU generation, or multi-GPU setups; the architectural mechanism, a fixed KV pool colliding with linear per-token growth, generalizes, but the specific throughput and cost numbers do not.

Does automatic prefix caching change this result?

The headline measurements in this article deliberately disable automatic prefix caching (--no-enable-prefix-caching) and use unique, non-repeating prompts, specifically so caching’s throughput benefit does not mask the context-length effect this piece is isolating. Real production traffic with repeated prefixes will see a smaller effective penalty at long context than this article’s no-reuse floor shows. That rescue effect, and by how much, is covered in the prompt-caching break-even companion piece.

Does DigitalOcean publish a maximum output token limit for this model?

No. DigitalOcean does not publish a max output token figure for mistral-3-14B on its pricing page, and this article does not estimate one.

Cost calculators price context length as a straight line because that is how the rate card is written, input tokens times a rate, output tokens times a rate, and ten times the tokens looks like ten times the cost. This article measured what actually happens underneath that rate card on one dedicated GPU: KV cache per token for Ministral 3 14B Instruct is genuinely linear, 160 KiB in BF16 and 80 KiB in FP8, derived directly from the model’s own config.json. What is not linear is how many of those per-token allocations fit in the fixed KV cache pool a GPU actually has, and that capacity falls hyperbolically as context grows, which is why effective cost per token on dedicated hardware rises with context length even though nothing about the per-token cache math changed. A second, independent non-linearity, quadratic prefill compute, exists alongside the memory effect. This article’s sweep is built around the memory effect specifically because it binds continuously for a request’s entire lifetime with no scheduler mitigation, unlike prefill compute, which is amortized across decode steps and is what chunked prefill exists to spread out. That asymmetry, not a claim that compute is negligible, is why the curve this article measures is a memory-effect curve.

That same mechanism explains why a flat per-token rate and a measured serving-cost curve cannot both be right about what long context costs. A rate that does not move with context length cannot track a cost that does, so it prices request shapes against each other instead, and long-context premium tiers are how a provider chooses to reintroduce the difference. The measured break-even context length in this article, where a dedicated H200 stops being the cheaper way to serve Ministral 3 14B traffic, is specific to being billed under a rate that stays flat across the whole sweep range. If DigitalOcean’s serverless rate for this model ever stepped at a context threshold the way OpenAI’s GPT-5.x family does on the same catalogue page, that break-even point would have to be recalculated against the stepped rate rather than the flat one.

The measured curve in the Results section, and the break-even context length it implies, is specific to Ministral 3 14B Instruct on a single H200 running vLLM v0.27.1 with prefix caching deliberately disabled. It is not a claim that every model on every accelerator crosses over at the same point, and the market’s own disagreement, OpenAI and Google still pricing a long-context step, Anthropic’s current models dropping it, is a sign that no single answer applies everywhere. What generalizes is the mechanism, and the mechanism is best understood as a statement about density rather than quantity: context length is priced on a rate card as a quantity, tokens times a rate, but it behaves on the hardware as a density, it determines how many requests share a fixed pool of GPU memory at once, and it is that density, not the rate card, that actually sets the cost of long context.

This article is the fifth in an inference-economics series that includes Long-Context Inference at Scale: The Hidden Infrastructure Cost (the direct predecessor, linked above in the Introduction), Token Economics Across Traffic Profiles on Dedicated GPUs, Why Spiky Inference Traffic Breaks the Dedicated GPU Math, and The Hidden Cost of Output Token Pricing. Related reading: How KV Caching Slashes LLM Inference Costs at Scale, Mixture-of-Experts Inference Cost, LLM Inference Optimization, How to Choose the Right GPU for vLLM Inference, and DigitalOcean’s own inference engine update notes and LLM inference benchmarking blog.

• DigitalOcean, Inference Pricing, last verified 24 August 2026
• DigitalOcean, Droplet Pricing, last verified 25 August 2026
• Google, Gemini Developer API Pricing, Standard tier, fetched 27 August 2026
• Hugging Face, mistralai/Ministral-3-14B-Instruct-2512 config.json, main branch, commit 1861cbb11d2a33d8107d82941d5662dedc5b04d8, read 20 August 2026
• vLLM, documentation home
• vLLM, release notes (v0.27.1, current stable as of 11 August 2026)
• vLLM Blog, vLLM V1: A Major Upgrade to vLLM’s Core Architecture, 27 January 2025
• vLLM, Automatic Prefix Caching (default-on behaviour for the v0.27.x line)
• vLLM, v0.8.0 release notes (V1 engine becomes the default, March 2025)
• Agrawal et al., “Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve,” USENIX OSDI 2024 (arXiv:2403.02310)
• Brenndoerfer, KV Cache Memory Calculation for LLM Inference, published 7 January 2026
• DigitalOcean Community, Long-Context Inference at Scale: The Hidden Infrastructure Cost
• DigitalOcean Community, Token Economics Across Traffic Profiles on Dedicated GPUs
• DigitalOcean Community, Why Spiky Inference Traffic Breaks the Dedicated GPU Math
• DigitalOcean Community, The Hidden Cost of Output Token Pricing
• DigitalOcean Community, How KV Caching Slashes LLM Inference Costs at Scale
• DigitalOcean Community, Mixture-of-Experts Inference Cost
• DigitalOcean Community, LLM Inference Optimization
• DigitalOcean Community, How to Choose the Right GPU for vLLM Inference
• DigitalOcean Community, Why your vLLM p99 latency blows up in production and how chunked prefill and scheduling fix it
• DigitalOcean Community, P50 vs. P99 Latency in LLM Inference
• DigitalOcean Community, Prompt-Caching Cost Break-Even
• DigitalOcean Blog, What’s New on Inference Engine
• DigitalOcean Blog, LLM Inference Benchmarking
• Benchmarking harness and raw sweep data, context-length inference cost sweep