Welcome to the production frontier of Generative AI. As open-weights models become increasingly powerful and competitive, engineering teams are transitioning from consuming shared, pay-per-token cloud APIs to deploying dedicated inference infrastructure.

Situated between CUDA and Kubernetes, inference engineering focuses on the second phase of a generative AI model's lifecycle: serving models in production with a strict focus on latency, reliability, and unit economics. Drawing from Philip Kiely’s book, Inference Engineering: Serving Generative AI in Production, this guide explores the technical architecture required to serve LLMs and multi-modal models at scale.

Table of Contents

  1. What Is Automated vs. What Requires Manual Engineering
  2. The Three-Layer Inference Stack
  3. Key Performance Metrics: TTFT vs. TPS
  4. Runtime Optimization Techniques
    • Continuous Batching
    • KV Cache Reuse & Prefix Caching
    • Quantization Strategies
    • Speculative Decoding
    • Model Parallelism Topologies
    • Disaggregated Serving
  5. Inference Engine Landscape: vLLM vs. SGLang vs. TensorRT-LLM
  6. Production Deployment & Operational Strategy
  7. Conclusion & Next Steps

What Is Automated vs. What Requires Manual Engineering

One of the most critical questions for teams building inference platform capabilities is establishing where software abstraction ends and manual engineering begins. Modern inference frameworks automate low-level kernel execution, but system-level orchestration, trade-off selection, and environment pinning still require active manual tuning.

As detailed in Philip Kiely's analysis, the table below maps out what modern inference software handles automatically versus what inference engineers must manually design and configure.

Operational Dimension What Is Automated Today What Requires Manual Engineering
Request Scheduling In-flight continuous batching; token-level request weaving; basic queue management within a single replica. Multi-replica load balancing; SLA-based priority queueing; cross-cloud capacity routing.
Memory Allocation Virtual memory paging for KV cache (e.g., PagedAttention); basic allocation of GPU VRAM per instance. Tuning total KV cache memory fractions; managing offloading to CPU RAM; preventing VRAM OOM under spike traffic.
Kernel Selection & Fusion Graph compilation (e.g., torch.compile); basic operator fusion for standard model architectures. Hand-writing custom CUDA kernels (e.g., FlashAttention, DeepGEMM); integrating non-standard model layers.
Model Optimization Runtime execution of pre-quantized weights (FP8/INT4/AWQ); basic prefix matching. Conducting post-training calibration (via ModelOpt); training specialized speculator heads (e.g., EAGLE draft heads).
Model Parallelism Execution of inter-GPU collective communications (NCCL) once tensor parallel sizes are declared. Deciding exact parallel topologies (TP vs. PP vs. EP vs. CP); matching topologies to interconnect bandwidth (NVLink vs. PCIe).
Deployment Lifecycle Base container runtime execution; process supervision inside single instances. Pinned dependency tree management; multi-cloud node provisioning; orchestrating 4-step canary deployments; engine compilation caching.

The Automation Boundary

Modern engines like vLLM, SGLang, and TensorRT-LLM have successfully automated token-by-token request scheduling and memory page mapping. However, inference optimization remains a highly experimental manual process. As Kiely illustrates through a real-world case study at Baseten, an inference engineer once benchmarked 77 distinct parameter and framework configurations via custom scripts before identifying a non-obvious setup that doubled throughput for a customer's code completion service.

The Three-Layer Inference Stack

Generative AI inference cannot be solved by simply running model weights on a GPU. Operating mission-critical workloads requires three tightly integrated layers:

+-----------------------------------------------------------------------+ | TOOLING LAYER | | Developer Abstractions, Observability, SDKs, Benchmarking & Evals | +-----------------------------------------------------------------------+ | INFRASTRUCTURE LAYER | | Multi-Cloud Routing, Capacity Management, Autoscaling, Containers | +-----------------------------------------------------------------------+ | RUNTIME LAYER | | CUDA Kernels, PyTorch, Inference Engines, Single-Instance Tuning | +-----------------------------------------------------------------------+ 
  1. Runtime Layer: Focuses on maximizing the performance of a single model running on a single GPU or a multi-GPU instance. It encompasses low-level CUDA kernels, PyTorch graph execution, and specialized engine servers.
  2. Infrastructure Layer: Handles scaling across multi-node clusters, availability zones, and cloud service providers. It addresses autoscaling, multi-cloud capacity management, container orchestration, and routing.
  3. Tooling Layer: Provides developer abstractions that balance granular manual control with operational productivity, including observability pipelines, evaluation benchmarks, and deployment SDKs.

Key Performance Metrics: TTFT vs. TPS

Optimizing inference requires navigating a fundamental trade-off between latency (user responsiveness) and throughput (cost efficiency per token). The two primary metrics governing LLM serving are:

  • Time to First Token (TTFT): The duration from request submission until the user receives the first streaming token. TTFT is dominated by the Prefill phase, which is compute-bound as the model processes the entire input prompt simultaneously to construct the initial Key-Value (KV) cache.
  • Tokens Per Second (TPS) / Inter-Token Latency (ITL): The generation rate of subsequent tokens during the Decode phase. Decode is memory-bandwidth bound, as each generated token requires reading every weight matrix from GPU VRAM back into compute registers.

Runtime Optimization Techniques

Continuous Batching

Traditional static batching waits for a fixed number of requests to arrive before launching inference, creating high latency for early arrivals. Continuous batching (or in-flight batching) operates at the individual token level: incoming requests are joined into the running batch on the next forward pass, and completed requests are evicted immediately upon generating an end-of-sequence token.

KV Cache Reuse & Prefix Caching

To prevent recomputing attention for previously processed tokens, LLMs store attention keys and values in GPU memory (the KV cache). Prefix caching identifies identical text prefixes across different incoming requests—such as long system prompts, agent instructions, or code context—and reuses their pre-computed KV cache, reducing TTFT from thousands of milliseconds down to tens.

Quantization Strategies

Quantization converts model weights and activations from high-precision floating-point formats (BF16/FP16) to lower-precision representations (FP8, INT4, or microscaling formats like MXFP4).

  • Prefill Acceleration: Lower precision unlocks higher Tensor Core FLOPS, directly speeding up compute-bound prompt processing.
  • Decode Acceleration: Halving the precision reduces memory payload sizes by 50%, directly mitigating memory bandwidth bottlenecks during token generation.
  • Post-training quantization tools like NVIDIA TensorRT Model Optimizer (ModelOpt) compute scale factors and perform calibration to preserve output quality.

Speculative Decoding

Because the decode phase leaves compute units underutilized while waiting for VRAM data transfers, speculative decoding leverages that idle compute to draft and verify multiple candidate tokens per forward pass.

  • Draft-Target Speculation: Uses a smaller, faster secondary model to propose candidate tokens, which the primary target model verifies in a single forward pass.
  • EAGLE Speculation: Attaches a lightweight, trained head directly to the target model's hidden states, achieving higher token acceptance rates with lower memory overhead.
  • N-gram Speculation: Builds a local dictionary from prompt context to guess repeated phrases without requiring a separate neural network model.

Model Parallelism Topologies

When models exceed the VRAM of a single GPU, parallelism strategies distribute execution across multiple devices:

  • Tensor Parallelism (TP): Splits individual matrix multiplication layers across GPUs within a single node. It reduces latency and expands available VRAM, but requires high-bandwidth interconnects like NVIDIA NVLink.
  • Pipeline Parallelism (PP): Distributes consecutive layers across GPUs or nodes, passing intermediate activation vectors sequentially.
  • Expert Parallelism (EP): Routes tokens to specific expert networks across devices in Mixture of Experts (MoE) architectures.
  • Context Parallelism (CP): Divides long sequence lengths across devices using algorithms like Ring Attention.

Disaggregated Serving

In standard deployments, prefill (compute-bound) and decode (memory-bound) share the same GPU instances, creating resource contention under high traffic. Disaggregated serving separates prefill workers from decode workers onto independent hardware clusters. Prefill nodes build the KV cache at maximum compute throughput and transfer the cached context over fast network backplanes (such as InfiniBand or high-speed Ethernet) to dedicated decode nodes.

Inference Engine Landscape: vLLM vs. SGLang vs. TensorRT-LLM

Choosing the right inference engine is one of the most consequential decisions in building a model server. As Philip Kiely notes, while engines share core features like continuous batching and PagedAttention, they differ in ease of deployment, model support depth, and maximum performance ceiling.

Feature / Metric vLLM SGLang TensorRT-LLM
Primary Focus Broad model/hardware support & quick setup High MoE throughput & frontend flexibility Maximum performance on NVIDIA hardware
Ease of Use High (pip-installable, CLI flags) Medium (Python frontend + CLI flags) Advanced (Docker containers, config.yaml)
Hardware Support NVIDIA, AMD, Intel GPUs, Google TPUs NVIDIA, AMD GPUs NVIDIA GPUs only
Performance Ceiling Solid out-of-the-box throughput Excellent for MoE & multi-node clusters Industry-leading latency and TPS
Multimodal Support Broad (vLLM Omni) Modular (SGLang Diffusion) TensorRT / PyTorch direct integration
Governance PyTorch Project (Linux Foundation) LMSYS Org NVIDIA (Open Source)

Engine CLI Execution Snippets

Here is how each engine is launched across an 8-GPU node:

1. vLLM

vllm serve deepseek-ai/DeepSeek-V3.1 \ --enable-expert-parallel \ --tensor-parallel-size 8 \ --served-model-name ds31 

2. SGLang

python3 -m sglang.launch_server \ --model-path deepseek-ai/DeepSeek-V3-0324 \ --speculative-algorithm EAGLE \ --speculative-num-steps 1 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 2 \ --trust-remote-code \ --tp 8 

3. TensorRT-LLM

trtllm-serve deepseek-ai/DeepSeek-R1-0528 \ --host 0.0.0.0 \ --port 8000 \ --backend pytorch \ --max_batch_size 1024 \ --max_num_tokens 3200 \ --max_seq_len 2048 \ --kv_cache_free_gpu_memory_fraction 0.8 \ --tp_size 8 \ --ep_size 8 \ --trust-remote-code \ --extra_llm_api_options ${EXTRA_LLM_API_FILE} 

Production Deployment & Operational Strategy

Moving an optimized inference runtime into production requires enterprise-grade operational practices:

1. Pinned Dependency Containerization

Because inference stacks combine CUDA toolkits, PyTorch builds, and fast-moving engine libraries, small version shifts can cause catastrophic runtime failures. Docker containers must pin every dependency explicitly:

# Pin base engine image and explicit Python requirements FROM vllm/vllm-openai:v0.7.2 COPY requirements.txt . RUN pip install --no-cache-dir -r requirements.txt 

requirements.txt MUST use exact equality bindings (transformers==4.56.2) rather than loose specifiers (transformers>=4.40.0).

2. Cold Start & Engine Cache Optimization

Downloading 100B+ parameter weights from remote storage during pod scale-up introduces massive network latency. Production setups cache model weights inside the local data center network. For engines that compile custom binary artifacts (like TensorRT-LLM), compiled engine binaries should be pre-built and cached in images matching the exact GPU hardware and CUDA driver version.

3. Canary Deployment Strategy

Because GPU infrastructure carries significant hourly costs, running full side-by-side active-active clusters for blue/green deployments is often cost-prohibitive. Instead, production platforms utilize a 4-step canary strategy:

  1. Build and verify the new inference container version.
  2. Spin up a minimal canary worker pool to receive a tiny percentage (e.g., 2–5%) of live production traffic.
  3. Monitor error rates, TTFT, and TPS metrics against production baselines.
  4. Shift traffic iteratively while scaling down old replicas.

Conclusion & Next Steps

Inference engineering is rapidly maturing into a critical engineering discipline. By mastering the interactions between the Runtime, Infrastructure, and Tooling layers, teams can transition away from generic API limits, take control of their unit economics, and deliver high-scale AI products with custom performance profiles.

To deepen your hands-on experience:

  • Set up local benchmarking using genai-bench or NVIDIA GenAI-Perf.
  • Profile CUDA kernel execution using PyTorch Profiler or NVIDIA Nsight Systems.
  • Benchmark FP8 quantization quality preservation using NVIDIA ModelOpt.