Designing the AI Request Pipeline: 8 Layers Between User Input and Your LLM

Most teams build their first AI feature in a weekend. A controller that takes user input, calls an LLM, and returns the response. Three files, maybe four. It works beautifully in the demo. Then production happens. The 11 PM alert is always the same: "AI feature returning garbage / not responding / costing $4,000 this week." And the root cause is always the same: the gap between what a demo LLM call needs and what a production AI system needs is enormous, and nobody mapped that gap before shipping. I've spent the last two years building AI features into enterprise Java applications. The systems that hold up in production all share the same structural pattern — what I call the AI Request Pipeline: eight distinct layers between user input and LLM response, each one solving a specific class of production failure. This post maps every layer, explains what breaks without it, and shows the Spring Boot implementation. [!NOTE] Code samples use Spring AI 1.x with Groq/OpenAI-compatible endpoints and PostgreSQL (pgvector) for the semantic cache. Architecture patterns apply regardless of your AI provider or language. The Pipeline at a Glance User Input │ ▼ ┌─────────────────────────────┐ │ 1. Input Guardrail Layer │ PII stripping, length limit, injection detection └──────────────┬──────────────┘ │ ▼ ┌─────────────────────────────┐ │ 2. Semantic Cache Layer │ Exact + fuzzy cache lookup before any LLM call └──────────────┬──────────────┘ │ (cache miss) ▼ ┌─────────────────────────────┐ │ 3. Model Router Layer │ Simple task → small model, complex → large model └──────────────┬──────────────┘ │ ▼ ┌─────────────────────────────┐ │ 4. Prompt Assembly Layer │ Context injection, system prompt, RAG retrieval └──────────────┬──────────────┘ │ ▼ ┌─────────────────────────────┐ │ 5. LLM Execution Layer │ Retry, circuit breaker, timeout, fallback └──────────────┬──────────────┘ │ ▼ ┌─────────────────────────────┐ │ 6. Output Validation Layer │ Schema check, safety check, hallucination guard └──────────────┬──────────────┘ │ ▼ ┌─────────────────────────────┐ │ 7. Response Cache Layer │ Store validated responses for future cache hits └──────────────┬──────────────┘ │ ▼ ┌─────────────────────────────┐ │ 8. Observability Layer │ Latency, tokens, cost, quality metrics └──────────────┬──────────────┘ │ ▼ LLM Response None of these layers are optional in production. Each one exists because something specific breaks without it. Let me walk through them. Layer 1: Input Guardrail Layer What breaks without it: PII in your LLM provider's logs. Prompt injection attacks. $10,000 in compute from one user sending 500KB documents. Legal exposure. The input guardrail layer runs before anything else — before the cache, before the router, before any LLM call. It's cheap and synchronous. @Component public class InputGuardrailService { private static final int MAX_INPUT_CHARS = 4_000; private static final Pattern PII_PATTERN = Pattern.compile( "\\b(\\d{3}-\\d{2}-\\d{4}|\\b[A-Z0-9._%+-]+@[A-Z0-9.-]+\\.[A-Z]{2,}\\b|\\b\\d{16}\\b)", Pattern.CASE_INSENSITIVE ); private static final List<String> INJECTION_MARKERS = List.of( "ignore previous instructions", "ignore all prior instructions", "disregard your system prompt", "you are now", "act as if you are" ); public GuardrailResult validate(String userInput) { if (userInput == null || userInput.isBlank()) { return GuardrailResult.reject("EMPTY_INPUT"); } if (userInput.length() > MAX_INPUT_CHARS) { return GuardrailResult.reject("INPUT_TOO_LONG") .withDetail("max", MAX_INPUT_CHARS, "actual", userInput.length()); } String normalized = userInput.toLowerCase(); for (String marker : INJECTION_MARKERS) { if (normalized.contains(marker)) { return GuardrailResult.reject("PROMPT_INJECTION_DETECTED"); } } // Strip PII before it touches logs or the LLM provider String sanitized = PII_PATTERN.matcher(userInput).replaceAll("[REDACTED]"); return GuardrailResult.allow(sanitized); } } A few design decisions worth explaining: Reject vs. strip. PII gets stripped (returned to the pipeline as a sanitized string) because stripping doesn't change intent. Injection markers get rejected outright because there's no safe version of "ignore your system prompt." Length limit. This is a business decision, not a technical one. For a chat feature, 4,000 characters is generous. For a document analysis feature, you might go higher but add a cost estimate shown to the user before they submit. What this layer does NOT do. It does not evaluate whether the input is on-topic (that's model routing), and it does not flag offensive content (that's a dedicated content moderation API if your use case needs it). One layer, one job. Layer 2: Semantic Cache Layer What breaks without it: Every LLM call costs money and adds latency. Users asking the same question — or very similar questions — each pay the full cost. A Q&A feature with 10,000 daily queries spends 10× what it needs to. The semantic cache checks whether a sufficiently similar question has been answered before. It returns the cached answer in milliseconds and costs nothing. @Service public class SemanticCacheService { private final EmbeddingModel embeddingModel; private final JdbcTemplate jdbc; private static final double SIMILARITY_THRESHOLD = 0.92; public Optional<CachedResponse> lookup(String input, String context) { float[] queryEmbedding = embeddingModel.embed(input); // pgvector cosine similarity query String sql = """ SELECT response, metadata FROM ai_response_cache WHERE context_key = ? AND 1 - (embedding <=> ?::vector) > ? AND expires_at > NOW() ORDER BY embedding <=> ?::vector LIMIT 1 """; return jdbc.query(sql, rs -> rs.next() ? Optional.of(new CachedResponse(rs.getString("response"), true)) : Optional.empty(), context, pgvector(queryEmbedding), SIMILARITY_THRESHOLD, pgvector(queryEmbedding) ); } public void store(String input, String context, String response, Duration ttl) { float[] embedding = embeddingModel.embed(input); jdbc.update(""" INSERT INTO ai_response_cache (input_text, embedding, context_key, response, expires_at) VALUES (?, ?::vector, ?, ?, NOW() + ? * INTERVAL '1 second') ON CONFLICT DO NOTHING """, input, pgvector(embedding), context, response, ttl.toSeconds() ); } } The similarity threshold is the most important tuning parameter. 0.92 is a good starting point for most Q&A workloads. Too low: you serve cached responses to questions that are actually different. Too high: you miss obvious cache hits. Tune it on a labeled sample: pick 50 query pairs, label them as "same intent" or "different intent," and find the threshold that separates them. TTL strategy by content type: Content type TTL FAQ answers 7 days Product information 24 hours Real-time data (prices, status) 0 — skip cache entirely Personalized responses 0 — skip cache entirely What the cache does NOT store. Don't cache user-specific, personalized, or time-sensitive responses. The context_key parameter lets you namespace the cache by feature (FAQ, support, onboarding) so different features don't cross-pollinate. Layer 3: Model Router Layer What breaks without it: You use GPT-4o / Claude Opus for everything. A simple yes/no classification question costs 50× what it should. Your monthly AI bill is 10× your estimate. The model router assigns each request to the cheapest model capable of handling it reliably. @Component public class ModelRouter { public record RoutingDecision(String modelId, String rationale) {} // Ordered from cheapest to most capable private static final String SMALL_MODEL = "llama-3.1-8b-instant"; private static final String MEDIUM_MODEL = "claude-haiku-4-5"; private static final String LARGE_MODEL = "claude-sonnet-5"; public RoutingDecision route(RoutingContext ctx) { // Classification tasks: low complexity, structured output if (ctx.taskType() == TaskType.CLASSIFICATION || ctx.taskType() == TaskType.EXTRACTION || ctx.taskType() == TaskType.SUMMARIZATION) { return new RoutingDecision(SMALL_MODEL, "structured low-complexity task"); } // Short responses with clear structure: medium model if (ctx.expectedResponseTokens() < 500 && ctx.taskType() != TaskType.CODE_GENERATION && ctx.taskType() != TaskType.REASONING) { return new RoutingDecision(MEDIUM_MODEL, "short-form task"); } // Code generation, multi-step reasoning, ambiguous intent: large model return new RoutingDecision(LARGE_MODEL, "complex task requires full capability"); } } The routing signal comes from your application, not the LLM. Your code knows what the user clicked, what feature they're in, what schema the output must match. Use that knowledge. Don't ask an LLM to classify the request — that's recursive and expensive. Measure routing quality. After routing a request to the small model, track user satisfaction signals (thumbs up/down, follow-up "can you try again?"). If the small model has low satisfaction on a task type, bump that task type to the medium model. The routing table should evolve based on production data. Layer 4: Prompt Assembly Layer What breaks without it: The LLM gets no context and hallucinates answers. Context is assembled inconsistently across different code paths. System prompts drift as engineers edit them without coordination. Prompt assembly is the layer where you compose the complete message that will go to the LLM: system prompt, retrieved context (if RAG), conversation history (if multi-turn), and the formatted user input. @Service public class PromptAssemblyService { private final RagRetrieverService ragRetriever; private final SystemPromptRegistry promptRegistry; private final ConversationHistoryService historyService; public Prompt assemble(AssemblyRequest req) { String systemPrompt = promptRegistry.get(req.featureId(), req.modelId()); // RAG: retrieve relevant context if this feature needs grounding String ragContext = ""; if (req.requiresGrounding()) { List<Document> docs = ragRetriever.retrieve(req.userInput(), req.featureId(), 5); ragContext = formatRagContext(docs); } // Conversation history (last N turns, budget-capped) List<Message> history = historyService.getRecentTurns( req.sessionId(), MAX_HISTORY_TURNS ); // Assemble: system → rag context → history → user input // IMPORTANT: most-important instructions go at START and END of context, // not the middle — attention is weakest in the middle of long contexts var messages = new ArrayList<Message>(); messages.add(new SystemMessage(buildSystemMessage(systemPrompt, ragContext))); messages.addAll(history); messages.add(new UserMessage(req.userInput())); return new Prompt(messages, buildOptions(req)); } private String buildSystemMessage(String systemPrompt, String ragContext) { if (ragContext.isBlank()) return systemPrompt; return systemPrompt + """ --- REFERENCE CONTEXT (use this to answer the user's question): %s --- Answer based only on the reference context above. If the answer is not in the context, say so. """.formatted(ragContext); } private ChatOptions buildOptions(AssemblyRequest req) { return ChatOptionsBuilder.builder() .model(req.modelId()) .maxTokens(req.maxResponseTokens()) .temperature(req.taskType() == TaskType.FACTUAL ? 0.1f : 0.7f) .build(); } } The key architectural decision here is SystemPromptRegistry. System prompts are not strings scattered across service classes. They are versioned, centrally managed, and model-aware (the same logical prompt may need different wording for different models). Store them in your database or a config file, not in Java string literals. This makes A/B testing and rollback of prompt changes possible without code deploys. Token budget management. The conversation history and RAG context are both dynamic. Before sending the final prompt, count the total tokens and trim if needed — history first (oldest turns), then RAG context (lowest-relevance chunks last). Never trim the system prompt or the user's current message. Layer 5: LLM Execution Layer What breaks without it: A single LLM provider outage takes down your entire feature. Transient 429s (rate limit) fail requests that would succeed on retry. Slow responses hold threads. This layer wraps the actual LLM call with retry logic, a circuit breaker, timeout enforcement, and fallback to an alternative model. @Service public class LlmExecutionService { private final ChatModel primaryModel; private final ChatModel fallbackModel; private final CircuitBreaker circuitBreaker; private final MeterRegistry metrics; public ChatResponse execute(Prompt prompt, String modelId, String requestId) { Timer.Sample timer = Timer.start(metrics); try { return circuitBreaker.executeSupplier(() -> executeWithRetry(prompt, modelId, requestId) ); } catch (CallNotPermittedException e) { // Circuit is open — use fallback model log.warn("[{}] Circuit open for {}, using fallback", requestId, modelId); metrics.counter("ai.circuit_breaker.fallback", "model", modelId).increment(); return fallbackModel.call(prompt); } finally { timer.stop(metrics.timer("ai.llm.latency", "model", modelId)); } } @Retryable( retryFor = { RateLimitException.class, ServiceUnavailableException.class }, maxAttempts = 3, backoff = @Backoff(delay = 1000, multiplier = 2, jitter = 500) ) private ChatResponse executeWithRetry(Prompt prompt, String modelId, String requestId) { log.debug("[{}] Calling model {}", requestId, modelId); return primaryModel.call(prompt); } } Circuit breaker configuration matters. For LLM calls: Failure threshold: 50% over a 10-second window (lower than typical services because LLM errors are expensive) Wait duration in OPEN state: 30 seconds (LLM providers recover faster than traditional databases) Half-open: allow 3 test calls before fully closing Timeout is non-negotiable. Set an explicit timeout on every LLM call. A hung LLM call with no timeout holds a thread (or blocks a virtual thread from making useful progress on other work). 30 seconds for streaming, 60 seconds for large completions. Never infinity. The fallback model is your safety net, not your primary. The fallback doesn't have to match quality with the primary. It just has to return something coherent. A degraded response is better than an error. Layer 6: Output Validation Layer What breaks without it: Schema mismatches crash downstream code. Hallucinated entity IDs cause 500s when your code tries to look them up in a database. Unsafe content reaches users. @Service public class OutputValidationService { private final ObjectMapper objectMapper; private final GroundingValidator groundingValidator; public ValidationResult validate(String rawOutput, ValidationSpec spec) { // 1. Schema validation (for structured outputs) if (spec.requiresJson()) { try { JsonNode parsed = objectMapper.readTree(rawOutput); List<String> violations = spec.jsonSchema().validate(parsed); if (!violations.isEmpty()) { return ValidationResult.fail(ValidationFailure.SCHEMA_MISMATCH, violations); } } catch (JsonProcessingException e) { return ValidationResult.fail(ValidationFailure.INVALID_JSON, List.of("Output is not valid JSON: " + rawOutput.substring(0, 100))); } } // 2. Business rule validation for (BusinessRule rule : spec.businessRules()) { RuleResult result = rule.evaluate(rawOutput); if (!result.passed()) { return ValidationResult.fail(ValidationFailure.BUSINESS_RULE, result.violations()); } } // 3. Grounding check (for RAG responses) if (spec.requiresGrounding() && spec.sourceDocuments() != null) { double groundingScore = groundingValidator.score(rawOutput, spec.sourceDocuments()); if (groundingScore < 0.7) { return ValidationResult.fail(ValidationFailure.HALLUCINATION_RISK, List.of("Response not sufficiently grounded in source documents")); } } return ValidationResult.pass(rawOutput); } } What to do when validation fails. You have three options, and the right one depends on the failure type: Retry with error context. For SCHEMA_MISMATCH: retry the LLM call with an additional user message: "Your response failed JSON schema validation with these errors: [errors]. Try again." Works well for schema failures, success rate ~85% on retry. Return a fallback response. For HALLUCINATION_RISK: return a canned response ("I don't have reliable information about this") rather than a potentially wrong one. This is correct for factual Q&A. Escalate to human. For safety violations: flag the conversation for review, return a neutral response to the user. Don't auto-retry. Grounding validation implementation. The groundingValidator compares the response against the source documents using embedding similarity. For each factual claim in the response, it checks whether a semantically similar statement exists in the source. This isn't perfect — it's a heuristic, not proof — but it catches the most egregious hallucinations. Layer 7: Response Cache Layer What breaks without it: You paid for a validated, high-quality response and immediately throw it away. The next identical query pays again. This layer stores validated responses. It runs after validation — you never cache a response that failed validation. @Service public class ResponseCacheService { private final SemanticCacheService semanticCache; private final Map<TaskType, Duration> TTL_BY_TYPE = Map.of( TaskType.FAQ, Duration.ofDays(7), TaskType.PRODUCT_INFO, Duration.ofHours(24), TaskType.SUMMARIZATION, Duration.ofHours(1), TaskType.CLASSIFICATION, Duration.ofHours(12), TaskType.PERSONALIZED, Duration.ZERO, // never cache TaskType.REAL_TIME, Duration.ZERO // never cache ); public void store(CacheStoreRequest req) { Duration ttl = TTL_BY_TYPE.getOrDefault(req.taskType(), Duration.ofHours(1)); if (ttl.isZero()) return; // explicitly not cached semanticCache.store( req.userInput(), req.contextKey(), req.validatedResponse(), ttl ); } } This layer should also handle cache invalidation. When your product information changes, you need to invalidate cached responses about that product. Design the context_key to include a version or category that you can invalidate in bulk: -- Invalidate all product info cache entries when catalog updates DELETE FROM ai_response_cache WHERE context_key LIKE 'product-info:%' AND expires_at > NOW(); Layer 8: Observability Layer What breaks without it: You can't answer the question "is our AI feature working?" You can't tell whether your model change improved quality. You can't debug why a user got a bad response. Observability for AI features needs four dimensions that don't exist in normal API observability: @Aspect @Component public class AiPipelineObservabilityAspect { private final MeterRegistry metrics; @Around("@annotation(TrackAiRequest)") public Object observe(ProceedingJoinPoint pjp) throws Throwable { String requestId = UUID.randomUUID().toString(); long start = System.currentTimeMillis(); AiRequestContext ctx = AiRequestContext.current(); try { Object result = pjp.proceed(); long latency = System.currentTimeMillis() - start; // 1. Latency (broken down by layer) metrics.timer("ai.pipeline.latency", "layer", ctx.currentLayer(), "model", ctx.modelId(), "cache_hit", String.valueOf(ctx.cacheHit()) ).record(latency, TimeUnit.MILLISECONDS); // 2. Token cost tracking if (result instanceof ChatResponse response) { int inputTokens = response.getMetadata().getUsage().getPromptTokens(); int outputTokens = response.getMetadata().getUsage().getGenerationTokens(); metrics.counter("ai.tokens.input", "model", ctx.modelId(), "feature", ctx.featureId()) .increment(inputTokens); metrics.counter("ai.tokens.output", "model", ctx.modelId(), "feature", ctx.featureId()) .increment(outputTokens); } // 3. Cache hit rate metrics.counter("ai.cache", "result", ctx.cacheHit() ? "hit" : "miss", "feature", ctx.featureId() ).increment(); return result; } catch (Exception ex) { // 4. Failure classification metrics.counter("ai.pipeline.error", "layer", ctx.currentLayer(), "type", ex.getClass().getSimpleName(), "feature", ctx.featureId() ).increment(); throw ex; } } } The four metrics that matter most: Metric Alert threshold What it tells you ai.pipeline.latency P95 > 5 seconds User experience degradation ai.cache.hit_rate < 20% for FAQ features Caching not working or queries too varied ai.tokens.cost_per_request > expected × 2 Prompt bloat, wrong model routing ai.pipeline.error by layer > 1% Which layer is breaking Trace individual requests. Every request gets a requestId that flows through all 8 layers and appears in every log line. When a user reports "the AI gave me a wrong answer," you can pull that request ID and see exactly: what input it received, whether it was a cache hit, which model was called, what the raw output was, whether validation passed, and what was returned. Without this, debugging production AI issues is guesswork. Putting It All Together Here's the orchestrator that wires all 8 layers: @Service public class AiRequestPipeline { private final InputGuardrailService guardrails; private final SemanticCacheService semanticCache; private final ModelRouter modelRouter; private final PromptAssemblyService promptAssembly; private final LlmExecutionService llmExecution; private final OutputValidationService outputValidation; private final ResponseCacheService responseCache; public AiResponse process(AiRequest request) { String requestId = UUID.randomUUID().toString(); // Layer 1: Guardrails GuardrailResult guardrail = guardrails.validate(request.userInput()); if (guardrail.isRejected()) { return AiResponse.rejected(guardrail.reason()); } String sanitizedInput = guardrail.sanitizedInput(); // Layer 2: Semantic cache lookup Optional<CachedResponse> cached = semanticCache.lookup(sanitizedInput, request.featureId()); if (cached.isPresent()) { return AiResponse.fromCache(cached.get().response()); } // Layer 3: Model routing ModelRouter.RoutingDecision routing = modelRouter.route( new RoutingContext(request.featureId(), request.taskType(), request.expectedComplexity()) ); // Layer 4: Prompt assembly Prompt prompt = promptAssembly.assemble(AssemblyRequest.builder() .userInput(sanitizedInput) .featureId(request.featureId()) .modelId(routing.modelId()) .sessionId(request.sessionId()) .taskType(request.taskType()) .requiresGrounding(request.requiresGrounding()) .build() ); // Layer 5: LLM execution (retry + circuit breaker) ChatResponse llmResponse = llmExecution.execute(prompt, routing.modelId(), requestId); String rawOutput = llmResponse.getResult().getOutput().getText(); // Layer 6: Output validation ValidationResult validation = outputValidation.validate(rawOutput, request.validationSpec()); if (validation.failed()) { if (validation.isRetryable()) { // One retry with error context injected rawOutput = retryWithErrorContext(prompt, validation, routing.modelId(), requestId); } else { return AiResponse.validationFailed(validation.failures()); } } // Layer 7: Cache the validated response responseCache.store(new CacheStoreRequest( sanitizedInput, request.featureId(), rawOutput, request.taskType() )); // Layer 8: Observability is handled by the @TrackAiRequest aspect return AiResponse.success(rawOutput, routing.modelId(), false); } } The pipeline reads linearly. Each layer either short-circuits (return early) or passes its output to the next layer. No layer knows about layers beyond its immediate next one. This makes each layer independently testable and independently deployable — you can upgrade the model router without touching the output validator. What This Architecture Costs You Honesty about overhead: this pipeline adds latency and complexity compared to a raw LLM call. Latency overhead per request (when all layers run): Layer P50 overhead Input guardrail 1–2 ms Semantic cache lookup (miss) 15–25 ms (embedding + pgvector query) Model routing < 1 ms Prompt assembly + RAG 20–50 ms LLM execution 800–3,000 ms (dominates) Output validation 5–20 ms Response cache write 10–15 ms (async — doesn't block response) Observability < 1 ms (async metrics) The LLM call dominates. Everything else is noise. The semantic cache layer adds 15–25 ms on a miss, but saves 800–3,000 ms on every hit. At 30% cache hit rate (achievable for most FAQ/support features), average latency drops significantly. Development overhead. This is real. You're building a pipeline instead of a function. The payoff comes at 3 AM when the LLM provider has a partial outage and your circuit breaker routes to the fallback model automatically while your on-call team sleeps. Which Layers to Skip in the Early Days If you're moving fast and not yet at scale, here's the priority order: Ship from day one: Layer 1 (guardrails), Layer 5 (retry + timeout), Layer 8 (basic latency/error metrics). These prevent the most damaging production incidents. Add in week 2–4: Layer 3 (model routing — this is the biggest cost lever), Layer 6 (output validation for structured outputs). Add at scale: Layer 2 + 7 (semantic cache — ROI is high only when you have enough traffic to get cache hits), Layer 4 (full prompt assembly with RAG — if your use case needs grounding). The architecture is meant to be grown into, not built all at once. The teams I've seen struggle with production AI aren't struggling because their models are wrong. They're struggling because they shipped a raw LLM call and are now retrofitting production concerns one emergency at a time. The pipeline structure gives you a place for each concern, a clear interface between layers, and the ability to improve one layer without touching the others. That's the architecture. What you build on top of it is up to you. All code samples are illustrative of the pattern — adapt types and error handling to your specific Spring AI version and AI provider. If you're building this pipeline and run into a specific layer that's giving you trouble, drop a question in the comments. Avaneesh Yadav is Engineering Manager at HashedIn by Deloitte. He writes about production AI architecture at buildingai.in.