A week after the customer service ticket extraction feature launched, the logs were full of JsonParseException. Sometimes the model wraps JSON in a Markdown code block; sometimes it prepends a line like "Sure, here are the results"; and the field name can even change from orderNo to order_no. Adding regexes, truncating strings, and tweaking prompts holds up for a couple of days before it breaks again. This article explains LangChain4j's structured output: how to push format constraints down to the protocol layer, and how to add business validation that the schema cannot handle. The code is a complete project; copy it and it runs.
1. What This Problem Actually Is: Why Does JSON Returned by LLMs Always Break
JSON parsing failures are not because the model is "disobedient"; they happen because you only ask it with prompts and do not constrain it at the protocol layer.
The five most common ways of asking a model to return JSON go wrong:
- Wrapped in a code block: it returns a json block wrapped in three backticks, and Jackson immediately reports "Unexpected character '`'".
- Prefixes and suffixes: "Sure, I've organized it for you as follows:" followed by JSON. You slice from
{to}, and it breaks again when curly braces appear inside string values. - Field drift: this time
orderNo, next timeorder_no, and the time after thatorder_number. - Type drift: the amount should be a number, but the model gives you
"399 元", and parsingBigDecimalblows up immediately. - Enums improvising: you define
REFUNDandLOGISTICS, and it returns"退款"and"refund_request".
Many people's first reaction is to change the prompt: add "must return pure JSON", add "do not explain", add few-shot examples. It works in the short term, then breaks again a couple of days later. The reason is no mystery:
- Prompts are soft constraints. When generating each token, the model can still choose any word.
- As context grows longer, earlier format requirements get diluted. Ticket conversations are often thousands of words long, so the model is more likely to "casually" add an explanation.
- Different models understand "JSON" differently. Some think code blocks count as JSON; some write numbers as strings.
- Even low temperature is not zero probability. As long as one token goes off track, the whole response is ruined.
Some people also think that setting temperature to 0 makes things stable. Temperature only affects the sampling distribution; it does not guarantee that the same input always produces the same output: server-side batching, model version canary releases, and changes in context length can all make results drift. What is more troublesome is "half-correct" output—four out of five fields are correct, parsing succeeds, and bad data is silently written to the database. A total failure at least triggers an alert; a half-correct result quietly pollutes reports.
Hand-written cleanup code is equally unreliable. Regexes can only cover the failure modes you have seen; substring(indexOf("{"), lastIndexOf("}")) truncates the JSON when } appears inside a string value; worse, swallowing parse exceptions and returning null pushes the null pointer to the business layer and doubles the debugging cost.
Cleanup code also has a hidden cost: it mixes "format problems" and "business problems" together. When parsing fails, you add a regex; when a field is wrong, you add if; six months later nobody dares to touch that code, because nobody knows which line is compensating for which model's which quirk. Structured output pulls this back into the type system: if something breaks, it is a compile error or a parse exception, and the boundary is clear.
Here the problem needs to be split into two layers:
LayerSpecific problemWho can solve itShapeIs it valid JSON, are the field names correct, are the types correct, are the enum values validJSON SchemaSemanticsIs the amount a refund amount or an order amount, is the classification correct, is the order number the one for this orderBusiness validation
The schema can guarantee that amount is a number, but it cannot guarantee that the number comes from "refund 399 yuan" rather than "original price 599 yuan". The schema can guarantee that category is one of the four enums, but it cannot guarantee that the model did not classify a logistics issue as a refund.
In one sentence: leave format to the schema, facts to validation, and failures to bounded retries.
2. How the Underlying Mechanism Actually Works: How JSON Schema Constrains LLM Output
Structured output compiles Java types into a JSON Schema, sends it to the model with the request, constrains the generation phase with the schema, and then deserializes the returned text back into an object.
The whole chain has three steps.
Step 1: Translate Java Classes into a Schema
LangChain4j uses reflection to read your return type and generates the schema field by field:
-
String→"type": "string" -
int/Integer→"type": "integer" -
BigDecimal/double→"type": "number" -
enum→"type": "string"plus theenumwhitelist -
List<T>→"type": "array"plusitems - Nested objects →
"type": "object"; recursive structures are closed off with$defsand$ref
The @Description on a field gets written into the schema's description. Don't underestimate this one line of text — it's mainly what the model relies on to decide "what should go in this field." The @JsonProperty(required = true) on a field goes into the required list.
This translation is recursive. List<TicketItem> produces an array, and the element objects get expanded in turn; self-referencing types don't recurse infinitely — once LangChain4j detects a cycle, it switches to $ref pointing at $defs. So you can put tree structures in your DTO and still get a valid schema, it's just that the deeper the nesting, the higher the odds the model fills something in wrong.
A schema generated from a ticket class looks roughly like this:
{ "type": "object", "properties": { "orderNo": { "type": "string", "description": "订单号，格式为 ORD- 加 8 位数字" }, "category": { "type": "string", "enum": ["REFUND", "LOGISTICS", "INVOICE", "OTHER"] }, "amount": { "type": "number", "description": "涉及金额，单位元；对话里没提金额就填 0" } }, "required": [] } Note that required here is empty. Only fields marked @JsonProperty(required = true) make it in. Before sending to OpenAI, strict mode does another rewrite pass: it stuffs all the keys into required and turns the types of fields not marked required into nullable forms, such as ["string", "null"] and ["number", "null"]. So "required" under strict mode only guarantees the key is present, not that the value is non-null. I'll dig into this trap later.
Step 2: Put the Schema into the Request
The request carries the schema via ResponseFormat:
ResponseFormat.builder() .type(ResponseFormatType.JSON) .jsonSchema(schema) .build(); The OpenAI adapter layer translates it into response_format: {"type": "json_schema", ...}.
If strict is enabled, the server enforces the constraint at the decoding stage: for every token generated, candidates that violate the schema get masked out outright. The model isn't "trying its best to get it right" — it "can't write it wrong." That's a whole different thing from asking it nicely in a prompt.
Capabilities vary widely across vendors:
- OpenAI's Structured Outputs supports a common subset of JSON Schema, with hard constraints in strict mode.
- Inference frameworks like vLLM use guided decoding and can also mask tokens according to the schema.
- Plenty of OpenAI-compatible services only support
json_object. It only guarantees "the whole thing is valid JSON" — field names, types, and enums are all ignored. - Some services just ignore
response_formatentirely — whatever you pass, they act like they never saw it.
json_object and json_schema often get conflated. The former only guarantees syntax; the latter governs field names and types as well. A concrete example: you ask it to extract {"amount": 399}, and json_object mode might return {"退款金额": "399元"} — perfectly valid syntax, and your DTO still can't handle it. Schema mode won't even let the key name amount be changed.
LangChain4j uses capability flags to decide which path to take. Only if the model's supportedCapabilities() includes RESPONSE_FORMAT_JSON_SCHEMA does it go with native schema; otherwise it appends the format instructions to the user message. The fallback mode isn't completely useless — it pastes the schema and field descriptions into the user message, so the structure the model sees is accurate, and most of the time it gets it right. The difference is between "most of the time" and "always."
There's a well-hidden trap here: OpenAiChatModel doesn't declare this capability by default. You have to explicitly declare .supportedCapabilities(RESPONSE_FORMAT_JSON_SCHEMA) for it to turn the capability flag on. Without that line, AiServices will still convert the result into an object, but under the hood it's "pasting a format requirement into the prompt," and stability drops off immediately. A lot of people say "I'm using AiServices, so why does it still fall over?" — and the answer is usually right here.
Step 3: Turn the Returned Text Back into an Object
After AiServices gets the response, it hands it to PojoOutputParser. The parser doesn't just readValue right away — it first tries the whole string; if that fails, it scans backwards for the last } or ], then matches braces going backwards, skipping over braces that sit inside strings. So code fences and surrounding chatter all get rescued.
This leniency is a safety net, not a design goal. If field names or types are wrong, it can't save you either — it just throws OutputParsingException.
There's another case it can't do anything about: the response gets truncated by max_tokens. The JSON is missing a closing brace, brace matching fails, and it throws an exception outright. You can catch this kind of failure early by checking finishReason — don't wait for Jackson to blow up. Setting max_tokens comfortably above your normal output size is a cheap and effective fix.
Three Things Schema Can't Cover
Semantics. Schema governs shape. Whose amount it is, whether the category is right, whether the order number belongs to this order — it doesn't concern itself with any of that.
Unknown values. The OpenAI adapter's strict mode puts all properties into required and also changes the types of fields not marked required into nullable form. The key must be present, and the value can be null. When the model encounters "the amount isn't mentioned in the conversation," it is more likely to give you null than to make up a number. So in the DTO, use wrapper types to receive it. If you use int, Jackson will silently turn null into 0, and validation sees fake data; if the business does not accept nulls, check it in validation, or add UNKNOWN to the enum and agree on 0 for the amount.
Cross-field rules. A constraint like "the refund amount cannot be greater than the order amount" can be written with conditional expressions in JSON Schema, but support varies widely among model vendors. Don't pin business rules on the schema; just write Java validation.
3. Hands-On: Writing LangChain4j Structured Output Code Step by Step
A minimal working project consists of four parts: dependencies, DTOs, the AI Service interface, and validation plus retry. The three examples below go from simple to complete, and each one is a full class.
Dependencies
<?xml version="1.0" encoding="UTF-8"?> <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd"> <modelVersion>4.0.0</modelVersion> <groupId>com.example</groupId> <artifactId>structured-output-demo</artifactId> <version>1.0.0</version> <properties> <maven.compiler.release>21</maven.compiler.release> <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding> </properties> <dependencies> <dependency> <groupId>dev.langchain4j</groupId> <artifactId>langchain4j</artifactId> <version>1.22.0</version> </dependency> <dependency> <groupId>dev.langchain4j</groupId> <artifactId>langchain4j-open-ai</artifactId> <version>1.22.0</version> </dependency> <dependency> <groupId>com.fasterxml.jackson.core</groupId> <artifactId>jackson-databind</artifactId> <version>2.22.4</version> </dependency> <dependency> <groupId>org.slf4j</groupId> <artifactId>slf4j-simple</artifactId> <version>2.0.20</version> </dependency> </dependencies> <build> <plugins> <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-compiler-plugin</artifactId> <version>3.16.0</version> </plugin> </plugins> </build> </project> Example 1: AiServices Automatically Generates the Schema
This is the least effort approach. Define an interface, set the return type to a DTO, and LangChain4j handles schema generation, request sending, and result parsing.
package com.example.structured; import dev.langchain4j.model.chat.ChatModel; import dev.langchain4j.model.openai.OpenAiChatModel; import dev.langchain4j.model.output.structured.Description; import dev.langchain4j.service.AiServices; import dev.langchain4j.service.UserMessage; import java.math.BigDecimal; import static dev.langchain4j.model.chat.Capability.RESPONSE_FORMAT_JSON_SCHEMA; public class TicketExtractionDemo { public enum Category { REFUND, LOGISTICS, INVOICE, OTHER } public static class Ticket { @Description("订单号，格式为 ORD- 加 8 位数字") public String orderNo; @Description("问题分类：REFUND 退款、LOGISTICS 物流、INVOICE 发票、OTHER 其他") public Category category; @Description("涉及金额，单位元；对话里没提金额就填 0") public BigDecimal amount; @Description("优先级，1 最低，5 最高；催得急或涉及投诉填 4 以上") public Integer priority; @Description("用户诉求摘要，不超过 30 个字") public String summary; } public interface TicketAssistant { @UserMessage(""" 从下面的客服对话中抽取工单信息。 只抽对话里明确出现的内容，不要推测。 对话： {{it}} """) Ticket extract(String dialogue); } public static void main(String[] args) { ChatModel model = OpenAiChatModel.builder() .apiKey(System.getenv("OPENAI_API_KEY")) .modelName("gpt-4.1-mini") // 这两行决定走原生 JSON Schema，而不是在 prompt 里贴格式说明 .supportedCapabilities(RESPONSE_FORMAT_JSON_SCHEMA) .strictJsonSchema(true) .temperature(0.0) .build(); System.out.println("模型能力: " + model.supportedCapabilities()); TicketAssistant assistant = AiServices.builder(TicketAssistant.class) .chatModel(model) .build(); Ticket ticket = assistant.extract(""" 用户：我上周买的耳机，订单 ORD-20261012，到现在还没发货。 客服：我帮您查一下物流。 用户：不用查了，直接退款，399 元，太慢了。 """); System.out.println("订单号: " + ticket.orderNo); System.out.println("分类: " + ticket.category); System.out.println("金额: " + ticket.amount); System.out.println("优先级: " + ticket.priority); System.out.println("摘要: " + ticket.summary); } } Let's break down the key behaviors:
-
{{it}}in@UserMessageis a placeholder corresponding to the method's only parameter. You don't need to manually concatenate strings. - The return type is
Ticket, notString. This single signature tells LangChain4j to generate a schema and deserialize the result. -
@Descriptionis not a comment; it goes into the schema. The more abstract the field name, the clearer the description must be. For example, ifprioritydoesn't say '1 to 5', the model will happily give you 10. -
temperature(0.0)reduces randomness, but don't expect it to replace the schema. -
System.out.println(model.supportedCapabilities())is for troubleshooting. If the output lacksRESPONSE_FORMAT_JSON_SCHEMA, you're in prompt fallback mode.
Example 2: Hand-Write the Schema Without Going Through the Interface Proxy
Some scenarios are not suitable for AiServices: the schema is assembled at runtime, you need to reuse a ChatModel, or you want to see exactly what request was sent. In these cases, construct ChatRequest directly.
package com.example.structured; import dev.langchain4j.data.message.SystemMessage; import dev.langchain4j.data.message.UserMessage; import dev.langchain4j.model.chat.ChatModel; import dev.langchain4j.model.chat.request.ChatRequest; import dev.langchain4j.model.chat.request.ResponseFormat; import dev.langchain4j.model.chat.request.ResponseFormatType; import dev.langchain4j.model.chat.request.json.JsonObjectSchema; import dev.langchain4j.model.chat.request.json.JsonSchema; import dev.langchain4j.model.chat.response.ChatResponse; import dev.langchain4j.model.openai.OpenAiChatModel; import java.util.List; import static dev.langchain4j.model.chat.Capability.RESPONSE_FORMAT_JSON_SCHEMA; public class RawSchemaDemo { public static void main(String[] args) { ChatModel model = OpenAiChatModel.builder() .apiKey(System.getenv("OPENAI_API_KEY")) .modelName("gpt-4.1-mini") .supportedCapabilities(RESPONSE_FORMAT_JSON_SCHEMA) .strictJsonSchema(true) .temperature(0.0) .build(); JsonSchema schema = JsonSchema.builder() .name("Ticket") .rootElement(JsonObjectSchema.builder() .addStringProperty("orderNo", "订单号，格式为 ORD- 加 8 位数字") .addEnumProperty("category", List.of("REFUND", "LOGISTICS", "INVOICE", "OTHER")) .addNumberProperty("amount", "涉及金额，单位元，没有就填 0") .addIntegerProperty("priority", "优先级，1 到 5") .addStringProperty("summary", "用户诉求摘要，不超过 30 个字") .required("orderNo", "category", "amount", "priority", "summary") .additionalProperties(false) .build()) .build(); ChatRequest request = ChatRequest.builder() .messages( SystemMessage.from("你是客服工单抽取器，只输出 JSON。"), UserMessage.from("订单 ORD-20261012 一直没发货，我要退款 399 元。")) .responseFormat(ResponseFormat.builder() .type(ResponseFormatType.JSON) .jsonSchema(schema) .build()) .build(); ChatResponse response = model.chat(request); System.out.println(response.aiMessage().text()); } } JsonObjectSchema.builder() provides a series of convenience methods; addStringProperty, addEnumProperty, and addNumberProperty all take two parameters: "name + description." required specifies which fields must be present, and additionalProperties(false) prevents the model from adding extras.
The advantage of hand-writing the schema is control: field names, enum values, descriptions, and required fields are all in your hands. The downside is that it easily drifts out of sync with the DTO. Change one side and forget the other, and parsing will still break. My advice: for scenarios the interface proxy can cover, prefer Example 1; only hand-write when the schema needs to be generated dynamically.
Example 3: Business Validation + Limited Retry
The first two examples solve shape; this one solves semantics. Core idea: after parsing succeeds, don't return immediately—run the rules first. If there's a problem, feed "which rule failed" back to the model and have it rewrite. Retry at most twice.
package com.example.structured; import com.fasterxml.jackson.databind.ObjectMapper; import dev.langchain4j.data.message.SystemMessage; import dev.langchain4j.data.message.UserMessage; import dev.langchain4j.model.chat.ChatModel; import dev.langchain4j.model.chat.request.ChatRequest; import dev.langchain4j.model.chat.request.ResponseFormat; import dev.langchain4j.model.chat.request.ResponseFormatType; import dev.langchain4j.model.chat.response.ChatResponse; import dev.langchain4j.model.openai.OpenAiChatModel; import dev.langchain4j.model.output.structured.Description; import dev.langchain4j.service.output.JsonSchemas; import java.math.BigDecimal; import java.util.ArrayList; import java.util.List; import java.util.regex.Pattern; import static dev.langchain4j.model.chat.Capability.RESPONSE_FORMAT_JSON_SCHEMA; public class ValidatedExtractor { private static final Pattern ORDER_NO = Pattern.compile("^ORD-\\d{8}$"); private static final ObjectMapper MAPPER = new ObjectMapper(); private static final int MAX_ATTEMPTS = 3; private final ChatModel model; public ValidatedExtractor(ChatModel model) { this.model = model; } public static class Ticket { @Description("订单号，格式为 ORD- 加 8 位数字") public String orderNo; @Description("问题分类：REFUND 退款、LOGISTICS 物流、INVOICE 发票、OTHER 其他") public String category; @Description("退款或涉及金额，单位元；没有就填 0") public BigDecimal amount; @Description("优先级，1 最低，5 最高") public Integer priority; @Description("用户诉求摘要，不超过 30 个字") public String summary; } public Ticket extract(String text) { ResponseFormat format = ResponseFormat.builder() .type(ResponseFormatType.JSON) .jsonSchema(JsonSchemas.jsonSchemaFrom(Ticket.class).orElseThrow()) .build(); List<String> problems = List.of(); for (int attempt = 1; attempt <= MAX_ATTEMPTS; attempt++) { String userText = text; if (!problems.isEmpty()) { userText = text + "\n\n上一次输出有以下问题，请逐条修正后重新输出：\n- " + String.join("\n- ", problems); } ChatRequest request = ChatRequest.builder() .messages( SystemMessage.from("只做信息抽取。不确定的字段按默认值填，不要编造。"), UserMessage.from(userText)) .responseFormat(format) .temperature(0.0) .build(); ChatResponse response = model.chat(request); String raw = response.aiMessage().text(); try { Ticket ticket = MAPPER.readValue(raw, Ticket.class); problems = validate(ticket); if (problems.isEmpty()) { return ticket; } System.out.println("第 " + attempt + " 次校验失败: " + problems); } catch (Exception e)Source：https://dev.to/nicholas1991/llm-json-parsing-always-fails-java-structured-output-in-practice-2pabRelated posts
Benchmarking Claude Haiku, Sonnet, and Opus on 64 Broken Data Pipelines
Why Your LLM Keeps Matching the Wrong Products (And Why a Better Prompt Won't Fix It)
Feature Theater: The AI Industry's Real Product Is Mediocrity at Scale
I Built a Tamper-Proof AI System. Then I Found the One Thing Anyone Could Change.
I Ran My AI Judge on a Decentralized GPU. The Entire Bill Was Four Cents.
