Most MCP tutorials stop at "it connects." But a model never sees your server code — it sees tool names, descriptions and JSON schemas. An MCP server is a user interface designed for a language model, and interfaces can be measured.

So I built one server twice, attacked both designs with the ten worst calls a model could plausibly make, and ran a model against each. The loose design rejected 0 of 10. The strict redesign rejected 10 of 10. Here is the full experiment — including the metric that pointed the wrong way.

Two designs, one notes store

The lab is a notes MCP server in pure Python stdlib, speaking JSON-RPC 2.0 over stdio, with initialize, tools/list and tools/call. The same notes store is exposed through two schema designs:

  • v1, deliberately loose: three vague tools, a free-form action string, no required fields, and bloated results.
  • v2, the strict redesign: one verb per tool, descriptions that say when not to use the tool, strict schemas, compact results, and structured errors.

The question was stated before any run: does schema redesign change how often models pick the wrong tool, produce malformed calls, or fail tasks? The pre-stated bar was that v2 counts as a win only if wrong-tool calls fall by at least 30% relative to v1 and task completion rises.

The adversarial audit

The deterministic audit sends 10 adversarial call classes — unknown ids, missing fields, wrong types, and invented actions. v1 rejected none of them and let bad calls change its data; v2 rejected all 10.

The model trials

Then a small local model (Qwen2.5-0.5B) ran 16 tasks against each design, with a chooser that sees only the tools/list output and a request, answers with one JSON tool call, and has that call executed for real against a fresh seeded store. Wrong-tool calls fell from 7 to 1, malformed calls from 4 to 1, and completed tasks rose from 6 to 13 of 16.

One honest failure is kept in the repo: a word-overlap "confusability" metric ranked the designs backwards (0.074 vs 0.289). Longer descriptions share more vocabulary; shared words are not shared meaning.

The point is not the notes app. It is that an MCP server is a UI for a model, and that interface can be measured — with an audit suite, a fixed task set, and a grading rubric — before you trust it in production.

Repo (server, both schemas, harness, audits — all runnable): https://github.com/ranadhir19/mcp-server-lab