From 6 to 12 November 2026, using dates as recorded in the logs, an AI agent (Claude Code) drove an open-source simulation app, NovaSolver Studio, through its MCP server. Structural, modal, buckling, thermal, flow, electrostatic and DC current problems. Every call and every reply was written to a log.
The headline numbers, all from those logs:
| MCP calls | 3,355 (49 log files) |
| Solves started | 187 |
| Accepted comparisons with a reference | 136 (from 118 analyses) |
| Median difference | 0.13% |
| Within 1% / within 3% | 105 / 130 |
| Above 5% | 2 |
The accepted set has 132 external-reference comparisons (formulas, series, correlations or independent 2D FEM) and 4 comparisons with another mesh of the same Studio model; 140 comparisons were recorded including reference-only results.
The interesting part is not the median. It is what got thrown away before those 136, and two types of setup mistake that passed a mesh or reference check.
What "the agent ran it" means here
It was not a chat where the model clicked around freely. Claude Code wrote Python scripts that act as an MCP client. Each script starts Studio's MCP server over stdio, calls the tools in a fixed order, and appends every call and reply to a .jsonl file. Claude Code also wrote the STEP geometry, the scripts that compute the reference values, and the scripts that compare the two.
The client is small. This is the core of mcpdrv.py, the shared driver:
async def call(self, name, **args): t0 = time.perf_counter() r = await self.s.call_tool(name, {"language": "ja", **args}) ... with io.open(self.log, "a", encoding="utf-8") as f: f.write(json.dumps({"tool": name, "args": args, "reply": text, "s": round(time.perf_counter() - t0, 2), "data": getattr(r, "structuredContent", None)}, ensure_ascii=False, default=str) + "\n") return text async def wait(self, text): m = re.search(r"Running as (\S+?)\.", text) if not m: return text while True: await asyncio.sleep(5) rep = await self.call("job_status", job_id=m.group(1)) if rep.startswith(("done", "failed", "stopped", "error")): return rep A typical case script then reads like a recipe: create_case or import_geometry, update_case with materials and boundary conditions, validate_case, check_convergence, get_results, verify_case, probe_results.
58% of the calls were waiting
job_status was called 1,949 times out of 3,355. Solves run as background jobs. run_simulation and check_convergence return a job id if the job is not done in about 20 seconds, and job_status waits up to 20 seconds per call before it answers.
So most of the traffic is the client asking "done yet?". The calls that carry engineering content are an almost flat band of about 170–200 each: set up, check, solve on three meshes, read results, compare.
The sum of all call durations is 42,086 seconds (about 11.7 hours). That is not wall time: it does not include the scripts' own time, and jobs that queued behind each other are counted in parallel.
From 206 cases to 171 results
Of 187 solves started:
| Outcome | Count |
|---|---|
| Out of memory (Studio's pre-check or during the solve) | 11 |
| Iterative solver did not converge | 3 |
| Tool error (the disk filled up) | 1 |
| Result not recorded | 1 |
| Results obtained | 171 |
| ...with a completed three-mesh test | 167 |
| ...mesh tests passed / failed | 145 / 22 |
| ...single-mesh pipe calculations | 4 |
Before any solve, validate_case returned RED 8 times. All eight were DC current cases where Joule heating was on by default and no face set a temperature. Four plate updates were rejected; the client continued, and one started job has no recorded completion. Eleven cases were only opened to read the geometry.
The accepted set excludes the 800 mm column, which failed the mesh test. Re-solves often covered a whole set after one case failed, to keep the mesh settings consistent.
Out of 23 topics, 13 needed at least one redo, design change or abandonment. One (a ruler vibrating as a cantilever) was dropped: the finest mesh of the convergence check needed 3.9–13.3 GB, with 3.7–6.1 GB free, in all four cases.
The mesh gate, in code
check_convergence solves on coarse, normal and fine meshes and computes Roache's Grid Convergence Index. This is the heart of it in Studio's source (run_worker.py):
e21, e32 = f2 - f1, f3 - f2 monotone = e21 * e32 > 0 and abs(e32) > 0 asymptotic = False if monotone: r = math.sqrt(r21 * r32) p = math.log(abs(e21 / e32)) / math.log(r) asymptotic = p >= 1.0 if not asymptotic: # error is not shrinking as the mesh refines: not in the asymptotic range p, fs = 1.0, 3.0 Studio uses observed order p ≥ 1 for its asymptotic flag; this is an implementation criterion, not a general definition of convergence. Passing requires all levels to converge, GCI at or below 5%, adjacent element counts to grow sufficiently, and either the asymptotic flag, a non-monotone sequence or a sufficiently small final change. Four chime rods with GCI around 1e-10% and two spinning disks at 0.0004% or less failed because adjacent element counts did not grow sufficiently (refined=False). The agent added face refinement and solved again; all passed.
Among the reference-only cases with a gate failure were: a sphere at Re = 1 (Cd went 28.78 → 28.96 → 29.54, GCI 7.6%) and a 0.5 mm electrical contact spot (GCI 4.6% but still drifting by about 0.8% per refinement).
How close, and where it missed
| |Studio − reference| / |reference| | Comparisons |
|---|---|
| under 0.1% | 58 |
| 0.1–1% | 47 |
| 1–3% | 25 |
| 3–5% | 4 |
| over 5% | 2 |
Reference assumptions and model differences are possible contributors to the largest gaps:
| Comparison | Difference | Possible contributors |
|---|---|---|
| Slit tube, L = 100 mm, twist vs Vlasov | +8.3% | The wall is about 20% of the radius; Vlasov assumes thin walls. 200 mm: +5.1%, 400 mm: +4.0% |
| Plate with a hole, d/W = 0.6, Kt vs Heywood | +3.9% | 2D plane-stress formula vs 3D von Mises. Against Peterson: +1.9% |
| Shelf with a mid divider, deflection vs beam formula | +3.6% | A 3D board compared with a beam formula |
| Sphere Re = 10, Cd vs standard drag curve | +3.1% | A fit to experiments. Studio's built-in Schiller–Naumann check: +5.79%, within its 8% |
| Glass pane 300×300 mm vs Navier series | +2.9% | Plate thickness explains only part (Mindlin: +2.5%); the 3D support condition may also contribute |
Where the reference is exact for the modelled problem, the numbers are tight. Six coaxial capacitances: within 0.016%. Six copper heat spreaders against the exact Bessel series: within 0.05%. Ten pin fins against the 1D fin formula: within 0.27%.
Two types of setup mistake the gate could not see
These are the most useful lines in the logs.
Plate with a hole. The agent attempted four updates combining the hole diameter and a mesh refinement on the hole; three cases have recorded results. On this template the refinement field is not editable, so the server rejected the whole change. The reply was a plain message (translated from Japanese): "This field cannot be changed: mesh.refine.hole." The script did not stop. The cases meant for 5, 10 and 15 mm were all solved at the default 10 mm.
They passed the template's Kt reference check; the saved replies do not record a GCI verdict. Kt came back as 2.5411 three times. The agent noticed only because the numbers were identical, then removed the refinement and re-ran. The fix in the script is now an assert:
r = await d.call("update_case", case_id=cid, expected_revision=1, changes={"geometry.params.hole": f"{D} mm"}) assert r.startswith("rev 2") or (D == 10 and r.startswith("rev 1")), r Shelf board. The plan was a board resting on two supports. The agent constrained a 10 mm strip on the underside at each end in z only. Preventing deflection across a finite-width strip also strongly restrained bending rotation, so the board behaved almost as if clamped: 0.346 mm maximum deflection, about a quarter of the simply supported beam formula's 1.467 mm. GCI was 0.99%, a pass. The agent caught it by comparing with the beam formula, then held the end faces in z only.
A mesh convergence check estimates discretisation uncertainty for its selected quantity. It cannot tell you that you set the wrong problem. Studio's validate_case says so in its own tool description, and returns a plain-language summary of the setup (which face carries what load, which way, what holds the part) for that reason.
The prose needed review too
The articles written from these runs were also drafted by Claude Code. A separate Claude agent fact-checked sets 4, 5 and 7; Codex (OpenAI) then reviewed the revised drafts read-only, and reviewed the other sets alone. The ledger for the old seven sets contains 135 review findings, including errors and optional improvements: 9 high, 61 medium and 65 low severity. Codex alone found 75, the Claude fact-checker alone 51, and both 9.
The worst was an extra eps*tau/sqrt(pi) term absent from Lee’s 1995 paper, shared by the heat-spreader draft and its reference script. The figure used the same error. At 20 mm it turned "0.47% high" into "4.3% high". The Claude fact-checker marked the formula as matching the paper; Codex found it.
Takeaways
- Over MCP, an agent can run a large, logged study: 187 solves, 136 accepted comparisons, median 0.13%, 130 under 3%
- Most tool calls are polling. A
job_statusthat waits on the server side (up to 20 s) keeps the client loop trivial - Of 167 completed mesh tests, 22 failed and 145 passed; setting checks blocked 8 unsolvable setups before they ran
- Two types of setup mistake passed a mesh or reference check. They were caught by comparing with a hand formula, and by noticing three identical numbers
- Treat a rejected
update_caseas an exception in your client, not as text
NovaSolver Studio is free and open source (GPL-3.0). It solves structural, thermal, flow and electrical problems in one app and can be driven from MCP clients.
- Overview and download: https://novasolver.jp/en/studio/
- Source code: https://github.com/novasolver/novasolver-studio
When your agent hands back a number from a tool, what does it have to show you before you believe it: the mesh study, a hand-calculation, the tool's setup summary, or something else?
