AI & ML
HydroMind: A Checkable Agent for Autonomous Hydrological Research
arXiv cs.SE0 views
arXiv:2611.23323v1 Announce Type: new Abstract: General-purpose coding agents have written code for hydrological and Earth-system models. In these studies the researchers decided in advance how the model would be evaluated, for example the test period, the scoring function, and the standard the model had to meet. No study has tested what happens when the agent makes these decisions itself: how many of its claims are supported by its own data, code, and results, and whether readers can check them. Both may depend on the agent's harness, that is, the tools it can call and the written instructions it follows. We compared three harnesses: the agent's own harness; this harness plus the modeling tools of HydroMind, a hydrological modeling system we developed for general-purpose agents, which run every model through fixed commands and store its settings, trained model, and predictions; and both plus HydroMind's research procedure, under which the agent writes a research design before running any model and records every change to it. Under each harness, the same agent studied four open questions in hydrological model development and was given only the data. Under the three harnesses, the experiments submitted 23, 20, and 41 claims, each stating that one model or approach is better than, worse than, or similar to another. A claim does not hold if the difference recomputed from the experiment's own code and predictions disagrees with it, or if the code did not compare the two models or approaches the claim names. Under the three harnesses, 30%, 20%, and 5% of the claims did not hold. A statistical test, confidence interval, or decision rule that lets readers judge the stated difference was reported for 48%, 53%, and 83% of the claims. With the language model unchanged, changing only the harness reduced the share of claims that do not hold and increased the share that readers can check.
Source:https://arxiv.org/abs/2611.23323PreviousYUL: Bringing Software Supply Chain Security to Coding AgentsNextA level is not a rank: an empirical audit of quantile-to-index conversion in 10 released conformal-prediction libraries
Related posts
The Songless Archive Is a Better Listening Log Than a Playlist
0dev.to
AI makes it easier to understand the whole project
0dev.to
ChatGPT Has No Secret Codes. These Features Are What Help
0dev.to
How to Check a Freelancer's AI Work Before You Pay Them
0dev.to
I scanned 277 public MCP servers. Here's what 90% of them miss.
0dev.to

The JSON Parsed. Is It Right? How to Score LLM Data Extraction
0dev.to