AI & ML
Evaluating LIM Trading Agents: A Critical Review of Backtesting Validity and Reporting Practices
arXiv cs.CR0 views
arXiv:2611.24133v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly used in investment research, and many studies report promising historical returns. However, such results are only credible if researchers can demonstrate that the model used only information available at the time, account for how strategies and configurations were selected, and explain how reported returns were generated. We identify six common threats to validity in LLM investment evaluations and apply them to 18 published studies. Across 108 assessments, we find substantial gaps in reporting. No paper fully addresses information timing, model-memory exposure, universe selection, trading costs, or configuration-selection bias. Only four studies provide minimum evidence that results hold across multiple market periods. Our analysis reveals that commonly used safeguards often address different problems than researchers assume. For example, using dated inputs does not guarantee that a pretrained model lacked knowledge of future events, and repeated experimental runs do not correct for strategy-selection bias. We therefore propose an evaluation protocol for future LLM investment research. Rather than judging whether LLMs can generate profits, our study examines whether existing evidence is sufficient to support such claims. We conclude that current reporting practices are generally insufficient for that purpose.
Source:https://arxiv.org/abs/2611.24133PreviousPIVOT: Distribution-Preserving Multi-BitWatermarking with Minimax-Optimal ScoringNextPULSAR-V: I/O-Aware Post-Update Attestation for RISC-V
Related posts
How to detect user frustration in chatbot conversations
0dev.to
Mình Đã Biến Gemini Thành Trợ Lý Cá Nhân Như Thế Nào (Lịch, Drive, Gmail và Những Lần Suýt Toang)
0dev.to
Im looking fr any free ai models / tools are there?, leaVE UR Lights💡
0dev.to
I checked 10 SEO agency websites for AI-search readiness. The real bugs were the boring ones.
0dev.to

How much GPU memory does an LLM need? The whole sum, worked out on real cards
0dev.to

Routing Upscaling Jobs by Output Size, Not Input Size
0dev.to