AI & ML
Monitoring Machine Learning Systems in Production: A Multivocal Literature Review
arXiv cs.SE0 views
arXiv:2611.23293v1 Announce Type: new Abstract: Monitoring is essential for maintaining reliable machine learning (ML) systems after deployment. However, knowledge about its practical implementation and effectiveness remains fragmented across academic and gray literature. We present a multivocal literature review of 94 sources (37 academic and 57 gray) covering both traditional ML and generative AI systems. We develop taxonomies covering monitored objects and properties, techniques, input signals, tools, automation, frequency, evaluation, and challenges. Our findings reveal an imbalanced monitoring process. Existing approaches provide broad and largely automated support for observing system behavior, detecting problems, and communicating findings, while monitor configuration, investigation, and response retain substantial human involvement. Monitoring is concentrated on inference, with no source covering the entire production ML system, limiting end-to-end diagnosis. Evidence of effectiveness is also limited: among 78 sources presenting an approach or tool, 45 provide no formal evaluation, 24 report offline or controlled evaluations, and only nine evaluate monitoring in live production. This gap is particularly evident for generative AI, where evidence is dominated by gray literature and vendor claims. We propose six directions for improving monitor design, cross-layer diagnosis, proxy validation, corrective responses, production-oriented evaluation, and organizational practices. Our review provides a structured basis for improving the detection, diagnosis, and resolution of production ML problems.
Source:https://arxiv.org/abs/2611.23293PreviousAn Empirical Study of Visual Artifacts in Pull Request DescriptionsNextFrom Ambiguous Tickets to Ranked Hypotheses: Multi-Agent Root-Cause Analysis for Autonomous Heavy-Vehicle Software
Related posts
Razorpay webhook not verified: anyone can fake a payment
0dev.to
From FAQPage Schema to Answerable AI Documentation
0dev.to
Wiring Android's MediaPipe LLM Inference API to a Multi-Turn Conversation Engine
0dev.to
What Is LLM Throughput? Tokens per Second, Concurrency, and Batching
0dev.to
Diese Woche haben mir zwei Tools etwas vorgeschlagen und es direkt abgeschickt.
0dev.to
AI Agent Observability: What to Log Before Production
0dev.to