Backend
Measuring What Matters in the Age of AI: The Research Software Metrics Landscape
arXiv cs.SE0 views
arXiv:2611.23253v1 Announce Type: new Abstract: Research software now serves as a primary instrument of science, but the methods the community uses to judge its quality, sustainability, reuse, and impact remain scattered across software engineering, open science, FAIR software, citation, and research-assessment literature. AI-assisted development sharpens the problem: code can be produced at rates that outpace human review capacity, provenance and attribution are harder to track, scholarly credit systems cannot keep pace, and existing metrics inherit assumptions about how research software gets developed that may no longer apply. To aid in navigating these challenges, we present a focused landscape synthesis of research software metrics based on 49 substantive references, organizing the recurring metrics into eight families described by data source, claimed construct, evidence of validity, automation potential, and susceptibility to misuse. Many widely cited research software metrics serve as partial signals but make weak standalone measures of scientific impact. The AI era opens or widens gaps around provenance, maintainability of generated code, reviewer effort, model and data dependencies, reuse and community-health signal inflation, and credit for human labor that AI does not perform. We argue for responsible, contextual, mixed-method use of research software metrics, and offer decision guidance for Research Software Engineers (RSEs), principal investigators, institutions, and funders selecting metrics that support better software, fairer credit, and more trustworthy science. All findings rest on the 49-reference focused review; a reproducible search-and-triage protocol and a 727-record triaged corpus accompany the paper as a transparency artifact for a future full systematic mapping study.
Source:https://arxiv.org/abs/2611.23253PreviousFrom Data Quality to Data Trustworthiness in AI-Driven Software-Intensive Embedded Systems: A Systematic ReviewNextGenerative AI in Software Testing: Practitioners' Perspectives on the Benefits, Challenges, Opportunities, and Risks
Related posts
Rent, License or Build a Multi-Asset Trading Terminal: What the Company Owns and When to Switch
0dev.to
The AI Verification Gap: Review AI Code Without Rubber-Stamping It
0dev.to
Why Your Next.js App Overflows PostgreSQL Connections (And How to Fix It)
0dev.to
A Practical Incident Runbook for AI Workflows
0dev.to
Local content zero-rating: keeping CDN traffic outside the data quota
0dev.to
Which RAGAS metrics can you run without ground truth?
0dev.to