AI & ML
Skill Roulette: Measuring Skill Invocation Rate Across LLM Coding Agents
arXiv cs.SE0 views
arXiv:2611.23266v1 Announce Type: new Abstract: Agent skills package procedures for large language model (LLM) coding agents. An agent selects a skill from its name and description before reading the procedure. It may complete the task without invoking a relevant skill. We investigate how reliably coding agents invoke relevant skills and which factors affect that decision. We estimate invocation rates by repeating trials with identical inputs; invocation rate is the proportion of trials in which the agent invokes the relevant skill. We test 9 non-directional hypotheses about skill metadata, competing skills, contextual instructions, and task wording and composition. Eight concern invocation; the ninth tests whether making a skill available changes the occurrence of an action that the task does not request. We use tasks from 4 software engineering categories in a synthetic repository, running each of 4 coding-agent harnesses with 2 models. The planned 20,400 trials include 60 repetitions per Baseline category and configuration, 60 more per category in Claude Code with Sonnet 5.5 that admit each category's task and skill, and 10 per experimental condition. Comparisons are specified before collection and evaluated within each model and harness configuration using Boschloo's exact test.
Source:https://arxiv.org/abs/2611.23266PreviousEscaping the Anthropomorphic Trap: Resolving the Cognitive Mismatch in Future AI Software DevelopmentNextAn Industry Perspective on Software Vulnerability Management: A Systematic Review of Approaches, Tools, Practices, and Challenges
Related posts
Razorpay webhook not verified: anyone can fake a payment
0dev.to
From FAQPage Schema to Answerable AI Documentation
0dev.to
Wiring Android's MediaPipe LLM Inference API to a Multi-Turn Conversation Engine
0dev.to
What Is LLM Throughput? Tokens per Second, Concurrency, and Batching
0dev.to
Diese Woche haben mir zwei Tools etwas vorgeschlagen und es direkt abgeschickt.
0dev.to
AI Agent Observability: What to Log Before Production
0dev.to