Sybil Solutions has published inference-optimisation-skill, an agent skill for tuning local LLM serving across decode, speculative decoding, prefill, KV-cache capacity, startup and time to first token. The project describes the procedure as hardware- and quantization-agnostic, but its immediate value for developers is narrower and practical: it replaces ad hoc flag changes with controlled experiments under a memory cap and full-context stability checks. The project is available on GitHub.

A benchmark loop with rejection criteria

The prescribed loop begins by calculating theoretical ceilings, then changes one variable at a time and compares the candidate with interleaved control runs. The queue runs control, candidate and control serially; the project calls for at least six samples and medians, and accepts a candidate only when it beats both neighbouring controls without regressing another measurement by more than 3%. The repository documents the complete workflow.

Those controls address a measurement problem identified by the project itself: it estimates benchmark noise at ±5–10% and warns that many apparent single-run gains are drift. Before publication, a selected configuration must also pass long-context needle retrieval, a concurrent full-KV soak, acceptance gates and verification from a clean, pinned build. These requirements are set out in the README.

The skill can run through Claude Code or another agent that loads SKILL.md-style instructions. The repository also supplies PROMPT.md for direct use, along with references covering ceiling calculations, measurement traps, optimization levers, stack-specific problems and result publication. The repository describes both integration paths.

What the harness measures

The bundled bench.py targets OpenAI-compatible servers. It measures startup, short-prompt TTFT, single-user and concurrent decode, cold prefill at several lengths, long-context needle retrieval and peak memory with a cap watchdog. Token counts come from server usage statistics rather than streamed chunks, while failed requests, looping output and battery-powered runs are flagged; results are written to ledger.jsonl. The benchmark behavior is documented here.

A separate ceiling.py estimates limits from measured bandwidth and FLOPs plus model component sizes. queue.sh serializes detachable control and candidate runs, while mem.sh attempts to include driver residency on unified-memory GPUs that the README says cgroups miss. The repository lists these scripts and their roles.

Reported gains stay tied to their machines

The README presents results by hardware, model, quantization and engine rather than as a universal speedup. On one Intel Arc B390 configuration running Ornith-1.5-35B-A3B with EXL3 4 bpw, SGLang XPU and MTP, it reports median single-user decode rising from 43.6 to 53 tokens per second. The same run moved 256K-token prefill from 297 to 628 tokens per second and reduced startup from 113 to 43 seconds. The configuration and measurements appear in the results table.

For that B390 run, the project estimates a practical single-user decode target of about 88–97 tokens per second and says the measured result reached roughly 55% of it. It reports a 256K prefill result of 628 tokens per second against an estimated ceiling of approximately 670–684. The README explains its interpretation of those ceilings.

Analysis: The defensible interpretation is that this project contributes an acceptance standard, not a portable fastest configuration. Repeated controls, retrieval tests, soak runs and clean-build qualification make experiments slower, but they also make it harder to promote a drift-driven or short-context gain as a robust improvement. The unresolved choice remains workload priority: because decode, concurrency, prefill, TTFT, startup and KV capacity are measured separately, a team still has to decide which constraint should determine the winning configuration. That interpretation follows from the project’s separate metrics and qualification gates.