A kernel can be correct and fast while the process used to evaluate it still produces unreliable training rewards.
For kernel RL, that matters. If recorded speedups become rewards, the evaluation conditions become part of the data recipe. More evaluations per second only helps when the resulting feedback is useful.
At Nodus, we are designing an experiment around a simple question: does a kernel's speedup label survive an isolated recheck on the same GPU profile?
The proposed protocol fixes candidate/reference pairs before measurement. It evaluates each pair in isolation, then replays it under declared GPU sharing and contention settings. Every comparison keeps the task, judge, candidate, reference, software stack and hardware profile fixed. A full GPU and a MIG slice are different targets; changing between them is a separate experiment.
We plan to report:
- Agreement on whether a candidate is faster than, slower than, or tied with its reference, with ambiguous results visible.
- Shared evaluations labeled fast that do not remain fast in isolation.
- Records whose labels agree with isolated rechecks per dollar, including failures, retries and validation overhead. Unknown costs stay unknown.
This is a research design. We have no measured Nodus training-uplift or provider-comparison result to report here. Timing repetitions are not additional independent tasks, and agreement on speedup labels does not establish agreement on every reward magnitude.
Public work gives us a useful starting point. KernelBench evaluates kernel generation. FlashInfer-Bench connects kernel definitions, workloads, solutions and evaluation traces. KernelBench-Verified focuses attention on the correctness of the evaluator. These address related but distinct parts of the problem.
We have also shared an independent CPU reproduction and regression cases for a normalization-input issue reported by zzw4257 in SOL-ExecBench. The synthetic example shows how identity inputs can conceal incorrect affine formulas. GPU validation and maintainer acceptance remain open. Read the credited reproduction.
The eventual learning test is separate: train from the same checkpoint under a matched budget, hold out workload families before training, and compare correctness-qualified performance on those untouched workloads. Faster kernels and trustworthy evaluations are necessary evidence along the way; neither alone proves that the data teaches transferable skills.
If your lab trains agents that write GPU kernels, we would like to compare evaluation protocols and scope a Nodus environment or data pilot around the improvement that would make it worth using. Discuss a kernel RL evaluation.
