AI assistance was used to draft and edit this article. The example uses synthetic closing prices.
A parameter search can keep its holdout dates sealed and still let tomorrow's price into today's signal. The signal function may have already read the whole array. A small test makes that failure visible. Pick a cutoff, run the signal on prices available at that cutoff, and compare the output with the same dates in a full-data run.
For example, imagine a rolling mean that was meant to use the two previous closes. An indexing slip puts the next close in the window. With the full array in memory the code still runs, and the finished chart can look entirely plausible.
Test both signal functions on the same data
def past_only(closes): return [ None if i < 2 else close > (closes[i - 2] + closes[i - 1]) / 2 for i, close in enumerate(closes) ] def one_bar_ahead(closes): return [ None if i < 2 or i + 1 == len(closes) else close > (closes[i - 1] + closes[i + 1]) / 2 for i, close in enumerate(closes) ] def first_changed_signal(signal, closes): full = signal(closes) for end in range(1, len(closes) + 1): prefix = signal(closes[:end]) if prefix != full[:end]: index = next(i for i in range(end) if prefix[i] != full[i]) return end, index, prefix[index], full[index] return None closes = [100, 101, 97, 104, 105] assert first_changed_signal(past_only, closes) is None assert first_changed_signal(one_bar_ahead, closes) == (3, 2, None, False) The example returns one value for each input bar, including placeholders during warm-up. The first failure is at a cutoff with three closes. At index 2, the prefix run lacks the next close, so the buggy signal returns None. The full run returns False for that same bar because it already sees 104. In a real engine, I would report the timestamp, strategy version and the first differing field alongside those two values. That gives you a regression test you can reduce to a handful of candles.
Run the check at each data boundary
- Warm-up. Compare missing values explicitly. A moving average becomes available only when its declared lookback is complete.
- Timeframes. If a four-hour signal reads a daily indicator, give it the last fully closed daily candle. Add a fixture with cutoffs just before and just after the daily close.
- Fills. Check signal generation and fill simulation separately. A signal may be known at the close while the first simulated entry occurs at the next open. If an open position remains at the cutoff, compare events through the cutoff and keep the closing-at-end assumption separate.
Use every cutoff on a small fixture. For a long history, sample timeframes, day boundaries and points where a signal changes. The full-history run and prefix runs must share the same fixed parameters for the comparison to mean anything.
Where this fits in a backtest
At AlyDoit we used this check while building Crypto AI. Our test report records candidate selection on 2022–2024 data and a separate evaluation from January 2025 through October 2026. The signal consistency check covers data availability. The held-out evaluation measures the selected rules on later data. The simulator enters at the next candle's open and deducts a modeled 0.25% round-trip cost. Those are three separate places where a reproducible result can go wrong.
If you add one cutoff fixture, make it a timeframe boundary where a higher-timeframe candle closes. That's where a join based on the wrong timestamp can silently give a strategy an extra bar of information.
