If you cross-validate a forecasting model with KFold(shuffle=True), the score you get is usually too good to be true. Rows from next month end up in the training set, rows from last month end up in the test set, and the model gets to interpolate instead of forecast. scikit-learn's TimeSeriesSplit fixes the ordering problem, and its gap parameter fixes a quieter one: when your label is built from a future window (the next 5 days, the next 20 days), the last training rows already contain information from the test period.
This tutorial shows how sklearn TimeSeriesSplit and gap work on a daily series: what the folds look like, how test_size, gap and max_train_size change them, how to check that no label window crosses into the test fold, how to plug the splitter into cross_val_score and GridSearchCV, and what breaks when you have several series per date (a panel of stocks, products or stores).
Code runs on scikit-learn 1.9, pandas 3.1 and NumPy 2.5. The data is synthetic and seeded: an autocorrelated daily index, the kind of thing you get from news counts, web traffic or sensor readings. Every number below comes from running the code.
A daily series with a forward-looking label
The setup is a typical forecasting table: features computed from the past, and a label that is the average of the next H days.
import numpy as np import pandas as pd from sklearn.model_selection import KFold, TimeSeriesSplit, cross_val_score from sklearn.ensemble import RandomForestRegressor rng = np.random.default_rng(42) n = 600 dates = pd.bdate_range("2024-01-01", periods=n) x = np.zeros(n) for i in range(1, n): x[i] = 0.9 * x[i - 1] + rng.normal() s = pd.Series(x, index=dates, name="x") H = 5 # label horizon in days df = pd.DataFrame({ "x": s, "ma5": s.rolling(5).mean(), "ma20": s.rolling(20).mean(), "chg5": s - s.shift(5), }) # mean of the NEXT H values (t+1 .. t+H) df["y"] = s[::-1].rolling(H).mean()[::-1].shift(-1) df = df.dropna() X, y = df[["x", "ma5", "ma20", "chg5"]], df["y"] print(df.shape, df.index[0].date(), "->", df.index[-1].date()) print(df.round(3).head(3)) (576, 5) 2024-01-26 -> 2026-04-10 x ma5 ma20 chg5 y 2024-01-26 0.474 0.402 -0.518 -0.338 0.246 2024-01-29 0.377 0.237 -0.499 -0.822 0.176 2024-01-30 0.154 0.224 -0.506 -0.065 0.080 The reversed rolling mean is a compact way to build a forward window: y on a given day is the mean of x over the next five rows. The first 19 rows are dropped because ma20 needs history, and the last 5 because their future is not known yet.
Shuffled KFold vs TimeSeriesSplit: the score you should not trust
model = RandomForestRegressor(n_estimators=200, min_samples_leaf=5, random_state=0, n_jobs=-1) for name, cv in [ ("KFold(shuffle=True)", KFold(n_splits=5, shuffle=True, random_state=0)), ("TimeSeriesSplit", TimeSeriesSplit(n_splits=5, test_size=60)), ]: r2 = cross_val_score(model, X, y, cv=cv, scoring="r2") print(f"{name:20s} R2 per fold {r2.round(3)} mean {r2.mean():.3f}") KFold(shuffle=True) R2 per fold [0.525 0.482 0.524 0.474 0.487] mean 0.499 TimeSeriesSplit R2 per fold [ 0.25 0.487 0.098 0.521 -0.431] mean 0.185 Same model, same features, very different answer. With shuffling, each test row has training neighbours from the day before and the day after, and because the series is autocorrelated those neighbours almost give the answer away. The walk-forward score is lower and much more uneven across folds, which is what live forecasting actually looks like. If your CV score drops sharply when you switch to TimeSeriesSplit, the drop is information, not a bug.
What TimeSeriesSplit folds look like
By default TimeSeriesSplit(n_splits=5) cuts the data into n_splits + 1 blocks, trains on everything before a block and tests on the block. The training set grows each fold (an expanding window).
def show_folds(cv, index): for k, (tr, te) in enumerate(cv.split(np.arange(len(index)))): print(f"fold {k}: train {index[tr[0]].date()}..{index[tr[-1]].date()} " f"({len(tr):3d}) test {index[te[0]].date()}..{index[te[-1]].date()} " f"({len(te)})") show_folds(TimeSeriesSplit(n_splits=5), df.index) fold 0: train 2024-01-26..2024-06-07 ( 96) test 2024-06-10..2024-10-21 (96) fold 1: train 2024-01-26..2024-10-21 (192) test 2024-10-22..2025-03-04 (96) fold 2: train 2024-01-26..2025-03-04 (288) test 2025-03-05..2025-07-16 (96) fold 3: train 2024-01-26..2025-07-16 (384) test 2025-07-17..2025-11-27 (96) fold 4: train 2024-01-26..2025-11-27 (480) test 2025-11-28..2026-04-10 (96) Three parameters change this picture:
-
test_sizefixes the length of each test block instead of letting it ben_samples // (n_splits + 1). Fixed blocks (for example 60 business days, about a quarter) make the folds comparable. -
gapdrops that many rows between the end of training and the start of the test block. -
max_train_sizecaps the training window, turning the expanding window into a rolling one.
show_folds(TimeSeriesSplit(n_splits=4, test_size=60, gap=5, max_train_size=250), df.index) fold 0: train 2024-05-20..2025-05-02 (250) test 2025-05-12..2025-08-01 (60) fold 1: train 2024-08-12..2025-07-25 (250) test 2025-08-04..2025-10-24 (60) fold 2: train 2024-11-04..2025-10-17 (250) test 2025-10-27..2026-01-16 (60) fold 3: train 2025-01-27..2026-01-09 (250) test 2026-01-19..2026-04-10 (60) Note what stays fixed: the test blocks are always the last n_splits * test_size rows, so changing gap or max_train_size changes only the training side. That makes it easy to compare settings on exactly the same test days.
Why gap matters when the label looks forward
With H = 5, the label of a training row on day t is built from days t+1 to t+5. If training ends the day before the test block starts, the last few training labels are averages of test-period values. The model is being fitted on numbers it will later be graded on. A small helper makes the overlap visible:
def label_overlap(cv, horizon): """Count training rows whose label window reaches into the test block.""" out = [] for tr, te in cv.split(np.arange(len(df))): last_needed = tr + horizon # last row each training label uses out.append(int((last_needed >= te[0]).sum())) return out for g in (0, 2, 5): cv = TimeSeriesSplit(n_splits=5, test_size=60, gap=g) print(f"gap={g}: leaking training rows per fold = {label_overlap(cv, H)}") gap=0: leaking training rows per fold = [5, 5, 5, 5, 5] gap=2: leaking training rows per fold = [3, 3, 3, 3, 3] gap=5: leaking training rows per fold = [0, 0, 0, 0, 0] The rule is simple: for a label that uses the next H rows, set gap >= H. With gap=5 no training label touches the test block. (If your label is the value exactly H steps ahead rather than an average, the same rule applies: row t still needs row t+H.)
How much does this change the score? Often less than people expect:
for g in (0, H): cv = TimeSeriesSplit(n_splits=5, test_size=60, gap=g) mae = -cross_val_score(model, X, y, cv=cv, scoring="neg_mean_absolute_error") print(f"gap={g}: MAE per fold {mae.round(3)} mean {mae.mean():.4f}") gap=0: MAE per fold [1.258 1.224 1.367 1.093 1.005] mean 1.1895 gap=5: MAE per fold [1.268 1.218 1.352 1.09 0.999] mean 1.1854 On this series the difference is in the third decimal. Only the first few test days of each block sit next to leaked labels, and a random forest on four smooth features cannot exploit them much. That is not a reason to skip the gap. The size of the leak grows with the horizon, with smaller test blocks, and with models that can memorise individual rows (nearest neighbours, deep trees, boosted models with many rounds). The gap costs you H training rows per fold and removes the question entirely. The big win in this example came from dropping the shuffle; the gap is what makes the remaining number honest.
Scaling and tuning inside the folds
Leakage also comes from preprocessing. If you fit a StandardScaler on the full table before cross-validation, the scaler has seen the test period's mean and variance. Put it in a Pipeline so it is refitted on each training fold, and pass the same splitter to GridSearchCV:
from sklearn.pipeline import make_pipeline from sklearn.preprocessing import StandardScaler from sklearn.linear_model import Ridge from sklearn.model_selection import GridSearchCV cv = TimeSeriesSplit(n_splits=5, test_size=60, gap=H) pipe = make_pipeline(StandardScaler(), Ridge()) grid = GridSearchCV(pipe, {"ridge__alpha": [0.1, 1, 10, 100, 1000]}, cv=cv, scoring="neg_mean_absolute_error") grid.fit(X, y) res = pd.DataFrame(grid.cv_results_)[["param_ridge__alpha", "mean_test_score", "std_test_score"]] res["mean_test_score"] = -res["mean_test_score"] print(res.rename(columns={"mean_test_score": "MAE", "std_test_score": "MAE_std"}).round(4).to_string(index=False)) print("best:", grid.best_params_) param_ridge__alpha MAE MAE_std 0.1 1.0772 0.0378 1.0 1.0771 0.0378 10.0 1.0797 0.0380 100.0 1.1073 0.0388 1000.0 1.3039 0.1297 best: {'ridge__alpha': 1} Notice that a plain ridge regression on the same four features beats the random forest from the earlier sections (MAE about 1.08 against 1.19): with honest folds, simple models often hold up well. Two details worth knowing. First, GridSearchCV refits the best model on all rows at the end (refit=True), which is what you want for the final model but means grid.best_score_ is the only out-of-sample number you have. Second, keep a final hold-out period that the grid search never sees; tuning on the same folds you report is a mild form of overfitting.
Panel data: several rows per date
TimeSeriesSplit splits rows, not dates. That is fine for one series, but with several series per day (one row per stock, product or store) a single date can land on both sides of a fold boundary.
tickers = ["AAA", "BBB", "CCC"] panel = pd.DataFrame( [(d, t, rng.normal()) for d in dates[:20] for t in tickers], columns=["date", "ticker", "f1"], ) cv = TimeSeriesSplit(n_splits=4, test_size=7) for k, (tr, te) in enumerate(cv.split(panel)): shared = set(panel.loc[tr, "date"]) & set(panel.loc[te, "date"]) print(f"fold {k}: test starts {panel.loc[te[0], 'date'].date()} {panel.loc[te[0], 'ticker']}" f" dates in both train and test: {[d.date().isoformat() for d in sorted(shared)]}") fold 0: test starts 2024-01-15 CCC dates in both train and test: ['2024-01-15'] fold 1: test starts 2024-01-18 AAA dates in both train and test: [] fold 2: test starts 2024-01-22 BBB dates in both train and test: ['2024-01-22'] fold 3: test starts 2024-01-24 CCC dates in both train and test: ['2024-01-24'] With 3 tickers and a test size of 7 rows, three of the four boundaries fall in the middle of a day; only fold 1 happens to start on a clean date. In practice this leaks same-day information: tickers move together on market-wide days, so training on AAA's Monday row and testing on BBB's Monday row is not a forecast. The fix is to split the unique dates and map the folds back to rows. A generator works anywhere cv= is accepted:
def date_time_series_split(dates_col, n_splits, test_days, gap_days=0): days = np.sort(dates_col.unique()) pos = dates_col.map({d: i for i, d in enumerate(days)}).to_numpy() tss = TimeSeriesSplit(n_splits=n_splits, test_size=test_days, gap=gap_days) for tr_days, te_days in tss.split(days): yield np.flatnonzero(np.isin(pos, tr_days)), np.flatnonzero(np.isin(pos, te_days)) for k, (tr, te) in enumerate(date_time_series_split(panel["date"], 3, 4, gap_days=1)): d_tr, d_te = panel.loc[tr, "date"], panel.loc[te, "date"] print(f"fold {k}: train rows {len(tr):2d} up to {d_tr.max().date()} " f"test rows {len(te):2d} {d_te.min().date()}..{d_te.max().date()} " f"overlap {len(set(d_tr) & set(d_te))}") fold 0: train rows 21 up to 2024-01-09 test rows 12 2024-01-11..2024-01-16 overlap 0 fold 1: train rows 33 up to 2024-01-15 test rows 12 2024-01-17..2024-01-22 overlap 0 fold 2: train rows 45 up to 2024-01-19 test rows 12 2024-01-23..2024-01-26 overlap 0 Now every fold holds whole days, gap_days is measured in days rather than rows, and the overlap is zero. Pass it as cv=list(date_time_series_split(...)) to cross_val_score or GridSearchCV; materialising the list lets the search reuse the same folds for every parameter setting.
A pre-flight check before you trust the score
Whatever splitter you use, a few assertions catch most mistakes:
def check_folds(folds, index, horizon): for k, (tr, te) in enumerate(folds): assert index[tr].max() < index[te].min(), f"fold {k}: train is not before test" assert tr.max() + horizon < te.min(), f"fold {k}: label window reaches test" return f"{len(folds)} folds OK" print(check_folds(list(TimeSeriesSplit(5, test_size=60, gap=H).split(X)), df.index, H)) try: check_folds(list(TimeSeriesSplit(5, test_size=60).split(X)), df.index, H) except AssertionError as e: print("AssertionError:", e) 5 folds OK AssertionError: fold 0: label window reaches test The check assumes rows are sorted by time, which TimeSeriesSplit also assumes and never verifies. Sort first (df.sort_index()), and sort panel data by date before building the folds.
FAQ
What does gap do in TimeSeriesSplit?
It removes gap rows between the end of each training set and the start of its test set. Use it when a label (or a feature) is computed from a window that would otherwise reach across the boundary.
How big should gap be?
At least the label horizon: for "mean of the next 5 days" or "value 5 days ahead", gap=5. If features use centred or forward windows, add those too, or better, fix the features.
Does gap shrink the test set?
No. Test blocks stay the same; the gap is taken from the end of the training data, so each fold trains on gap fewer rows.
Is TimeSeriesSplit the same as purged cross-validation?
Close, for forward-looking labels: a gap of H rows purges the overlapping training labels. Purged k-fold with an embargo goes further by also testing on blocks in the middle of the sample; TimeSeriesSplit only ever trains on the past.
Why is my TimeSeriesSplit score so much worse than KFold?
Usually because KFold was leaking. On autocorrelated data shuffled folds let the model interpolate between neighbouring days. The walk-forward number is the one to plan with.
Conclusion
For time-ordered data, KFold(shuffle=True) answers the wrong question. TimeSeriesSplit trains only on the past; test_size keeps the test blocks comparable; gap set to at least the label horizon stops forward-looking labels from leaking into the test period; and max_train_size gives you a rolling window when old data stops being relevant. Keep scaling inside a Pipeline, split on unique dates when there are several series per day, and run a one-line assertion on the folds before reading the score.
News-flow attention is a common example of a daily feature with exactly these properties: autocorrelated, published per stock and per day, and easy to misalign with a forward return window. Lava's stock heat monitoring page shows how that kind of per-stock and per-sector heat data is laid out for A-shares.
中文简介:本文用一条固定种子的合成自相关日度序列,讲解 scikit-learn 的 TimeSeriesSplit 与 gap 参数:先对比 KFold(shuffle=True) 与按时间切分的 R²,说明打乱切分在自相关数据上会严重高估;再展示 test_size、gap、max_train_size 对折的影响(测试块不变,只改训练端);用一个小函数统计“标签窗口跨进测试块”的训练行数,给出 gap ≥ 标签预测期 H 的规则,并如实说明本例中 gap 对得分影响很小但能消除泄漏;随后把 StandardScaler 放进 Pipeline 配合 GridSearchCV 调参;针对多只股票同一天多行的面板数据,说明按行切分会把同一天拆到训练和测试两边,给出按唯一日期切分的生成器;最后提供折检查断言。

