Case Study · ML · Statistics
Walk-Forward, or How Not to Cheat at Time Series
The idea
My STAT 4188 final project, Stock_ML_Regressor_Models, was, on paper, a model bake-off. The proposal in the README lays out the plan: pull historical stock data, build multiple regression models (linear regression, random forest, XGBoost, SVR, an MLP, an LSTM), fine-tune each, score them on common metrics, and crown the optimal one. It even carried a hypothesis worth the name: that a model fitted to AAPL would generalize, well to other tech stocks and poorly to other sectors. The data side was straightforward: a short script against the Alpha Vantage API fetched twenty-five years of daily AAPL prices, November 1999 through November 2024, about 6,300 trading days.
The models mostly taught me what every such project teaches: gradient-boosted trees are strong, neural nets are fussy, and grid search takes all night. What stuck was a single clause in the proposal, “I will split my data using time series specific techniques (e.g. walk-forward validation)”, because it turned out to be the one methodological decision the whole project stood or fell on. This page is about that clause: why the ordinary way of validating a model quietly cheats on time series, and what validation that cannot cheat looks like instead.
The plate below is the argument in miniature. A fixed synthetic price series (a seeded random walk with drift and a little seasonality, so the figure is identical on every load) is split both ways, and the same deliberately simple predictor (the last training value before , plus the drift fitted on the training set) is scored under each scheme. Under walk-forward it must genuinely extrapolate, and errors compound across the test horizon. Under shuffled K-fold there is almost always a training point one step behind every test point, so the “forecast” collapses to a one-step hop, and the reported RMSE comes out several times smaller for the identical model. The ochre hairlines show why: each test point is tied back to training points from its own future.
With JavaScript enabled this plate splits a fixed synthetic price series (140 points, seeded random walk with drift) two ways and scores the same naive drift predictor under each: shuffled 5-fold cross-validation reports RMSE 1.02, while honest walk-forward validation reports 5.51; the shuffle flatters the model by 82%, because its folds train on the test set's future.
The approach
K-fold cross-validation is the default habit: shuffle the data, cut it into parts, hold out each part in turn. Its guarantee rests on an assumption so standard it’s easy to forget it’s there: that observations are i.i.d., or at least exchangeable, so that which points you hold out doesn’t matter. A price series violates this about as thoroughly as data can. Successive days are strongly autocorrelated, the level trends over decades, and, the part that actually bites, time has a direction. When you shuffle before splitting, every fold trains on points from its test set’s future. The task the model is graded on is no longer forecasting; it’s interpolation between neighbors it has already seen, which is dramatically easier and completely unavailable at deployment time, when the future is the one thing you don’t have.
Walk-forward validation is the repair, and it’s simple to state precisely. Order the data in time. For fold , train on an initial segment and test on the block immediately after it:
so the train window expands (or slides) and the test block marches forward. Every fold satisfies the one constraint that matters:
no training point postdates any test point. Shuffled K-fold violates this constraint almost surely, in every fold. And the constraint has to hold for the whole pipeline, not just the model fit: any statistic computed over the full series (a normalization constant, a feature scaling) smuggles the future in through the side door.
The repo honored the proposal on this point. The 80/20 holdout split is chronological; the test
set is the last five years, untouched; and hyperparameter search runs GridSearchCV over
scikit-learn’s TimeSeriesSplit, which is exactly the expanding walk-forward scheme formalized
above. Five model families were tuned and scored this way, ridge regression and the tree
ensembles among them.
What the split found, and what a second pass changes
What the project got right is the thing this page celebrates: the split. And the split earned its keep, because it delivered an ugly, informative result. On the chronological test set (2020 through 2024, when AAPL traded far above anything in the training years) the tree ensembles collapsed: test MSE of 2.52 for the random forest, 2.66 for XGBoost, 2.57 for AdaBoost, in standardized units where the full series has variance one. Worse than predicting a constant. That isn’t a bug in the trees; it’s what trees are, piecewise-constant functions that cannot output values beyond their training range. A shuffled split would have hidden this completely, since every test point would have had training neighbors at its own price level. Walk-forward surfaced the failure exactly as deployment would have.
A second pass holds the rest of the pipeline to the same constraint the folds already satisfy, in roughly ascending order of consequence:
- Predict returns, not prices. Prices are famously nonstationary; the interesting (and much harder) object is the return series. An RMSE on a trending price level is dominated by whether the model tracks the trend at all, and the do-nothing baseline, predict yesterday’s price, is nearly unbeatable. Scoring that baseline first gives every other metric a floor to stand on.
- Fit the scaler inside the fold. A
StandardScalerfit on the full twenty-five years before the split scales the training-era features with the test era’s mean and variance. It is a mild leak next to the fold-design one, and it violates the same constraint; the scaler belongs inside the cross-validation loop, fit on each training window alone. - Lag every feature. The target was the same-day close, and a feature such as
Change(close minus open, of that same day) or a 14-day moving average whose window includes the current close carries the answer with it. Ridge regression’s test MSE of 0.0068 is the tell: near-perfect on a five-year holdout is a symptom rather than a success, and the rule that prevents it is that every feature at time is computed from data available strictly before . - Run the hypothesis. The generalization-across-sectors question, the actual scientific content of the proposal, is the experiment the repository is set up for and the one worth running, with split adjustments pulled from a proper adjusted-close series rather than a hand-maintained table.
The pattern in that list is the lesson. Getting the fold structure right is necessary and nowhere near sufficient: leakage re-enters through scaling, through feature construction, through any choice informed by data the model shouldn’t have yet, so the constraint has to bind the whole pipeline rather than just the splitter. The plate above is the exam to give a time-series project on day one, and the reason my later from-scratch work (the from-scratch course) starts from baselines and validation before models.