DEV Community: Quantooox

Walk-forward testing with an embargo: keeping time in order

Quantooox — Mon, 02 Nov 2026 10:48:25 +0000

train_test_split(X, y, shuffle=True) is fine for photos of cats. For anything that happens in time, it quietly lets the model learn from the future: Tuesday's training row sits next to Monday's test row, and markets on neighbouring days are related.

Walk-forward

Keep time in order:

Train on everything up to a date.
Test on the period right after it.
Move forward and repeat.

Our schedule for the stock models:

Training window: the trailing 756 trading days (about three years).
Refit: every 63 trading days (about a quarter).
Label: each stock's result versus SPY from the next open to the open 20 trading days later.

The gap (embargo)

There's a second, subtler leak. The label looks 20 days ahead. So the last training rows have labels that overlap with the first test days. The model has, in effect, seen part of the test period's outcome.

The fix is an embargo: leave a gap of 20 trading days between the end of the training data and the start of the test period, so no training label reaches into the test window.

def walk_forward(dates, train=756, step=63, embargo=20):
    start = train + embargo
    while start < len(dates):
        tr = dates[start - embargo - train : start - embargo]
        te = dates[start : start + step]
        yield tr, te
        start += step

What it costs

Fewer training rows, more refits, and results that look worse than a shuffled split. Every one of those is a more honest picture of what would have happened in real time.

If a time-series result was produced with a random split, treat it as unverified.

From Quantooo Lab, the makers of quantooo.com, a free, open and educational quant lab: every formula, every test and every failure is public. See it here: https://quantooo.com/method

Educational, not investment advice.

Out of how many? Multiple testing and the deflated Sharpe ratio

Quantooox — Sun, 01 Nov 2026 11:25:18 +0000

Flip a fair coin 10 times. Getting 8 heads is unusual, about a 5% chance. Now get 100 people to do it. A few of them will hit 8 heads, and none of them has a special coin.

Backtesting works the same way. Each feature set, model, lookback window and threshold you try is another person flipping coins. The more you try, the better your best result gets, even if nothing works.

The question to ask

When someone shows you a great backtest, ask: out of how many?

If it's the best of 200 variations, the bar for "impressive" is much higher than if it was the only thing tested.

Correcting for it

The deflated Sharpe ratio (Bailey and López de Prado) asks: given how many trials were run, and how much their results varied, what Sharpe ratio would the best of them reach by luck alone? Then it measures how far the actual best result is above that bar, also accounting for skew and fat tails.

The idea in plain words:

Count every configuration you tried, including the ones you threw away.
Estimate how big the best result would be if none of them had skill.
Only call it a finding if the real best clears that bar with room to spare.

What we do

Every model and setting we try for the stock-ranking models is logged as a trial. The leaderboard shows the trial count next to the results, and the deflated Sharpe of the best one. Across our twelve model families, none cleared the bar.

A habit worth keeping

Keep a trial log, even a plain CSV with one row per run: date, config, result. It costs nothing and it's the only honest way to answer "out of how many?", including to yourself.

From Quantooo Lab, the makers of quantooo.com, a free, open and educational quant lab: every formula, every test and every failure is public. See it here: https://quantooo.com/models

Educational, not investment advice.

Random referees: two controls every ranking model should beat

Quantooox — Fri, 30 Oct 2026 11:19:23 +0000

A model that ranks stocks with an information coefficient (IC) of 0.02 sounds like something. Is it? The quickest way to find out is to compare it with things that can't have skill.

We run two kinds of random referee next to every result.

1. Shuffled rankings (is the result better than chance?)

Keep everything about the strategy the same (number of stocks, rebalancing dates, costs), but pick names at random. Do it many times. You get a distribution of results that pure luck produces with your exact setup.

Two flavours are useful:

Cross-sectional shuffle: each day, shuffle scores across stocks. Tests whether the ranking itself carries information.
Time-block shuffle: shuffle whole blocks of dates. Keeps the time structure, so it catches results that come from one lucky period.

If your model sits inside the cloud of random results, it hasn't shown skill, whatever the headline number.

2. Shuffled labels (does the pipeline leak?)

Train the full model, with the same features, the same walk-forward schedule, the same everything, on labels that have been randomly shuffled. There is nothing real to learn. The IC should come out around zero.

If it doesn't, something in your pipeline is leaking the answer: a feature computed with future data, an overlapping window, a join on the wrong date.

For our LightGBM model, the shuffled-label version scored about zero, which is what you want to see.

What we found

Our score and the twelve model families we tried didn't clearly beat the random referees. Seeing that side by side is more useful than any single number: it shows how big "nothing" looks in this data.

Try it

Before you trust any ranking model, ask two questions: how does it compare with random picks, and what does it score on shuffled labels? If nobody ran those, the result isn't finished.

From Quantooo Lab, the makers of quantooo.com, a free, open and educational quant lab: every formula, every test and every failure is public. See it here: https://quantooo.com/research

Educational, not investment advice.

The sealed holdout: why we looked at our test data exactly once

Quantooox — Thu, 29 Oct 2026 11:17:53 +0000

Here's a loop most of us have run:

Build a model.
Check it on the test set.
Not great. Tweak a feature.
Check again.

After twenty rounds the test score looks good. But you've been fitting the test set, one decision at a time. The number no longer estimates how the model does on new data.

Sealing it

For our stock score we did something stricter:

Development data (2020–2025): build, explore, change things freely.
Final year: sealed. Nobody looks at it.
Freeze: every rule written down and versioned (v1) before the seal is broken: features, weights, the universe, the measure, the significance test.
Open once. Report whatever comes out.

Ours came out negative (IC −0.0585, t −2.47). We published it, and we did not try a "v1.1" on the same year. A new version has to be announced first and tested on data that arrives after the announcement.

What counts as a peek

More than you'd think:

Plotting the holdout to "sanity-check" it.
Choosing a hyperparameter grid after hearing how a colleague's model did on it.
Re-running with a different random seed because the first one looked unlucky.
Dropping a stock from the holdout because "its data looked odd".

Each one leaks a little information from the test set into your choices.

A cheap version for your own projects

You don't need special tooling:

Put the holdout in a separate file, and don't load it in your notebooks.
Write the frozen config to a file and commit it (a hash in git is a timestamp).
Write the evaluation script before you run it, and run it once.
Record the result even if you hate it.

It's uncomfortable. It's also the only way your test number means what you think it means.

From Quantooo Lab, the makers of quantooo.com, a free, open and educational quant lab: every formula, every test and every failure is public. See it here: https://quantooo.com/method

Educational, not investment advice.

Look-ahead bias: the backtest bug that uses tomorrow's newspaper

Quantooox — Wed, 28 Oct 2026 11:17:01 +0000

Imagine testing a strategy for January 2024 using the unemployment rate for January 2024. Sounds fine, until you remember that number was published in early February. Your backtest just read tomorrow's newspaper.

That's look-ahead bias: using information that didn't exist yet at the moment of the decision.

Where it hides

Economic data. Most series are indexed by the period they describe, not the day they were released. The gap is often weeks.
Company reports. A quarter ends on 31 March; the filing may arrive in May. Joining on the quarter-end date leaks six weeks of the future.
Revisions. The GDP number you download today is not the one people saw at the time.
Index membership. Using today's list of index members for a past date (that one has its own name: survivorship bias).

The fix: store when you knew it

For every number we keep two dates: the period it describes, and the moment it became public. Then every join is an "as of" join on the publication time.

import pandas as pd

# facts: one row per value, with the time it was actually published
facts = facts.sort_values("published_at")
days = days.sort_values("decision_time")

# for each decision, take the latest value published *before* it
known = pd.merge_asof(days, facts, left_on="decision_time", right_on="published_at",
                      by="ticker", direction="backward")

For US company filings we use the SEC's acceptance timestamp. (Gotcha we hit: in the SEC submissions feed, that timestamp is Eastern time but marked as UTC, so it has to be corrected.) For economic series we use release calendars. Where a release date is unknown, we apply a conservative lag rule rather than the period date.

The cost

Doing this makes backtests look worse. Usually noticeably worse. That's the point: the "better" version was never achievable in real time.

If a backtest's author can't tell you when each input was published, assume it leaks.

From Quantooo Lab, the makers of quantooo.com, a free, open and educational quant lab: every formula, every test and every failure is public. See it here: https://quantooo.com/economy

Educational, not investment advice.

Our stock score ranked stocks backwards on the one test that mattered

Quantooox — Tue, 27 Oct 2026 11:16:51 +0000

We built a simple score for about 100 large US stocks. It combines five things: trend, stability, company fundamentals, the market backdrop and news tone. Each part is a percentile against the other stocks on the same day, and the five parts are averaged with equal weights.

The question we cared about: do stocks with a higher score do better over the next month than stocks with a lower one?

The test

We split the data in two:

Development, 2020–2025. This is where we built and checked the score.
A sealed final year. Locked away. We froze every rule first, then opened it once.

The measure is the information coefficient (IC): the rank correlation between today's scores and each stock's result over the following month, relative to the S&P 500. Zero means no skill.

The result

Period	IC	t-statistic
Development	+0.0085	0.46
Sealed year	−0.0585	−2.47

In development the score was indistinguishable from random. On the sealed year it was significantly negative: the stocks the score liked tended to lag.

What we did with it

We published it, and we changed how the site talks about the score. It now says plainly: the score describes a stock; it does not predict it. The bands on the site read "above / around / below average", never anything that sounds like a recommendation.

We also did not reopen the sealed year to try a better version. That's the whole point of sealing it: one look, one answer. A second look turns the test set into a training set.

Why a negative result is worth sharing

Most backtests you see were the best of many attempts, checked on data the author had already looked at. A result that survives that process tells you very little. A frozen rule tested once on unseen data, published whatever it says, tells you a lot, even when the answer is "no".

The full table, the controls and the code are public.

From Quantooo Lab, the makers of quantooo.com, a free, open and educational quant lab: every formula, every test and every failure is public. See it here: https://quantooo.com/research

Educational, not investment advice.

We tested 12 machine-learning models on the stock market. None of them worked, and we published everything.

Quantooox — Mon, 26 Oct 2026 11:15:33 +0000

Most finance content shows you what worked. We built a free site, quantooo.com, to do the opposite: test ideas properly, in public, and publish the result whether it is good or bad.

Here is what we found, and how we tested it.

The setup

The site gives about 100 large US stocks (and the Nifty 50 in India) a score from 0 to 100, built from five plain measures: trend, stability, company fundamentals from official filings, the market backdrop, and the tone of the news. Every step of every score is shown on the page, with its source and date.

The question was simple: do stocks with a higher score do better over the next month than stocks with a lower one?

How we tested it

Only the past. Every model was trained on earlier data and judged on later data it had never seen (walk-forward testing).
A sealed final year. The last 12 months were locked away. We froze the rules first, then opened that year once.
Random referees. Every result was compared with random rankings. If a coin flip does as well, the model isn't smart.
Counting every try. Try 100 ideas and one will look brilliant by luck, so every configuration we tried is counted and the results are corrected for it.

What we found

The score: in development (2020–2025) its link with next-month results was +0.0085, statistically indistinguishable from random (t = 0.46). On the sealed final year it was −0.0585 (t = −2.47): higher-scored stocks actually did worse.

Twelve model families went through the same test: simple rules, ridge regression, LightGBM, neural networks, kNN, LSTM, GRU, a regime-switching model and more. None showed a significant edge. As a check on the pipeline, a model trained on shuffled labels scored about zero, so the test isn't leaking information.

India, same method, with company results used only from the minute they were published: +0.009 (t = 0.21), no skill either.

The one thing that helped, a little

Predicting direction failed. Predicting turbulence did a bit better. A volatility model for the S&P 500 had lower forecast error than the standard HAR model on 2025–26 (QLIKE 0.816 vs 1.047, p = 0.047). In an exploratory follow-up, the decades-old GARCH model did best of all.

That matches the research paper the site is built on, Where Machine Learning Helps: machine learning is far more useful for measuring risk than for picking winners.

Why publish a failure?

Because a result you only hear about when it works tells you nothing. Every night the scores are saved before the market opens, fingerprinted, and timestamped on the Bitcoin blockchain, so nobody, including us, can quietly edit them later. Mistakes go into a public corrections log.

We are also running hypothetical paper portfolios for one year from 19 October 2026, with rules registered before the first trade. We'll report the result either way. (Hypothetical, not a claim of returns; they could easily lose.)

Check our work

The leaderboard of all 12 models: https://quantooo.com/models
The backtest, controls and the sealed year: https://quantooo.com/research
Every formula: https://quantooo.com/method
The track record nobody can edit: https://quantooo.com/track-record

It's free, with no ads, no fees and no broker links. If you see a flaw in the method, we'd genuinely like to hear it in the comments.

Written by Quantooo Lab, the makers of quantooo.com. Educational, not investment advice.