DEV Community: 王伟

I Costed Futures Like Crypto, and Half the Loss Was Made Up

王伟 — Sat, 31 Oct 2026 01:00:04 +0000

A cost model is not a detail. It can be larger than the effect you are trying to measure.

I ran a 1-minute futures strategy and it lost 94% of its capital. That looked like a decisive answer: the strategy does not work.

It was not decisive. Half of that loss was invented by my cost model.

What I assumed

I used 5 basis points per side, which is 10 bp round trip. That number came from crypto, where I had been working. It felt conservative: surely paying more cost than reality is the safe direction.

That intuition is wrong, and not mildly wrong.

What it actually costs

Instrument	measured round trip	assumption was
NQ (Nasdaq futures)	0.30 bp	33× too high
MGC (micro gold)	0.55 bp	18× too high
BTC perpetual	10.2 bp	about right

Note the last row. The number I used was correct — for the asset class I took it from. That is the trap: a cost assumption is a fact about a specific market, and it does not travel.

Futures on a liquid index are extraordinarily cheap to trade. Crypto perpetuals are not. They differ by more than an order of magnitude, and I applied the expensive one to the cheap market.

Why it mattered more than usual

Here is the part that makes this a systemic problem rather than a slip.

With 1-minute entries, the per-trade risk on this strategy was only about 6.2 bp. So the cost-to-risk ratio was:

10 bp cost ÷ 6.2 bp risk  ≈ 162%

I was charging more than one and a half times the risk of the trade as friction. The result was that every trade lost 3.16R on average, of which 2.16R was invented fees and only 1.0R was the strategy.

That is not a strategy being measured. That is a cost model being measured.

Correcting the number to 0.30 bp moved the same run from −94.0% to −45.5%.

The rule of thumb

Before rejecting a strategy on cost grounds, compute one ratio:

round-trip cost ÷ per-trade risk

If it exceeds about 5%, stop. Do not conclude the strategy is bad — conclude that you do not know your costs, and go measure them. A ratio above 100% (as mine was) means the cost term dominates the result entirely and the backtest is telling you about your assumptions, not about the market.

The corollary is just as important: a strategy whose edge is small relative to friction is not necessarily broken. It may be correctly measured and simply too small — which is a different conclusion with a different next step (find cheaper execution, trade less often, or move to a higher timeframe) and not "throw it away".

How to measure instead of guess

I did not have a perfect cost model afterwards either. What I had was something better than a guess:

Use the exchange's own fee schedule, not a number from another asset class.
Add the spread separately. It is usually larger than commission on liquid futures and it varies by time of day. Sample it, don't assume it.
Sanity-check against a fill-quality sample. Take 50 real fills, compare your simulated price to the actual one. The gap is your slippage estimate.
State the number and its source in the results. "Costs: 0.30 bp round trip, measured from NQ fill samples over the sample period" is a claim someone can audit. "5 bp" is a habit.

The uncomfortable asymmetry

An optimistic cost assumption makes a bad strategy look good — and you will find out in production, expensively.

A pessimistic cost assumption makes a good strategy look bad — and you will never find out, because you will have moved on. That failure is silent. It cost me a line of research, and I only noticed because a 94% loss looked too clean to be real.

So the discipline is symmetric: do not treat high costs as the safe direction. Both directions are errors, and the pessimistic one is the one nobody catches for you.

The cost-per-risk check is one of the static checks in a small open-source linter (GitHub, MIT, no dependencies). It flags a fee column that is missing or zero, and reports cost against per-trade risk so the ratio is visible before you draw conclusions.

If you would rather have this as a download than a repo: the same tool, plus the
demo trade logs (one with five bugs planted, one clean), is *$0** on Gumroad —
pay what you want, and $0 is a valid answer: https://weiwave23.gumroad.com/l/backtest-lie-detector*

One Missing Call Site Turned -5.7% Into +17.9%

王伟 — Fri, 30 Oct 2026 01:00:03 +0000

A one-line structural bug, worth 23.6 points of return and most of a 74% win rate.

I want to start with the number, because the number is the point:

−5.7% when exits are checked every bar. +17.9% when they are only checked on signal bars.

Same strategy, same data, same period, same costs. The difference is where one function was called from.

The bug

My backtester had a loop that looked roughly like this:

for i, bar in enumerate(bars):
    signal = strategy.check(bar)
    if signal:
        strategy.execute(bar)      # <- entry AND exit logic live in here

execute() contained both the entry and the exit check. Putting the exit check inside a function that only gets called on signal bars means the exit is only evaluated on bars where a signal fired.

On every other bar, the position sat there. Stops were not checked. Targets were not checked. Time-based exits were not checked. The strategy effectively had permission to hold through anything, as long as no new signal appeared.

The fix is not a change to the strategy at all:

for i, bar in enumerate(bars):
    signal = strategy.check(bar)
    if signal:
        strategy.execute(bar)
    strategy.manage(bar)           # <- exits, every bar, unconditionally

That is the entire change: move exit management out of the signal branch.

Why it flatters you

This one is directionally biased, and the bias is not subtle.

Losers get held longer than they should. A position that would have been stopped out keeps living until the next signal happens to fire. Sometimes it recovers. Sometimes it recovers because you were not checking — which is not a strategy, it is a delay.
The win rate inflates. My original run reported a 74.1% win rate. Fixing the call site moved it by 6 to 8 points. That is the difference between "this looks like a professional system" and "this looks like a coin flip with costs".
It shows up as fewer trades reaching their stop. If you log exits, you will see stop-loss exits that cluster in time — they fire in bursts on signal bars instead of being spread evenly. That is the fingerprint.

The mechanism is worth stating plainly: you were not simulating a strategy, you were simulating a strategy plus a free option to defer every adverse outcome. That option is worth a lot, and it is worth it in the direction you want.

How to find this in your own code

Three checks, in increasing order of effort:

1. Count exits by type over time. If stop-outs cluster on the same bars as entries, your exits are gated by the signal path. They should be roughly uniform in time, driven by price, not by signal timing.

2. Search for the phrase you are about to write. In almost every backtester, exits end up inside a function the entry path also calls. Grep for execute, handle_bar, on_bar, step — and ask, for each call site, "does this run on every bar or only sometimes?"

3. Print the bar index of every exit and every entry, then diff. If the sets of indices overlap far more than chance, exits are coupled to signals.

4. The structural fix beats the test. Make exit management a function that is called unconditionally, from the main loop, once per bar, and make it impossible to call it from the signal branch. If the API shape permits the bug, the bug will come back.

What it cost me

The strategy was negative before the fix. That is the uncomfortable part: this bug kept a losing strategy alive for months. If I had run it honestly I would have dropped it, and I would have been right to.

Bugs that make results worse get found. Bugs that make results better get shipped. This is the structural asymmetry of backtesting: your attention is drawn to things that look broken, and a false positive looks like progress.

Which is why the check has to be mechanical instead of attentional. I could not have caught this by staring harder at the strategy — the strategy was fine. I caught it by asking a question that had nothing to do with signals: where is this function called from?

The general shape

This belongs to a family I keep running into: the backtester's control flow is itself a modelling assumption.

You think you are modelling a strategy. You are actually modelling a strategy inside a simulation harness, and the harness decides when your code gets to see the world. Every asymmetry in that decision — which bars you get called on, whether you see a bar before or after it closes, whether the exit path is privileged — silently becomes part of the strategy.

So when an audit turns up nothing in the strategy, the next place to look is the harness. Mine was wrong in a way no amount of strategy review could have found.

The call-site check is one of the static checks in a small open-source linter (GitHub, MIT, no dependencies). It can flag exits that cluster with entries; it cannot read your control flow, which is why the structural fix matters more than the check.

There is also a *$0** download on Gumroad that bundles the tool with the demo
logs, if you want something to test against before pointing it at your own data —
name your price, and $0 is a valid answer: https://weiwave23.gumroad.com/l/backtest-lie-detector*

"Enough Trades" Is Not a Number, It Is a Ratio

王伟 — Thu, 29 Oct 2026 01:00:04 +0000

Two configs, same strategy, same data. One becomes provable in under a year. The other cannot be proven in a working lifetime.

"Am I overfitting?" gets all the attention. The more common problem is quieter and harder to fix: you do not have enough data to tell, and no amount of cleverness changes that.

I had two configurations of the same breakout strategy. Same code, same decade, different instrument and parameter set.

	trades	per-trade vol	t
Gold, k=0.75, 1:1	2,352	0.75%	2.93
BTC, k=0.50, 4R	1,506	2.16%	1.10

Both looked reasonable. Both had positive expectancy. One was nearly provable; the other was not going to be provable at all.

The formula

There is a closed form for the sample size you need to reach a given t-statistic, and it has no free parameters:

n = (3 × per-trade volatility ÷ per-trade edge)²

Read it as: the number of trades you need is the square of three divided by your signal-to-noise ratio. Not "how many trades did I run". Not "how many years of data". The ratio.

That is why "I have 1,500 trades, that's plenty" is not a statement about evidence. 1,500 trades with noisy returns is worth less than 300 with clean ones.

Applying it

From the table, the observed t and n give the ratio directly: edge/vol = t / sqrt(n).

	t	n	edge/vol	trades needed	still needed	at this rate
Gold	2.93	2,352	0.060	2,466	114	~0.9 yr
BTC	1.10	1,506	0.028	11,202	9,696	~37 yr

Gold needs about nine more months of trading. BTC needs thirty-seven years — and that assumes the edge and the volatility both stay exactly where they are, which is the optimistic case.

Both are the same strategy. The difference is that BTC's per-trade volatility is 2.9× gold's.

Why noise dominates silently

Look at the right panel of that figure: n grows with the square of the inverse ratio. Halve your signal-to-noise and you need four times the data. Drop it to a third and you need nine times.

BTC's ratio is less than half of gold's, so the requirement is roughly (0.060/0.028)² ≈ 4.5× more trades — and BTC also produces more trades per year, so the year count should improve. It does not, because the ratio entered as a square while the trade frequency is only linear. Frequency cannot outrun a squared penalty.

This is why a strategy with a great-looking annualised return can be the worst candidate for deployment. BTC had the highest annualised figure of anything I tested and the weakest evidence by a wide margin. The return was real in the sample. The evidence that it would repeat was not.

What I do differently now

1. Compute the required n before the strategy is finished. If the answer is thirty-seven years, I want to know that on day one, not after six months of work. The calculation takes one line and it has changed which projects I start.

2. Treat "needs a lifetime" as a rejection. Not "promising but unproven". If the required sample is longer than the instrument's plausible lifespan, the hypothesis is unfalsifiable in practice, and an unfalsifiable strategy is indistinguishable from a random one for as long as I would be trading it.

3. Prefer high signal-to-noise instruments for a first test. Same logic as the gold/BTC split: if you must prove something, prove it where the arithmetic permits it. Testing on the noisiest instrument first is the most expensive possible ordering — you spend the most time and learn the least.

4. Report n and the ratio, not just the t. A t-statistic alone hides whether the result is 300 clean trades or 3,000 noisy ones. The ratio makes the difference visible in one line.

The uncomfortable corollary

Some of the things I believed in earlier were not tested — they were untestable, and I mistook the absence of proof for the presence of evidence.

That is a worse error than a bug, because a bug can be fixed by editing a line. An unfalsifiable hypothesis can only be abandoned, and abandoning it costs whatever I already invested in it.

The formula is unforgiving in a useful way. It does not care how much I like the strategy, or how plausible the story is. It just says: with this much noise and this much edge, you need this much data. If you cannot get the data, you cannot get the answer.

The fold, sizing and selection helpers, plus this arithmetic, are in a small open-source kit (GitHub, MIT, no dependencies). If you have a config where the required n came out short and the strategy still failed, that is the case I want to hear about — it means the ratio was not the binding constraint.

After 455 Parameter Sets, a t-Stat of 2.18 Means Nothing

王伟 — Wed, 21 Oct 2026 16:44:56 +0000

The significance test you learned assumes you ran one test. You ran hundreds.

I had a strategy with a t-statistic of 2.18 on 400 trades.

Under the textbook reading, that is significant. The conventional threshold for 5% two-sided is about 1.96, and 2.18 clears it. I had spent weeks on the parameter grid before that number appeared, evaluating roughly 455 parameter sets along the way.

So I asked the question that killed it: if all 455 of those sets had zero edge, how big would the best of them look?

The test nobody runs

The standard t-test answers: "if the true mean were zero, how often would a sample like mine show a t of 2.18 or larger?" That question is conditional on one sample.

I did not take one sample. I took 455, threw away 454 of them, and kept the largest. The quantity I am actually looking at is not a t-statistic from one draw — it is the maximum of 455 t-statistics. The distribution of a maximum is nothing like the distribution of a single value.

The arithmetic rule of thumb people quote is: expected false positives = N × α. With 455 sets at 5%, that would predict about 22.8 sets "significant" by luck. That number is right about the count and useless about the threshold, because it does not tell you how big the biggest one gets.

For that, simulate.

The simulation

I generated 455 strategies, each with:

exactly zero edge (mean 0)
400 trades, same volatility as mine
the same t-statistic formula

then recorded the maximum t across the 455, and repeated that 2,000 times. Everything below is real output, not a sketch.

Best-of-455 t under the null:
  median   2.99
  p90      3.51
  p95      3.71
  p99      ~4.0

P(best-of-455 t >= your 2.18) = 0.999

The median best-of-455 t under pure zero edge is 2.99. My 2.18 did not clear the median. It was below what search produces on a normal day.

Why the bar moves so much

The maximum of N draws grows, but slowly — roughly like sqrt(2 log N) for well-behaved tails. Two consequences worth internalising:

The usual t = 3 bar is not conservative enough when N is in the hundreds. People treat t = 3 as "strict". Under a 455-wide search at my sample size, the median best is 2.99. You would need to clear something like the 95th percentile — 3.71 — to say anything at all.

The bar depends on your sample size. More trades per strategy tightens the single-test distribution but does not shrink the selection effect; with 400 trades the null max still sits near 3. The two effects do not cancel, and it is easy to add trades while also adding grid points, netting out to nothing.

The honest accounting

Here is what I do now, and it is mechanical:

Write down N before searching. Not after. If I planned 20 sets and ended up evaluating 455, the number I report is 455.
Compute the selection-adjusted p-value, not the naive one. Simulate the null (or use a formula) and ask what the best of N looks like at my sample size.
Report N alongside the t-statistic, every time. A t-statistic without N is not interpretable once you have searched.

The third one is the cheapest and it is the one nobody does. "t = 2.18" sounds like a result. "t = 2.18 after 455 evaluations, where the null median is 2.99" is a different sentence entirely, and both are one line.

What this does not mean

It does not mean the strategy is bad. It means the evidence I had was insufficient to distinguish it from noise produced by searching.

Those are different conclusions with different next actions:

Conclusion	Next action
"The strategy has no edge"	Drop it
"The evidence is insufficient"	Collect more data, or search less
"The edge is real but unproven"	Paper-trade with small size while accumulating

Mine was the second. The honest move was to stop adding filters (each filter is a new search, and it would need its own out-of-sample) and instead increase trades per strategy while holding the grid fixed.

The uncomfortable arithmetic

Every additional parameter set you evaluate buys you a slightly higher chance of a good-looking result, and charges you a slightly higher bar to clear. Early on, the first effect dominates: going from 1 to 10 sets is clearly worth it, you learn something. Later, the second dominates: going from 400 to 455 mostly just raises the bar.

There is a crossover, and it is usually much earlier than people think. Mine was somewhere in the low tens.

The number of things you tried is not a detail of your process. It is a parameter of your test, and leaving it out is how a search gets reported as a discovery.

The simulation is part of a small open-source kit (GitHub, MIT, no dependencies): backtest_kit.py selection --tried N runs the null for your own N and sample size. It reports a probability, not a verdict — the verdict is yours.

My Walk-Forward Said +29.2 Per Trade. Honest Out-of-Sample Said -4.6.

王伟 — Wed, 21 Oct 2026 16:44:29 +0000

A parameter set that won on folds 1-3 lost on folds 4-5. Same strategy, same data.

I ran the standard walk-forward check. Three parameter sets, five sequential folds, about 400 trades. For each fold I picked the set that was best on the previous folds only, then scored it on the fold it had never seen.

Here is the whole output:

fold   picked      oos net    oos t
2      k0.75        436.5     0.88
3      k0.75        827.9     1.56
4      k0.75       -941.2    -2.51
5      k1.00       -813.2    -1.96

Stitched out-of-sample: n=107  net=-489.99  mean=-4.58  t=-0.51
In-sample best mean   :  +29.17 per trade
Optimism gap          :   33.75 per trade

The parameter set k0.75 looked best for three consecutive folds. It was the worst set for the next two.

+29.17 per trade in sample. −4.58 out of sample. The gap is 33.75 per trade, which is larger than the entire in-sample edge — the sign flipped.

What "out-of-sample" actually means

Most people treat out-of-sample as a split. You cut the data at some date, fit on the left, evaluate on the right, and call the right side honest.

That is not what makes it honest. Out-of-sample is a procedure, not a file. What makes it honest is that no choice in the evaluation was informed by the data being evaluated.

Here is the distinction, concretely:

What you did	Is the score honest?
Optimised on 2015-2020, tested on 2021-2023, and you never looked at 2021-2023 while designing	yes
Optimised on 2015-2020, tested on 2021-2023, then went back and tweaked because 2021-2023 looked bad	no — that file is now in-sample
Picked the parameter that maximised the whole period, then reported the second half	no — the second half informed the pick
Picked on folds 1..i-1, scored on fold i, never revisited	yes

The third row is the one that catches people. If you choose a parameter using any statistic computed over the test period — total return, Sharpe, whether the equity curve "looked right" — the test period has leaked into the choice, and the score is in-sample no matter which file it lives in.

The gap is not a haircut

The natural instinct is to treat the optimism gap as a discount: knock 20% off the in-sample number and move on.

That does not work, because the gap is not proportional to the edge. It is roughly proportional to how much selection freedom you had. In this example:

in-sample best mean: +29.17
honest out-of-sample: −4.58

The gap (33.75) was bigger than the edge (29.17). There was no discount that turns this back into a positive number, because the honest number is negative.

When the gap exceeds the edge, what you measured was not a weak strategy. You measured the selection process.

Why this happens

Three mechanisms, all mundane:

1. Fold-to-fold variance is large. A fold with 20-25 trades has a standard error of roughly sd / sqrt(25). If per-trade sd is around 100, the standard error of the mean is about 20 per trade. So a "best mean" of 29 is roughly one to two standard errors above zero — entirely reachable by luck. With three candidate sets, picking the top one at each step reliably picks the noisiest winner.

2. Regime change is not announced. The demo I used has the edge decaying after ~60% of the sample on purpose, which is a clean version of what real markets do more subtly. A parameter that suits the first regime is not obliged to suit the second.

3. The selection is repeated, not once. You do not choose a parameter once. You choose at fold 2, again at fold 3, again at fold 4. Each choice is a fresh chance to pick the lucky one, and the errors compound across the stitched result.

How to run this yourself

You need one thing beyond a trade log: a column saying which parameter set produced each trade.

entry_time, exit_time, side, entry_price, exit_price, size, fee, param_set
2024-01-02T09:30:00,...,BUY,100.12,101.40,10,2.00,k0.75
2024-01-03T14:05:00,...,SELL,101.85,100.90,10,2.00,k1.00

With that column, the procedure is mechanical:

Split trades into K sequential folds by time.
For fold i (starting at i=2), compute each parameter set's mean on folds 1..i−1.
Pick the best one.
Score it on fold i only.
Stitch the fold-i results. That stitched series is your honest number.
Compare it to the best in-sample mean. The difference is the gap.

python backtest_kit.py folds trades.csv --k 5

Without the column, all you can do is split validation, which is weaker — it tells you about stability over time, not about selection. The tool says so rather than pretending otherwise.

The part that stings

I did not set out to fool myself. I had a parameter grid, I picked the winner on the development data, and I reported the number. That is the standard workflow, and it is exactly the workflow that produces this gap.

The gap is not a sign of carelessness. It is the expected output of selecting on noise. Which means the fix is not "be more careful" — it is to make the selection procedure mechanical and to report the stitched number, including when it is negative.

The uncomfortable question is not "was my gap large". It is: how many strategies do I still believe in because I never computed this number for them?

The fold walk and the gap calculation are in a small open-source kit (GitHub, MIT, no dependencies). If you have run this on your own grid and the gap came out smaller than the edge, I would like to see the setup — that is the interesting case.

The One-Line Fix That Took My t-Stat From 6.22 to 1.04

王伟 — Wed, 21 Oct 2026 16:42:36 +0000

A breakout strategy passed all six parameter cells. It was a fill assumption, not an edge.

I had a breakout strategy on 1-minute data come in at t = 6.22.

Six parameter cells, all six passing. That is the kind of number that makes you start thinking about position size and funding an account. It is also the kind of number that should make you suspicious, because in ten years of doing this I have almost never seen a real edge survive a clean audit with a t-stat that high.

It was not real. The strategy was fine. The fill assumption was wrong, and it was wrong in exactly the direction that flatters you.

Here is the whole story, because the same bug is probably sitting in a lot of backtesters, including yours.

The bug

My breakout entry looked like this:

if high[j] >= buy_level:
    entry = buy_level

buy_level came from the previous session's high. The logic reads fine: price trades up through my level, I get filled at my level. That is what a limit order does.

The problem is the gap. When the new session opens with a gap straight through buy_level, bar j satisfies high[j] >= buy_level on its very first tick — at the open. The open is already above my trigger, and I record a fill at buy_level, a price that never traded in that session.

I was gifting myself the gap. Systematically, on every gap-through day, in the direction that made the equity curve look better.

The X is the recorded fill. The dotted line is that bar's low. The fill sits 1.48 below the lowest price of the bar — a price that, by construction, no one could have transacted at. The backtester did not complain, because nothing in the code knows what a plausible fill looks like.

The fix

One line:

entry = max(buy_level, open[j])   # the worse of the two, for a long

If the bar opened above my trigger, I get the open — the price I could actually have paid. If it opened below, I get my trigger, as intended. Same for shorts with min(...).

That is the entire change. Not a filter, not a parameter, not a new feature.

Same trades, same period, same parameter grid. 6.22 → 1.04. Passing cells: 6 of 6 → 0 of 6.

The entire edge lived in the fill.

Why this is hard to catch

Three reasons, and they compound.

It is plausible. entry = buy_level reads like correct code. It is the natural way to express "I got filled at my level". Nothing about it looks like a mistake, so code review slides right over it.

It inflates results in the direction you want. There is no error, no crash, no unusual log line. The only symptom is a nicer equity curve, which is precisely the thing you were hoping for. Your brain is not going to flag good news as a bug.

It hides in the gap. If you trade a market that does not gap, you may never see it. It only appears on days where the open jumps past the trigger — and on those days it silently substitutes a favourable price for the one you would have got.

How often it actually bites

I measured the share of entries where the recorded fill fell outside the bar's [low, high]:

The pattern is exactly what you would expect: the more a market gaps overnight, the worse it gets. Index futures on a session-based breakout are the worst case — roughly one entry in six.

And note what this does to the interpretation of a backtest. On NQ, 17.4% of entries were priced at a level the market never traded. If your strategy's edge is on the order of a few basis points per trade, and one entry in six is being handed a free gap, then the measured edge is mostly the gap.

How I found it

Not by reading the code. I had read that code a dozen times.

I found it by looking at when the largest 1-minute moves happened. I ran a data-quality pass that printed the twelve biggest 1-minute jumps with their timestamps, and all twelve landed between 18:00 and 18:05 New York time.

That clustering was the tell. Twelve enormous 1-minute moves, all in a five-minute window, every time — those are not market moves. That is the session boundary. The "jumps" were cross-session gaps being recorded inside a session-labelled dataset.

Once I was looking at gaps specifically, the fill assumption fell apart in about ten minutes.

The general lesson here is worth more than the bug: when you audit a backtest, look at the time distribution of anomalies, not just their size. A 2% one-minute move at a random time is a market event. A 2% one-minute move that always happens at 18:00 is a data-structure artefact, and it is telling you that your data is not shaped the way your code assumes.

What this generalises to

Fill assumptions are where backtests lie most often, because they are the one part of the simulation that has no external check. Your signal logic produces a number you can inspect. Your fill price is just... whatever you wrote.

Three specific forms to look for:

Gap-through fills — the one above. Trigger levels set from prior session data, filled at the trigger regardless of the open.
Look-ahead on exits — if execute() is only called on bars where a signal fired, then exits are only checked on those bars. I had this one too: fixing it moved a different strategy from −5.7% to +17.9%, and inflated the win rate by 6–8 points. The original reported a 74.1% win rate; most of that was the bug.
Cost assumptions borrowed from another asset class — two futures strategies of mine were costed at 5bp per side. Actual round-trip cost was closer to 0.30bp (NQ) and 0.55bp (MGC). That is an 18–33× overstatement. Correcting it moved one of them from −94.0% to −45.5%. Rule of thumb: if cost ÷ per-trade risk exceeds about 5%, suspect the cost model before you reject the strategy.

Three checks you can run on your own log

You do not need my code for these; you need your trade log and your price bars.

1. Is every fill inside its bar?

For each trade, find the bar containing the entry timestamp and check
low <= entry_price <= high. Anything outside never traded. This single check catches the gap-fill bug directly, and it is the highest-yield check I know.

2. Is the fill better than the bar's open?

For breakout entries, if a long fill is below the open, either a limit order filled later in the bar (fine, but then model it that way) or you have gifted yourself the gap (not fine). This one has false positives — limit orders legitimately fill better — so treat it as a prompt to look, not a verdict.

3. Do your anomalies cluster in time?

Print your largest N moves with timestamps and histogram the hour. If the biggest moves all sit at a fixed time of day, you are looking at session boundaries, and your session handling is probably wrong.

The tool

I turned the checks I now run into a small script, because doing them by hand every time is how you stop doing them. It is one file, standard library only, no pandas:

python backtest_lint.py trades.csv --prices prices.csv

That image is real output from the tool on a demo log that has five bugs deliberately planted in it. It scores 1/6 and names each one with the row number and the actual numbers. The same repo also ships a clean control that scores 6/6 — a checker that always says "fail" would be worthless, so both directions are testable before you trust it.

Code: https://github.com/wei498766-ai/backtest-lie-detector — MIT, no dependencies.

The uncomfortable part

The uncomfortable part is not that I had a bug. Everyone has bugs.

The uncomfortable part is that the bug made the result better, and the result looked like a discovery. I did not find this because I was careless. I found it because I got suspicious of a number that was too good, and then went looking for how it could be too good.

That is not a scalable process. It depends on being suspicious at the right moment, about the right number — and the numbers you most want to believe are exactly the ones you are least likely to question.

So the process has to be external. Cheap, mechanical checks that run whether or not you are in the mood to doubt yourself.

That is the whole argument for this kind of tooling. Not that it makes you a better trader, but that it does not care what you want to be true.

If you have hit something in this family — particularly anything where the backtester's assumption is invisible in the output — I would like to hear about it. The ones that do not throw errors are the ones worth collecting.