Since early 2026 I’ve been building a research platform for US equities as a side project. It’s written in Go, runs on my own infrastructure, and can pull data, run backtests, paper trade and show me the market in real time.
The part I’m proudest of decides whether a trading idea deserves to be believed.
Every backtest looks good. You only ever see the ones you kept. The hard part isn’t finding a curve that goes up; it’s knowing when to trust one. So the pipeline rests on one assumption: a backtest is guilty until proven innocent.
This post is about that method, from “I have a hunch” to “this could risk real money”. It isn’t about strategies or results; those stay private.
Why a backtest is guilty until proven innocent
Backtests mislead in predictable ways:
- Optimistic fills and missing costs. A backtester that assumes you always get the better of two prices, or free trading, invents profit out of thin air.
- Multiple testing. Try a hundred variants and the best looks brilliant by chance alone. The more you search, the less your best result means.
- Survivorship. Keep only today’s companies and you’ve quietly dropped every one that went bankrupt or was delisted. An idea that buys weakness can look fantastic until you put the casualties back.
- A few lucky days. If a handful of sessions drive the result, you don’t have a strategy; you have a lottery ticket that already paid out.
- Fitting the past. Enough parameters fit any history, which says nothing about the future.
The single most dangerous line in a backtester is the one that decides your fill price.
There’s a newer reason too: AI coding agents write a new variant in minutes. When trying things is nearly free, the bar for believing them has to go up, not down.
Decide the bar before you look
The most important rule comes from science, not finance: pre-registration. Clinical trials register outcomes before collecting data, so nobody can move the goalposts.
Every idea becomes a numbered program, with a dated document committed before any result exists. It must contain:
- A thesis and a mechanism. Why should this edge exist, and who’s on the other side of the trade? “It backtests well” isn’t a mechanism; an edge needs a birth certificate.
- A frozen design: universe, time windows, parameters and the exact set of variants. Nothing is added later.
- Frozen bars: numeric pass and fail criteria, including what gets closed on a fail. An uncomputed bar is a fail.
- Stop points where the run waits for my explicit go-ahead, always including before any gate run.
- Trial accounting: every variant reaching the gates is charged to a running ledger. That number matters later.
Writing the bars down first feels like bureaucracy. It’s the cheapest protection against fooling yourself: once you’ve seen a result, every bar you set afterwards is shaped by it.
The lifecycle: from hunch to one-shot gate run
Stage 1 is a cheap pre-test: a quick gross-to-net screen for whether there’s anything there, with every cell reported, not just the good ones. One robustness check: a real effect shouldn’t live only in odd or only in even years (allowing for crash-heavy ones).
The verdict is written the same day and committed. If the idea dies, it gets an autopsy, because why it failed often teaches more than the program itself.
There’s no same-day iteration. A tweak suggested by today’s result is shaped by it, so it becomes a new program, registered fresh and charged to the ledger like any other.
Only then comes the gate run: one shot, a frozen command, my explicit go, and every trade and equity curve saved.
The nine gates
The gate run is a checklist: every item must pass, and each exists because of a specific way backtests lie.
- G1Honest economicsOptimistic fills, cost blindness, strategies that only work in calm markets
- G2Deflated SharpeMultiple testing and non-normal returns
- G3Probability of backtest overfittingOverfitting to history
- G4Walk-forwardLeaking the future into the past
- G5Tail robustnessLucky days and understated tail risk
- G6SurvivorshipSurvivor-only history
- G7CapacityEdges too small to matter
- G8Live confirmationSimulation that doesn't match reality
- G9Reserved holdoutEverything above, one last time
Show the full table
| Gate | What it asks | What it catches |
|---|---|---|
| G1 Honest economics | Is it still profitable with realistic slippage and fees, fills capped at real limit prices, costs at most a third of gross profit, and drawdowns survivable through the dot-com crash and 2008? | Optimistic fills, cost blindness, strategies that only work in calm markets |
| G2 Deflated Sharpe | Is the Sharpe ratio still meaningful after accounting for how many things I tried? | Multiple testing and non-normal returns |
| G3 Probability of backtest overfitting | How often does the in-sample winner end up below average out of sample? | Overfitting to history |
| G4 Walk-forward | Does it hold up across time when each period picks its own variant? | Leaking the future into the past |
| G5 Tail robustness | Does it survive a Monte Carlo of plausible histories, and losing its five best days? | Lucky days and understated tail risk |
| G6 Survivorship | Does it still work on point-in-time data that includes delisted companies? | Survivor-only history |
| G7 Capacity | How much money can it handle before its own trades move the price? | Edges too small to matter |
| G8 Live confirmation | Do at least 50 paper fills, across at least one bad market, match the backtest within its cost band? | Simulation that doesn't match reality |
| G9 Reserved holdout | Does it pass, once, on a reserved period it has never seen? | Everything above, one last time |
A few deserve a closer look.
G2, the Deflated Sharpe Ratio. The best Sharpe ratio out of many trials is biased upwards, more so with every trial. Bailey and López de Prado’s deflated Sharpe ratio corrects for that, and for fat-tailed returns. The catch: you need the true number of trials, and nobody counts the ideas they tried in their head. So the pipeline deflates over the larger of the ledger and a floor of 1,000.
G3, the Probability of Backtest Overfitting. Combinatorially symmetric cross-validation splits history into blocks, tries every split into in-sample and out-of-sample halves, and asks how often the in-sample champion lands below the median out of sample. More than half the time means the selection is fitting noise.
G4, non-forced walk-forward. Picking the best variant over the whole history, then “walking it forward” through the folds, quietly uses the future to choose. Here each fold picks its own variant using only its own past, and at least 75% of eligible folds must be profitable after costs.
G5, Monte Carlo with a stationary block bootstrap. Markets have memory: bad days cluster. Resampling single days at random destroys that and makes tails look safer than they are; Politis and Romano’s stationary bootstrap resamples blocks of random length instead, which keeps the clustering. Across 10,000 alternative histories, the 5th-percentile Calmar ratio must stay above zero. Separately, the pipeline removes the five best sessions and checks the edge survives. If five days are the whole story, the story isn’t good enough.
- median
- 5th percentile
G6, point-in-time data. Each date’s universe is what was tradable on that date, including companies that later went bust, with splits, dividends and other corporate actions. A surprising amount of apparent edge evaporates here.
G9, the reserved holdout. A period of history is set aside, untouched by research, and each idea gets one look. It must run up to the edge of the available data, so nobody can reserve an old, convenient decade. Once used, it’s spent.
Monte Carlo and power: when “no answer” is the honest answer
Not every question can be answered with your data, and pretending otherwise is its own kind of overfitting.
Before a forward test, a Monte Carlo power analysis simulates how many observations it takes to detect an edge worth having, at the confidence I want. It also works out the time to a readout up front: qualifying events per week, so weeks until there’s enough data to decide. Some ideas die right there: the answer would take years.
An experiment that ends short of that sample gets neither “fail” nor “pass”, but “UNPOWERED: no verdict.” It sounds like a non-answer, but it’s the most honest one: a lucky streak isn’t promoted, and an unlucky one doesn’t kill a good idea.
The same logic has a hard edge. Strategies that earn in rare, huge jackpots have tails so fat that certifying them needs a longer track record than you can realistically collect. That isn’t a judgement on the strategy; it’s a limit on what can be known.
The reporting rules
Results are only as honest as their reporting, so the pipeline has rules for that too:
- Lead with the realistic curve: capped and fully costed. An optimistic version only appears labelled as such.
- Always state N. A Sharpe ratio without the number of trials behind it is meaningless.
- Make every number reproducible: pinned to its run and a hash of its parameters. An unpinned number can’t be cited.
- Report concentration: the share of profit from the best regime and the top five sessions, and the share of positive sessions. It’s a result, not a footnote.
- An uncomputed gate is a fail, and the report says so.
Making the rules executable
Rules that live in a document get bent at 11pm, so wherever possible they live in code.
The statistics (deflated Sharpe, PBO, the block bootstrap, the holdout power tables) sit in a small, tested validation package. The holdout tool refuses to reuse a spent window. Pre-registrations are committed before the code that runs them, so the history shows which came first. And because I build much of the platform with AI coding agents, enforcement in code matters even more. An agent will happily produce a plausible backtest; it won’t argue with a gate that refuses to pass.
Beyond trading
None of this is specific to finance; I bring the same habits to engineering:
- Decide how you’ll judge something before you build it.
- Measure the people it’s for, not your own output.
- Kill ideas early, and write down why.
- Accept that some answers need calendar time you can’t code around.
That last one may be the project’s biggest lesson. When writing another experiment takes an evening and waiting for evidence takes months, it’s tempting to keep building; building feels like progress. Often the most rigorous thing to do is wait for the answer you’ve already set up.
Further reading
- David H. Bailey and Marcos López de Prado, The Deflated Sharpe Ratio: Correcting for Selection Bias, Backtest Overfitting and Non-Normality, Journal of Portfolio Management, 2014.
- David H. Bailey, Jonathan Borwein, Marcos López de Prado and Qiji Jim Zhu, The Probability of Backtest Overfitting, Journal of Computational Finance.
- Dimitris N. Politis and Joseph P. Romano, The Stationary Bootstrap, Journal of the American Statistical Association, 1994.
- Robert Carver, Systematic Trading, Harriman House, 2015, on why costs, not cleverness, decide most strategies.