Why I'm Done Trusting Backtests I Can't Re-Run
Introducing Skolfin Research: trading strategies from vendors, papers, and the community — run through walk-forward, cluster, and Monte Carlo testing, with every verdict published free, pass or fail.
I spent the first few years of my trading journey searching for the holy grail strategy just like everyone else starts out doing. I analyzed countless free YouTube strategies, strategies from newsletters, free courses, and paid training. I read many books on trading systems and methods and started to see a pattern with the strategies being offered both paid and for free. Very seldom was there any mention of validation and if there was, it was not at the level needed to determine if there was an identifiable edge.
For example, I bought a set of trading strategies from a professional vendor to be traded as a portfolio of strategies. Real money — the kind of purchase you make after reading a polished sales page with impressive equity curves and a wall of green checkmarks.
Then I did something most buyers never do: I ran every strategy through walk-forward analysis, multi-cluster analysis and null testing.
Only one passed and had an identifiable, repeatable edge.
That experience — and a dozen like it across years of building and testing systematic strategies — is why this publication exists. Retail systematic trading has a trust problem. Strategies sell for $500 to $1,500 on the strength of backtests you can’t inspect, “validated” by checkmarks the seller awarded themselves. The buyers are usually people early in their journey — learning RealTest, EL/PL and Python Script — who have no way to tell a genuine edge from a curve fit to the answer key. As a matter of fact, very few beginning traders actually understand what curve fitting actually is. I know, because I was one.
Somewhere along the way I stopped trusting anything I couldn’t reproduce, and built the tooling to test claims properly: walk-forward efficiency, cluster analysis across twenty in-sample/out-of-sample configurations, Monte Carlo null tests, the Deflated Sharpe Ratio with the trial count disclosed — the published methodology (Pardo; Bailey & López de Prado) that institutional quants use to catch overfitting, applied to the strategies retail traders actually get sold.
Skolfin Research is where that testing happens in public.
What kind of place this is
This is a community for people who’d rather know if a strategy actually has an edge than be sold a dream — traders learning a scripting platform, tired of hype, who want to see how honest validation actually works so they can do it to anything, including my work.
The standing invitation: reply to any email and tell me what strategy, vendor claim, or paper you want tested. Reader requests feed the queue. The comment section is for people who check each other’s work.
What you’ll get here
Replications. Vendor claims, research papers, YouTube strategies, and community ideas — rebuilt on survivorship-bias-free data with honest costs, and run through the gates. Every verdict published, pass or fail.
When something FAILS, you get the code. Failed replications ship with the full test script so you can re-run the failure yourself. If you can’t re-run it, it’s marketing.
The validation standard, taught. A series explaining exactly how the testing works, using real strategies as the worked examples — so the checkmarks here are ones you can audit. Look for a future post on the validation scope for each strategy and what types of strategies do not benefit from the specific type of validation pipeline we will be discussing here.
Building a multi-strategy portfolio. The long-form series on how a systematic portfolio actually gets constructed — including the graveyard of everything that died along the way.
Cadence: roughly weekly. Some weeks the honest result is “everything I tested died” — those get published too, because the failures are what make the passes believable.
Everything above is free. Later, if this publication earns it, there will be a paid tier for the code behind strategies that PASS the gates, and occasionally a fully-validated strategy for sale with its complete evidence dossier. But the verdict — every verdict — stays free, forever. That’s the deal.
What’s coming first
The opening post is already written: there’s a whole genre of Nasdaq-100 mean-reversion strategies selling for around $1.2K on the strength of 20%+ backtests. I rebuilt the trade they’re all selling — same universe, honest costs — and ran the numbers: full period, decade by decade, and through 2022, the exact sideways year the genre is marketed for. The gap between the sales pages and the honest archetype is the entire reason this publication needs to exist. It goes up this week, replication script included.
If that’s the kind of reading you want in your inbox, subscribe — it’s free, and every post arrives there in full.
Nothing here is investment advice. All results are backtested or historical, verified where stated, and reproducible from included code. Trade nothing you haven’t tested yourself.