Turning an observation into a falsifiable rule

Every system starts as a sentence in natural language, and natural language is where most of them die. "Stocks tend to bounce after they gap down." "The franc catches a bid when equities sell off." "Breakouts fail on Fridays." Each of these feels like a hypothesis. None of them is. They are observations dressed as claims, and the gap between the two is where a research process either becomes serious or becomes theatre.

A claim is falsifiable when there exists a specific, statable outcome that would make you abandon it. Karl Popper made this the boundary of scientific work, and the criterion transfers to markets almost unchanged. If you cannot describe in advance the result that would kill your idea, you are not testing it; you are decorating it. This lesson is about the translation step — turning a sentence into an object with enough structure that reality can say no to it.

The four things a hypothesis must pin down

An observation becomes testable when it commits to a population, an event, a horizon and a measurement. Miss any one and the claim stays slippery.

Population. Which instruments, over which period, under which listing conditions? "Swiss stocks" is not a population. "Constituents of the Swiss Performance Index as of the close of the prior month, excluding any with a twenty-day median traded value below CHF 1 million" is a population. Note that the definition must be evaluable at each point in the past using only information available then — the reason for that constraint is the subject of the sibling lessons on survivorship and look-ahead bias.

Event. What exactly triggers an observation? "Gaps down" needs a threshold and a reference: the opening auction price is more than 2% below the previous day's official close. Now it is countable.

Horizon. Over what window do you measure the response? One day, five days, until an exit condition fires? The claim "stocks bounce" is compatible with almost any data if the horizon floats.

Measurement. What number is compared to what benchmark? Raw return, return in excess of the index, return per unit of realised volatility? These give materially different answers, and a claim that only works under one of them is a narrower claim than the one you started with.

Once those four are fixed, the sentence has become a rule, and a rule can be counted.

Worked translation

Take the observation: "the Swiss market opens badly on Monday after a bad Friday in the US, and then recovers."

Written as a testable specification:

• Universe: the SMI index, via a tradable proxy, from 2005 to the present.
• Condition: the S&P 500 closed down 1.5% or more on the previous US trading day.
• Entry: at the SIX opening auction price on the next Swiss trading day, but only if that price is at least 1% below the prior Swiss close.
• Exit: at the same day's closing auction price.
• Measurement: return from open to close, in CHF, net of an assumed round-trip cost, compared against the unconditional average open-to-close return of the same instrument on all other days in the same sample.

That last clause is doing more work than anything else. It is the null hypothesis: the thing you are trying to reject. Without it you will "confirm" your idea whenever the market drifts up, because the average day drifts up. A great deal of published strategy research amounts to rediscovering the equity risk premium in a new costume.

Notice also that the specification forces two decisions you might not have realised you were making. The SMI is a price index — it excludes dividends — while the Swiss Performance Index is a performance index that includes them. If your rule holds positions overnight or across ex-dividend dates, testing on the SMI understates the return of actually holding the basket. And SIX runs an opening and a closing auction, so "the open" and "the close" are auction prints, not the first and last trade of a continuous session. Getting these two details wrong is enough to change a conclusion.

Choose the null before you look

The order of operations matters. Write down the benchmark, the direction of the expected effect, and the size of effect that would interest you — before you run anything. This is a private version of pre-registration, the practice medical trials adopted precisely because researchers who chose their endpoints after seeing the data reliably found effects that later evaporated.

Concretely, keep a specification document per idea, with: the sentence in plain English, the mechanism you believe produces it, the four elements above, the null, the sample you will test on, the sample you will hold back, and the result that would cause you to discard the idea. Date it. Do not edit it after seeing results; if the idea changes, write a new one and count it. The count is the point — the number of specifications you have tried is the number you must correct for when you judge the survivor.

The mechanism clause is not optional

The strongest predictor of whether an apparent edge survives is whether you can say why it exists in terms of somebody else's constraint. Markets are not machines that emit patterns; they are people and institutions with obligations. A durable edge usually corresponds to somebody being paid to bear a risk, somebody being forced to transact at a bad time, or somebody being slow to update.

So the specification should include a sentence of the form: "this exists because ___ must ___ at ___". For example: index funds must buy an added constituent near the close of the effective date, regardless of price. Pension funds rebalance on a calendar, not on value. Leveraged holders must reduce when margin requirements rise, which is exactly when prices are falling. Each of those is a mechanism, and each generates a testable prediction about when the effect should be strongest — a second edge you can use to check the first. An effect that has no mechanism and shows no sensible variation across conditions is more likely to be an artefact of your search.

The corollary is that a mechanism claim is itself falsifiable. If your story is "index funds are forced buyers", then the effect should be larger for additions that represent a bigger share of the stock's float and smaller for tiny additions. If it is not, your story is wrong even if your numbers are right, and a rule with the wrong story will not tell you when it has stopped working.

How much evidence would settle it

Before you test, work out whether your sample could possibly answer the question. The arithmetic is simple and it disqualifies most ideas immediately.

For a series of trades, the t-statistic of the mean is approximately the per-trade edge divided by the per-trade standard deviation, multiplied by the square root of the number of trades:

t ≈ (mean / standard deviation) × √N

Suppose the true per-trade edge is 0.05 standard deviations — a realistic figure for a decent liquid-market rule. To reach t = 2 you need √N = 2 / 0.05 = 40, so N = 1,600 trades. If your rule fires forty times a year, you need forty years of data to distinguish it from zero at a conventional threshold. If instead the edge is a fat 0.20 standard deviations per trade, √N = 10 and N = 100 trades will do.

The same relationship in annual terms is worth memorising: for a strategy with Sharpe ratio SR, the t-statistic of its mean return after T years is roughly SR × √T. A strategy with a true Sharpe of 0.5 needs T = (2 / 0.5)² = 16 years to clear t = 2. A true Sharpe of 1.0 needs 4 years. This is why a two-year track record tells you almost nothing, and why the honest response to "it has worked for eighteen months" is that eighteen months was never going to be enough.

To connect the two: a per-trade edge of 0.05 standard deviations with 250 trades a year implies an annualised Sharpe of about 0.05 × √250 = 0.79, before costs. Costs are where most such edges go to die, which the sibling lesson on transaction costs takes up in detail.

And the threshold itself deserves scepticism. Harvey, Liu and Zhu argued in 2016 that, given how many candidate predictors the finance literature has tested, the conventional t = 2 hurdle is far too lax and something closer to t = 3 is appropriate for a newly proposed effect. Your own research has the same problem in miniature: if you have tried thirty specifications, the best of them will look impressive by chance alone.

Signs you have written an unfalsifiable rule

Some formulations cannot lose, and they are seductive precisely for that reason.

A rule with a free parameter chosen after the fact — "it works with the right stop" — is unfalsifiable until the stop is fixed in the specification. A rule whose failures are explained by an unstated condition — "it works except in choppy markets" — is unfalsifiable until "choppy" is a number computable from data available at the time. A rule stated as a tendency without a horizon — "price eventually returns to value" — is compatible with any drawdown of any depth for any duration, which means it forbids nothing.

The test is mechanical: hand the specification to a competent stranger with the same data. If they cannot produce the identical trade list without asking you a question, the rule is not yet a rule. That standard sounds harsh, and it is the standard everything in the rest of this course assumes.

What to take away

• A hypothesis becomes testable only when it fixes a population, a trigger event, a holding horizon and a measurement, each computable from information available at the time.
• The null hypothesis must be chosen before testing and must be something other than zero, or you will repeatedly rediscover the market's average drift and call it an edge.
• A mechanism clause — who is forced to do what, and when — makes the idea doubly falsifiable, because the effect should then vary predictably with the strength of that constraint.
• Sample size is decisive: at a per-trade edge of 0.05 standard deviations you need roughly 1,600 trades for t = 2, and a strategy with a true Sharpe of 0.5 needs about sixteen years to prove itself.
• If a competent stranger cannot reproduce your exact trade list from the written specification, you have an observation, not a rule.

Sources

• Karl Popper, The Logic of Scientific Discovery (English edition 1959; originally Logik der Forschung, 1934) — the falsifiability criterion.
• Campbell R. Harvey, Yan Liu and Heqing Zhu, "…and the Cross-Section of Expected Returns", Review of Financial Studies (2016) — multiple-testing correction and the case for a higher significance hurdle.
• David H. Bailey, Jonathan Borwein, Marcos López de Prado and Qiji Jim Zhu, "Pseudo-Mathematics and Financial Charlatanism: The Effects of Backtest Overfitting on Out-of-Sample Performance", Notices of the American Mathematical Society (2014).
• David R. Aronson, Evidence-Based Technical Analysis (2006) — on converting subjective chart claims into objective, testable rules.
• Robert Carver, Systematic Trading (2015) — on specifying rules with few free parameters and fixing them in advance.
• Marcos López de Prado, Advances in Financial Machine Learning (2018) — on research protocol, multiple testing and the discipline of holding data back.
• SIX Swiss Exchange, index rulebooks and trading guides — definitions of the SMI as a price index, the SPI as a performance index, and the opening and closing auction mechanics.