Naya Trader Member sign in

Method

Compared to what? Your control group decides the answer

"+0.3% per trade" means nothing until you know what a random trade at the same minute would have made.

29 October 2026 · 6 min read

"60% win rate." "Average +0.3% a trade." Compared to what? On a morning when the whole market rises, almost any purchase looks clever. So every signal we test is compared with a control: every other stock you could have bought at that same minute, held by the same rule, paying the same costs. The edge is the difference between the two.

Our own screen lost to random

We started with the conditions a lot of intraday screens use: up on the day, above VWAP, above the 20-period EMA, busy in the last ten minutes, busier than yesterday. Over 126 sessions, we compared the stocks meeting each condition with every other stock trading over ₹2 crore, at the same minute:

ConditionEdge vs controlt (by day)
Up 1% or more from today's open−0.10%−3.64
Above VWAP−0.05%−1.06
Above EMA 20−0.04%−3.48
Heavy volume in the last 10 minutes−0.13%−6.62
All conditions together−0.17%−3.99

The stocks that looked strongest did worse than a random stock bought at the same minute. Without the control, you'd never see it.

The control has to be everyone

Later we tested a pullback setup, and the control was accidentally built only from stocks the screen had already flagged. Those were momentum stocks, which tend to drift up anyway. The results:

Control drawn fromEdgetVerdict
Only stocks the screen flagged−0.33%−6.12Much worse than random
Every liquid stock−0.073%−1.33No different from random

Same trades, a different comparison, and the opposite conclusion. The headline number moved by 4.5 times. Note the direction: here the biased control was too good, so the setup looked worse than it was. You can't predict which way the bias will push, so you can't correct for it afterwards. You have to build the control right.

Noise has a t-statistic too

Test enough ideas and some will clear t = 2 by pure luck. To find out how lucky, we shuffled each day's returns randomly across stocks, which destroys any real signal, and ran the same 45-feature search again, many times over:

So for a search that size, a result needs |t| of 3.44 or more before it counts as evidence. A short-selling result that had looked promising at t = 2.4 was beaten by noise in 80% of the shuffles.

Subtract the market

A short strategy tested in a falling market collects the fall. In one sample the market went down on 62% of days. Of the 0.24% to 0.37% a day that shorting stretched stocks earned before costs, about 0.18% was just that, and a random short at the same minute earned it too. A good control removes it automatically.

The checklist we use

  1. The control is every stock that was tradeable at that minute and passed the same liquidity filter. It is recorded before, and independently of, whether the signal fired.
  2. Same minute, same holding rule, same costs on both sides.
  3. Report the difference and its t-statistic day by day, not pooled across trades, so a few big days can't carry the result.
  4. Raise the bar for "significant" with the number of ideas tried.
  5. Tune on one period. Judge once, on a later period the rule has never seen.

← Five measurement bugs that fake an edge
All write-ups: Research →