← Research Library
New to these reports? Start here
  • Dotted-underlined words have a plain-English definition — hover or tap them. Every term is also on the glossary page.
  • "Null" means we found nothing, not that something broke. Most reports here are negative results, on purpose — knowing an idea doesn't work is the point.
  • Two questions get asked separately. First, is the effect real? Second, is it already priced into the betting odds? An effect can be completely real and still useless to bet on.
  • A "calibration" row is a self-check. It runs the same method on something already known to be true. If that fails, the whole report is unreliable — so it's reported alongside the findings.
  • If a confidence interval includes zero, the real effect might be nothing at all, so no claim gets made.

FDR-Controlled Retrosheet Split Miner -- Findings

Data: 4,706,329 plate appearances, seasons 2000-2025. Discovery <= 2018, validation after. BH q=0.05, discovery cell floor=100 per side per outcome. Runtime 462.9s.

The funnel (the multiple-comparisons trap, made visible)

  • Real hypotheses generated (pre-floor): 632,415
  • Excluded by min-cell floor: 462,049
  • Real hypotheses tested: 170,366
  • Placebo hypotheses tested: 100
  • Naive raw p<0.05 (all): 18,651 (expected by chance alone: ~8,523.3; real only: 18,646)
  • Survive BH q<=0.05: 2,688 (q<=0.01: 1,555; real only q<=0.05: 2,688)
  • OOS CONFIRMED: 626 | FAILED_OOS: 551 | INSUFFICIENT_OOS: 1,511

Placebo results (should ~all die)

  • Placebo hypotheses: 100
  • Placebo raw p<0.05: 5
  • Placebo surviving BH q<=0.05: 0
  • Placebo OOS-confirmed: 0

Known-truth gates (must survive)

check rate C rate ~C disc effect disc p BH q val effect verdict
league_platoon_HIT 0.2672 0.2533 +0.0139 4.70e-149 1.15e-144 +0.0082 CONFIRMED
league_platoon_OB 0.3412 0.3145 +0.0266 0.00e+00 0.00e+00 +0.0181 CONFIRMED
league_platoon_K 0.1780 0.1965 -0.0185 0.00e+00 0.00e+00 -0.0118 CONFIRMED
league_platoon_HR 0.0342 0.0296 +0.0046 2.04e-104 1.93e-100 +0.0047 CONFIRMED
coors_HIT 0.2928 0.2588 +0.0340 8.00e-139 1.52e-134 +0.0402 CONFIRMED
coors_HR 0.0359 0.0306 +0.0052 1.63e-22 2.15e-19 +0.0025 CONFIRMED

Confirmed catalog (top 25 by |shrunk effect|)

# split outcome rate C rate ~C raw eff shrunk eff disc p BH q val eff nC n~C
1 igler001: vs LHB vs vs RHB K 0.196 0.339 -0.1429 -0.1335 7.2e-08 2.2e-05 -0.0553 494 678
2 hughj001: vs LHB vs vs RHB K 0.089 0.212 -0.1231 -0.1140 4.7e-10 2.0e-07 -0.1226 542 1,101
3 smitw002: vs LHB vs vs RHB K 0.351 0.231 +0.1200 +0.1093 3.9e-06 7.6e-04 +0.0759 541 679
4 roget001: vs LHB vs vs RHB K 0.322 0.198 +0.1235 +0.1051 4.4e-04 3.3e-02 +0.0966 286 333
5 greec003: vs LHB vs vs RHB K 0.421 0.286 +0.1345 +0.1040 4.7e-04 3.5e-02 +0.0645 233 423
6 bundd001: vs LHB vs vs RHB K 0.171 0.279 -0.1082 -0.1027 2.4e-07 6.4e-05 -0.0774 704 969
7 renfh001: vs LHP vs vs RHP K 0.192 0.302 -0.1100 -0.1027 3.0e-04 2.5e-02 -0.0338 313 643
8 corbp001: vs LHB vs vs RHB K 0.310 0.202 +0.1077 +0.1014 4.2e-11 2.1e-08 +0.0467 881 2,666
9 marqg001: vs LHB vs vs RHB K 0.193 0.289 -0.0966 -0.0911 3.0e-05 4.2e-03 -0.0342 617 791
10 bracb001: vs LHB vs vs RHB K 0.202 0.291 -0.0897 -0.0863 6.5e-05 7.7e-03 -0.1860 595 1,050
11 mcfat001: vs LHB vs vs RHB K 0.194 0.103 +0.0910 +0.0844 7.5e-06 1.3e-03 +0.0921 566 648
12 lynnl001: vs LHB vs vs RHB K 0.181 0.266 -0.0848 -0.0832 1.1e-10 5.3e-08 -0.0685 1,705 2,603
13 abadf001: vs LHB vs vs RHB K 0.251 0.164 +0.0868 +0.0800 2.9e-04 2.4e-02 +0.0961 574 573
14 wietm001: vs LHP vs vs RHP K 0.247 0.167 +0.0804 +0.0774 4.5e-09 1.7e-06 +0.1194 1,100 3,104
15 mahlt001: vs LHB vs vs RHB BB 0.151 0.040 +0.1115 +0.0769 9.8e-06 1.6e-03 +0.0208 205 302
16 schem001: vs LHB vs vs RHB K 0.252 0.330 -0.0776 -0.0768 3.2e-13 1.9e-10 -0.0651 3,429 3,908
17 morga001: vs LHB vs vs RHB K 0.258 0.173 +0.0849 +0.0765 5.0e-04 3.6e-02 +0.1248 422 767
18 mcfat001: vs LHB vs vs RHB OB 0.292 0.389 -0.0970 -0.0741 4.1e-04 3.1e-02 -0.0881 559 646
19 blacc001: at home vs on the road OB 0.403 0.316 +0.0875 +0.0730 1.5e-08 5.3e-06 +0.0593 1,923 1,927
20 milla002: vs LHB vs vs RHB K 0.322 0.244 +0.0779 +0.0728 1.5e-05 2.3e-03 +0.1319 945 1,727
21 chacj001: vs LHB vs vs RHB K 0.149 0.223 -0.0741 -0.0725 3.1e-10 1.4e-07 -0.0866 1,940 2,636
22 lynnl001: vs LHB vs vs RHB OB 0.367 0.290 +0.0767 +0.0703 1.5e-07 4.2e-05 +0.0408 1,698 2,574
23 lynnl001: vs LHB vs vs RHB BB 0.132 0.060 +0.0720 +0.0681 4.8e-16 3.6e-13 +0.0279 1,697 2,598
24 eickj001: vs LHB vs vs RHB OB 0.355 0.270 +0.0852 +0.0679 4.6e-04 3.4e-02 +0.1280 699 753
25 schem001: vs LHB vs vs RHB OB 0.317 0.246 +0.0704 +0.0672 2.3e-11 1.2e-08 +0.0661 3,409 3,880

Best DEBUNKED examples (impressive in discovery, died out-of-sample)

# split outcome disc eff disc p BH q val eff val p(1-sided) verdict
1 LEAGUE: at park DET05 vs elsewhere K -0.0164 3.2e-46 1.6e-42 -0.0028 0.101 FAILED_OOS
2 milla002: innings 7-9 vs other innings K +0.2024 1.3e-37 4.6e-34 -0.0490 0.815 FAILED_OOS
3 LEAGUE: at park CHI12 vs elsewhere HR +0.0066 3.2e-34 9.4e-31 +0.0002 0.403 FAILED_OOS
4 LEAGUE: at park ARL02 vs elsewhere K -0.0139 4.1e-34 1.2e-30 -0.0052 0.159 FAILED_OOS
5 LEAGUE: innings 7-9 vs other innings OB -0.0064 4.6e-33 1.1e-29 -0.0005 0.282 FAILED_OOS
6 LEAGUE: at park HOU03 vs elsewhere K +0.0134 5.2e-31 1.1e-27 +0.0027 0.111 FAILED_OOS
7 LAD: opposite-hand matchup vs same-hand K -0.0265 3.8e-28 7.2e-25 -0.0008 0.427 FAILED_OOS
8 TB: opposite-hand matchup vs same-hand K -0.0264 1.5e-26 2.7e-23 -0.0047 0.155 FAILED_OOS
9 CHC: innings 7-9 vs other innings K +0.0252 1.1e-24 1.7e-21 +0.0039 0.195 FAILED_OOS
10 SEA: opposite-hand matchup vs same-hand OB +0.0299 4.2e-24 6.2e-21 +0.0067 0.091 FAILED_OOS

On this page

Source

backtests/split_miner/REPORT.md
updated 2026-07-05 21:09