New to these reports? Start here
- Dotted-underlined words have a plain-English definition — hover or tap them. Every term is also on the glossary page.
- "Null" means we found nothing, not that something broke. Most reports here are negative results, on purpose — knowing an idea doesn't work is the point.
- Two questions get asked separately. First, is the effect real? Second, is it already priced into the betting odds? An effect can be completely real and still useless to bet on.
- A "calibration" row is a self-check. It runs the same method on something already known to be true. If that fails, the whole report is unreliable — so it's reported alongside the findings.
- If a confidence interval includes zero, the real effect might be nothing at all, so no claim gets made.
Two dead knobs: one was worth fixing, one was worth measuring and leaving off
Date: 2026-08-30 · Harness: evaluate_fantasy_mae.py --set <knob>=true
Both were on the "recommended, never implemented" list. Neither had been measured. One turned out to matter a lot and one not at all.
1. Weekly rosters — ✅ SHIPPED ON
The engine built its player list from load_rosters(max(season)) filtered to
status == 'ACT' — the roster as it stood at the end of the season. Anyone
who finished the year on IR or inactive was erased entirely, including every
week they actually played, and their final status was back-dated onto those
weeks.
Size of the hole, 2025: 1,063 of 4,866 productive player-weeks (21.8%)
across 161 players — not fringe names. Bo Nix (17 weeks, 17.9 PPRpoints per receptionA fantasy scoring format that awards a point for every catch, which raises the value of high-volume receivers./wk) ends
the season RES; De'Von Achane (16 weeks, 20.2) and Chris Olave (16 weeks,
16.8) end INA. All three were invisible to the projection engine.
Fixed by using load_rosters_weekly, taking each week's own status.
| coverage | ||
|---|---|---|
| 2024 | 81.0% → 93.6% | |
| 2025 | 79.7% → 93.5% |
Absolute MAEmean absolute errorAverage size of the miss, ignoring direction. If a projection is off by 3 one week and -5 the next, the MAE is 4. Lower is better. rises slightly, which is expected — the added player-weeks are harder. The metric that is population-fair is the margin over the naive baseline on the same rows, and that improves nearly everywhere:
| stat | 2024 vs naive | 2025 vs naive |
|---|---|---|
| passing_yards | −2.44 → −4.53 | −6.17 → −7.57 |
| passing_attempts | −0.59 → −0.74 | −0.69 → −0.82 |
| rushing_yards | −0.11 → −0.61 | −0.20 → −0.35 |
| rushing_attempts | +0.07 → −0.02 (flips to winning) | +0.08 → +0.10 |
| receiving_yards | −0.77 → −0.85 | −0.77 → −0.86 |
| receptions | −0.03 → −0.04 | −0.03 → −0.04 |
2. Pace adjustment — ❌ MEASURED, LEFT OFF
pace_factor was hardcoded to 1.0, so it multiplied every stat by nothing.
Implemented properly from nfl_team_game_epa.plays (both teams' trailing
plays-per-game, strictly prior weeks, normalised to a 63-play league average
and clipped to ±10%).
⚠️ The first A/B came back byte-identical to the baseline. pace_factor
was only read inside _run_monte_carlo_simulation, and analytic_projection
(the live path since v4.0) bypasses that branch entirely — so a "working"
implementation still multiplied nothing. It had to be applied to the adjusted
baseline, on volume stats only, since pace scales opportunity and not
efficiency.
Once it actually applied, it lost:
| worse | better | |
|---|---|---|
| 2024 | 7 of 8 | 1 |
| 2025 | 5 of 8 | 3 |
passing_attempts — the stat pace should most obviously improve — is worse in
both years (+0.12, +0.25). The likely reason is double-counting: the player's
own trailing volume baseline already reflects how fast his team plays, so
multiplying by a team pace factor applies the same information twice.
Kept as pace_adjustment=False with this document, so the next person to notice
the dead knob does not reimplement it blind.