NFL Hierarchical Usage Model — Evaluation vs Baselines

Date: 2026-07-14 · Seed: 20260714 · Harness: model_usage.py → results/usage_model_results.json + results/usage_model_predictions.parquet (88,724 walk-forwardwalk-forwardEvaluating week by week using only what was knowable before each week, mimicking how the model would actually have been used at the time. player-week-prop rows, 88 folds, 2021–2024 test seasons, 3,000 MC samples/row).

House rules honored: walk-forward only; bootstraps clustered by game_id (MAEmean absolute errorAverage size of the miss, ignoring direction. If a projection is off by 3 one week and -5 the next, the MAE is 4. Lower is better., Brier) or by player (market join); no ROIreturn on investmentProfit as a percentage of the money wagered. +2% means $2 profit per $100 bet. fabricated; negative results reported as-is. Metrics below are on the model's own held-out player-weeks — the same rows the baseline is scored on.

TL;DR verdict

• Point-estimate MAE: the distribution-aware model does NOT lower MAE. Its posterior mean is significantly worse than a naive trailing-8-game mean on 6 of 10 props (game-clustered CI entirely negative), and statistically tied on the other 4. It never significantly beats the naive baseline on MAE.
• Probabilistic calibrationcalibrationA deliberate sanity check on the method itself: run it on something already known to be true. If it fails to detect the known thing, the method is broken and its other results mean nothing. is genuinely good — pooled ECE 0.024, and the P(over) forecast beats a coin-flip (Brier < 0.25) with significance on the six discrete count/TD props (targets, receptions, carries, all 3 TD props); attempts is the lone count prop whose skill CI straddles zero. This is the real payoff of carrying a distribution. But for the three continuous yards props the self-generated line sits at the predictive median, so P(over) ≈ 0.50 by construction and the "Brier < 0.25" test is vacuous there.
• Market-relative: essentially no usable real-close data. Only 2024 overlaps the model window. After mapping to devigged closes, just 167 rows are standard two-way markets (88 players, one season, engine-selected bets — not a random prop sample). On that sample the model's P(over) is indistinguishable from the devigged market (Brier 0.2533 vs 0.2511; market−model diff −0.0022, player-clustered 95% CI95% confidence intervalThe range the true value is plausibly in. If this range includes zero, we cannot rule out that the real effect is nothing at all. [−0.0244, +0.0188] spans zero). This is calibration-only, NOT CLVclosing line valueWhether you got a better price than the market settled at. Widely used as a faster signal of skill than profit, which takes ages to measure.. No ROI computed.

This matches the expected pattern (literature + the MLB-prop precedent): distribution machinery can improve calibration but does not improve point MAE here and shows no market edge on the thin real sample.

1. Point-estimate MAE (vs naive trailing-mean baseline, same rows)

Baseline = trail8_<prop> (8-game trailing mean of the stat), scored on the identical matched player-weeks. diff = baseline_MAE − model_MAE (positive ⇒ model better); 95% CI is a 2,000-rep bootstrapbootstrapRe-running a calculation on thousands of resampled versions of the data to see how much the answer wobbles. The spread of those answers becomes the confidence interval. clustered by game_id.

Restricting to active player-weeks (trailing usage above a role floor) does not rescue it: attempts +1.50%, rushing_yards +0.45%, passing_yards +0.55%, passing_tds +1.23% turn marginally positive, but targets (−2.73%), receptions (−1.22%), receiving_yards (−1.40%), receiving_tds (−4.10%), rushing_tds (−4.16%) stay negative. The small "active" flips are exactly the low-n props whose full CI already straddled zero.

Why: modeling E[stat] = volume × efficiency with each layer regressed to its own prior adds variance to the point forecast that a single well-tuned decayed mean avoids. The volume props (targets/receptions/carries) — where a trailing mean is already near-optimal — lose the most.

vs the incumbent production engine (MAE_REPORT.md) — context, not head-to-head

The incumbent real_fantasy_engine was never run on these 2021–2024 folds; its published numbers are 2024 dev + 2025 hold-out on a 4-game naive and slightly different per-category definitions, so this is directional only, not the same held-out player-weeks:

• Incumbent receiving_yards (2025 hold-out): 18.31 vs its naive 18.93 → beats naive by 0.61. The usage model's receiving_yards: 19.20 vs its naive-8 18.72 → loses to naive by 0.47.
• Incumbent passing_yards (2025 hold-out): 63.10 vs naive 68.20 → beats naive by 5.10. Usage model passing_yards: 70.55 vs naive 70.24 → tie/slightly worse.

Directionally the incumbent beats its trailing-mean baseline on the yardage props where the usage model loses to its own. On point MAE the distribution model is a step backward, not forward.

2. Probabilistic calibration — the real payoff of a distribution

P(over) is evaluated at the model's self-generated line (floor(median)+0.5). Reference forecast is the constant 0.5 (Brier 0.2500 / log-loss 0.6931).

Caveat that guts the yards rows: for continuous stats the line is placed at the predictive median, so P(over) ≈ 0.50 mechanically and Brier ≈ 0.25 — there is no information in the yards Brier numbers. The count props (discrete) do carry information: because count distributions are right-skewed, the median-centered line sits at an over-probability below 0.5 and the model correctly forecasts that (and the TD props are dominated by the correctly-modeled P(>=1)). The negative-binomial dispersion is justified at the distribution level even though it doesn't help point MAE: NB count log-loss beats Poisson on every volume stat (targets 2.119 vs 2.164, carries 2.958 vs 3.420, attempts 4.321 vs 5.633).

Calibration table (pooled, all 88,724 rows, deciles of predicted P(over))

Pooled ECE = 0.0240. Calibration is good, with a mild, consistent overconfidence (predicted slightly exceeds realized in every bin) — the model would benefit from a light shrink of P(over) toward 0.5 before it is ever used against a real line. PIT means cluster near 0.47–0.50 and 80% coverage runs 0.75–0.92 by prop (JSON dispersion_by_fold / summary.*.cov80), consistent with a distribution that is close but slightly too narrow on the yards props.

3. Market-relative (real devigged closes) — thin, selection-biased, NOT CLV

Data reality. stat_props.db non-synthetic = 1,121 rows, of which only 1,059 are 2024 (the sole season overlapping the model's 2021–2024 window; the 62 real 2025 rows and all of player_props.db, whose snapshots are Dec-2025 -> Feb-2026, are outside the model's prediction range and cannot be scored). Of those 1,059, only 207 are standard two-way prices (over odds in −140…+140); 852 are longshot alt/milestone lines (over >= +300, under ~ −110) whose two implied probabilities sum to less than 1 (median "hold" −0.30) — devigging them is meaningless. They are also engine-selected bets, not a random prop cross-section.

Scoring the overlap. Mapping names -> gsis (nflverse 2024) and joining on player-week-stat gives 579 matched rows; recomputing the model's P(over) at the book's line (normal approx from pred_mean/pred_sd — flagged) and devigging the close:

On the only defensible sample — 167 standard two-way NFL props, 88 players, one season — the model's P(over) is statistically indistinguishable from the devigged market close (the market is 0.0022 Brier better, CI spans zero in both directions). The model neither beats nor is beaten by the market here. Log-loss tells the same story (model 0.7052 vs market 0.6953).

This is calibration-adjacent, explicitly NOT CLV, and far too thin + selection-biased to support any ROI claim. No ROI is computed or implied.

4. Clustered-CI / honesty notes

• All MAE and Brier CIs above are 2,000-rep bootstraps clustered by game_id (never iid over correlated player-weeks within a game); the market join clusters by player (no game_id in stat_props.db).
• Nothing ROI-like is reported because there is no honest closing-line sample to build it on; the one usable market comparison is a calibration diff with a CI spanning zero.
• The self-generated-line Brier is a demonstration of the over/under machinery, not evidence of market skill — for yards it is degenerate by construction.

Honest verdict

The hierarchical usage x efficiency x game-environment model fails to lower point MAE — it is significantly worse than a naive trailing-8 mean on 6 of 10 props and never significantly better. Its distributional output is genuinely well-calibrated (ECE 0.024) and carries real information for discrete count/TD props (NB > Poisson, Brier < 0.25 with significance), which is the model's one defensible advantage over the single-mean incumbent. But on the only real devigged-close sample that exists — 167 two-way 2024 props, one season, engine-selected — its P(over) is indistinguishable from the market (CI spans zero), so there is no evidence of a market edge. Recommendation: do not ship it as a point-projection replacement; if anything is salvageable it is the calibrated P(>=1) TD / discrete-count distribution, and only after a light overconfidence shrink and a real (non-selected, two-way) closing-line capture to judge it on CLV. Exactly the outcome expected from the literature and the MLB-prop precedent: better calibration, no better MAE, no demonstrated beat of the market.