Upgrade Experiments 2026-07 — Item 5 Synthesis

Gate discipline reminder (house rule): walk-forwardwalk-forwardEvaluating week by week using only what was knowable before each week, mimicking how the model would actually have been used at the time. only, log-loss/MAEmean absolute errorAverage size of the miss, ignoring direction. If a projection is off by 3 one week and -5 the next, the MAE is 4. Lower is better. wins that worsen market-relative numbers don't ship, three clean FAILs would be a perfectly good outcome. All three experiments below cleared their pre-registered gates on the letter of the rule or honestly failed them — no post-hoc gate-loosening was found by any checker.

1. MLB starter-pitcher blend weight (mlb-blend) — ADOPT (mechanism), unproven at 95% CI

Gate verdict: PASS (checker confirmed: gate_verdict_confirmed: true, overall verdict "sound")

Pooled numbers (n_val=4949, n_test held-out 2025-2026, odds subsample n=330):

Robustness (game-clustered paired bootstrapbootstrapRe-running a calculation on thousands of resampled versions of the data to see how much the answer wobbles. The spread of those answers becomes the confidence interval., n=3906 test games, n_boot=5000, seed=20260712): mean diff +0.000153, 95% CI95% confidence intervalThe range the true value is plausibly in. If this range includes zero, we cannot rule out that the real effect is nothing at all. [-0.000416, +0.000709] — spans zero, P(challenger better)=0.709. Same pattern the NHL v3 de-snooping campaign flagged for its own +4.32%: technically clears the gate, statistically indistinguishable from noise on this snapshot.

Checker findings: (major) gate condition (a) — "wins on validation log loss over the full grid" — is near-tautological because the full-grid argmin search space is a strict superset of the blend_w=0.5-restricted search, so it will pass whenever any of the newly-added 600 combos edges out 0.5 by any margin; only conditions (b) and (c) carry real discriminating power. (minor) VALIDATION_SEASONS/TEST_SEASONS share season 2025 — pre-existing, disclosed, symmetric across incumbent/challenger. (minor) promotion-spec line numbers are stale by 25-55 lines (targets correct).

Verdict: ADOPT the mechanism, not today's specific numbers. The promotion spec deliberately avoids hardcoding blend_w=0.25/w=65/prior=8 as new defaults — it adds pitcher_blend_w as a 4th grid dimension (default 0.5, i.e., a no-op until the next scheduled retrain runs) and lets the existing retrain cadence discover the argmax itself, exactly like it already does for w/half_life/prior. This mirrors how NHL v3 was shipped: honestly labeled unproven, judged forward on live data, not retroactively hand-picked. Recommend the owner apply the spec and watch the next few retrain cycles' moneyline_backtest_metrics.json rather than manually pinning today's snapshot values.

Promotion steps (apply to [redacted] and scripts/mlb/train_moneyline.py; NOT applied by me): 1. moneyline_model.py: add DEFAULT_PITCHER_BLEND_W = 0.5 near the other DEFAULT_PITCHER_* constants. 2. moneyline_model.py, MLBEloV2Model.__init__: add pitcher_blend_w: float = DEFAULT_PITCHER_BLEND_W param, store as self.pitcher_blend_w. 3. moneyline_model.py, _update_pitcher_side: replace value = 0.5 * v_results + 0.5 * v_skill with value = self.pitcher_blend_w * v_results + (1.0 - self.pitcher_blend_w) * v_skill. 4. moneyline_model.py, to_state(): add 'pitcher_blend_w': self.pitcher_blend_w, to the hyperparams dict. 5. moneyline_model.py, from_state(): add pitcher_blend_w=hp.get('pitcher_blend_w', DEFAULT_PITCHER_BLEND_W),. 6. scripts/mlb/train_moneyline.py, grid_search_v2(): add blend_grid = [0.0, 0.25, 0.5, 0.75, 1.0], fold into the product(...) loop, thread pitcher_blend_w=blend into walk_forward_probs_v2(...), record 'blend_w': blend in best. 7. Same file, walk_forward_probs_v2() and the final = MLBEloV2Model(...) construction in main_v2(): pass pitcher_blend_w=best.get('blend_w', DEFAULT_PITCHER_BLEND_W). 8. Do not hardcode today's snapshot values as new defaults — let the next scheduled retrain persist whatever it finds to moneyline_model_v2.json, same as it does today. 9. Verification: rerun python3 scripts/mlb/train_moneyline.py, confirm the printed test log loss / new backtest metrics reflect the 4-D argmax; re-check the blend_w=0.5 equivalence proof against pre-change code before the state file is overwritten.

2. NBA correction-layer head-to-head (nba-features) — ADOPT variant (b) only; REJECT variant (c)

Gate verdict: PASS for both (b) and (c) (checker confirmed via full independent re-execution, exact reproduction to 4 decimals including bootstrap CIs since the seed is deterministic)

Pooled held-out metrics (n=7757, odds-joined n=4294, 2019-20…2025-26 walk-forward):

Gap-to-market narrows modestly: 0.0234 (a) → 0.0216 (b) → 0.0200 (c) — all three still lose to market, consistent with "close the gap, don't beat the market."

Per-season stability (log-loss wins vs incumbent, 7 held-out seasons): - (b): wins 5/7 (71%), improvement broadly spread (2020 -0.0044, 2023 -0.0041, 2024 -0.0037, 2025(partial,n=91) -0.0220 ≈11% of pooled gain — not dominant). - (c): wins only 4/7 (57%), and season 2020 alone (n=1337) contributes more than 100% of (c)'s entire net pooled improvement (2023 actively loses to incumbent, +0.0064) — classic noise-chasing pattern.

Checker findings: (minor) NaN-row footnote off (203 actual vs 390 claimed) — cosmetic, doesn't touch the reported metrics since all NaNs are filled with asserted defaults. (minor) tune_elo() selects EloElo ratingA rating system, originally from chess, that moves a team up or down based on results and the strength of the opponent. hyperparameters using log loss over the entire held-out window rather than a strictly prior split — pre-existing production quirk, applied identically to all 3 variants so it doesn't biasbiasWhether the misses lean consistently one way. A projection can have a good average error size but still be biased if it is almost always too high. Bias is often the more fixable problem. the head-to-head, but mildly optimistic for the absolute model-vs-market gap reported everywhere including the incumbent's own production metrics.

Verdict: ADOPT (b) only. (c)'s technical gate pass is not trustworthy — its pooled win is a single-season artifact. (b)'s win is thin (CI crosses zero) but broadly distributed and directionally consistent with "modest, try-upgrade" expectations from the methodology review. Ship with the same "unproven, judge forward on live CLVclosing line valueWhether you got a better price than the market settled at. Widely used as a faster signal of skill than profit, which takes ages to measure./log-loss-vs-close" framing as everything else this cycle — do not claim it beats the market.

Promotion steps for (b) (apply to [redacted] and scripts/nba/train_moneyline_v1.py; NOT applied by me): 1. moneyline_v1.py: import build_features, FEATURE_NAMES as EXPANDED_FEATURE_NAMES, get_team_recent_stats, build_feature_row from feature_engineering.py. Define COMBINED_FEATURE_NAMES = FEATURE_NAMES + EXPANDED_FEATURE_NAMES (45 entries, order load-bearing); keep existing FEATURE_NAMES for backward compat. 2. Add build_expanded_matrix(records) (port from backtests/nba_eval/feature_merge_headtohead.py, including its exact _DEFAULTS NaN-fill dict — load-bearing for parity). 3. Extend the live prediction path (features_for_matchup, NBAMoneylineV1._win_prob, generate_predictions, backfill_and_settle) to build the 39 expanded features per-matchup from completed-games history at prediction time (new engineering, not copy-paste — the eval harness only validated an offline batch path). 4. scripts/nba/train_moneyline_v1.py: replace the 6-feature training matrix with np.concatenate([elo6_matrix, expanded_matrix], axis=1). 5. Replace bare LogisticRegression(max_iter=1000, C=1.0) with make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000, C=1.0)) for this variant — scaling is load-bearing (expanded features are unscaled 0-220, elo6 features are 0-1; unscaled fit was never tested). 6. Persist the full sklearn Pipeline in STATE_PKL (drop-in for predict_proba at runtime). 7. Bump MODEL_VERSION (e.g. nba_ml_v1_expanded) — auditable rollout, not a silent swap. 8. Re-run trainer + market_comparison; confirm reproduction lands near 0.6309 / 64.5% / -2.44% pooled before trusting it in production. 9. Do not port variant (c)'s XGBoost path. 10. After deploy, kill -HUP $(cat logs/[redacted].pid) per standing deploy memory (batch-job model, so the main requirement is re-running the retrain, not a route reload).

3. RB rushing volume/yards regression (fantasy-rb) — REJECT

Gate verdict: FAIL, confirmed (checker independently recomputed gate booleans from raw MAE numbers: {ridge: False, lightgbm: False}, matching exactly)

Binding gate: ships only if it beats BOTH v4 AND naive-trailing-4-game on 2025 holdoutholdoutData deliberately set aside and never looked at while developing an idea, then used once at the end as a fair test. Peeking at it first would defeat the purpose. MAE for BOTH rushing_yards AND rushing_attempts (RB-only, n=721).

Subgroup detail (committee_back, n=353 — the hypothesized hard case):

Because both stats must clear, and rushing_attempts never beats naive (worst in the exact committee-back subgroup this experiment targeted), the regression does not ship. Market-relative check correctly skipped per pre-registration (never run when the MAE gate fails).

Checker findings: none — verdict "sound," gate pre-registration confirmed genuine (predates the experiment by ~13.5 hrs in docs/METHODOLOGY_REVIEW_2026_07_FULL.md), no leakage found (RidgeCV/LightGBM tuned only on 2024 dev weeks, 2025 holdout touched once), naive baseline reused verbatim from scripts/evaluate_fantasy_mae.py, v4 numbers pulled from the real production engine (not reimplemented).

Verdict: REJECT. Honest negative — this is exactly the outcome the house framing says is fine. rushing_yards is a decisive win in isolation but the pre-registered joint gate correctly blocks shipping a partial win. No promotion spec.

Out-of-scope bug surfaced (not fixed, flagged for the owner): [redacted]'s nfl_data_py fallback 404s on any data_years window including season 2025; the swallowed exception makes get_matchup_multiplier silently degrade to a no-op constant 1.0. Confirmed def_matchup_mult_rb was constant 1.0 across the entire 2025 holdout used both by this experiment and by the published MAE_REPORT.md table — the v4 engine's defensive-matchup adjustment has likely been a silent no-op for all of the 2025 holdout, for every def-matchup-sensitive category, not just RB rushing. Does not change today's gate verdict (both benchmarks used the same broken multiplier symmetrically) but worth investigating separately.

Summary Table

No production files, trainers, configs, or [redacted] were modified in the course of this synthesis or the underlying experiments. No commits were made.