CFB-2a promotion spec — market arm in scripts/calibrate_models.py

Status: SPEC ONLY — not applied. Ground rule for this task forbids editing production files; this is the precise diff for the owner to apply by hand.

What this adds

Today scripts/calibrate_models.py --sports cfb reports the CFB model's baseline/Platt accuracy-Brier-log loss-ECE walk-forwardwalk-forwardEvaluating week by week using only what was knowable before each week, mimicking how the model would actually have been used at the time., with no market comparison anywhere in the file (confirmed by reading the whole CFB path before writing this spec). This diff makes every weekly retrain also emit, on the identical walk-forward test-game set (test_seasons=(2022,2023,2024), train_window=4):

• market_ml — no-vig implied home win prob from cfb_games.home_ml/away_ml (2021+, ~82% coverage of the 2022-2024 test set) — accuracy/Brier/log-loss.
• market_spread — market-spread-implied SU win prob (2018+, ~99% coverage), sigma fit walk-forward on strictly prior seasons — accuracy/Brier/log-loss.
• roi_vs_market_ml — model (baseline + Platt) vs devigged ML, flat + quarter-Kelly ROIreturn on investmentProfit as a percentage of the money wagered. +2% means $2 profit per $100 bet., game-clustered 95% bootstrapbootstrapRe-running a calculation on thousands of resampled versions of the data to see how much the answer wobbles. The spread of those answers becomes the confidence interval. CIs, per season + pooled.

All of the market math is already gate-verified: backtests/cfb_eval/market_math.py imports the canonical [redacted] (devig_multiplicative), and backtests/cfb_eval/replicate_spike_gate.py reproduces experiments/cfb_spike.json's overall market-aware numbers exactly (acc 0.697423, Brier 0.197621, flat ROI −5.3088%, n=2499 — see backtests/cfb_eval/replicate_spike_gate_result.json, "pass": true). This diff does not reimplement that math — it wires backtests/cfb_eval/market_baseline.py's already-built, already-verified functions into the weekly retrain via a local import.

Why a local import, not a copy

backtests/cfb_eval/market_baseline.py already contains load_games_with_meta / walk_forward_with_game_ids / add_market_arms / per_season_and_pooled / roi_backtest_block — all built specifically to mirror prep_cfb's game set (verified byte-identical game counts per season via verify_matches_prep_cfb, itself importing prep_cfb from this very file). Copying the ~230 lines into scripts/calibrate_models.py would fork the two implementations; a local import keeps ONE source of truth. If the owner would rather not have scripts/ depend on backtests/, the fallback is to promote backtests/cfb_eval/market_math.py + the four market_baseline.py helper functions above into app/core/ev_engine/ instead and import from there in both places — noted as an alternative below, not this diff's default.

The import is wrapped in try/except so a backtests/ issue can never break the other 4 sports' weekly retrain (cfb is the only sport this touches).

Diff

--- a/scripts/calibrate_models.py
+++ b/scripts/calibrate_models.py
@@ -223,6 +223,42 @@ def _walk_forward_oos_two_season(feat_by_season, y_by_season, train_window, te
             {'chold_raw': last_chold_raw, 'chold_y': last_chold_y, 'meta': last_meta})


+def _cfb_market_arm(test_seasons, train_window):
+    """CFB market baseline arm -- devigged no-vig ML (2021+) and
+    market-spread-implied SU win prob (2018+) accuracy/Brier/log-loss, plus
+    model-vs-market ROI (flat + quarter-Kelly, game-clustered 95% CIs), on
+    the IDENTICAL walk-forward test-game set prep_cfb() builds for this
+    sport (test_seasons=(2022,2023,2024), train_window=4 by default).
+
+    Lifted from backtests/cfb_eval/market_baseline.py -- gate-verified
+    against experiments/cfb_spike.json's overall market-aware numbers
+    (see backtests/cfb_eval/replicate_spike_gate.py /
+    replicate_spike_gate_result.json, "pass": true) before being trusted.
+    House rule: judged vs devigged closing lines, walk-forward only, ROI
+    bootstrap clustered by game (never iid).
+
+    Local import + broad except so a backtests/ issue never breaks the
+    other 4 sports' weekly retrain -- this is CFB-only.
+    """
+    try:
+        from backtests.cfb_eval.market_baseline import (
+            load_games_with_meta, walk_forward_with_game_ids, add_market_arms,
+            per_season_and_pooled, roi_backtest_block,
+        )
+        feat_by_season, y_by_season, meta_by_season = load_games_with_meta(
+            test_seasons, train_window)
+        df = walk_forward_with_game_ids(feat_by_season, y_by_season, meta_by_season,
+                                         train_window, test_seasons)
+        df, sigma_by_season = add_market_arms(df, feat_by_season, y_by_season,
+                                               meta_by_season, test_seasons, train_window)
+        return {
+            'n_games': len(df),
+            'n_ml_coverage': int(df['has_ml'].sum()),
+            'n_spread_coverage': int(df['market_spread'].notna().sum()),
+            'spread_sigma_by_test_season': {str(k): v for k, v in sigma_by_season.items()},
+            'market_ml': per_season_and_pooled(df, 'market_ml', mask_col='has_ml'),
+            'market_spread': per_season_and_pooled(df[df['market_spread'].notna()], 'market_spread'),
+            'roi_vs_market_ml': {
+                'model_baseline': roi_backtest_block(df, 'model_base'),
+                'model_platt': roi_backtest_block(df, 'model_platt'),
+            },
+        }
+    except Exception as e:
+        print(f"  [cfb market arm] ERROR — {e}")
+        import traceback; traceback.print_exc()
+        return None
+
+
 def prep_ncaab(test_seasons=(2022, 2023, 2024), train_window=2):
     from app.sports.ncaab.models.feature_engineering import (
         FEATURE_NAMES, build_features, feature_matrix,
@@ -531,6 +567,20 @@ def main():
             r = run_sport(sport, cfg['prep'], cfg['test'], cfg['window'], cfg['cal_path'],
                            walk_forward_fn=cfg.get('walk_forward_fn'))
             if r:
                 results[sport] = r
+                if sport == 'cfb':
+                    market = _cfb_market_arm(cfg['test'], cfg['window'])
+                    if market:
+                        results['cfb']['market'] = market
+                        pooled_ml = market['market_ml'].get('pooled')
+                        pooled_sp = market['market_spread'].get('pooled')
+                        print("\n  --- CFB market baseline (devigged consensus) ---")
+                        if pooled_ml:
+                            print(f"  market_ml     n={pooled_ml['n']:<5} "
+                                  f"acc={pooled_ml['accuracy']:.4f}  brier={pooled_ml['brier']:.4f}")
+                        if pooled_sp:
+                            print(f"  market_spread n={pooled_sp['n']:<5} "
+                                  f"acc={pooled_sp['accuracy']:.4f}  brier={pooled_sp['brier']:.4f}")
+                        roi_b = market['roi_vs_market_ml']['model_baseline']['pooled']['ml_flat']
+                        print(f"  model ROI vs market ML (flat, pooled): "
+                              f"n_bets={roi_b['n_bets']}  roi={roi_b['roi']:+.2%}")
         except Exception as e:
             print(f"\n{sport}: ERROR — {e}")
             import traceback; traceback.print_exc()

What this diff deliberately does NOT do

• Does not persist a JSON report. scripts/calibrate_models.py today has no JSON output for any sport (console-only) — this diff matches that existing style. Wiring the market arm into a persisted, dashboard-facing table (/cfb/performance) is Track 2b (backtests/cfb_eval/walkforward_regen.py, separate task), not this one.
• Does not change calibrator persistence. run_sport()'s calibrator_path / ECE-improvement-gated pickle.dump(...) to app/sports/cfb/models/trained/cfb_calibrator.pkl is untouched — this diff only adds a read-only reporting arm alongside it, called after r is already computed.
• Does not change train_window/test_seasons defaults, thresholds, or any model hyperparameter. Pure reporting addition.

Verification before applying

1. git show cd3b537:scripts/cfb_spike.py + backtests/cfb_eval/replicate_spike_gate.py → gate PASS (already run, see replicate_spike_gate_result.json).
2. python3 backtests/cfb_eval/market_baseline.py → confirms verify_matches_prep_cfb (game counts match prep_cfb for all 7 seasons) and produces sane per-season numbers (market beats model every season 2022-2024 on accuracy/Brier — the expected literature-consistent honest result, not a bug).
3. After applying, run python3 scripts/calibrate_models.py --sports cfb once by hand and confirm the new --- CFB market baseline --- block prints without exceptions and without touching any file under app/sports/cfb/models/trained/ other than the pre-existing cfb_calibrator.pkl persistence path (unchanged by this diff).
4. pytest scripts/unit (existing house convention for touched routes/scripts).

Alternative considered (not this diff)

Promote backtests/cfb_eval/market_math.py's no-vig/edge/ROI helpers into app/core/ev_engine/ so neither scripts/ nor backtests/ forks the market math, and both import from the same canonical module devig.py already lives in. Slightly cleaner long-term layering, more invasive (touches a second production path) — deferred to the owner's judgment, not spec'd here.