Did the Weather + FTN Features Actually Earn Their Keep?

Measurement of commit 7ba65e1, which wired real Open-Meteo weather and trailing FTN charting features into the fantasy projection engine and the betting feature builder.

Verdict: modest real gain on fantasy (concentrated exactly where weather should bite, with a clear biasbiasWhether the misses lean consistently one way. A projection can have a good average error size but still be biased if it is almost always too high. Bias is often the more fixable problem. correction). No confirmed betting edge.

TL;DR

Fantasy side — measured on the real walk-forward harness

scripts/evaluate_fantasy_mae.py, season 2025, weeks 5–18, strict walk-forwardwalk-forwardEvaluating week by week using only what was knowable before each week, mimicking how the model would actually have been used at the time., A/B on the weather_adjustments config flag. 3,328 projected player-weeks (2,654 with a stat line).

Unlike the injury multipliers, weather is not gated behind _is_future_week — the archive has real observations for completed games — so the standard harness measures it directly.

MAE by category

The pattern is the point, not the headline number. Every improvement is in an efficiency/production stat where weather physically acts (passing yards, passing TDs, receiving yards, receiving TDs). Every regression is in a volume stat (attempts, targets, receptions) and is ≤ +0.23%. That is what a correctly-located signal looks like; a spurious one would scatter.

Bias — the clearer result

All four major categories improved. The engine was systematically over-projecting passing, and real weather removes a meaningful share of it. For a projection used to set lineups, removing a persistent 7-yard-per-QB-game over-projection matters more than a 1% MAE move.

Betting side — the two-stage frame

Stage AStage AThe first question: is the effect real at all? Measured against what actually happened, ignoring betting markets.: is there a real effect on the outcome? Stage BStage BThe second question: is the effect already reflected in the betting odds? An effect can be completely real and still worthless to bet, because the price already includes it.: is it already in the closing lineclosing lineThe final odds right before a game starts. It reflects everything the betting market knows, which makes it the hardest benchmark to beat. (formal null E[market_error | feature] = 0)? Books have had forecasts and charting for years, so "priced" was the prior.

⚠ A unit error in the first pass nearly produced a false positive, and is worth recording. Slopes were initially reported per 1.0 unit of feature and compared against a points-denominated tradeable floortradeable floorThe minimum edge needed to overcome the bookmaker's cut. Anything smaller is real but unprofitable.. But wx_pass_mult spans only ~0.85–1.05, so "per 1.0 unit" is ~5× its entire observed range — which made several features look wildly "TRADEABLE". Re-expressed per +1 SD of the feature, most of them collapsed. Same class of error as the earlier starter-misidentification artifact: the statistics were fine, the units were not.

Weather vs the market (per +1 SD, develop 2015–2021)

Spread error: wind and precipitation both null.

The pre-registered holdout test — and why it still fails

Wind was the one survivor, so it got exactly one holdout test:

The continuous slope does survive. But three things kill it as an edge:

1. The actual bet doesn't clear the vigvigThe bookmaker's built-in cut. It is why a coin-flip bet at -110 needs you to win about 52.4% of the time just to break even.. In the holdout, games at wind ≥15 mph: n=44, mean total error −0.35, under hit rate 50.0% against the 52.4% needed. Betting it would break even at best.

2. The linear fit is not describing the tail. Mean total error by wind bin:

The 20+ bin rests on 22 games total, 16 of them in develop, and its −11.88 develop mean is a huge leverage point on the linear slope. The holdout tail reverses (16–20 is +1.20). The slope predicts ~−2.2 points at 15 mph; the observed windy-game mean is −0.35. The estimate and the reality disagree by 6×.

3. It explains almost nothing. corr(wind, total error) = −0.104, so wind accounts for 1.08% of the variance in market error. Overall mean total error is +0.022 — the market is essentially unbiased in aggregate.

The one genuinely interesting pattern is a gradient rather than a level bias: calm games trend over (+1.35 at 0–4 mph) and mid-windy games trend under. That is directionally consistent across both splits and is worth watching forward, but it is not a bet at these magnitudes.

FTN vs the market (per +1 SD, develop 2022–2023, n=447)

FTN features move the outcome decisively (Stage A: sum-catchable +20.7 pts on actual totals, sum-drop −34.1) — they clearly measure something real about offensive quality. But essentially none of it survives against the closing line. The two spread effects are statistically solid and economically too small; the pressure effect's confidence intervalconfidence intervalA range of values the true answer is plausibly in. If it includes zero, the effect might genuinely be zero, so we cannot claim it is real. reaches nearly to zero.

The holdout for FTN was not opened, since nothing cleared the floor in-samplein-sampleMeasured on the same data used to build or tune the idea. Nearly always looks better than reality..

Caveats

• FTN covers only 2022–2025, so its develop split is 447 games. UnderpoweredunderpoweredNot enough data to detect an effect even if it is really there. An underpowered null means "we could not tell", which is very different from "there is nothing there". for anything but a large effect.
• The fantasy A/B is one season (2025). The direction is consistent and the improvements are correctly located, but a single season of walk-forward is not a strong test of a ~1% MAE move.
• Weather Stage B tests the closing line only; nothing here can detect a stale-opening-line or line-movement edge, since nflverse carries no openers.
• Wind and wx_pass_mult are collinear by construction (the multiplier is a function of wind), so they are one finding, not two.

Bottom line

Keep the weather features on the fantasy side. They cut a systematic passing over-projection by ~42% and improve exactly the categories where weather physically acts, at negligible cost elsewhere.

Do not build a weather or FTN betting model expecting edge. The market prices both. Wind shows a real, holdout-surviving gradient against the closing total, but it explains 1% of variance, its magnitude is inflated by ~22 extreme games, and the tradeable version of it hits exactly 50% out of sample. The features remain worth having in the betting feature set for calibrationcalibrationA deliberate sanity check on the method itself: run it on something already known to be true. If it fails to detect the known thing, the method is broken and its other results mean nothing., not for edge — and any future claim of edge from them needs a forward test, not another backtest.

Reproduce:

python3 scripts/evaluate_fantasy_mae.py --season 2025 --weeks 5-18 \
    --variant current --set weather_adjustments=0 --tag wx_off_2025
python3 scripts/evaluate_fantasy_mae.py --season 2025 --weeks 5-18 \
    --variant current --set weather_adjustments=1 --tag wx_on_2025
python3 backtests/weather_ftn_value/measure_betting_value.py