Research · published Oct 21, 2026
Market vs models: Kalshi's prices were the better forecast
Almanac replayed 60 days of Kalshi's daily high-temperature markets in 24 cities and asked one question: which put more weight on the bracket that won, the market's prices or Almanac's model? The prices did, a day ahead and on the day.
A day ahead
Log loss on the bracket that won (lower is better)
- Market prices
- 1.26
- Almanac's model
- 1.48
1,417 city-days, Aug 20, 2026 to Oct 18, 2026. Brier score 0.108 for the market, 0.123 for the model.
On the day
Log loss on the bracket that won (lower is better)
- Market prices
- 0.92
- Almanac's model
- 1.25
1,419 city-days, Aug 20, 2026 to Oct 18, 2026. Brier score 0.086 for the market, 0.110 for the model.
What it means
In these markets, over this window, the prices already carried what 12 weather models said, and more. Buying where the model disagreed with the price lost 3.6 to 6.3¢ a contract on average, at every threshold tried. Almanac shows its model numbers as context, next to this record, and never as a signal. This doesn't show that nobody can do better than the market; it shows that this model, tested this way, didn't.
How the test worked
- Two decision times per city-day: 17:00 UTC the day before (forecasts at least two days old, so no run published after the decision could leak in) and 17:00 UTC on the day (forecasts at least a day old, plus the high so far from that morning's METARs).
- The market: each bracket's mid price at the decision time, from Kalshi's hourly price history, the six scaled to add to 1.
- The model: 12 forecasts from Open-Meteo, each corrected for its own bias at that station using only days already settled, combined by past accuracy into one probability per whole degree.
- The score: log loss on the bracket that won, averaged over city-days, and the Brier score over every bracket. The truth: how each market actually settled.
Source: Prices and results: Kalshi's public market data; Forecasts as issued: Open-Meteo previous model runs; Official highs: NWS climate reports via IEM (Central Park's shown)
Buying where the model disagreed
Each time the model's probability for a side was at least the threshold above that side's ask plus the fee (as for a 10-contract order), the replay bought one contract at the ask and held it to settlement (asks from 3¢ to 97¢, spreads up to 10¢).
| When | Gap of at least | Buys | Won | Average result |
|---|---|---|---|---|
| A day ahead | 3¢ | 3,596 | 33% | −4.8¢ |
| A day ahead | 5¢ | 2,901 | 32% | −4.9¢ |
| A day ahead | 8¢ | 2,082 | 31% | −4.6¢ |
| A day ahead | 12¢ | 1,298 | 29% | −5.6¢ |
| A day ahead | 15¢ | 845 | 29% | −6.3¢ |
| On the day | 3¢ | 3,268 | 28% | −4.2¢ |
| On the day | 5¢ | 2,933 | 27% | −4.2¢ |
| On the day | 8¢ | 2,459 | 26% | −4.1¢ |
| On the day | 12¢ | 1,916 | 26% | −3.6¢ |
| On the day | 15¢ | 1,578 | 25% | −4.3¢ |
Average result is per contract, after the fee: what one contract paid at settlement, minus its ask and fee. The first and second halves of the window were each negative at every threshold too (in the CSV).
How often the model's probabilities came true
Every bracket a day ahead, grouped by the probability the model gave it, against how often it won. A well-calibrated forecast wins about as often as it says.
| The model said | Brackets | Won |
|---|---|---|
| 0% to 10% | 3,393 | 4% |
| 10% to 20% | 1,609 | 15% |
| 20% to 30% | 1,934 | 24% |
| 30% to 40% | 1,287 | 37% |
| 40% to 50% | 216 | 39% |
| 50% to 60% | 43 | 37% |
| 60% to 70% | 11 | 55% |
| 70% to 80% | 7 | 71% |
| 80% to 90% | 2 | 50% |
What to keep in mind
- The forecasts had to be one to two days old to rule out lookahead, while the market traded on fresher runs, so the replay may have been harsher on the model than live use would be.
- One window, Aug 20 to Oct 18, 2026: late summer into autumn. A second model design (each corrected model as its own component) scored worse: 1.54 a day ahead and 1.31 on the day.
- The model and the replay are Almanac's own; the code that ran it is the engine's backtest.
An independent study
An independent study points the same way. Alexander W. Crosier's "Prediction Markets Beat the Weather Forecast on Tomorrow's High Temperature" (arXiv:2609.23969, posted Sep 21, 2026, a preprint that hasn't been peer reviewed) used five years of Kalshi data for seven US cities and found the market-implied forecast more accurate than the National Blend of Models, the most accurate single public forecast, in six of the seven, by about 10% in root-mean-square error after the market's first hour of trading.
Source: arXiv:2609.23969 (abstract, v1)
The data
Every number on this page, as one file: the scores for both decision times and both model designs, the buys at each threshold (all days, and each half), and the calibration table. Computed Oct 19, 2026.
Why raw model numbers mislead · Log loss, in one paragraph · Can these markets be gamed?