WeatherMergeWhich weather forecast is actually right?
For 4 years, a homemade project called WeatherMerge pulled the forecast from every weather source it could find, every few minutes, and logged it next to a backyard weather station in West Seneca, NY. Here is how each one actually did.
The scorecard
Each source's morning forecast for the next day, scored against what the station measured. Temperatures are the average miss in degrees, so lower is better. Rain is a skill score from 0 (no better than guessing the usual odds) to 1 (perfect).
| Source | High | Low | Rain | Now | Days |
|---|---|---|---|---|---|
| 1. WGRZ (Channel 2) | 2.6° | 3.3° | 0.36 | 2.9° | 1,010 |
| 2. Weatherbit | 2.7° | 2.9° | 0.41 | 1.9° | 1,106 |
| 3. Apple Weather | 2.9° | 2.6° | 0.45 | 2.2° | 1,063 |
| 4. WeatherMerge blendaverage of all | 2.9° | 3.0° | 0.47 | — | 1,012 |
| 5. WKBW (Channel 7) | 2.9° | 3.3° | 0.36 | 3.1° | 1,012 |
| 6. AccuWeather | 3.1° | 3.5° | 0.43 | — | 708 |
| 7. Visual Crossing | 3.2° | 2.8° | 0.50 | 1.6° | 1,342 |
| 8. OpenWeatherMap | 3.4° | 2.9° | 0.30 | 1.5° | 1,343 |
| 9. Pirate Weather | 3.5° | 3.4° | 0.43 | 2.1° | 1,316 |
| 10. Weather Unlocked | 4.0° | 3.0° | 0.19 | 2.8° | 897 |
| 11. Tomorrow.io | 4.1° | 3.4° | 0.12 | 2.0° | 1,343 |
| 12. myweather2 | 4.1° | 3.6° | — | 2.9° | 894 |
The WeatherMerge blend averaged every source and still only ranked #4: averaging smooths out the bad days, but it also dilutes the best source. The local TV stations held their own, averaging 2.8° against 3.4° for the apps.
The further out, the worse it gets
Average miss on the daily high, from today out to a week ahead. Hover or tap a day to see every source.
When it says 70% chance of rain, does it rain?
Each dot groups the days a source forecast a given chance of rain, and shows how often it actually rained in the backyard. On the dashed line, the percentages mean exactly what they say. Tomorrow.io was the most optimistic: on days it called 10% or less, it rained 18% of the time. Rain is scored April to November, because the backyard gauge cannot measure snow.
What's the temperature right now?
Every source's "current temperature" compared with the backyard thermometer at the same minute. The National Weather Service reading, off by 2.9° on average, comes from the nearest official station, miles away, and it shows.
Rain in the next hour
Four sources forecast rain minute by minute. The trade-off is clear: Apple Weather saw 92% of the rainy minutes coming but was right only 29% of the times it said rain. The best sit up and to the right. This is the next 15 minutes, April to November, from September 2023.
Hour by hour
Average miss on the temperature 1, 6 and 12 hours ahead, for the sources that published hourly forecasts.
| Source | 1 hour | 6 hours | 12 hours |
|---|---|---|---|
| Apple Weather | 2.1° | 2.2° | 2.3° |
| Pirate Weather | 2.2° | 2.3° | 2.4° |
| Dark Sky | 2.2° | 2.4° | 2.5° |
| OpenWeatherMap | 2.2° | 2.8° | 2.8° |
| Tomorrow.io | 2.3° | 2.5° | 2.6° |
| WGRZ (Channel 2) | 2.7° | 2.7° | — |
| WKBW (Channel 7) | 2.9° | 2.9° | 2.8° |
A backyard has its own weather
Every source ran cooler than the backyard on afternoon highs. The yard heats up fast in the morning sun, about 2.1° warmer than forecast between 8 and 11am, and cools off fast at night, about 1.3° cooler between 7 and 10pm. Forecasts describe a neighborhood; a thermometer describes one spot in it.
Spring is the hardest season to call
Average miss on next-day highs, all sources, by month. April was the toughest (4.8°), December the easiest (2.4°).
The biggest busts
The worst next-day high forecasts in four years. Tap one to see what every other source said that day.
Backyard records
What the station itself saw.
Nailed it: Apple Weather got both the high and the low within 3° on 40% of days, the best of any source.
How this was scored
- Ground truth is one Ambient Weather station in a West Seneca, NY backyard, read every minute. Daily highs and lows use 15-minute averages, so a single gust of sun does not count as the high.
- Daily scores use each source's morning forecast: the median of everything it published between 6 and 10am, so a single glitched update cannot count as the forecast. Sources disagree on what a "high" is (the daytime high or the whole calendar day's) and what a "low" is (tonight's or the day's), so each was scored against the definition it actually uses.
- Rain means at least 0.01" in the gauge, April through November only. Rain skill is the Brier skill score against the backyard's own rain frequency.
- Impossible readings were dropped (Pirate Weather occasionally reported -459.67°F, which is absolute zero). Sources came and went over the years, and Dark Sky shut down too soon after the station went up to rank, so the "Days" column shows each source's sample size.
- This is one backyard, and your neighborhood may crown a different winner. Cobalt itself uses Apple WeatherKit, which ranked third on next-day highs, first on lows, first on hourly temperatures and second on rain.