You're right that 8 wards is too small to claim statistical validity, and the calibration was derived from those same 8 wards. That's overfitting until proven otherwise.
That's exactly why we are pre-registered predictions for 136 councils on May 7. I'll publically announce our predictions on May 1st.
If the calibration generalises, we have something. If it doesn't, we have a well-documented failure. I hope for the former, I can accept the latter :)
So to summarise the 2 potential questions... Is it any good? Does it work? The honest answer is: I don't know yet.
YouGov doesn't poll council elections. Nobody does. There's no commercial incentive to poll 5,000 individual wards at £10-50 per respondent.
That's the gap.
YouGov's MRP models predict at constituency level for general elections. We predict at council ward level for local elections.
Different product, different market.
On cost: 65,000 synthetic respondents cost us roughly £35 in GPU compute. A 1,000-person
YouGov poll costs £5,000-15,000. The accuracy isn't the same yet (we're at 75% winner accuracy on by-elections vs YouGov's 90%+ on generals), but we're predicting contests nobody else attempts.
The "in" is that local elections, by-elections, and ward-level prediction are completely unserved.
We're not replacing YouGov. We're filling the space below where polling is commercially viable.
And more than anything, we're testing and learning in public. If our panels get close to the real result after 7th May election, then there is something here... if not, well... I accept all your criticism :D
a sample size of 8 after 9 rounds of overfitting?
You're right that 8 wards is too small to claim statistical validity, and the calibration was derived from those same 8 wards. That's overfitting until proven otherwise.
That's exactly why we are pre-registered predictions for 136 councils on May 7. I'll publically announce our predictions on May 1st.
If the calibration generalises, we have something. If it doesn't, we have a well-documented failure. I hope for the former, I can accept the latter :)
So to summarise the 2 potential questions... Is it any good? Does it work? The honest answer is: I don't know yet.
The paper says as much. May 8 will tell us.
The competition in this particular space is YouGov. Who have the same or higher accuracy rating, and are already being employed by all the incumbents.
This seems to get about the same results as a random sampling poll, but with AI instead of people. So higher infrastructure costs.
... Where's the "in", here?
YouGov doesn't poll council elections. Nobody does. There's no commercial incentive to poll 5,000 individual wards at £10-50 per respondent.
That's the gap.
YouGov's MRP models predict at constituency level for general elections. We predict at council ward level for local elections.
Different product, different market.
On cost: 65,000 synthetic respondents cost us roughly £35 in GPU compute. A 1,000-person
YouGov poll costs £5,000-15,000. The accuracy isn't the same yet (we're at 75% winner accuracy on by-elections vs YouGov's 90%+ on generals), but we're predicting contests nobody else attempts.
The "in" is that local elections, by-elections, and ward-level prediction are completely unserved.
We're not replacing YouGov. We're filling the space below where polling is commercially viable.
And more than anything, we're testing and learning in public. If our panels get close to the real result after 7th May election, then there is something here... if not, well... I accept all your criticism :D
I predict it with 50% without any data whatsoever. 75% does not impress me.
thats fair.