It took me a while, but here’s what I gather about this (I’m pretty sure it’s correct, but I’m not an expert).
A calibrated forecast means that if you say there is a 20% chance of rain, then it actually rains 20% of the time.
It’s a desired feature, but not the only one (e.g. you could be calibrated by stating: Chick-fil-A is open every day except Monday, but your forecast will always be wrong on Sunday and Monday).
So if
1. you are Bayesian (you state your beliefs)
2. and coherent (the laws of probability apply, so e.g. if P(A) = 0.4, then P(not A) cannot be anything other than 0.6):
and you are predicting something (e.g rain tomorrow), then
if you believe it will rain with probability 0.7 but you are 80% sure of your belief, you won’t say 0.7; you will say something else: 0.8 × 0.7 + 0.2 × something_you_believe = 0.58.
Coherence forces you to collapse your uncertainty into your probability at each forecast.
This theorem shows that, over many forecasts, in your belief system you are certain to be producing a calibrated forecast: your current beliefs assign probability 1 to the proposition that your future forecasts will be calibrated.
But that can’t be, which is the paradox. So Bayesianism is too strong compared to how scientists reason, because scientists always think their model can have an error.
Must ensure that this quote is present whenever Cromwell's Rule is mentioned - "I beseech you, in the bowels of Christ, think it possible that you may be mistaken."
It took me a while, but here’s what I gather about this (I’m pretty sure it’s correct, but I’m not an expert).
A calibrated forecast means that if you say there is a 20% chance of rain, then it actually rains 20% of the time. It’s a desired feature, but not the only one (e.g. you could be calibrated by stating: Chick-fil-A is open every day except Monday, but your forecast will always be wrong on Sunday and Monday).
So if
1. you are Bayesian (you state your beliefs)
2. and coherent (the laws of probability apply, so e.g. if P(A) = 0.4, then P(not A) cannot be anything other than 0.6):
and you are predicting something (e.g rain tomorrow), then if you believe it will rain with probability 0.7 but you are 80% sure of your belief, you won’t say 0.7; you will say something else: 0.8 × 0.7 + 0.2 × something_you_believe = 0.58. Coherence forces you to collapse your uncertainty into your probability at each forecast.
This theorem shows that, over many forecasts, in your belief system you are certain to be producing a calibrated forecast: your current beliefs assign probability 1 to the proposition that your future forecasts will be calibrated.
But that can’t be, which is the paradox. So Bayesianism is too strong compared to how scientists reason, because scientists always think their model can have an error.
Title a reference to The Well-Tempered Clavier?
(1982)
Must ensure that this quote is present whenever Cromwell's Rule is mentioned - "I beseech you, in the bowels of Christ, think it possible that you may be mistaken."
This is a classic!