The Field Guide
Weather Inside the Head: A Field Guide to Probability, Calibration and Intellectual Humility
Nine short chapters. Read them in order, or jump to the case files. Each claim is labelled with how firmly it stands.
1 A vocabulary for uncertainty
Most of what we believe is neither certain nor absurd. It lives in between, and the trouble starts when that middle ground has no language. A useful vocabulary separates four ideas that ordinary speech keeps blurring.
- Probability
- A number between 0 and 1 attached to a claim. Two traditions interpret it differently. On a frequency reading, it describes how often something happens across a long run of similar cases. On a subjective or Bayesian reading, it describes a degree of belief held by a particular person given particular evidence. Lichtenstein, Fischhoff and Phillips (1982) open their classic review from the subjectivist position: a degree of belief is internal, with no single “correct” probability sitting in reality to compare it against. Established (as a description of the debate)
- Calibration
- The bridge between the two readings. You cannot check one degree of belief against one outcome, but you can check many. A forecaster is calibrated if, among all the claims she rated 70%, about 70% came true; among her 90% claims, about 90%. Calibration is sometimes called reliability. Established
- Resolution (discrimination)
- Calibration is not the whole of good judgment. Someone who always forecasts the average rainfall frequency is perfectly calibrated and nearly useless. Resolution measures whether your confidence moves with reality — whether your 90s are right more often than your 60s. Murphy (1973) showed how the Brier score splits into reliability, resolution and uncertainty, and the simulator above reports all three. Established
- Proper scoring rule
- A way of scoring forecasts under which honesty is the best policy: your expected score is highest when you report exactly what you believe. The Brier score and the logarithmic score are both strictly proper (Gneiting & Raftery, 2007). A rule that just counts hits is not; it rewards rounding every belief up to certainty. Established
- Overconfidence — three kinds
- Moore and Healy (2008) argue that research has used one word for three phenomena: overestimation (thinking you performed better than you did), overplacement (thinking you did better than others) and overprecision (being too sure your beliefs are accurate, as when confidence intervals are too narrow). They report that the first two can reverse on different kinds of task, while overprecision is the most persistent. The simulator measures something close to overestimation on two-choice questions. Established distinction
- Intellectual humility
- In philosophy, Whitcomb, Battaly, Baehr and Howard-Snyder (2017) describe it as properly attending to, and owning, one’s intellectual limitations. In psychology, Leary and colleagues (2017) studied it as the degree to which people recognise that their beliefs might be wrong, measured by a self-report scale. The two framings overlap but are not identical, and a 2022 review by Porter and colleagues notes that definitions still vary across subfields. Definitions contested
2 Where miscalibration comes from
If people were simply unable to assign probabilities, calibration research would be short. Instead it finds systematic patterns, and several mechanisms compete to explain them.
Missing or slow feedback
Calibration is a skill learned from outcomes. A forecaster who learns tomorrow whether it rained gets thousands of graded repetitions; a manager estimating whether a strategy “will work” may get one ambiguous outcome a year, mixed with luck and other people’s choices. Lichtenstein and colleagues’ review found that experienced weather forecasters were well calibrated while most other groups studied were not, and that training helped only to a limited extent. A feedback explanation fits that contrast well, though the review itself could not isolate feedback from other differences between forecasters and laboratory volunteers. Interpretation of established findings
The hard–easy effect
The same review reported that overconfidence grows with task difficulty: on hard question sets people tend to claim more than they deliver, while on easy sets they can become underconfident. The simulator tags questions “plain” or “tricky” so you can look for this in your own session, with the warning that a dozen questions per tag is far too few to see it reliably. Established pattern Its causes are contested — see chapter 3
Precision that feels like knowledge
Lichtenstein and colleagues also observed that people act as though they can make finer distinctions in their degree of uncertainty than they really can. Moore and Healy’s “overprecision” names a related habit: holding beliefs in intervals too narrow for the evidence. A practical symptom is choosing 90% and 100% for answers you would have struggled to justify aloud. Established
Random error, sorted the wrong way
Any judgment contains scatter. When scattered judgments are grouped by the confidence the judge stated, the extreme bins collect the lucky overshoots, and their hit rates sag toward the middle. That sag looks exactly like overconfidence. Plate II lets you produce it with zero bias. Established as a statistical mechanism Its share of real overconfidence is contested
Motivated confidence
People may sometimes prefer to feel sure: certainty can be pleasant, persuasive and socially rewarded. This is plausible and widely discussed, but the studies cited here do not establish how much of measured miscalibration it explains, so it is labelled accordingly. Hypothesis
3 Two lenses on overconfidence
For decades, overconfidence was described as one of the sturdiest findings in judgment research. Then a rival tradition asked whether some of it was manufactured by how the questions were chosen and how the data were sorted. Both lenses remain in use, and a careful reader keeps both open.
Lens A · A real bias in human judgment
The heuristics-and-biases tradition, represented by the Lichtenstein, Fischhoff and Phillips review in the 1982 Cambridge volume Judgment under Uncertainty, documented confidence exceeding accuracy across many tasks and populations. Moore and Healy (2008) refined rather than rejected this picture: overprecision in particular looks robust. Later forecasting research, such as Mellers and colleagues’ (2014) tournament work, treated overconfidence as a real error worth training against.
If this lens is right: practise, keep score and deliberately widen your uncertainty, especially on hard questions.
Lens B · Partly an artefact of method
Gigerenzer, Hoffrage and Kleinbölting (1991) proposed a Brunswikian theory: people’s confidence is adapted to the natural environment of questions they meet, so when researchers select tricky or unrepresentative items, apparent overconfidence follows. Erev, Wallsten and Budescu (1994) added that random error alone can produce over- and underconfidence in the same data, depending on the analysis.
If this lens is right: before concluding that a person is biased, ask how the questions were sampled and how the data were grouped.
Where this leaves us. The lenses are not mutually exclusive. Brenner (2000) argued that error models do not show observed overconfidence to be spurious; subsequent analyses by Budescu and colleagues reported overconfidence above what error alone predicts in at least one study. A reasonable reading is that some measured overconfidence is real, some is produced by item selection and analysis, and the mix differs by task. That reading is an editorial synthesis, not a consensus statement. Contested
This laboratory is itself exposed to Lens B. Its questions were chosen by one author, many deliberately “tricky”. If your gauges show overconfidence, part of the reason may be that the deck is unrepresentative of the questions you meet in ordinary life.
4 Case files: the forecasters and the phrase
Case file A · United States, 1950–1977
The forecasters who learned to mean what they said
Context. In 1950 Glenn Brier, a statistician at the U.S. Weather Bureau, published a three-page paper in Monthly Weather Review proposing a way to verify forecasts stated as probabilities. The score that bears his name penalises the squared distance between a stated probability and what happened. By the 1970s, subjective precipitation-probability forecasts were part of U.S. operational practice.
What was documented. Murphy and Winkler (1977) summarised operational and experimental programmes in the United States and concluded that weather forecasters could formulate subjective probability forecasts of precipitation and temperature in a reliable — that is, calibrated — manner. Lichtenstein and colleagues’ 1982 review singled out weather forecasters as the conspicuous exception to generally poor calibration among the groups studied. Established (documented)
Why it may have worked. Forecasters made similar judgments daily, received unambiguous outcomes within a day, and were scored with a rule that rewarded honesty. They also had statistical guidance to lean on. Which of these conditions mattered most is not settled by these sources. Interpretation
Limits of the lesson. Calibration in a repetitive, fast-feedback domain does not show that calibration transfers to one-off judgments such as elections, wars or career choices. And calibration alone is not skill: a forecaster who always announced the seasonal average would also be calibrated, while telling you almost nothing about tomorrow.
Case file B · Washington, March 1951
“A serious possibility”
Context. In March 1951 the U.S. intelligence community issued National Intelligence Estimate 29-51 on whether the Soviet bloc might attack Yugoslavia that year. Its key judgment called an attack in 1951 a “serious possibility”. Sherman Kent, of the CIA’s Office of National Estimates, later told the story in a 1964 essay, “Words of Estimative Probability”, since declassified and published by the CIA’s Center for the Study of Intelligence.
What was documented. By Kent’s account, a senior State Department planner asked him a few days later what odds the phrase meant. Kent’s own answer was roughly 65 to 35 in favour of an attack; the planner and his colleagues had read it as much lower. When Kent polled the members of the Board of National Estimates who had agreed the wording, their private odds ranged from about 20% to about 80%. The same five words had concealed a disagreement the size of the whole probability scale. Primary historical source
Kent’s remedy. He and his colleague Max Foster proposed a short chart pairing words with numbers. Kent wanted estimators to show what he called a decent reticence before the unknowable
— and also to stop hiding behind vague words.
| Words | Probability | Give or take |
|---|---|---|
| Certain | 100% | — |
| Almost certain | 93% | about 6% |
| Probable | 75% | about 12% |
| Chances about even | 50% | about 10% |
| Probably not | 30% | about 10% |
| Almost certainly not | 7% | about 5% |
| Impossible | 0% | — |
Kent added a rule: “possible”, unmodified, should signal only that something is neither certain nor impossible, with no odds attached — and should never be inflated into “serious possibility”. He listed “likely” as a synonym of “probable”.
The opposition. Kent reported resistance from colleagues he nicknamed the “poets”, who held that rhetoric could only convey the general drift of a judgment, against his “mathematicians”. He conceded that readers who think like poets may not receive a precise message however carefully it is written. This is a genuine counterargument, not a strawman, and it survives in debates about whether numbers create false precision. Contested
Limits of the lesson. This is one episode recounted by one participant, written thirteen years later. It shows that a shared phrase can hide disagreement; it does not by itself show how often that happens, or whether numbers would have changed the policy.
5 Intellectual humility, properly sized
Calibration is humility you can count. It turns a vague virtue — “know what you don’t know” — into a checkable habit: claims at 90% should come true nine times in ten. But humility is broader than any score, and it has two failure modes, not one.
The limitations-owning account of Whitcomb and colleagues places intellectual humility between two vices. Intellectual arrogance ignores or denies one’s limitations. Intellectual servility overweights them — or underweights one’s genuine strengths — and defers when it shouldn’t. A scale built on this account, by Haggard and colleagues in 2018, describes humility as the mean between the two. Philosopher Ian Church (2017) has criticised the account for limitations of its own, which is a useful reminder that the concept is still being argued over. Philosophical account; contested
On the psychological side, Leary and colleagues (2017) reported that people scoring higher on their intellectual humility scale also scored higher on openness, curiosity and tolerance of ambiguity, and lower on dogmatism. These are correlations from self-report questionnaires; they do not show that humility causes better thinking, and self-reports of humility have an obvious awkwardness: the boastful and the modest may both misreport. Correlational findings Causal direction unknown
The underconfidence side deserves equal billing. In the simulator, gauges filled above their lines are a miss too. Underconfidence wastes information you have, cedes decisions to whoever speaks most loudly, and can keep good ideas out of the room. Porter and colleagues’ 2022 review lists “can someone be too intellectually humble?” as a question still open. Open question
6 Evidence and myth
| Common belief | What the evidence supports | Status |
|---|---|---|
| “The forecast said 70% and it didn’t happen, so the forecast was wrong.” | One outcome cannot verify one probability. A 70% forecast should fail about three times in ten; only a run of forecasts can be checked. | Established |
| “A calibrated forecaster is a good forecaster.” | Calibration is necessary, not sufficient. Always forecasting the base rate is calibrated with zero resolution (Murphy, 1973). | Established |
| “Everyone is overconfident about everything.” | Direction depends on the task and the definition: overestimation and overplacement can reverse on easy versus hard tasks; overprecision is the most persistent (Moore & Healy, 2008). | Established nuance |
| “Overconfidence is just a statistical illusion.” | Noise and item selection can create apparent overconfidence (Erev et al., 1994; Gigerenzer et al., 1991), but critics argue real overconfidence remains (Brenner, 2000). | Contested |
| “Training reliably fixes calibration.” | Probability training was associated with better accuracy in a large forecasting tournament (Mellers et al., 2014); a 2025 reanalysis argued those effects shrink, vanish or reverse once other variables are controlled (Hauenstein et al., 2025). The 1982 review found training gains limited. | Contested |
| “Everyone knows what ‘likely’ means.” | In Kent’s 1951 episode one agreed phrase hid private readings from about 20% to 80%. | Documented case |
| “Humble people don’t hold strong views.” | Philosophical accounts treat humility as owning limitations, not abandoning conviction; servility is a failure too (Whitcomb et al., 2017). | Definitions contested |
7 Counterarguments worth keeping
- Unique events have no frequency. “Will this marriage last?” happens once. A frequentist can deny that a probability means anything here. Calibration offers a partial answer — judge the forecaster across many unique events — but it evaluates the person, not the single claim.
- Numbers can create false precision. Kent’s “poets” had a point: “68%” can sound more knowledgeable than the evidence allows. Ranges, or Kent’s own coarse bands, are one compromise.
- Confidence can be useful. Some argue that overconfidence motivates effort or persuades others. Don Moore’s 2020 book Perfectly Confident argues against relying on that, but the broad question of when miscalibrated confidence pays off socially is not settled by the sources here. Open question
- Quiz skill may not transfer. Calibration on trivia is a different task from calibration about your own career, health or relationships, where feedback is slow and motives are strong.
- Humility can be performed or demanded unfairly. Announcing uncertainty can become a style rather than a practice, and calls for humility are sometimes directed only at people with less power.
8 Ethical limits
- No grading of people. The scores here are illustrative and unvalidated. Do not use them to assess students, employees, candidates or partners.
- No diagnosing motives. A gap between confidence and accuracy on one quiz says nothing about why. “You’re just overconfident” is often a way of winning an argument rather than examining one.
- Humility is not false balance. Being open to error does not oblige anyone to treat every claim as equally supported. Calibration means high confidence where evidence is strong, too.
- Uncertainty can be weaponised. Emphasising doubt can be used to delay decisions that the evidence already supports. Asking “how sure, and on what basis?” guards against both false certainty and manufactured doubt. Interpretation
- Privacy. Nothing you enter leaves your browser. There are no analytics, accounts or forms that send data anywhere.
9 What this laboratory does not prove
- Your gauges do not measure a stable trait. They summarise a small, author-chosen set of questions answered once, today.
- A gap inside the luck band is not evidence of good or bad calibration; even outside it, the deck may simply be unrepresentative (Lens B).
- The Noise Machine shows that error can mimic overconfidence. It does not show how much real overconfidence is noise.
- The Phrase Prism compares you with one analyst’s 1964 proposal. It does not show how professionals read these words today.
- Nothing here establishes that intellectual humility causes better decisions; the cited studies are largely correlational or conceptual.
- The scores are not validated psychometric instruments and must not be used for selection, grading or diagnosis.