She had met him three times. The conversations were consistent - thoughtful, warm, considerate about small things. The picture that had formed was clear and favourable. She felt like she understood what kind of person he was.

Eight months later, the picture had become more complicated. Patterns that were not visible in three conversations were now visible in eight months. The coherence she had initially felt - the sense that she had a real read on him - had been confidence built on too small a sample, arranged into too neat a story.

What the illusion of validity is

The illusion of validity is the tendency to overestimate the accuracy of one's judgments, particularly when the available information feels coherent and internally consistent. When the pieces fit together into a clear narrative, the mind takes the coherence itself as evidence of accuracy - as though a story that makes sense must be a true one.

The problem is that coherence and accuracy are not the same thing. A small set of consistent data points can build a compelling picture while missing the dimensions that would complicate or contradict it. The feeling of having a clear read on a person, a situation, or an outcome is real. What is not guaranteed is that the read reflects the full picture.

The mechanism

The bias operates through narrative construction. When information is internally consistent - when one piece of evidence reinforces another, when the details align into a coherent account - the brain experiences this as a signal of truth. The coherence itself produces confidence. There is a sense of things falling into place, of a pattern being visible.

What this feeling does not track is how much of the relevant picture is missing. A coherent account built from three data points can be highly consistent internally while missing the fifteen data points that would revise it significantly. The consistency of what is seen tells you nothing about the significance of what is unseen.

Expertise does not protect against this. Experienced professionals - clinicians, interviewers, financial analysts - are also susceptible when the information they encounter is consistent. In fact, expertise can increase exposure to the bias because greater familiarity with patterns makes those patterns more compelling when they appear, regardless of sample size.

A recognition story

A hiring manager interviews a candidate. The candidate's preparation is thorough, their responses are articulate, and their answers cohere with each other in a way that feels genuine. By the end of the hour, the manager has formed a clear impression: this is a high performer.

The impression feels solid. It is based on a real interaction with a real person. The information is consistent. The narrative is compelling.

What the manager cannot see is that the interview - seventy minutes under optimal conditions, with a person who has prepared specifically for this setting - is a thin and highly curated slice of a much larger picture. The consistency that feels like evidence is internal to the sample, not a measure of the sample's completeness.

Research on structured versus unstructured interviews has consistently found that interviewers' confidence in their post-interview assessments exceeds the predictive validity those assessments actually have. The illusion of validity is one reason why.

Why it matters

In high-stakes contexts - hiring, diagnosis, investment, criminal justice - the illusion of validity generates confident conclusions from inadequate evidence. The decisions that follow are made with a degree of certainty that the underlying information does not support.

People tend to overestimate the accuracy of their judgments when information appears consistent or inter-correlated. The inter-correlation is part of the trap: when variables move together (a candidate who speaks confidently also answers precisely and presents themselves consistently), the redundancy feels like reinforcement. In reality, correlated signals carry less independent information than they appear to.

What it is not

The illusion of validity is not just overconfidence. Overconfidence is a broad tendency to be too certain. The illusion of validity is specific: it is confidence elevated by the coherence of available information, not confidence in general. Incoherent information can leave people appropriately uncertain; coherent information creates the feeling of certainty whether or not the certainty is warranted.

It is not eliminated by expertise. Experienced practitioners in fields requiring judgment - interviewers, diagnosticians, investment managers - are susceptible. The expertise provides a richer repertoire of patterns to recognise; the bias operates when those patterns appear and feel compelling regardless of the adequacy of the data.

It is not the same as the narrative fallacy. The narrative fallacy is the tendency to create causal stories to explain events. The illusion of validity is the tendency to mistake story coherence for predictive accuracy. The two overlap but are distinct.

Sources