Three customers complained. The product manager called an emergency review. The team spent a day reassessing a feature that had been working well for thousands of users.

Two customers praised it the following week. The emergency review was quietly deprioritised.

Both responses treated three data points as if they were a meaningful signal. The actual pattern - across the full population of users - had not changed either time. What changed was the last small sample visible to the room.

What it is

Insensitivity to sample size is the tendency to underestimate the variability that can occur in small samples, leading to overconfident judgments based on limited data. When a small sample produces a result that looks like a pattern, people tend to treat it as informative regardless of the sample's size - as if three observations carry the same evidential weight as three hundred.

The effect runs in both directions: people draw strong positive conclusions from a small number of good results, and strong negative conclusions from a small number of bad ones. In neither case do they sufficiently discount for the fact that extreme outcomes are expected to occur frequently in small samples even when the underlying population is entirely average.

The mechanism

The bias arises from the representativeness heuristic: individuals judge the likelihood of an event by how closely it resembles their mental model, ignoring the effect of sample size on sampling variability. If a sample looks like a representative outcome - even though it is tiny - people treat it as confirming evidence of a broader truth.

Research on judgment and decision-making demonstrated that participants' probability judgments were insensitive to sample size, treating small samples as if they were as informative as large ones. The mental move is to ask "does this sample look like what I would expect?" rather than "given this sample size, how much random variation should I expect?" The first question leads people astray because small samples can easily produce extreme results by chance - and those extreme results can look convincing without being reliable.

The common mistake

A manager observes a new sales approach used three times. It succeeds all three times. They roll it out to the whole team.

What they have is three data points. Three successes in a row is consistent with a 50% success rate - that outcome has a 1-in-8 chance of occurring by chance even if the approach works only half the time. It is also consistent with a 90% success rate. The three observations do not discriminate between these possibilities in any meaningful way. Yet they feel like evidence.

The mistake is not observing. The mistake is drawing a conclusion with a confidence that the sample size does not support.

The better move: treat small samples as preliminary evidence that warrants attention rather than conclusive evidence that warrants action. Ask: Given this sample size, what range of outcomes would I expect to see even if nothing has changed? If the observed result falls within that range, the sample is not telling you much.

Where it shows up

Medical and clinical judgments. A clinician who sees three unusual cases in quick succession may revise their mental model of how common a condition is - when the run may be coincidence. Small case clusters look like patterns even when they are not.

Performance evaluation. Evaluating an employee after a short run of visible successes or failures gives the impression of a meaningful signal when the sample may be too small to separate genuine performance from chance variation.

Consumer and market research. A product tested with a small group of users produces results that are reported with a precision that the sample size does not support. The confidence intervals on small samples are wide - a finding that looks like a 20-point difference might swing 30 points in either direction with a larger sample.

Everyday social judgment. Encountering two or three members of a group who behave in a particular way leads to an inference about the whole group - when those two or three observations carry almost no inferential power about a population.

What it is not

It is not a belief that small samples are always better. The bias is not that people explicitly prefer small samples. It is that they fail to discount for the increased variability that small samples produce. A person who knows intellectually that small samples are unreliable may still respond to a small sample finding with the same confidence they would apply to a large one.

It is not the same as the availability heuristic. The availability heuristic concerns how easily examples come to mind. Insensitivity to sample size concerns how much weight is given to whatever sample is available, regardless of its size.

Sources