Sample Size Neglect

Sample size neglect occurs when an individual infers too much from a small new sample of information. It is described by behavioral finance as a cognitive error, and can lead people to make poor judgments based on inadequate information. The underlying cause of sample size neglect is that much of the time people fail to understand that more variance is likely to occur in smaller samples. Knowing what a large enough sample size is can be an issue for people who do not have a full understanding of statistical methods.

When a sample size is too small, accurate and trustworthy conclusions cannot be drawn. An example is an experiment where people were presented with the following scenario:

A person draws from a sample of five balls, and finds that four are red and one is green
A person draws from a sample of 20 balls, and finds that 12 are red and eight are green
Which sample provides better evidence that the balls are predominantly red?
Most people said that the first, smaller sample provides much stronger evidence because the ratio (4:1) was so much higher. But, in fact, the high ratio is outweighed by the much smaller sample size. The sample of 20 actually provides much stronger evidence.

In another oft-referenced experiment people were presented with the following:

A certain town is served by two hospitals — In the larger hospital an average of 45 babies are born each day, and in the smaller hospital about 15 babies are born. Roughly 50% of all babies are boys. However, the actual percentage fluctuates from day to day. During one year, each hospital recorded the days on which more than 60% of the babies happened to be boys. Which hospital recorded more such days?

Statistically, the larger sample will show less total variability than a smaller sample. Therefore, the smaller hospital will observe more days on which the number of boys born deviates significantly from the expected average. However, 22% of respondents said the larger hospital would experience more such days, and 56% said that the results would be the same for both hospitals. Most people responded incorrectly, and half of those respondents had the incorrect intuition that larger samples exhibit more variability.