You are thinking about trying out the Enigmatic Egg - a fancy new restaurant in town. Since your friend Charlotte said she ate there and loved it you put it on your watch list. Later, when Tim said the same thing, you put it on your “must do” list. It feels like a sure bet because everybody says they love it.
But it’s not a sure bet because not everybody loves it. In the absence of other data points, two seems sufficient and does, in fact, feel like everybody.
The incorrect belief that a small sample is highly representative of a population is called the law of small numbers, or LOSN. It’s not actually a law but a cognitive bias. (The term was coined as a riff on the law of large numbers, which is a real statistical law).
In our example, you didn't make a decision based on a single data point (Charlotte’s opinion) but when you had two data points it suddenly felt like solid evidence. Unfortunately, a sample size of two is still too small for making a solid inference.
Don’t get put off by the term “sample size” which brings to mind polls, surveys and nerdy statisticians. We pull sample sizes from data sets every day to make decisions, and we usually only look at data that is readily available to us, such as what we have heard or experienced recently. When the data we consider is just a tiny subset of the whole, your brain creates the illusion that it is much more meaningful than it truly is.
It’s easy to see that bigger is better when it comes to sample sizes - the more data we have the better the chance that it accurately reflects the overall data set. But when it comes to small numbers, our intuitive response to overestimate its accuracy becomes a liability.
More irrationality
Another problem is that small samples produce extreme results more often, leading us to vastly incorrect conclusions. For example, let’s say our fictional restaurant is actually rather hit or miss and only half the people who eat there would give it a thumbs up. If you ask 2 people, there is a 50% chance that both will say the same thing and there is an equal chance of that response being thumbs up or thumbs down. The chance of 3 random people agreeing drops to 25%, 4 = 12.5%, 5 = 6.25% and so on.
The smaller the sample number, the more likely it is that your data is an anomaly instead of the rule.
In his popular book, Thinking Fast and Slow, Daniel Kahneman devotes a chapter to LOSN. He shows how we are not adequately sensitive to sample size because our subconscious brain (System 1) favors certainty over doubt. It “creates a rich image on the basis of scraps of evidence.”
Kahneman also says we humans have “a widespread misunderstanding of randomness.” If you flip a coin 6 times and it comes up heads every time, it is very difficult not to believe the coin has a proclivity to land heads up. Our lizard brains expect the outcomes to be more balanced, so it sees a pattern that doesn’t exist. We expect more evenness in our randomness, and small numbers are more likely to be uneven.
What to do about it
Of course, not all decisions require hunting down more data points, but the more impactful the decision, the more important it is to avoid the law of small numbers. Here a couple of things to think about that will help you do just that:
When you (or your team) are making a big decision, think or talk about how many data points are influencing that decision. Is it a small number relative to the total population? For example, two or three people in a meeting may convey that they have heard a complaint about a product. Before assuming the product is faulty, get a bigger objective measure of customer experience.
Think about probabilities. Sometimes you do need to make a decision based on a small amount of anecdotal data because that is all that is reasonably available. Fine, but understand that there is a chance the data is not representative and may even reflect the opposite of what is in the larger population.
So go ahead and break the law! When it comes to the law of small numbers, be an outlaw. Resist your brain’s desire for certainty and your decision performance will improve.
Think well and be well.
- Steve Haffner
Want to learn more about improving your decision performance?
Click here for my free book, 7 Strategies for Making Better Decisions