Predicting the Improbable

From VoxEU:

Japan’s trio of tsunami, earthquake, and nuclear disaster has left the world stunned. As this column points out, even the experts were shocked. But while these events were highly unlikely, they were still possible. This column uses evidence from the Danish lottery to show that people tend to adjust their expectations of future events based on only small pockets of recent experience, often at their cost.

Important events are hard to predict – a fact that is particularly hard-felt when it comes to low probability events with dramatic consequences. Nuclear catastrophe, financial crisis and the like are things that even experts struggle to predict. The difficulty stems from a lack of understanding of the underlying factors and complex interactions among causes (probabilities are not independent but conditional on other events).
Experts are thus to some extent forced to base their predictions on inference from observing the past. A difficult issue is to know when a model should be revised given that an event that has been deemed to be highly improbable happens to occur. The issue is most relevant for policy recommendations. For example, what recommendations should experts provide for the regulation of nuclear power in the wake of the Fukushima disaster or for the regulation of banks in the light of the recent financial crisis?
While experts struggle to predict such events accurately, the average person is often simply baffled. They tend to misperceive randomness in a variety of ways, especially when it comes to rare events.

• One common tendency is to see patterns in random data when there are none.

This can lead to a tendency to overreact to recent events, allowing their occurrence to change beliefs about future events in exaggerated ways. More specifically, many people tend to over-infer characteristics of the underlying probability distribution when observing a small number of random events. A literature pioneered by Tversky and Kahneman (1971) has identified the belief in the “law of small numbers” as the source of such over-inference.

Why do people “over infer” from recent events?

There are two plausible but apparently contradicting intuitions about how people over-infer from observing recent events.

• The “gambler’s fallacy” claims that people expect rapid reversion to the mean....MORE

HT: Simoleon Sense