Saturday, May 1, 2021

Winton's Chief Science Advisor: What are the chances? Can We Summon the Improbable?

 From Aeon, June 16, 2014:

David Hand is Emeritus Professor of Mathematics at Imperial College, London, and chief scientific advisor to Winton Capital Management. His latest book is The Improbability Principle (2014). 

What are the chances?
From a big lottery win to a disaster like the Titanic, unlikely stuff happens all the time. But can we summon it at will? 

Statisticians tell us that the chances of winning the lottery are incredibly small – for the UK National Lottery, for example, around one in 14 million per ticket. That’s about the same probability as seeing a flipped fair coin come up 24 heads in a row, and far less than your chances of being killed by a meteorite. And yet, week after week, people do win, supplying newspapers with a constant flow of personal-interest stories into the bargain. What’s going on? How can something that has such an incredibly small chance keep on happening?

The explanation is, of course, straightforward. The chance that your ticket is the winner is indeed small. But you’re not the only person entering the draw. In fact, many people buy lottery tickets each week. Often, they buy more than one. So, overall, a very large number of tickets are bought. And while each ticket might have a very small chance of winning, if we add up that very large number of very small chances, it soon amounts to something sizeable. With enough people buying enough tickets, we should actively expect to see someone win.

This distinction – between the chance that you (or, indeed, any other particular person) will win the lottery and that someone will win – is a manifestation of what I call the law of truly large numbers. If a large enough number of people each buy a lottery ticket, then the probability that someone will win becomes substantial. It grows so large, indeed, that someone wins almost every week.

This law is part of what I have called ‘the improbability principle’. The principle states that extremely improbable occurrences are, in spite of the odds against them, actually quite common. It says we should expect to see events that we might regard as incredibly unlikely – such as someone winning the lottery. The improbability principle consists of five elements, of which the law of truly large numbers is just one.

Allow me to introduce its partners in – not crime exactly, but… Well. You’ll see.

The law of inevitability says that some outcome must occur – one of the 14 million sets of six numbers from one to 49 must be chosen when the lottery balls drop. So, if you bought all possible combinations, you’d be certain to hold the jackpot-winning ticket. That sounds trivial but, of course, people have still found a way to make money out of it.

The law of selection says, in effect, that while prediction might be hard, postdiction is easy. It’s easy to look back and see the causal chain that led inexorably to disaster. It’s not so easy to choose among the multitude of possible chains that lead into the future.

The law of near enough says that you can dramatically increase the chances of a coincidence if you broaden what you mean by a coincidence. You would be surprised to encounter an old friend in a strange town, perhaps, but you might be almost as surprised if you met a friend of a friend, even though friends of friends heavily outnumber friends.

Finally, the law of the probability lever says that slight changes can make highly improbable events almost certain. Thus we encounter financial crashes, positive results in ESP experiments, people being repeatedly struck by lightning and so on....