Exploring the Mathematics of Luck
With the next US election a few days away, you would be hard put to find an expert who can be persuasively certain about the outcome. Despite all the data crunching and AI smarts, uncertainty shadows our lives, as it does the fate of the planet.
But rarely has an academic paid such singular attention to the topic. And not just with a dry statistical eye, but with a poet’s empathy for how uncertainty can unsettle us. David Spiegelhalter, currently a Professor at the University of Cambridge, starts with his own birth. Of how chancy it was. Of how his grandfather miraculously survived many near-death encounters.
As a 35-year-old gas officer inspecting a danger-riddled landscape in 1918 – when World War I was still raging – there were many occasions on which David’s grandpa evaded being ripped into smithereens. On the 29th January of that year, he wasn’t as lucky. He was, as he later recorded in a diary, “blown up.” If you think his luck had run dry, Spiegelhalter suggests that the short-term misfortune might have been the opposite: a life saver. Because that injury permitted him to stay behind the scenes, and survive the war. His life was marked by other such freakish escapes.
Then there was the unlikely meeting of David’s parents during the war, and his father’s emergence from tuberculosis and an almost plane crash. Till Spiegelhalter could claim, “and here I am.”
The Role of Micro-Contingencies
Such “micro-contingencies” or chance events in a particular order have led to all our births. If we each exhume our pasts, we would be astonished that we came to be. “But what underlies or drives this fragile chain of events?” David’s less interested in examining whether everything is pre-ordained or utterly random. Either way, a common aspect threaded through all scenarios is uncertainty: “This constant state of uncertainty is an essential part of the human condition.”
Uncertainty Through History
The past wasn’t different. In the Roman era, life expectancy was about 25 years. Globally, some eras have been more “volatile,” like the 1930s. Right now, there are new causes for jitters: wars, pandemics, climate change, threats to democracy. Folks also obsess about rising prices, health care, crime, unemployment, and poverty.
To top it all, jobs have become more unstable. When David graduated, he was optimistic about bagging a job and staying in it for many years. Which he did, working at the Medical Research Council for 32 years: “Now, job insecurity and the gig economy are the norm.”
Uncertainty Is a Relationship
As a mathematician, Speigelhalter has spent his life studying uncertainty, how to predict the future better, and manage what we can’t know. As he puts it, uncertainty is not a quantity or property that exists out there, ready to be captured and grappled with. Rather uncertainty is a “relationship between someone…and the outside world, so it depends on the subjective perspective and knowledge of the observer.” Folks react differently to it: some are excited by unknowns, while others get anxious.
The Puzzle of Luck
Probability emerged as a field of study only during the Renaissance. The Professor is surprised it took so long. After all, we’re fascinated by luck and coincidences. Take the case of Mr. and Mrs. Huntrodds of Whitby – their births, marriage, and deaths all occurred on the same date: September 19th.
Luck, too, can be sorted into types. The most important type of luck is “constitutive luck” – “who you were born as.” In general, we can agree more easily on what has happened but not as much on why.
Handling Unknown Unknowns
There’s not much point worrying about known unknowns, as there are also a more tremble-inducing category of “unknown unknowns.” The US Secretary of State, Donald Rumsfield defined the latter in 2002 as “the ones we don’t know we don’t know.”
Since uncertainty is girded by ignorance, we can distinguish between aleatory uncertainty – which centers around a future we cannot know. As opposed to epistemic uncertainty which hinges on what we currently do not know. For instance, if you’re already done with an exam, but don’t know the results yet, you would be dwelling in an epistemic sort of nail biting. Aleatory fears would have to do with an exam that you haven’t attempted yet.
The Psychology of Risk
Our fear of risks or hazards (defined as “the characteristics of a possible event”) tends to be more psychological than analytical or logical. We worry less about whether something will happen and more about what it will feel like if it does.
Assigning Numbers to Uncertainty
Words are subjective. Words like ‘likely’ can be very unclear. What does “likely” really mean? “Extremely likely” and “somewhat likely” mean different things to different people. In medicine, side effects felt by 1-10% of patients in the UK are termed “common”. It makes sense then to turn all this vagueness into numbers, which, at the very least, impart a collective understanding.
During the Bay of Pigs Cuban Missile Crisis, Brigadier General David Gray wrote a memo saying the U.S. invasion of Cuba had a “fair chance” of success. The CIA had actually estimated only a 30% chance of success. If President Kennedy had known this number, he might have decided not to invade Cuba.
Iffy Decisions: The Bin Laden Operation
Much later, when Obama’s advisors suggested invading Abbottabad in Pakistan to find Osama Bin Laden, they gave different probabilities about his presence there. Some said there was a 30-40% chance, while others believed it was an 80-90% chance. Obama concluded it was probably a 50-50 chance, meaning they weren’t sure if Bin Laden was there.
The next morning, Obama decided to go ahead with the mission to kill Bin Laden. The attack was precise and successful. Bin Laden was at the site, proving the higher probability estimates were correct.
This situation shows that even with numbers, there can be differences. David suggests that advisors proffering various probabilities might have helped Obama wrestle with uncertainty and make a more informed decision. Net-net, the operation was better planned than Bay of Pigs, which rested on hazy descriptors.
Elusive Odds
We should always remember that we can’t measure probability with a tool. It’s not something we can find exactly. Probability is a judgment and is always subjective.
In the 2016 American election, Nate Silver said Donald Trump had about a 28% chance of winning. This was a higher chance than most other pollsters gave Trump. However, Silver did not say Trump had a 70 or 80% chance of winning. After Trump won, people criticized Silver for not predicting the outcome accurately. But he was actually closer to the final result than many others.
David also wades into other stuff, like the Bayes Theorem, which calculates the conditional probability of an event – like when new evidence might reconfigure the chances of something occurring. Like if a dog is barking at night, is there a thief around? What if the dog always barks when a thief is around, but also sometimes when a thief is not around? Spiegelhalter makes all this refreshingly accessible, even for the innumerate or math-challenged.
Thinking Like Hedgehogs or Foxes
In his popular book Thinking Fast, Thinking Slow, Daniel Kahneman explains that we have two modes of thinking. System 1 is fast, emotional, and instinctual. System 2 is slower, logical, and more reflective.
While Spiegelhalter suggests we should adopt System 2 when it comes to uncertainty, he also uses a different analogy to divide thinkers. That of a fox or a hedgehog. Hedgehogs have one big view of the world. For example, they can be very religious, strongly libertarian, or extremely conservative. They filter observations through this one big idea. Because of this, they may not handle surprises or uncertainties well.
Foxes, on the other hand, are flexible. They are willing to change their minds and adapt. They stay open to different possibilities. Interestingly, uncertain foxes make better predictions than certain hedgehogs. Foxes tolerate events not unfolding as expected, while hedgehogs stick to their one big idea.
What Makes a Good Forecaster?
In his book Future Babble, journalist Dan Gardner discusses what makes someone good at predicting the future. Key habits or features include:
- Aggregation: Gathering and combining information from many sources.
- Metacognition: The ability to think about your own thinking.
- Humility: Being humble enough to admit when you’re wrong and quickly changing your views based on new evidence.
Playing With Uncertainty
Speigelhalter also conducts fun experiments. He uses a simulator to see how long it would take for monkeys to compose Shakespeare’s works (seems like too long to be feasible!). He evaluates if top football teams are bolstered by skill or luck. He explores ancient gambling by ordering a lamb chop from a butcher, to hurl a knucklebone in different directions.
Will this book make you more certain? Not sure. Regardless, we ought to adopt the mindset of the physicist Richard Feynman, who said: “I can live with doubt and uncertainty and not knowing.”
References
David Spiegelhalter, The Art of Uncertainty: How to Navigate Chance, Ignorance, Risk and Luck, Pelican, 2024