From Aesop's fable: the hedgehog and the fox
I’ve always been skeptical of people who act confident they know what’s next. Prophesying anything but the simplest extrapolations always sounded alarm bells in my mind. The world seemed far too complex and opaque to be able to peer into the future with any degree of accuracy, to the point where I’ve often wondered if there was something I was missing and that maybe I was the only one who thought of this exercise to be mostly nonsense. This is why I’m attracted to thinkers like Nassim Taleb and why books like Phillip Tetlock’s “Expert Political Judgement” always speak to me. They reconfirm my skepticism about prediction and keep me from falling into the predictive trap which narrows the possibilities in our mind to the point where we’re stunned when the unexpected happens.
Tetlock’s book is the most thoughtful and thorough attempt to judge predictions I’ve ever come across. While most write short academic papers, Tetlock went above and beyond by creating a 250 page book detailing his findings and objections, followed by 100 pages of appendices in which he explains his methods in great statistical detail. The study itself is massive: 20 years of predictions from over 1,500 experts in politics and economics around a variety of real world political events, from the fall of the Soviet Union to the possible secession of Quebec from the rest of Canada. Tetlock is rigorous in both method and accountability, and gives ample opportunity (some might say too much) for those that argue against its conclusions.
Before we get to the results, we must make a crucial distinction about how people think. In fact, one of the principal discoveries of Tetlock’s book is that how people think trumps what people think in terms of judging who is a good forecaster. It made no difference if a person was liberal or conservative, or whether a person was a boomster or doomster when judging forecast accuracy. What did matter is the distinction between what Tetlock calls “foxes” and “hedgehogs”, an idea borrowed from an essay by 20th century philosopher Isaiah Berlin. The fox is the thinker who knows many things. He accepts the abstractness of the future and considers ambiguity and contradictions as inevitable features of life. The hedgehog on the other hand knows one big thing. He is less tolerant of abstraction and works devotedly within one tradition, reaching for “formulaic solutions to ill-defined problems.”
We just can't predict
Tetlock’s study concludes that when forecasting, it is better to think like a fox than a hedgehog. Foxes dominated hedgehogs in forecast accuracy in most comparisons. This result does not surprise me. It is a known fact that our world, especially our social world, is becoming more complex each passing year as we become more interconnected. The result is that there are far more variables to predict than there used to be. Foxes are better able to accept this ambiguity and avoid making bold claims based on relatively shallow cause and effect. They better realized how little they actually know and deal with ambiguity in the messy fashion it must be dealt in. Hedgehogs deal with randomness by trying to eliminating it. They displace ambiguity by applying the theory they trust best to the events of the real world, and as Nassim Taleb has demonstrated, going from theory to reality instead of the inverse is dangerous and leads to more errors combined with more confidence (overconfidence).
The other principal discovery Tetlock makes, which to me was just as important than the distinction in thought processes, is the stubborn persistence that when it comes to predicting the future we all suck, from cab drivers to top political scientists. Both foxes and hedgehogs barely outrank the proverbial chimp throwing darts (assigning each outcome the same chance of occurrence), with hedgehog predictions coming dangerously close to being no better than random. When basic predictive algorithms based on extrapolation are tossed in the mix, it gets even uglier: the algorithms soundly beat the experts in every measure of forecast accuracy Tetlock puts forth. The results, while leaving some room for debate, are quiet convincing: we just can’t predict.
So why do experts still get paid so much to make predictions? If it’s clear that their predictions hold such little value, why are we still listening? Some proposed reasons include:
- There’s an intuition in all of us to accept something that appears to limit uncertainty: One of the fundamental needs of humans is to simplify complexity and abstractness into concrete statements. When someone tells us a convincing story containing extensive knowledge (an expert’s prediction) our first response is to believe, not increase our skepticism. It takes mental effort to be skeptical and avoid narrative traps, and most people would rather believe in the certainty of future events than to believe the world is too random for us to properly deduce.
- Environments seem less complex than they really are: We have trouble recognizing that as we add variables to the equation, we need to be more and more precise in the possible actions of these variables to reach a confident forecast. Once an environment reaches the complexity of an international political outcome, where many different variables enter into the equation, small errors in the forecasted properties of these variables multiplied by numerous interactions can result in huge error rates, which is why economic forecasts studied for accuracy are found to have massive error rates.
- Seersucker theory: a paper by Armstrong (1980) proposes that “No matter how much evidence exists that seers do not exist, suckers will pay for the existence of seers.” This argument basically says we humans are suckers for prophesy and will continue our refusal of empirical evidence. No matter how much evidence we stack up against forecasters, they will continue to thrive because our minds have a weak spot for certainty, no matter how false that certainty is in reality.
- It’s easy to be “right” and hard to be “wrong” in forecasting: Tetlock documents numerous excuses invoked by forecasters when their predictions came up empty and all of them are designed to hide the fact that they were wrong. Not only is there a massive asymmetry between accounts of getting it right and accounts of getting it wrong (people yell from rooftops if they nail a prediction and are silent when they whiff) but even when an expert gets it wrong they have a plethora of tools to help them avoid ridicule. The common excuses Tetlock documents include the “close call counter-factual” where experts claim the situation they predicted was tantalizing close to becoming reality or the “I made the right mistake” excuse where experts admit they may have been wrong, but it was much more prudent to be wrong where they were. This excuse was invoked numerous times by forecasters who predicted the Soviet Union to be more powerful than they turned out to be; forecasters insisted it was better to be prepared for a strong U.S.S.R than to dismiss them as weak. The last common excuse was the “off on timing” excuse, where the expert insisted that while the timing of their prediction may have been off, the prediction itself was correct and will be validated in the future.
Tetlock’s book accomplishes a considerable amount. He sheds some light on how we predict, how well we predict, the diminishing returns of increased knowledge in prediction, common excuses for prediction error, and a starting point for additional research to refine or refute his study. As it stands, Tetlock sets the current standard for judging judgement in the social sciences, and puts some hard quantitative data to this vague business of prediction. If you rely on experts forecast’s as part of your decision making, you owe it to yourself to learn from Tetlock’s account.
