The most recent addition to my read shelf on GoodReads is a thick book called Signal and Noise by Nate Silver. Due to its length (450 pages of text, over 100 pages of references) and large hard cover Czech version it isn’t a book you can read on a train so it takes time get through it.

The book was recommended to me in a context of a discussion about Gleick’s Information: History. Theory. Flood. and Taleb’s books last year. I had absolutely no idea who the author is but apparently Silver is some sort of public figure with his own share of controversy and criticism.

The topic of the book are predictions in all fields and forms imaginable. Silver opens with a brief analysis of the economic crisis which covers poor predictions by investors and private forecasters, absurd position of rating agencies and poor predictions of impacts of measures taken by public authorities. The key message is that it’s very hard to predict things that have never happened in the past (black swans) especially if you’re personally interested in not recognizing them and riding the way as far as possible no matter what.

The following chapter covers his current topic of interest: predictions in politics. Three signs of bad predictions are covered:

1) Black & white predictions 2) Lack of revisions of the predictions 3) Individual predictions far away from from the mainstream just for the sake of publicity

Another point that gets repeated throughout the book is a lack of accountability. No one scores TV commentators on accuracy of their predictions. If they’re wrong nobody cares, if they’re correct they can bask in the limelight (especially when they have an outlandish prediction).

Next, unwieldy fields of politics and economy are contrasted with a very structured baseball which offers significantly better conditions for predicting the careers of individual players, results of games or the whole teams. I know next to nothing about baseball but the argument about structured environment, rules and the emphasis of mental aspects (teams in the highest competitions apparently play several times a week) of the players make sense.

Independent weather forecasting is the winner of the book. Sort of. Even though the mechanisms behind weather are pretty well known on small scale, accurate forecasting is hard due to the complexity and dynamic nature of the problem (Lorentz, butterfly effect, etc.). Many fold growth of the computing power used for weather forecasting has translated into more accurate short term forecasts in the last decades. However, the improvement is in the range of tens of percents, not hundreds. An important aspect of commercial weather forecasts is their tendency to predict more rain with a true pessimistic logic - if it rains, we’re covered, if it doesn’t people are happy.

Silver moves from weather to earthquakes. The large scale predictions and expectations based on the Gutenberg-Richer law are incredibly precise but only in the sense that earthquakes of any given magnitude come on average so and so often. The law doesn’t help in predicting when and where will the next earthquake occur. Every model so far has failed by either predicting earthquakes that never came or by missing some that did. Many of those fail into the trap of overfitting, i.e. trying to explain all past data points as much as possible and losing the signal in the noise.

The following chapter takes a stab at economists and their inability to predict macroeconomic indicators. Silver identifies inaccurate data, too black & white sources and complex, dynamic and highly social structure of the economy as the root causes. As a solution, he suggests prediction markets which would help realign the interests of people and organizations predicting the future and consumers of their predictions. The social aspect of systems is discussed more thoroughly in relation with the spread of diseases. Any prediction about a system with a social aspect which is made public to the people who constitute the system can become self fulfilling or self depreciating by people acting upon that information. Even the growing computational power doesn’t help much with agent based modelling of disease outbreaks.

The second part of the book turns from description of fields where predictions are more or less successful to the future and ways how to make predictions better. Bayes’ theorem stands out as a clear winner. Silver reiterates the benefits of stating one’s beliefs prior to evaluating evidence. Just making the prior belief explicit helps consumers of the prediction understand where it’s coming from as opposed to guessing as usual. He also goes through the typical examples of positive/negative test results in medical diagnosis and exemplifies Bayes’ theorem on basketball betting.

Some people love to play chess. There are computers. Therefore, people play against computers. It seems obvious today but the lure of beating the mechanical opponent has existed even before the first computer was constructed (and people played against the Mechanical Turk). The interesting point about this is that computers and people have diametrically different strengths and need to adjust their styles when playing each other. Since chess is very structured and doesn’t change much there’s a long record of games on all levels of proficiency which human and computer players use for learning the game. The weakness of human players is that they can’t access the database in a perfect way. They remember strengths of different openings and defenses against each other (in form of tactics and probabilities of different outcomes) but they don’t necessarily know all the games played over decades move by move. Computers do. Similarly human players know the endings, which combinations of pieces can win and even the blueprint for reaching the desired outcome. Computers can compute these precisely in milliseconds. The one part where humans have the upper hand is the middle game. Computers are lost in the number of possible moves in each turn and have to prune their options in that phase. Humans have too but they are equipped with imagination and have a feel for the game which enables them to concentrate only on the best moves. Still it’s very hard to beat a computer.

Poker is a great example of probability based game which enjoyed a huge boom in the recent years. It allowed some people to get rich and gave the possibility of getting rich to others. Silver illustrates the power and perils of prediction on playing poker. The game has clear rules but chance plays a huge role. So much so that it takes thousands of games to see whether a player is skilled or just lucky. The whole environment is filled with both natural noise and noise created by other players trying to cover their signals. Even though other fields might not be exactly the same the crushing noise is common.

In the next chapter Silver comes back to financial markets which are in a sense prediction markets. He uses Eugene Fama’s arguments to explain the theory of efficient markets by which markets reflect all information available and therefore no one can consistently outperform them. He criticizes technical analysis (staring at charts) and even fundamental analysis. The only approach discussed in a positive light is Shiller’s long term investment based on P/E ratio. To add to the mix he covers herd mentality, high volume trading (burdened by transactional costs) and other common topics. He concludes that financial markets aren’t perfect but they’re the best we have.

In the next to the last chapter Silver tackles the predictions of climate and climate change. He goes through a number of them and tries to measure their accuracy. His key point is that scientist don’t doubt the mechanism of global warming but due to the complexity of the global environment, quality of data and the length of the period any reasonable prediction must look ahead it’s impossible to confidently predict anything other than basic ideas. There’s a structural uncertainty caused by the complexity, uncertainty stemming from poor quality of input data which looking further and further away smooths down but is replaced by uncertainty of the development and impacts of any actions that might be taken in the meantime. Politicizing of the topic which forces clear cut predictions doesn’t help either and causes more confusion.

Silver circles back to finding signal in the noise in the last chapter. The example he starts off with is the attacks on Pearl Harbor and goes on to 9/11 both of which are very clear in the hindsight but weren’t predicted ahead of time because of wrong assumptions and lack of imagination (why would Japan wake the sleeping giant? planes are planes, not weapons). Based on the number of terrorist attacks and the number of victims he finds similarities with the Gutenberg-Richter law and a success story in the form of Izrael which managed to avoid large scale attacks, maybe thanks to its policies. The ultimate problem with preventing terrorism is confusing improbable with unknown. It’s generally easier to deal with known unknowns than unknown unknowns which are of course those that (smart) terrorists will use.

The ultimate message of the book is to become aware of one’s assumptions by stating them out loud ahead of time and adjusting them in the light of new evidence. The world isn’t simply black & white and probabilities are everywhere. Some of them are near certainty and some of them close to zero - those two types can bite the most. Expressing uncertainties in predictions isn’t a sign of weakness but of understanding the limits of one’s knowledge. Bad predictions are created by people who are motivated to make them bad for their profit or who are forced to make strictly binary predictions. Probabilistic approach doesn’t feel natural but is the best way when dealing with anything that actually matters.

What I liked about the book is the focus on assumptions and motivations of people who can cause havoc and explain a lot. Some of the examples were foreign to me (sports) but everything was described very clearly and understandably. I consider this book a good read and I’d recommend it to anyone interested in the world.