On making predictions

If you're up for reading an economics post from the most verbose writer on the Internet, try this post from Scott Sumner.

Yes, but if we don’t have standards, if we aren’t going to hold people to their words, then what do we really have? Suppose I said; “I predict a major 20% to 30% drop in the S&P500 within the next 4 years. And if it doesn’t happen, but happens sometime later, I should still get credit.” Would you take me seriously? People don’t seem to understand that unless a prediction is both accurate and timely, it really isn’t of much value.

I would phrase this differently. I would say that the timeliness of a prediction is part of its accuracy.

If I predicted every year from 1919 through 2004 that the Boston Red Sox would win the World Series, should anyone give me credit for a successful prediction in 2004 despite being wrong in each and every other year?

The answer, of course, is that it depends. If my predictions were based on an objective model with independently demonstrated accuracy, then, yes, I should be given credit. Of course, this is an extremely unlikely scenario, since we would probably never be able to identify a model that fails in a such a spectacular fashion as this one as actually being accurate. Nevertheless, it's the process that's important, not the results, because the process is what we can control ex ante. If I am known to have the correct (or most correct, given available information) prediction process, then the results don't matter; I have made the best prediction I could.*

On the other hand, if my predictions are unsystematic and wildly subjective, then I should be vigorously laughed at for making such predictions. I should receive no credit for a successful prediction in 2004. As the saying goes, "Even a stopped clock tells the right time twice a day." We don't give the stopped clock any credit for this.

I think most people know this intuitively, but I think a couple things happen that cause people to take unsystematic predictions more seriously than they ought to:

1. We have a cognitive bias that causes us to remember successful predictions more often than unsuccessful predictions. Unsuccessful predictions are everywhere. Successful predictions, especially successful predictions of really spectacular events, stand out to us. We then grant undue expert status to the successful predictor causing us to overweight his analysis in the future.
2. We are fooled into perceiving patterns and systems where none exist, giving us the illusion that we are operating systematically. For example, there were people who argued that the Yankees would never win a World Series with Alex Rodriguez for any number of reasons. Most if not all of the reasoning in these predictions involved extrapolating from small sets of data to proclaim large significant patterns. We see how well that worked out.

So you have to have a system, but you also have to let the system speak for itself. You can't say, "Well, I predicted the Red Sox would win sometime between 2000 and 2003, but they won in 2004... hey, I was pretty close!" If your system did not predict this, then you cannot call it a success because your system provided no useful information ex ante.

It's important to keep all of this in mind when you hear experts going on and on making wild predictions, whether those predictions are about baseball or economics. We must insist that we operate with in an objective, systematic framework or we will find ourselves falling victim to a whole host of epistemological charlatans and stopped clocks.

* Of course, the best way to evaluate or process is by the results it produces. A process that consistently produces poor results must eventually be rejected. The key is that any one outcome of a process is not sufficient evidence. We must have a large sample of unbiased outcomes before we can make a correct determination on the efficacy of a particular process.

**EDIT** See also: Robin Hanson. He's talking about economics, but the lessons are applicable everywhere.