What is probability?

I haven't seen any particularly interesting (or unusually egregious) uses of statistics in the last few days, so it's time to have a more philosophical outing.  Of course, if you see anything in the news you think is worthy of a post, email me (evans [at] stats [dot] ox [dot] ac [dot] uk).

So to the title of the post - what is probability?  Probability can be viewed as just a mathematical construct, which we'll explore some other time, but even with the mathematical rules in place, there are various interpretations of what a probability actually is.  At school we usually think about rolling dice and flipping coins, so let's start there.  I flip a 50p coin: what's the probability that it comes up heads?  You'd probably [sorry] say it's 'evens', 'a half', 'fifty-fifty', or (as mathematicians generally prefer), '0.5'.

Frequentism

What do we mean by 0.5?  Well a standard explanation might go something like: 'if we flip the coin a large number of times (e.g. 1000), approximately half the time it will come up heads (about 500 times)'.  This is usually called a frequentist interpretation of probability, and when applied to the example of a coin, it seems very natural.  The essence is that if a more or less identical experiment is repeated a large number of times, the probability of an outcome, O (e.g. O = {'the coin shows heads'}), is the proportion of times we expect to see O occur.

It's easy to see how this interpretation applies to flips of a coin or rolls of a die, but what about (in my view) less mundane events?  How would I assess the probability that Chelsea win their next game?  If I were entirely naive about the nature of football, or have no information about Chelsea's opponents, the importance of the game, and so on, then I'd want to know how often Chelsea have won recently.  As it happens, they have won 34 out of their last 61 games, so a plausible sounding estimate for the probability of them winning the next game might be 34/61 = 0.557.

A bookmaker might not be too impressed with this methodology; it doesn't take into account Chelsea's opponents, for example, which we know is very important.  Suppose I tell you it's Swansea City; Chelsea have played Swansea twice in recent times, they won once, and drew once.  Does that mean there's a 50% chance Chelsea will win this time?  And no chance whatsoever of them losing?  That would not be a satisfactory prediction, and it goes against the 'large number of times' aspect of frequentism that we mentioned earlier.

Indeed what if were Chelsea playing a team they've never played before - do we have any information with which to predict the result?  What if their star player is out injured?  Can we take this into account?  The answer is, of course, that we can, but we would have to construct a model; this is a mathematical framework which somehow describes (in a highly simplified manner) which aspects of Chelsea, and of their opponents, and of the particular match, we think make them more or less likely to win.  We'll talk about modelling another time.

The frequentist paradigm starts to seem rather less natural when we are considering 'one-off' events.  It's possible to imagine Chelsea and Swansea playing each other lots of times, under similar circumstances, and that some proportion of the time Chelsea will win.  But in reality, these teams only play each other a few times a year, and after a while their line-ups will have changed considerably, so the circumstances aren't really the same each time.

More extreme examples are easy to construct.  What's the probability that Usain Bolt will break both the men's 100m and 200m records at the 2012 Olympics?  According to William Hill, it's '7 to 2', or a probability of 2/(2+7) = 2/9 = 0.222.... (aside: interestingly they reckon he's more likely to break both records than to break precisely one).  Surely this event is in some sense unique - the 2012 Olympics occur only once.  In addition few other events are, to an athlete, anything like the Olympics, so there is pretty much no replication of the conditions under which he can break the record.  In addition, if he breaks once it will alter the chances of him doing it on another occasion, since the record will have changed!

A die-hard frequentist might argue that one can imagine an infinite population of different possibilities for what will happen in the 2012 Olympics, and that in some proportion of these purely hypothetical scenarios, Bolt breaks the record.  This proportion is the probability we should assign to the event.  It's not an explanation I find particularly convincing or aesthetically pleasing.

Subjective Probability

An alternative view says that a probability is just the personal, subjective opinion about the likelihood that an event will occur; this is the subjective or Bayesian interpretation.  Suppose you ask me the probability that it will rain in Seattle tomorrow.  Now, I'm not a weather forecaster, but having lived in Seattle for three years, I can say that it rains about 50% of the time (i.e. it rains some amount on half of all days), and so I might say 0.5.

My friend Charles still lives in Seattle, and he knows that today it didn't rain.  He knows that today's weather is a pretty good predictor for tomorrow's weather, so he can add this to his knowledge that it rains about half the time in Seattle.  Using this information, he comes to an estimate of about 0.3.  My other friend Will also used to live in Seattle (so he knows it rains half the time), but now he works at the National Center for Atmospheric Research in Boulder, Colorado.  He has access to weather data for today (so he also knows that it didn't rain today), but he also knows that a front of low pressure is moving in the direction of Seattle from the Pacific.  Using all this information, he thinks the chance of rain is 0.65.

Of the three subjective observers, none is 'wrong', but some have more information than others.  It's likely that Will would make better predictions on average than me, if I just said 50% chance of rain every time.  We might consider a forecast to be good if it's both well calibrated (i.e. 60% of the times you predict an event will occur with probability 0.6, the event happens) and sharp (the probabilities are generally close to 0 or 1).  (Probabilistic weather forecasting is terrifically useful, see the UW's excellent Probcast page for the Pacific North West of the US.)

The Great Divide

It would be perfectly possible to assign a frequentist interpretation in the weather example, but in my view it seems less natural: I'm sampling from the hypothetical population of all days, Charles from all days following a dry day, and Will from only those days which arise after observing precisely the weather data he has in front of him.  In addition to the subjective and frequentist ideas, there are other less common views of what probability is.

The Bayesian vs frequentist interpretation of probability (and consequently statistics) has become the prominent division in modern statistical thinking.  Personally, I don't think this apparent chasm in paradigms is as important as is sometimes suggested, and many statisticians are, like me, quite happy to use whichever seems most convenient at the time.  As we shall see in the future, each method has certain practical advantages and nice theoretical properties.

[This post was edited on 12th October 2014 to change the email address given.]