What do we mean when we say that an event has a 50% chance of happening?
One straightforward interpretation is that if we played out some situation 100 times, we’d expect that the event would occur 50 times. This approach, called the frequentist interpretation in the philosophical literature, is an intuitive way of thinking about, for example, the chance that a fair coin will land heads up when flipped. We can easily imagine flipping a coin 100 times to check empirically how often it comes up heads. Note that each flip is similar to the last, but not entirely the same–assuming the coin flipping process is deterministic, if we held the initial conditions fixed, we would expect the same result every time. From the Stanford Encyclopedia of Philosophy’s entry on the Philosophy of Statistics:
This leads to a central problem for frequentist probability, the so-called reference class problem: it is not clear what class to associate with an individual event or item (cf. Reichenbach 1949, Hajek 2007). One may argue that the class needs to be as narrow as it can be, but in the extreme case of a singleton class of events, the chances of course trivialize to zero or one.
(Footnote about radioactive decay and quantum processes). That said, if we’re not being rigorous, it’s not too taxing to take these minor variations (height of the flip, wind conditions, ambient temperature, etc.) in stride and assume that we can all agree on what it means to flip a coin 100 times.
The same cannot be said for more complex (or time-sensitive) events, such as an earthquake hitting the San Francisco Bay area in the next 10 years, or someone surviving a cruise ship catastrophe. How would we reproduce these happenings? What would we need to hold fixed, and what could we change? Are such events amenable to a probabilistic description to begin with? If they are, whatever probabilities we ascribe to them don’t seem to make sense under the frequentist model described above.
In these cases, we can look to the models that produced those numbers to begin with to see what they mean. For instance, if our Titanic survival model made use of a Naive Bayes algorithm, we’d be mightily justified in making sense of it under a Bayesian (or subjective, or epistemic) interpretation. That is to say, the probability in question doesn’t represent some frequency of a physical process; rather, it’s more a measure of how strongly we believe in the outcome. Again from the SEP:
Probabilities may be taken to represent doxastic attitudes in the sense that they specify opinions about data and hypotheses of an idealized rational agent. The probability then expresses the strength or degree of belief… a view that places probabilities somewhere in the realm of the epistemic rather than the physical, i.e., not as part of a model of the world but rather as a means to model a representing system like the human mind.
There are other interpretations, e.g. ones involving a notion of “propensity”, but this is confusing enough as it is, so anyone interested can head to this Wikipedia article for a better explanation.
A somewhat orthogonal consideration is how to measure the accuracy of models that output probabilities… for instance a model could say person A has a 68% chance of surviving, which is enough to classify them as a survivor, and we can say the model itself is 76% accurate based on how it performs on historical data… but that’s a discussion for a different post.
P.S. My preferred take on this episode of Black Mirror (spoiler alert, sort of?) has it engaging with the frequentist interpretation as it pertains to relationships; maybe not saying much about it, but engaging with it nonetheless. The system in this episode could, on a very positive reading, be used to create couplings that have the best qualities of love-based relationships (e.g., a sense of agency, getting to pick who you want), and arranged marriages (enumerated in this article). But again, I digress.