Frequentist and Bayesian approaches to probability theory
One of the fascinating discussions concerns the Bayesian versus the Frequentist interpretations of probability.
To be clear, there isn't a clear-cut “correct” approach.
Mathematically, both are sound and self-consistent; both follow rigorous derivations, and practitioners from each will agree that the theorems from the other camp were correctly proven, to the extent that mathematical fact can be proven true.
Where they diverge is in how one interprets a result or observation. Better yet, with regard to which questions one can ask of probabilities. Let me bring this into more concrete terms.
Bayesian theory models its results from the point of view of what a rational individual ought to believe, if presented with the set of observations seen so far.
Say there are two competing explanations, hypothesis H1 and hypothesis H2. If we consider a set of observations “data”, a Bayesian will be interested in asking which is more probable:
P(H1 | data) vs P(H2 | data)
i.e., with the observed data being a given--it was already observed and, reasonably, cannot change--which one of the hypotheses is more likely to be true? One may then proceed to assign probabilities to H1 and H2, conditional on the data observed.
Frequentists, on the other hand, ask almost the inverse question. According to this way of thinking, what was observed was just a sample of what could have been observed. Moreover, hypotheses cannot be assigned probabilities; they can only supposed to be true or false. A Frequentist will, therefore, compare:
P(data | H1) vs P(data | H2)
i.e., first, let's suppose that hypothesis H1 is true; in this case, what would be the likelihood that I would have observed “data”? Conversely, let's then assume that H2 is true; in this case, what would be the likelihood of having observed “data”?