Quantification of Statistical Evidence: P-Values, Likelihood Principle and Bayes Factors

1. Introduction

This paper presents a statistical philosophy that is focused on the quantification of evidence derived from observed data. The format of the paper departs from traditional presentations of statistical methodology. Our focus will be on the reasoning underlying the methods, offering a fresh development of several familiar statistical approaches. In particular, our development will eschew the notion that a null hypothesis is formulated merely to be rejected and will eschew the notion that type I and type II error probabilities are meaningful measures of evidence. Rather, we will conceptualize a hypothesis as a statistical model, with that hypothesis being judged according to its capacity to provide an accurate prediction. The remainder of this study is structured to systematically bridge frequentist and Bayesian perspectives on evidence. We begin in Section 2 by evaluating the p-value’s role as a metric for statistical evidence, followed by an exposition of the Likelihood Principle in Section 3 to establish a theoretical foundation. Section 4 examines the inferential implications of the likelihood ratio, which provides the necessary framework for Section 5 to formally link p-values with Bayes factors. After detailing the core properties of Bayes factors in Section 6, the paper concludes in Section 7 with a discussion on the versatile and often underestimated utility of p-values in broader hypothesis testing contexts.

2. P-Value as a Measure of Evidence

The p-value is a widely used tool for statistical inference. Nevertheless, the role of the p-value in formulating scientific conclusions is often misused. In this section, we shall elucidate how the p-value is optimally interpreted as a measure of evidence.

2.1. Statistical Evidence

We can say that data favor a hypothesis when that hypothesis provides a reasonable explanation for what has occurred. For example, let us consider a population of students in a school. For the general population, IQ scores follow a normal distribution with mean 100 and standard deviation 15. We may be interested in testing the hypothesis that IQ scores in the student population are distributed the same as in the general population.

To test the hypothesis that the mean IQ score for the student population matches the general population, a random sample of 25 students is selected and their IQ scores are measured.

Let x denote the sample mean IQ. If the hypothesis is correct, we predict that x will be generated from a normal distribution with mean ${\mu }_x=100$ and ${\sigma }_x={\displaystyle \frac{15}{\sqrt{25}}}=3.$ The observed data is then compatible with the hypothesis when the sample mean x occurs in a region of high probability under the predictive distribution.

Conversely, the data provides evidence against a hypothesis when what has been observed deviates from what has been predicted. We would not predict data x in the tail region of low probability if the hypothesis were true. Observing such data would provide evidence against the hypothesis. The further into the tail, the more our data deviates from the predictive distribution, and the stronger the evidence against the hypothesis becomes. Thus, a tail probability calculation would serve as a measure of evidence. Data incompatible to the hypothesized distribution is characterized by a small p-value.

Figure 1. Normal distribution for the general population.

Figure 1. Normal distribution for the general population.

For our example, suppose we observe $x=109$ as the sample mean IQ score for the students. The p-value will be calculated as the tail probability, p-value $=2\times P(z>3)=2[1-P(z\le 3\left)\right]=2[1-0.99865]=2\left(0.00135\right)=0.0027$ .

Figure 2. Predictive distribution and p-value calculation.

Figure 2. Predictive distribution and p-value calculation.

2.2. Evidence Versus Belief

Scientific evidence differs from scientific belief. A strong degree of belief is not built until multiple sources of evidence support or contradict a hypothesis. Since belief emerges from multiple studies and multiple researchers, the language for disseminating an advancement of scientific knowledge is the language of evidence. Therefore, quantification of evidence represents a core principle in statistical science.

The p-value represents a probabilistic measure of evidence, but not a probabilistic measure of belief. One must avoid the desire to interpret p as the probability that the hypothesis under examination is true. However, this leaves open the question of how to represent a p-value scale of evidence. Fisher recommended the scale displayed in Table 1. See [2] for further discussion.

The Fisher scale appears to be consistent with common p-value interpretations. In our example, we can assert there is between strong and overwhelming evidence against the hypothesis that the distribution of student IQ scores aligns with the general population.

3. The Likelihood Principle

A measure of evidence is said to satisfy the likelihood principle if the measure depends solely on the observed data and not on the experimental plan. The argument supporting the likelihood principle is that evidence measures should depend exclusively on the relevant information contained in the experimental results and not on the experimenter’s thoughts and intentions. The name derives from the fact that an evidence measure must depend exclusively on the likelihood function to satisfy the likelihood principle.

3.1. P-Value and Sampling Distribution

Since a p-value is computed based on the sampling distribution derived from the hypothesis being tested, the p-value as a measure of evidence depends on the experimental plan. Let us see with an example.

Consider a scientist investigating binomial probability $\theta$. Suppose $\theta$ is the probability that an experimental drug leads to a successful patient outcome. The goal is to test the hypothesized probability $\theta ={\displaystyle \frac{1}{2}}$. In the experimental study, 12 subjects are observed with 3 successes. To quantify the evidence from the experiment, we look to compute a p-value. If the experimental plan were to sample 12 subjects, the p-value would be computed based on the binomial distribution. The lower tail probability is computed for

$$X\sim \mathrm{BIN}\left(12,{\displaystyle \frac{1}{2}}\right)$$

as

$$P_0(X\le 3)={\displaystyle \sum _{i=9}^{12}}\left({\displaystyle \genfrac{}{}{0pt}{}{12}{i}}\right){\left({\displaystyle \frac{1}{2}}\right)}^{12}=0.0730$$

giving the two-tail p-value as

$$p=2\left(0.073\right)=0.146.$$

Using the Fisher scale, the evidence against the hypothesized probability of success is unremarkable. But instead of a predetermined sample size $n=12$, suppose that the experimenter’s intention was to sample until $x=3$ successes were observed. Under this scenario, the p-value is computed based on the negative binomial distribution. The upper-tail probability is computed for $N\sim NB\left(9,{\displaystyle \frac{1}{2}}\right)$ as

$$P_0(N\ge 12)={\displaystyle \sum _{i=12}^{\infty }}\left({\displaystyle \genfrac{}{}{0pt}{}{i-1}{2}}\right){\left({\displaystyle \frac{1}{2}}\right)}^i=1-{\displaystyle \sum _{i=3}^{11}}\left({\displaystyle \genfrac{}{}{0pt}{}{i-1}{2}}\right){\left({\displaystyle \frac{1}{2}}\right)}^i=0.033$$

giving the two-tail p-value as

$$p=2\left(0.033\right)=0.066.$$

Now, we have evidence (borderline to moderate based on the Fisher scale) against the hypothesized success probability. Equivalent hypotheses, tested from equivalent data, reach different levels of evidence as measured by the p-value. The computation of the p-value is not invariant to the sampling distribution, even though the experimental plan seems like it should be unrelated to the evidence provided from the observed data.

3.2. The Likelihood Ratio Statistic

The development of an evidence measure which does satisfy the likelihood principle proceeds as follows. Let $L\left(\theta \right)$ denote the likelihood as a function of an unknown parameter $\theta$. For simplicity, we take the single parameter case. Nuisance parameters and parameter vectors can be handled with slight adjustments to the development. Let $\widehat{\theta }$ denote the maximum likelihood estimate. We consider the problem of testing the hypothesis $\theta ={\theta }_0$ under the likelihood inference framework. Define the likelihood ratio as

$$\begin{array}{cc}\hfill LR\left({\theta }_0\right)& ={\displaystyle \frac{L\left({\theta }_0\right)}{L\left(\widehat{\theta }\right)}}.\hfill \end{array}$$

Then $0<LR\left({\theta }_0\right)<1$. As $LR\left({\theta }_0\right)$ decreases, the data evidence against the hypothesis increases. In this sense, $LR\left({\theta }_0\right)$ provides a measure of evidence against the hypothesis in the same spirit as a p-value. However, $LR\left({\theta }_0\right)$ as an evidence measure satisfies the likelihood principle.

We return once more to the problem of testing the success probability $\theta$. The likelihood ratio is invariant to the sampling distribution. So, the measure of evidence is the same whether the data comes from a binomial or negative binomial.

Suppose $X\sim BIN(n,\theta ).$ Then

$$\begin{array}{cc}\hfill L\left(\theta \right)& =\left({\displaystyle \genfrac{}{}{0pt}{}{n}{x}}\right){\theta }^x{(1-\theta )}^{n-x}.\hfill \end{array}$$

Suppose $N\sim NB(x,\theta ).$ Then

$$\begin{array}{cc}\hfill L\left(\theta \right)& =\left({\displaystyle \genfrac{}{}{0pt}{}{n-1}{x-1}}\right){\theta }^x{(1-\theta )}^{n-x}.\hfill \end{array}$$

In either case, we write the likelihood ratio as

$$\begin{array}{cc}\hfill LR\left({\theta }_0\right)& ={\displaystyle \frac{L\left({\theta }_0\right)}{L\left(\widehat{\theta }\right)}}={\displaystyle \frac{{\theta }_0^x{(1-\theta )}^{n-x}}{{\widehat{\theta }}^x{(1-\widehat{\theta })}^{n-x}}},\hfill \end{array}$$

where the sample proportion $\widehat{\theta }={\displaystyle \frac{x}{n}}$ is the maximum likelihood estimate. For testing ${\theta }_0={\displaystyle \frac{1}{2}}$ with observed data $x=3$, $n=12$, we compute $\widehat{\theta }={\displaystyle \frac{1}{4}}$ and $LR\left({\theta }_0\right)=0.208.$

Adherence to the likelihood principle through the use of the likelihood ratio statistic, provides a measure of evidence which is a function only of the observed data. We can say the data supports the hypothesized success probability at about $20\%$ of the level of support to the maximum likelihood estimate. Although we were successful in creating a measure of evidence that satisfies the likelihood principle, we have lost the familiarity of interpreting evidence on a p-value scale.

Neath $\left(2017\right)$ [5] established a connection between the p-value evidence scale and a likelihood ratio scale. From Table 2, we can say the strength of evidence against the hypothesized success probability is between borderline and moderate, no matter whether the experimenter planned binomial sampling or negative binomial sampling.

3.3. P-Value and Likelihood Ratio Under Censored Data

The p-value as a measure of evidence depends on the experimental plan. Thus, the computation of the p-value depends on the plan for censoring data regardless of whether the data observed was actually censored or not. Let’s see this with an example based on an idea from [1].

An experimenter is interested in the temperature of a particular surface. Available for performing the experiment is a device for measuring surface temperature. The device involves measurement error. The reading is distributed $N(\theta ,1)$ where $\theta$ is the actual temperature. Suppose that it is hypothesized that the actual surface temperature is ${100}^{\circ }$. The device returns a reading of $98.{35}^{\circ }$. A p-value is then computed based on a normal distribution. The lower tail probability is computed for $X\sim N(100,1)$ as

$$\begin{array}{cc}\hfill P_0(X\le 98.35)& =\mathsf{\Phi }(-1.65)\hfill \\ \hfill & =0.05\hfill \end{array}$$

giving the two tail p-value as

$$\begin{array}{cc}\hfill p& =2\left(0.05\right)=0.10.\hfill \end{array}$$

The evidence against the hypothesized temperature is only borderline on the Fisher scale.

The experimenter is happy to have an answer. The next day, he notices that the device is broken and returns a truncated value ${100}^{\circ }$ whenever the temperature measurement exceeds ${100}^{\circ }$. Now it is the case that a p-value should be computed based on a truncated normal distribution. The lower tail probability based on the observed reading $98.{35}^{\circ }$ is computed to be $0.05$ as before. However, evidence against the hypothesized temperature can only arise for $x<{100}^{\circ }$. So the p-value based on the truncated normal is one-sided, giving $p=0.05$. The evidence against the hypothesized temperature has increased after learning that the device was broken. A quantification of evidence based on the p-value depends not on the observed data alone, but also on the irrelevant question of whether the device is working for potential readings that were not observed. The experimenter is now confused by the results.

Let’s investigate a measure of evidence which does satisfy the likelihood principle. The likelihood function for this problem is based on a truncated normal distribution. We can write the likelihood in two parts, depending on whether we observe $x<{100}^{\circ }$ or the truncated value of $x={100}^{\circ }$

$$\begin{array}{cc}\hfill L\left(\theta \right)& =\left\{\begin{array}{c}{\displaystyle \frac{1}{\sqrt{2\pi }}}{e}^{-{\displaystyle \frac{1}{2}}{(x-\theta )}^2},x<100\hfill \\ 1-\mathsf{\Phi }({100}^{\circ }-\theta ),x=100.\hfill \end{array}\right.\hfill \end{array}$$

Since we observed $x=98.{35}^{\circ }$, we have

$$\begin{array}{cc}\hfill L\left(\theta \right)& ={\displaystyle \frac{1}{\sqrt{2\pi }}}{e}^{-{\displaystyle \frac{1}{2}}{(98.{35}^{\circ }-\theta )}^2}.\hfill \end{array}$$

Since $\widehat{\theta }=x$ is the maximum likelihood estimate,

$$\begin{array}{cc}\hfill LR\left({\theta }_0\right)& ={\displaystyle \frac{L\left({\theta }_0\right)}{L\left(\widehat{\theta }\right)}}\hfill \\ \hfill & =e^{-{\displaystyle \frac{1}{2}}{(-1.65)}^2}.\hfill \end{array}$$

So $LR\left({\theta }_0\right)=0.256$, and the evidence against the hypothesized temperature is borderline as we saw initially. The experimenter is happy once again to have an answer that makes sense.

4. Consequences of the Likelihood Principle

In the last section, we saw examples where the likelihood ratio statistic provides what seems to be a more relevant measure of evidence than that which was provided by the p-value approach. In this section, we look at some consequences of adhering to the likelihood principle that may not be satisfactory.

4.1. Testing Multiple Hypotheses

In the previous sections, we considered problems where a single hypothesis $\theta ={\theta }_0$ is being tested. Now, let’s consider the problem of testing multiple hypotheses, ${\theta }_1={\theta }_{10},\dots ,{\theta }_M={\theta }_{M0}.$ For example, it may be that we are testing several medical treatments, where each treatment effect is represented in its own hypothesis.

Suppose we observe $z-$statistics for each of the M tests. Because we are testing multiple hypotheses, our focus may be on those results providing the strongest evidence. For example when testing multiple treatments, our focus would naturally be on those treatments offering the strongest evidence of an effect. In measuring evidence using the p-value, we must take note of the experimental plan. When the focus is on the test featuring the strongest evidence, then the focus is on the most extreme test result. The p-value is then computed based on the distribution for ${max}_{\ell =1,\dots ,M}\mid Z_{\ell }\mid$, and not on the $N(0,1)$ distribution as it would be for testing a single hypothesis. We will compute the p-value for the distribution on $max\mid Z_{\ell }\mid$ through simulation. Figure 3 below shows the plots of some predictive distributions for multiple testing. Note how the distribution shifts to larger $z-$values as the number of tests performed increases.

Our interest is on how evidence strength for the landmark $z-$values on the Fisher scale change as the number of hypotheses being tested increase. Table 3 summarizes the results.

Remember that a p-value is based on how the observed data conforms to the predictive distribution derived from the hypothesis being tested. In our case here, the predictive distribution on $max\mid Z_{\ell }\mid$ is derived from multiple hypotheses, under the model where all stated hypotheses are true. The evidence for observing large Z diminishes in the sense that it becomes more probable to observe large Z for at least one of the multiple tests being considered. For example, suppose $M=20$ medical treatments are being tested simultaneously, and that $max\mid z_{\ell }\mid =2.58$ is observed. From Table 3, we see $p=0.18$ is the computed p-value. The evidence against the model where all stated hypotheses are true is unremarkable as it is not unlikely to observe a Z as large (or larger) for at least one of the $M=20$ tests.

Let us now consider a measure of evidence based on the likelihood ratio statistic. Remember that an evidence measure satisfying the likelihood principle does not depend on the experimental plan. Thus, the evidence against the hypothesis ${\theta }_i={\theta }_{i0}$ will be the same no matter what other hypotheses are under consideration. In our example, the evidence against the hypothesis where $z_{\ell }=2.58$ was observed is interpreted as strong evidence on the likelihood ratio scale. The likelihood ratio measure of evidence gives a very different interpretation than the p-value measure. So we have two very different measures of evidence. Unlike the examples from the previous chapter, it is not so clear which measure is preferable.

Let us think again about testing several medical treatments. For $M=20$ treatments being studied, it would not be unlikely for a $z-$statistic from one of the treatments to be as large, or larger than the observed $max\mid z_{\ell }\mid =2.58.$ Under this argument, evidential support for a treatment effect somewhere among the medical treatments being studied is lacking. But we did observe $z=2.58$ for testing one of the treatment effects. This should indicate evidence in favor of an effect for that specific treatment. Under the likelihood principle, a measure of evidence should depend on the data observed (here, $z_{\ell }=2.58$), and not on whether it was the experimenter’s intention to test other hypotheses.

We can reach an agreement between the two arguments by noting that the p-value is measuring evidence against the model where all of the hypotheses are true, and the likelihood ratio is measuring evidence against specific pieces of that model. Nevertheless, we may not be comfortable with the likelihood approach in a setting where the number of hypotheses being tested is very large. From Table 3, for $M=100$ we see it is a near guarantee that at least moderate evidence against one of the hypotheses will be attained. It is unsettling to know before an experiment is conducted that the data to be observed will reach a particular strength of evidence.

4.2. Stopping Rules

We are back to the problem of testing a single hypothesis. Suppose we have data $x_1,\dots ,x_n$ from a $N(\theta ,1)$ distribution. For simplicity, suppose we are testing the hypothesis that $\theta =0.$ Let $z_n=\sqrt{n}\mid {\overline{x}}_n\mid$ be the $z-$statistic after n observations.

Suppose that after $n=25$ observations, the computed $z-$ statistic is $z_{25}=2.0$, which seems to represent moderately strong evidence against the hypothesis. However, the two members of the experimental team have different views on the plan of experimentation. One member of the team saw the plan as stopping after $n=25$ observations, no matter the result. The other member saw the plan as stopping at this point only if the evidence attained from the data reached at least moderate strength $(i.e.\mid Z\mid >1.96)$. If the evidence was not yet strong enough, an additional 25 observations were to be taken before computing a $z-$statistic once again. The second team member was unsure how long observations were to be taken before the experiment was deemed futile. No matter which team member is correct in their description of the experimental plan, the observed data would be the same. However, since the p-value is computed based on the experimental plan, the team members would assign different strengths of evidence. See Table 4 for a display of how the evidence measure depends on the planned number of "looks" at the data. Figure 4 shows the plots of how the predictive distribution on the observed $\mid Z\mid$ changes with respect to the sample size when the plan is to continue sampling until we get a $z-$statistic larger in magnitude than $1.96$. Under the first member’s view, only one look at the data was in plan. Here $p=0.05$. If the plan was to take four looks, after $25,50,75,100$ observations, stopping early if $\mid Z\mid >1.96$, then the correct evaluation gives $p=.1339$. The p-value measure of evidence strength requires knowledge of why an experiment was stopped.

Under the likelihood principle, the team members would be in agreement as to the strength of evidence. As we have stated earlier, the likelihood ratio statistic as an evidence measure depends only on the data observed, and not on the experimental plan. It seems that the likelihood principle frees us from an unnecessary concern over stopping rules. But as discussed in [6], we can take this feature of the likelihood principle to an undesirable extreme. Suppose the plan is to compute $Z_n=\sqrt{n}\mid {\overline{x}}_n\mid$ after every new observation, stopping as soon as we get $\mid Z_n\mid >1.96$.

Table 5 gives the p-values at the evidence thresholds on the Fisher scale, based on the maximum sample size. Figure 5 shows the plots of the predictive distributions under this sampling scheme. If our maximum sample size were $n=500$, there would be about a $50\%$ chance of having a $z-$statistic exceeding $1.96$ level somewhere in the sequence.

Since such a result is quite compatible with the hypothesis, the evidence against the hypothesis would be unremarkable. If there were no maximum sample size, it can be shown that a $Z_n$ would exceed the $1.96$ level at some point. For this experimental plan, the p-value would be computed as

$$\begin{array}{c}\hfill p=P(\mid Z_n\mid >1.96\mathrm{for}\mathrm{some},n=1,2,\dots )=1.\end{array}$$

Thus, before the experiment begins, it is certain that we will observe $\mid Z_n\mid >1.96$. This certainty is displayed by a p-value measure giving no strength at all to this data evidence.

For the maximum likelihood ratio measure, which satisfies the likelihood principle, the experimental plan, no matter how extreme, is irrelevant for observed $\mid Z_n\mid >1.96$. It must be that $LR\left({\theta }_0\right)<0.146$. To see this, write

$$\begin{array}{cc}\hfill L\left(\theta \right)& ={\mathsf{\Pi }}_{i=1}^n{\displaystyle \frac{1}{\sqrt{2\pi }}}{e}^{-{\displaystyle \frac{1}{2}}{(x_i-\theta )}^2}\hfill \\ \hfill & ={\left(2\pi \right)}^{{\displaystyle \frac{-n}{2}}}{e}^{{\displaystyle \frac{-n}{2}}{(\overline{x}-\theta )}^2}·e^{-{\displaystyle \frac{1}{2}}{\sum }_{i=1}^n{(x_i-\overline{x})}^2}.\hfill \end{array}$$

Since $L\left(\theta \right)$ is maximized at $\widehat{\theta }=\overline{x}$, then

$$\begin{array}{cc}\hfill LR\left({\theta }_0\right)& ={\displaystyle \frac{L\left({\theta }_0\right)}{L\left(\widehat{\theta }\right)}}={\displaystyle \frac{e^{-\frac{n}{2}{\left({\overline{x}}_n\right)}^2}}{1}}=e^{-{\displaystyle \frac{1}{2}}{z}_n^2}.\hfill \end{array}$$

Because we have $\mid Z_n\mid >1.96$, then

$$\begin{array}{c}\hfill LR\left({\theta }_0\right)<e^{-{\displaystyle \frac{1}{2}}{\left(1.96\right)}^2}=0.146.\end{array}$$

Before any data is collected, it is certain that at least moderate evidence against the only hypothesis being tested will be attained. Since the argument holds for any $z-$threshold, we can create an experimental plan where any strength of evidence is guaranteed to be met. As in the problem with multiple testing, we are in a very uncomfortable position of knowing before an experiment is conducted that the observed data will reach any strength of evidence we desire, according to the likelihood ratio statistic.

5. Bayes Factor

The goal of a Bayes factor is to provide a measure of evidence based on how belief in a hypothesis changes after observing the data. See [4] for an in-depth discussion. To motivate the derivation of a Bayes factor, suppose we observe $x_1,\dots ,x_n\mathrm{from}N(\theta ,1)$, or equivalently observe ${\overline{x}}_n\sim N\left(\theta ,{\displaystyle \frac{1}{n}}\right)$. Supposed that under the hypothesized model, $\theta =0$. The development of a Bayes factor requires an alternative hypothesis stated as a predictive model.

Before the creation of an alternative model, let’s think about the interpretation of $\theta$. Since we set $Var\left(X\right)=1$ and ${\theta }_0=0$, then $\theta$ is on a standard deviation scale. A parameter defined in such a way corresponds to Cohen’s effect size. A scale commonly used for interpreting Cohen’s effect size is displayed below.

$$\begin{array}{cc}\hfill \theta & =0.2\mathrm{Small}\mathrm{effect}\hfill \\ \hfill \theta & =0.5\mathrm{Medium}\mathrm{effect}\hfill \\ \hfill \theta & =0.8\mathrm{Large}\mathrm{effect}\hfill \end{array}$$

The reason we need to know typical values for $\theta$ is that we need to model $\theta$ under the alternative hypothesis. We will define the effect size for our alternative model using a Bayesian prior probability distribution. We write

$$\begin{array}{c}\hfill \theta \sim N(0,{\sigma }_{\theta }^2),\end{array}$$

where ${\sigma }_{\theta }^2$ can be chosen to correspond to the expected effect size. If ${\sigma }_{\theta }^2$ is relatively small, the alternative is modelling a small effect size. If ${\sigma }_{\theta }^2$ is relatively large, the alternative is modelling a large effect size. Since the data distribution is modelled as

$$\begin{array}{c}\hfill {\overline{X}}_n|\theta \sim N\left(\theta ,{\displaystyle \frac{1}{n}}\right),\end{array}$$

then the predictive distribution under the alternative becomes

$$\begin{array}{c}\hfill {\overline{X}}_n\sim N\left(0,{\displaystyle \frac{1}{n}}+{\sigma }_{\theta }^2\right).\end{array}$$

Let ${\pi }_0,{\pi }_1({\pi }_0+{\pi }_1=1)$ represent Bayesian prior probabilities on the respective hypotheses / models. That is,

$$\begin{array}{cc}\hfill {\pi }_0& =P\left(M_0\right),\hfill \\ \hfill {\pi }_1& =P\left(M_1\right),\hfill \end{array}$$

where $M_0$ and $M_1$ denote the competing models. Using Bayes Rule, we write the posterior probability on model $M_0$ as

$$\begin{array}{cc}\hfill P\left(M_0\right|D)& ={\displaystyle \frac{P\left(D\right|M_0)P\left(M_0\right)}{P\left(D\right|M_0)P\left(M_0\right)+P\left(D\right|M_1)P\left(M_1\right)}}.\hfill \end{array}$$

The above probability statement leads to

$$\begin{array}{cc}\hfill {\displaystyle \frac{P\left(M_0\right|D)}{P\left(M_1\right|D)}}& ={\displaystyle \frac{P\left(M_0\right)}{P\left(M_1\right)}}·{\displaystyle \frac{P\left(D\right|M_0)}{P\left(D\right|M_1)}}.\hfill \end{array}$$

Let $P\left(M_0\right)/P\left(M_1\right)$ denote the prior odds. Let $P\left(M_0\right|D)/P\left(M_1\right|D)$ denote the posterior odds. Then we define the Bayes Factor as

$$\begin{array}{cc}\hfill B{F}_{01}& ={\displaystyle \frac{P\left(D\right|M_0)}{P\left(D\right|M_1)}}.\hfill \end{array}$$

We will write $P\left(D\right|M)$ as predictive densities evaluated at the observed ${\overline{x}}_n$. To make the connection to a likelihood ratio statistic, we will use the notation for the likelihood function. Let

$$\begin{array}{cc}\hfill L\left(\theta \right)& =\sqrt{{\displaystyle \frac{n}{2\pi }}}{e}^{-{\displaystyle \frac{n}{2}}{({\overline{x}}_n-\theta )}^2}.\hfill \end{array}$$

Then $P\left(D\right|M_0)$ can be represented as $L\left({\theta }_0\right)$ and $P\left(D\right|M_1)$ can be represented as

$$\begin{array}{c}\hfill \int L\left(\theta \right)P\left(\theta \right)d\theta ,\end{array}$$

where $P\left(\theta \right)$ represents the prior on $\theta$ under the alternative model $M_1$. Now we can write

$$\begin{array}{cc}\hfill B{F}_{01}& ={\displaystyle \frac{L\left({\theta }_0\right)}{\int L\left(\theta \right)P\left(\theta \right)d\theta }}.\hfill \end{array}$$

Note how the BF₀₁ depends on the data only through the likelihood function. Thus, a Bayes factor provides a measure of evidence which satisfies the likelihood principle. Rather than using

$$\begin{array}{cc}\hfill L\left(\widehat{\theta }\right)& =ma{x}_{\theta \in \mathsf{\Theta }}L\left(\theta \right)\hfill \end{array}$$

for the denominator, as in the definition of a likelihood ratio statistic, the Bayes factor uses a weighted average of the likelihood function, where the weights are determined by prior distribution used in defining the alternative model. Since maximizing the denominator for the Bayes factor leads to the likelihood statistic, we see that the likelihood ratio statistic overstates the evidence against the hypothesized model. Both evidence measures are formed as ratios of the likelihood function, so both can be interpreted on the same scale.

Refer back to Fisher’s scale for interpreting evidence strength displayed in Table 1 and Table 2. Instead of Fisher’s scale, Jeffreys’ scale is more often used for interpreting strength of evidence measured by a Bayes factor [3]. As with the p-value and the likelihood ratio statistic, small values of BF₀₁ represent evidence against the hypothesized model.

Table 6. Jeffreys’ evidence scale.

$0.3$    to    1	not worth a bare mention
$0.05$    to    $0.3$	positive
$0.005$    to    $0.05$	strong
$0.000$    to    $0.005$	very strong

Table 6. Jeffreys’ evidence scale.

$0.3$    to    1	not worth a bare mention
$0.05$    to    $0.3$	positive
$0.005$    to    $0.05$	strong
$0.000$    to    $0.005$	very strong

Jeffreys’ evidence scale is more conservative than Fisher’s scale. We can take this difference in interpretation as a reminder that there is no unique answer for scaling a measure of evidence strength.

6. Properties of Bayes Factors

Before examining some important properties, let’s derive BF₀₁ for our problem using familiar hypothesis testing notation. We will begin by writing the Bayes factor as a ratio of predictive density functions,

$$\begin{array}{cc}\hfill B{F}_{01}& ={\displaystyle \frac{f_0\left({\overline{x}}_n\right)}{f_1\left({\overline{x}}_n\right)}},\hfill \end{array}$$

where

$$\begin{array}{cc}\hfill f_0\left({\overline{x}}_n\right)& =\sqrt{{\displaystyle \frac{n}{2\pi }}}{e}^{-{\displaystyle \frac{n}{2}}{\left({\overline{x}}_n\right)}^2}\hfill \end{array}$$

is the $N\left(0,{\displaystyle \frac{1}{n}}\right)$ density function, and

$$\begin{array}{cc}\hfill f_1\left({\overline{x}}_n\right)& =\sqrt{{\displaystyle \frac{n}{2\pi (1+n{\sigma }_{\theta }^2)}}}{e}^{-{\displaystyle \frac{n}{2\left(1+n{\sigma }_{\theta }^2\right)}}{\left({\overline{x}}_n\right)}^2}\hfill \end{array}$$

is the $N\left(0,{\displaystyle \frac{1}{n}}+{\sigma }_{\theta }^2\right)$ density function. We used the algebraic identity

$$\begin{array}{cc}\hfill {\displaystyle \frac{1}{\frac{1}{n}+{\sigma }_{\theta }^2}}& ={\displaystyle \frac{n}{1+n{\sigma }_{\theta }^2}}\hfill \end{array}$$

in writing $f_1\left({\overline{x}}_n\right)$. Let $z_n=\sqrt{n}\left({\overline{x}}_n\right)$ denote the standardized test statistic for our hypothesis testing problem. Now we can write

$$\begin{array}{cc}\hfill B{F}_{01}& =\sqrt{1+n{\sigma }_{\theta }^2}·{\displaystyle \frac{exp\{-\frac{1}{2}{z}_n^2\}}{exp\{-\frac{1}{2\left(1+n{\sigma }_{\sigma }^2\right)}{z}_n^2\}}}\hfill \\ \hfill & =\sqrt{1+n{\sigma }_{\theta }^2}·exp\{-{\displaystyle \frac{1}{2}}{\displaystyle \frac{n{\sigma }_{\theta }^2}{1+n{\sigma }_{\theta }^2}}{z}_n^2\}.\hfill \end{array}$$

Equivalently, we could write BF₀₁ as a ratio of predictive densities for the test statistic z. Under the hypothesized model, $z\sim N(0,1)$. Under the alternative model,

$$\begin{array}{c}\hfill z\sim N(0,1+n{\sigma }_{\theta }^2).\end{array}$$

6.1. Bayes Factor Support for the Hypothesized Model

Evidence as measured by a p-value or a likelihood ratio statistic can only be stated against the hypothesized model. A Bayes factor, however, can specify evidence either against or in favor of the hypothesized model. If BF₀₁ $>1$, then $f_0\left(z\right)>f_1\left(z\right)$ representing data more compatible with the hypothesized model than the alternative model. We can solve the inequality BF₀₁ $>1$ as

$$\begin{array}{c}\hfill exp\left\{{\displaystyle \frac{1}{2}}{\displaystyle \frac{n{\sigma }_{\theta }^2}{1+n{\sigma }_{\theta }^2}}{z}^2\right\}<\sqrt{1+n{\sigma }_{\theta }^2}\\ \hfill {\displaystyle \frac{1}{2}}{\displaystyle \frac{n{\sigma }_{\theta }^2}{1+n{\sigma }_{\theta }^2}}{z}^2<log\sqrt{1+n{\sigma }_{\theta }^2}\\ \hfill z^2<ln(1+n{\sigma }_{\theta }^2)·{\displaystyle \frac{1+n{\sigma }_{\theta }^2}{n{\sigma }_{\theta }^2}}.\end{array}$$

For example, let’s take ${\sigma }_{\theta }^2=1$. Figure 6 plots the threshold $\sqrt{ln(1+n{\sigma }_{\theta }^2)·{\displaystyle \frac{1+n{\sigma }_{\theta }^2}{n{\sigma }_{\theta }^2}}}$, separating support between the competing models, as a function of the sample size n.

Note that as n gets larger, we are finding z values supporting the hypothesized model under the Bayes factor which provided evidence against the hypothesized model under the p-value framework and likelihood ratio framework. Let’s investigate one of these instances. Suppose we have a sample of size $n=50$. Under the hypothesized model, $z\sim N(0,1)$, as it always is. Under the alternative model, $z\sim N(0,51)$ giving a standard deviation for z of about 7. Figure 7 plots the predictive densities for both models. The density for the model appears nearly uniform over the plotted interval, due to its extreme variability. We can see from the plot how $f_0\left(z\right)>f_1\left(z\right)$, $(B{F}_{01}>1)$ when $\mid z\mid <2$.

So while observing $z=2$ gives a p-value indicating moderate evidence against the hypothesized model, the Bayes factor gives equal evidence to the hypothesized model as it gives the alternative. Figure 8 plots the Bayes factor as a function of z for this example. It is not until $\mid z\mid$ is much larger than the thresholds for evidence on the Fisher’s scale before we find evidential support for the alternative from the Bayes factor.

Before leaving this example, let’s understand why $\mid z\mid$ must be much larger than the Fisher p-value threshold in order to state evidence against the hypothesized model. Note that $\overline{x}$ serves as an estimate for Cohen’s effect size. The prior variance ${\sigma }_{\theta }^2=1$ serves as a specification for the effect size under the alternative model. The Bayes factor measures whether the observed effect size $\overline{x}$ is more compatible with an effect size of zero, as specified under the hypothesized model, or with the potentially large effect size specified under the alternative model. Since $z_n=\sqrt{n}{\overline{x}}_n$, then $z_n/\sqrt{n}$ is the observed effect size. As n gets larger $z_n$ must get larger as well to support the larger effect size under the alternative model. This phenomenon is even more pronounced in this example due to the large prior variance. Unlike p-value and likelihood ratio measures of evidence, it is not enough for the data to be incompatible with the hypothesized model. We also need data to be compatible with the alternative model before finding evidence against our hypothesis. Under the Bayes factor framework, we have two competing models under consideration, not just a single hypothesized model.

6.2. Bayes Factor Dependence on Sample Size

The $z-$statistic, when applicable, determines the strength of evidence as measured by a p-value, or a likelihood ratio statistic, in a manner that is not dependent on the sample size. The same is not true for a Bayes factor. Table 7, Table 8 and Table 9 display the Bayes factor BF₀₁ for the Fisher’s evidence threshold values of z for sample sizes $n=1,25,50,100$ and variances ${\sigma }_{\theta }^2=0.2^2,0.5^2,0.8^2$ (so that ${\sigma }_{\theta }^2$ matches small, medium and large Cohen’s effect sizes)

The Jeffreys evidence scale highlight the entries in the above tables, shown as one star for positive evidence, two stars for strong evidence and three stars for very strong evidence. When the sample size is small, the Bayes factor does not provide much evidence in favor of either of the competing hypotheses. This is because there is very little distinction between the predictive distributions for the two models. For example, consider the prior variance ${\sigma }_{\theta }^2=0.2^2$. For $n=1$, the predictive density is graphed in Figure 9.

Because both predictive models are nearly the same, no standard of evidence towards either model can be attained from observed data. It is not until the sample size grows larger that a Bayes factor can provide an informative measure of evidence.

Let’s note how data which provides moderate evidence against the hypothesized model on Fisher’s scale $(z=1.96,p=0.05)$ never even registers positive evidence on Jeffreys’ Bayes factor scale. Furthermore, note how very strong evidence is not yet attained at the z values presented in the tables. To see what observed values are needed for very strong evidence against the hypothesized model, let’s solve,

$$\begin{array}{cc}\hfill B{F}_{01}& =0.005\hfill \end{array}$$

to get

$$\begin{array}{cc}\hfill z^2& =(2ln\left(200\right)+ln(1+n{\sigma }_{\theta }^2)){\displaystyle \frac{1+n{\sigma }_{\theta }^2}{n{\sigma }_{\theta }^2}}.\hfill \end{array}$$

Figure 10 plots the z value needed for very strong evidence from a test against an alternative model with a medium effect size $({\sigma }_{\theta }=0.5)$.

Figure 10. Test against an alternative model with a medium effect size.

Figure 10. Test against an alternative model with a medium effect size.

7. Conclusions

Let us end our paper by thinking of how a test statistic and p-value are interpreted when teaching and learning statistics. It would be typical for the p-value to simply be compared to some type I error level, with a decision of reject $H_0$ or fail to reject $H_0$ to follow. But our discussion in this paper demonstrates how there is so much more detail to consider for even a simple example. Our paper discusses further questions for an analyst to pursue. We looked at issues involving how sample size determination, stopping rules, multiple hypotheses, and effect sizes influence the quantification of statistical evidence.

Simulations were done in R. See [7]. The relevant R code is shown below.