Prior Density Selection and Statistical Models

1. Introduction

Mathematical and population-based probability models are often used to summarize, simplify and make interpretable scientifically relevant patterns in data [1]. In some settings, pre-existing prior information for the parameters in these models, typically interpreted as population characteristics, can also be expressed on a probability scale. This information can be incorporated into the statistical analysis via Bayesian statistical models [2].

The subjective Bayesian approach to statistical modeling uses a probability density $p\left(\theta \right)$ to represent and describe existing baseline beliefs regarding the unknown and unobservable model parameters $\theta$. After assuming a population-based probability model and collecting the data $x$, the beliefs represented by $p\left(\theta \right)$ are updated to include this new information via the posterior density $p\left(\theta \right|x)$, using the likelihood function $L\left(\theta \right|x)$ and application of Bayes theorem. Once an experiment is conducted, $p\left(\theta \right|x)$ provides the new, updated probability weighting for each value of $\theta$. Note that from the Bayesian perspective, there is no assumption of a “true” value for $\theta$ to formally test or estimate, rather $p\left(\theta \right|x)$ gives a probability weight for each possible $\theta$ value. The researcher can interpret this directly, or compare distinct values of $\theta$ using measures such as the posterior odds ratio or the Bayes factor [3].

The likelihood function, can be written:

$$L\left(\theta \right|x)=k\cdot \prod _{i=1}^{n}f(x_i\left|\theta \right)$$

where $f\left(x_i\right|\theta )$ is the probability model or density for the $i^{th}$ independent subject response $x_i$ and the constant $k$ emphasizes the fact that the likelihood is a function of $\theta$, not necessarily a density for $\theta$. The likelihood function is the key source of information to be drawn from a given model-data combination. Often the mode of the likelihood function, the maximum likelihood estimator, is the basis of frequentist inference. A function of the local curvature of the log-likelihood function about its maximum value provides an estimate of sampling variation [4].

Many applications of Bayesian methods view the selection of0 $p\left(\theta \right)$ as primarily subjective in nature, reflecting the ideas and beliefs of the individual researcher [5]. From this perspective there are no limitations on the shape or entropy of $p\left(\theta \right)$ apart from the need to select a function that technically integrates to one and has no negative values. Indeed, a common choice is a non-informative density, a flat prior which may not technically integrate to one [6]. In some model settings, the need to employ Monte Carlo based methods to estimate marginal posterior elements supports the selection of convenience priors to aid in convergence [7].

Other justifications of $p\left(\theta \right)$ are more formal and less subjective. For example, some argue for the use of meta-analysis and a choice of $p\left(\theta \right)$ that reflects existing scientific understanding of the possible values for $\theta$ [8]. The choice of subjective prior may also be guided by the need to respect specific aspects of the statistical model and related inferential approach.

For example, the idea of matching priors, choosing prior densities for $\theta$ that deliberately structure the posterior density in a manner that yields tail areas similar to those underlying frequentist p-value tail area calculations. This limits the choice of $p\left(\theta \right)$ to a specific form [9].

The selection of prior density may also be guided by the need to maintain specific properties of the likelihood function and related model structure, for example likelihood invariance or model symmetry [10,11]. Note that the model parameter $\theta$ is defined and takes its meaning from the context of the statistical model.

Here, prior density selections reflecting specific model related and matching properties are reviewed. It is shown that when these considerations are extended to include the matching of likelihood and posterior density higher order derivatives, they provide new exclusionary rules for the selection of prior densities. In addition, the class of related acceptable prior densities are non-informative on a scale related to the Pratt-Arrow utility-based measure of relative risk

2. Bayesian Inference

As noted, the Bayesian approach or perspective is based on the joint posterior density $p\left(\theta \right|x)$ which can be expressed as:

$$p\left(\theta \right|x)=c\cdot p(\theta )\cdot L(\theta \left|x\right)$$

or

$$\mathit{log}p\left(\theta \right|x)=c^{\text{'}}+\mathit{log}p(\theta )+\mathit{log}L(\theta \left|x\right)$$

where $p\left(\theta \right)$ is the prior density, $L\left(\theta \right|x)$ the likelihood function and $c$ the constant of integration. All three functions of $\theta$ can be viewed as weighting the model parameter space, with prior and posterior densities restricted to a probability scale.

Once the joint posterior density $p\left(\theta \right|x)$ is obtained, integration is employed in the Bayesian setting to obtain marginal posterior densities for any given ${\theta }_i$. For example,

$$p\left({\theta }_1\right|x)=\int \cdot \cdot \cdot \int p(\theta \left|x\right)d{\theta }_{2}d{\theta }_3\cdot \cdot \cdot d{\theta }_p$$

gives the marginal posterior for ${\theta }_1$ alone. The central region of this density is a Bayesian credible region which can be used for estimation regarding ${\theta }_1$. With the advent of modern methods for numerical integration [12], accurate calculations in many Bayesian settings are available. Posterior odds ratios, Bayes factors and predictive distributions can be used for inference [3].

Note that, in general, re-scaling variables can sometimes induce information into the Bayesian setting, affecting nominally non-informative priors and making them informative. It is assumed here that once the appropriate scaling is found for the parameters $\theta$ in the likelihood function, the likelihood is expressed in terms of this parameter scaling and prior densities expressed in a similar manner.

3. Bayesian and Frequentist Similarities

Limitations on the choice of subjective prior can arise. The idea of matching priors, choosing prior densities for $\theta$ that deliberately structure the posterior density in a manner that yields tail areas similar to those underlying frequentist p-value tail area calculations, limits the choice of prior density. While similarities between Bayesian and frequentist statistics exist in large samples due to the presence of central limit theorems along with specific assumptions, they also occur in smaller samples where specific model structures are a key consideration. In these settings, limitations on the choice of subjective priors arise.

4. Matching Prior Selection

As noted above, in large samples and subject to regularity conditions [4], as $n\to \infty$,

$${\theta }_{mle}\left(x\right)\sim N(\theta ,I^{-1}(\theta \left|x\right))$$

where ${\theta }_{mle}\left(x\right)$ is the maximum likelihood estimate and $I^{-1}\left(\theta \right|x)$ the Fisher information.

From the related Bayesian perspective, with a non-informative prior independent of the sample size $n$, and subject to regularity conditions, as $n\to \infty$,

$$\theta |x\sim N({\theta }_{mle}\left(x\right),J^{-1}\left(\theta \right|x\left)\right)$$

with $J\left(\theta \right|x)$ the observed Fisher information with $J^{-1}\left(\theta \right|x)\to I^{-1}(\theta \left|x\right)$ in probability.

Note that these results both have multivariate versions that are very similar:

$${\theta }_{mle}\left(x\right)\sim N_p(\theta ,I^{-1}(\theta \left|x\right))$$

$$\theta |x\sim N_p({\theta }_{mle}\left(x\right),J^{-1}\left(\theta \right|x\left)\right)$$

This close technical relationship between large sample Bayesian and frequentist inferential probability has motivated investigations regarding which choice of $p\left(\theta \right)$ leads to equivalence in tail area probabilities and central regions related to inference [9]. Examples include [13,14,15]. Usually this involves matching tail areas of the sampling and posterior distributions to the first or second order of approximation.

Results can be formulated in terms of the accuracy of the Bayesian tail area in frequentist terms. The prior density $p\left(\theta \right)$ is defined to be an $r^{th}$ order probability matching prior if its use satisfies the condition:

$$p_{{\theta }_i|x}({\theta }_i\le {\theta }_{i;1-\alpha })=1-\alpha +o\left(n^{-r/2}\right)$$

where $p_{\theta |x}({\theta }_i\le \theta )$ is the cumulative distribution function of the marginal posterior density for ${\theta }_i$ with a frequentist probability coverage of $1-\alpha$ with an relative error of $o\left(n^{-r/2}\right)$. Typically, accuracy at the $r=1,2$ level are considered.

So-called matching priors for $\theta =({\theta }_1,{\theta }_2,...,{\theta }_p)$ that satisfy this type of condition for say ${\theta }_1$ are often of the form:

$$p\left(\theta \right)\propto g({\theta }_2,...,{\theta }_p)\left[i_{11}\right(\theta ){]}^{1/2}$$

where $i_{11}\left(\theta \right)$ is the first entry in the Fisher information, $g(\cdot )$ is an arbitrary smooth function, and the Fisher information has a specific assumed structure, for example the correlations along the first row or column of the Fisher information are assumed to be zero, ie. ${\theta }_1$ is orthogonal to $({\theta }_2,...,{\theta }_p)$. Note that this prior is not unique.

The overlap of Bayesian and frequentist perspectives show a conceptual link going beyond tail area similarity, especially where the prior is independent of the sample size $n$. The overlap links the informative or learning aspect of the Bayesian method to the large sample minimal sufficiency of the $m.l.e.$ and the likelihood based score function, in units given by the Fisher information and is further discussed below.

5. Fiducial or Transferred Probability

It is interesting when discussing the Bayesian and frequentist overlap in large samples to consider briefly the concept of fiducial probability, the idea that in certain settings, often in smaller samples, probability intervals derived on the sample space can be transferred or matched to intervals on the parameter space without use of a prior density and Bayes theorem, relying instead on the structure of the statistical model or a matching condition.

While this approach can be linked to the likelihood concept, it often reflects the importance of symmetry in defining a statistical model.

The fiducial approach is primarily justified by the inversion of significance tests, namely the region of the sample space not included in a rejection region for a test of the null hypothesis $H_0\hspace{0.33em}\hspace{0.33em}:\hspace{0.33em}\hspace{0.33em}\theta ={\theta }_0$ with Type I error $=\alpha$, based on minimal sufficient statistics, and viewed as a region of supported values for $\theta$. The level of this supported region is $1-\alpha$ and this level is justified by repeating the significance test for each possible value of $\theta$, keeping those values of $\theta$ (usually in the initial form of a pivotal quantity) that are not rejected by the significance test.

The supported region is sometimes viewed as the central region of a fiducial or confidence distribution. This distribution is built using the tail area of the minimal sufficient statistic distribution, taking into account the pivotal quantity structure that underlies most significance tests (the minimal sufficient statistic is standardized to give a pivotal quantity) with this pivotal quantity having a known distribution. Note that in exponential family probability models the likelihood statistics are the minimal sufficient statistics.

The fiducial distribution can easily be derived in one dimension. Remember that a probability density $f\left(x\right)=(d/dx)F\left(x\right)$ where $F\left(x\right)=P(X\le x)$. In significance testing, the tail area $(1-F(x\left)\right)$ is required and it follows that $f\left(x\right)=-d/dx(1-F(x\left)\right)$. To include the pivotal structure, this is usually written in terms of $T(x,\theta )$, where typically $T(x,\theta )=\left(S\right(x)-\theta )/\sqrt{Var\left(S\right(x\left)\right)}$ and $S\left(x\right)$ is the minimal sufficient statistic. If $\left|dT\right(x,\theta )/dx|=\left|dT\right(x,\theta )/d\theta |$ is assumed to hold, then there is a symmetry between $S\left(x\right)$ and $\theta$ (they are interchangeable) within $T(x,\theta )$, and $d/dT(1-F(T(x,\theta ))$ is equivalent to $d/d\theta (1-F(T(x,\theta )\left)\right)$ .

On this basis it can be claimed that the probability distribution can be transferred from the sample space to the parameter space through the symmetry of the pivotal, without assuming a prior density for $\theta$ and assuming $S\left(x\right)$ is an observed value. Note that the likelihood ratio itself can be used as a pivotal quantity, so aspects of the likelihood function, apart from the likelihood statistics, may be involved.

More practically, the significance test requires both an observed value of $S\left(x\right)$ and a null value for $\theta ,$ say ${\theta }_0$. If the rejection region of the test is well defined (easiest in one dimension with a scalar $\theta$) then the probability of the rejection region will alter as ${\theta }_0$ alters. As ${\theta }_0$ goes from $-\infty$ to $\infty$ (the defined parameter space) the probability of the rejection region based on the observed minimal sufficient statistic or likelihood statistic will range from $0$ to $1$ with each value greater than zero. So the linking of the ${\theta }_0$ values to the respective coverage of the rejection region can be viewed as defining a distribution for $\theta$.

There are serious issues regarding how this relates to formal probability theory, especially in multivariate settings where integrating out unwanted ${\theta }_i$ values is required and ordered mappings are difficult to define [16]. The fiducial “transfer” is often not stable in terms of probability and cannot necessarily be interpreted as a Bayesian posterior density [17].

This issue arises since the mapping from the sample space to parameter space through the pivotal is often not unique [18], as typically $T(x,\theta )=g\left[\right(S\left(x\right)-\theta )/\sqrt{VarT\left(x\right)}]$ and the inverse function $g^{-1}\left(x\right)$ are not $1-1$. The procedure is also subject to selection bias as the observed p-value is a component of the construct. But this can be viewed as a potentially positive aspect; a frequentist approach reflecting important aspects of the observed data, rather than only average, pre-planned frequentist concepts.

Where this mapping is $1-1$, for example by imposing a group theoretic perspective (ie. symmetry) on the model as a first step in model definition and conditioning formally on the observed data (usually in the form of residuals), the fiducial argument has better support. This is accomplished in the work of Fraser’s [19] structural inference and applies primarily to location scale models.

There have been recent attempts to generalize the fiducial concept [20], and the relative weighting of the parameter space given by the fiducial approach may be of interest in some settings, as it reflects a frequentist inferential procedure conditional on aspects of the observed data.

6. Location-Scale Statistical Models

Note that even in settings with small sample size, outside of improper non-informative prior selection, isomorphic mappings exist between Bayesian and conditional frequentist methods in linear models, when multivariate symmetry is imposed, yielding pivotal inversions and related probability mappings from the sample space to the parameter space that give identical Bayesian and frequentist inference regions for $\theta$. The symmetry in question typically requires the selection of a right invariant Haar prior density from the Bayesian perspective and imposition of a specific type of symmetry as a basic property of the linear model from both frequentist and Bayesian perspectives [18].

In the pivotal quantity based setting above, if applying a standard Bayesian linear model with a right invariant Haar prior density to preserve a certain type of model symmetry, the conditional frequentist model, for example Fraser’s structural inference [19], is mathematically identical (isomorphic) to the Bayesian model, in small sample linear models. Namely the central regions estimating $\theta$ will be identical. A formal proof of this connection was developed [21].

To state this more practically, in the linear model $y\sim N(X\beta ,{\sigma }^{2}I)$, the conditional distribution of the least squares estimator $b=(X^{\text{'}}X{)}^{-1}{X}^{\text{'}}y$, conditional on the observed residuals $(y-Xb)$, is isomorphic (in $p$ dimensions) to the Bayesian posterior distribution of $\beta$ (again conditional on the observed data), if the prior for $(\beta ,\sigma )$ is the right invariant Haar prior. This choice preserves a symmetry of the linear model under the location-scale group action and the induced group acting on the parameter space in the Bayesian setting. Note that a justification for choosing such a prior can be the imposition of model symmetry itself.

These direct small sample connections between the Bayesian and frequentist settings are more limited in generalized linear models using exponential family densities where mean and variance elements are not independent and the resulting residuals harder to interpret. These models are inherently non-linear and not easily prone to group theoretic representation. The likelihood function is the basis of inference in such settings, often using large sample properties of the maximum likelihood estimate [4].

In large samples in general, due to the central limit theorem, this type of pivotal symmetry often gives similar frequentist and Bayesian results, since in large samples, with non-informative priors and additional assumptions, the Bayesian central limit theorem and the frequentist central limit theorem for the maximum likelihood estimate often overlap [22] and the connection between Bayesian credible regions and frequentist confidence regions is direct for the overall joint distributions. Note that inference for individual parameters ${\theta }_i$ may still differ due to integration of the posterior density and related shrinkage effects.

7. Jeffreys Prior: Preserving Likelihood-Based Invariance

When discussing Bayesian prior selection in general it is useful to consider the Jeffreys prior. This is closely linked in concept to the likelihood function, but within a Bayesian setting. It is primarily justified as preserving the invariance of the likelihood function under rescaling of the parameter space, in the context of the posterior density [10]. However, it is not based on prior existing information, as it is a function of the sample size and likelihood function for the upcoming experiment.

If the Bayesian asymptotic result $\theta \sim N\left({\theta }_{mle}\right(x),J^{-1}(\theta \left|x\right))$ can be assumed to hold, the Jeffreys prior here can be viewed as the inverse of the estimated Cramer-Rao bound $J\left(\theta \right|x{)}^{1/2}=\left(J^{-1}\right(\theta \left|x\right){)}^{-1/2}$. Assuming that $I\left(\theta \right|x)$ is a positive definite symmetric (Hermitian) matrix, this is essentially the inverse of the local curvature of the log likelihood function and will behave locally as the inverse of asymptotic variation; it will be relatively flat where the likelihood is pronounced and non-informative in the sense that it preserves the local shape of the likelihood function about its mode as a key element of the shape of the posterior density, even under transformation of scale $\theta \to g\left(\theta \right)$.

In the multivariate setting, the Jeffreys prior is assumed to be $p\left(\theta \right)=\left|I\right(\theta \left|x\right){|}^{1/2}$ [2]. This has the same basic interpretation regarding invariance, but as a determinant. The determinant of a square semi-positive definite matrix $A$ can be written in several forms useful for statistical application.

In general, for a positive, semi-definite matrix $A$, the following properties hold:

$$tr\left(A\right)=\sum _j{\lambda }_j\left(A\right)$$

$$\mathit{log}det\left(A\right)=\mathit{log}\left(\prod _j{\lambda }_j\left(A\right)\right)=\sum _j\mathit{log}{\lambda }_j\left(A\right)$$

where ${\lambda }_j\left(A\right)$ are the eigenvalues of $A,$ $tr\left(A\right)$ its trace. The multivariate Jeffreys prior can therefore be expressed:

$$\mathit{log}p\left(\theta \right)=(1/2)\mathit{log}|det\left(I\right(\theta \left|x\right)\left)\right|$$

$$=(1/2)\mathit{log}\left(\prod _j{\lambda }_j\left(I\right(\theta \left|x\right))\right)$$

$$=(1/2)\sum _j\mathit{log}{\lambda }_j\left(I\right(\theta \left|x\right))$$

where $I\left(\theta \right|x)$ is a positive-definite symmetric matrix. This links the eigenvalues of the information matrix to the invariance of the posterior density in regard to the scaling of the likelihood function.

8. Prior Density and Matching Information as a Non-Informative Scale

The likelihood function summarizes the information present in the model-data probability structure describing possible outcomes of the experiment in question. The need to consider likelihood properties in relation to prior density selection follows as the model parameter $\theta$ takes its meaning within the context of the probability model and related likelihood function.

If the goal of a non-informative prior is to ensure that the information regarding $\theta$ in the posterior density is dominated by the statistical information in the likelihood function, then requiring the second order derivative based condition:

$${\displaystyle \frac{{\partial }^2\mathit{ln}p\left(\theta \right|x)}{\partial {\theta }^2}}{|}_{{\theta }_{mle}}={\displaystyle \frac{{\partial }^2\mathit{ln}L\left(\theta \right|x)}{\partial {\theta }^2}}{|}_{{\theta }_{mle}}+c$$

matches the curvature of the log-likelihood function about its mode to the curvature of the resulting log posterior density, up to an additive constant. This assumption indirectly imposes restrictions on the form and scale of $p\left(\theta \right)$.

Definition 1: For a scalar parameter$\theta$, define the concept of posterior information as the local curvature of the log-posterior about its mode:

$${\displaystyle \frac{{\partial }^2\mathit{ln}p\left(\theta \right|x)}{\partial {\theta }^2}}={\displaystyle \frac{{\partial }^2}{\partial {\theta }^2}}[\mathit{ln}c+\mathit{ln}p(\theta )+\mathit{ln}L(\theta \left|x\right)]$$

$$={\displaystyle \frac{{\partial }^2}{\partial {\theta }^2}}\mathit{ln}p\left(\theta \right)+{\displaystyle \frac{{\partial }^2}{\partial {\theta }^2}}\mathit{ln}L\left(\theta \right|x)$$

$$={\displaystyle \frac{{\partial }^2}{\partial {\theta }^2}}\mathit{ln}p\left(\theta \right)+J\left(\theta \right|x)$$

where $J\left(\theta \right)$ is the observed Fisher information $J\left(\theta \right|x)=-{\displaystyle \frac{{\partial }^2}{\partial {\theta }^2}}\mathit{ln}L(\theta \left|x\right)$.

Definition 2: Let$p\left(\theta \right|x)$be a unimodal posterior density function. A matching restriction on derivatives resulting in similar information is defined as:

$${\displaystyle \frac{{\partial }^2\mathit{ln}p\left(\theta \right|x)}{\partial {\theta }^2}}\hspace{0.33em}{|}_{\theta ={\theta }_{mle}}={\displaystyle \frac{{\partial }^2\mathit{ln}L\left(\theta \right|x)}{\partial {\theta }^2}}\hspace{0.33em}{|}_{\theta ={\theta }_{mle}}+c$$

where$c$is a constant, not a function of$\theta$, and${\theta }_{mle}$(x) is the mode of the likelihood. This implies that the local shape of the posterior density is given by the shape of the likelihood function.

Theorem 1: If the condition defined above holds, the following second order condition is true for the prior density:

$${\displaystyle \frac{{\partial }^2}{\partial {\theta }^2}}\mathit{ln}p\left(\theta \right){|}_{\theta ={\theta }_{mle}}=c$$

where $c$ is a constant.

Proof: The definition of the posterior density gives:

$${\displaystyle \frac{{\partial }^2\mathit{ln}p\left(\theta \right|x)}{\partial {\theta }^2}}={\displaystyle \frac{{\partial }^2}{\partial {\theta }^2}}[\mathit{ln}c+\mathit{ln}p(\theta )+\mathit{ln}L(\theta \left|x\right)]={\displaystyle \frac{{\partial }^2}{\partial {\theta }^2}}\mathit{ln}p(\theta )+{\displaystyle \frac{{\partial }^2\mathit{ln}L\left(\theta \right|x)}{\partial {\theta }^2}}$$

Assuming similar second order derivatives:

$${\displaystyle \frac{{\partial }^2\mathit{ln}p\left(\theta \right|x)}{\partial {\theta }^2}}={\displaystyle \frac{{\partial }^2\mathit{ln}L\left(\theta \right|x)}{\partial {\theta }^2}}+c$$

and it follows that:

$${\displaystyle \frac{{\partial }^2}{\partial {\theta }^2}}\mathit{ln}p\left(\theta \right)=c$$

Note that this implies a similar condition for all higher order derivatives.

For example,

$${\displaystyle \frac{{\partial }^3\mathit{ln}p\left(\theta \right|x)}{\partial {\theta }^3}}={\displaystyle \frac{\partial }{\partial \theta }}{\displaystyle \frac{{\partial }^2\mathit{ln}p\left(\theta \right|x)}{\partial {\theta }^2}}={\displaystyle \frac{\partial }{\partial \theta }}\left({\displaystyle \frac{{\partial }^2\mathit{ln}L\left(\theta \right|x)}{\partial {\theta }^2}}+C\right)={\displaystyle \frac{{\partial }^3\mathit{ln}L\left(\theta \right|x)}{\partial {\theta }^3}}$$

and more generally, for $m>2$ :

$${\displaystyle \frac{{\partial }^m\mathit{ln}p\left(\theta \right|x)}{\partial {\theta }^m}}={\displaystyle \frac{{\partial }^m\mathit{ln}L\left(\theta \right|x)}{\partial {\theta }^m}}$$

and for the prior:

$${\displaystyle \frac{{\partial }^m\mathit{ln}p\left(\theta \right)}{\partial {\theta }^m}}=0$$

The second order condition above can be used to derive a general form for related priors.

Corollary 1: The family of non-excluded priors when matching higher order log-likelihood and log-posterior derivatives are probability densities of the form:

$$\mathit{ln}p\left(\theta \right)=a{\theta }^2+b\theta +d$$

$$p\left(\theta \right)=\mathit{exp}\left(a{\theta }^2+b\theta +d\right)$$

where $d$ is a rescaling of the norming constant, $a$ and $b$ are hyper-parameters. Some of these can be zero.

This defines a family of possible priors which includes the normal distribution. A reasonable restriction on the parameters $a,b,d$ is to require the prior to be well defined within the context of large sample regularity conditions.

Examples of acceptable prior densities in this context are all prior densities $p\left(\theta \right)$ such that the second derivative of $\mathit{ln}p\left(\theta \right)$ is a constant. The normal, exponential and uniform distributions satisfy this restriction. Technically, so do flat or highly non-informative priors on restricted support in the parameter space.

Theorem 2: Priors that do not meet the matching information restriction above include:

$$p\left(\theta \right)\propto \mathit{exp}({\theta }^m),m>2$$

$$p\left(\theta \right)\propto {\theta }^m,m>2$$

$$p\left(\theta \right)\propto \mathit{sin}(g\left(\theta \right))$$

$$p\left(\theta \right)\propto \mathit{cos}(\mathit{exp}({\theta }^3\left)\right)$$

$$p\left(\theta \right)\propto \sum a_m{\theta }^m;m\ge 3$$

Proof: Follows from applying the previous result.

Prior densities with third or higher order non-zero derivatives are ruled out in this setting. Interestingly, if a selected prior density is based on a continuous but non-differentiable function, for example a fractal prior, which may occur if belief is assessed by physical measurement of individual responses, for example observed EEG assessments, rather than simple betting behavior, then according to the Weierstrass theorem this can be approximated by a higher order polynomial which is differentiable, but would not be acceptable here if the order of the polynomial were $\ge 3$.

9. Similarities to the $\mathit{A}\mathit{R}\mathit{A}$ Utility Based Measure

It is interesting to note that probability as a measure that processes information is not unique. In regard to providing a measure of belief for a specific set of $\theta$ values, the probability measure $p\left(\theta \right)$ is equivalent to $c\cdot p\left(\theta \right)$, where $c$ is any constant, since this measure gives the same set of preferences in regard to possible $\theta$ values as $p\left(\theta \right)$. This is clearly seen on a logarithmic scale. The use of a probability scale $(c=1)\hspace{0.33em}$is a choice with many advantages, but it is only one choice among many to measure relative levels of belief in relation to information for $\theta$.

The use of Bayesian probability is very similar to the broader concept of utility where outcomes are ranked in terms of relative perceived satisfaction [5]. Here, weighting the perceived probabilities of occurrence for possible values for $\theta$ are of interest. Interestingly, utility is itself an ordinal, not a cardinal or absolute scale [23]. It is more similar to likelihood in interpretation when using probability measures to assess information. Note that some of the technical limitations of a utility related theory when a model uses many individual model parameters ${\theta }_i$ in the statistical model are adjusted for in the Bayesian setting by the technical requirement of exchangeability [2].

The form of non-excluded information similar priors given above supports a scale of non-informativity related to utility. Taking the information similar prior density to be of the general form $p\left(\theta \right)=\mathit{exp}(a{\theta }^2+b\theta +d)$, this satisfies the following general condition:

$${\displaystyle \frac{p^{″}\left(\theta \right)}{p^{\text{'}}\left(\theta \right)}}=k$$

or

$${\displaystyle \frac{\partial }{\partial \theta }}{\mathit{ln}p}^{\text{'}}\left(\theta \right)=k$$

a restriction on the form of the prior that results from the initial information prior restriction. In this sense, information similar prior densities are constant and non-informative on this scale.

Interestingly, the utility-based Pratt-Arrow $ARA\hspace{0.33em}$relative risk aversion measure [24] is similar in structure and can be defined as:

$$ARA\left(u\right)=-{\displaystyle \frac{d}{du}}\mathit{log}[{\displaystyle \frac{d}{du}}I\left(u\right)]=-{\displaystyle \frac{\frac{d^2}{d{u}^2}I\left(u\right)}{\frac{d}{du}I\left(u\right)}}$$

where $I\left(u\right)$ is a function representing utility and $I^{\text{'}}\left(u\right)>0$ for all $u$.

If the $ARA$ measure of relative risk is constant, a similar condition is obtained:

$$ARA\left(u\right)=-{\displaystyle \frac{\frac{d^2}{d{u}^2}I\left(u\right)}{\frac{d}{du}I\left(u\right)}}=K$$

A constant value for the $ARA\hspace{0.33em}$measure implies a stabilization in terms of risk avoidance behavior, interpreted in relation to

$${\displaystyle \frac{p^{″}\left(\theta \right)}{p^{\text{'}}\left(\theta \right)}}=k$$

choosing a prior density that avoids differences between the statistical information obtained using the second-order properties of the posterior and the second-order properties of the likelihood function.

Note that when employing the likelihood function and model structure to guide the selection of $p\left(\theta \right)$ a consideration of sample size is necessary. The use and interpretation of frequentist sampling distributions to define likelihood functions requires caveats regarding large sample effects, including central limit theorems. For example, distributions such as the gamma and Weibull distributions, once standardized, excluded in small samples, are not excluded in large samples.

10. Kullback-Liebler Distance

Kullback-Liebler distance [25] is a measure of the distance between two probability densities $p$ and $q$. It is closely related to measures of entropy. It can be written:

$$D(p,q)=\int \mathit{log}{\displaystyle \frac{p\left(x\right)}{q\left(x\right)}}p\left(x\right)dx$$

$$=E_p\left[\mathit{log}{\displaystyle \frac{p\left(x\right)}{q\left(x\right)}}\right]$$

It can be used here to assess the effects of different prior densities on the resulting posterior density. It is especially useful if the densities in question are members of the exponential family $p\left(x\right|\theta )$. Note that the difference between two densities in the exponential family defined by the K-L distance is typically called the deviance.

For example, taking a common likelihood and two prior densities $p_1$ and $p_2$, the distance between the two resulting posterior densities $p$ and $q$ is given by

$$D\left(p\right(\theta \left|x\right),q\left(\theta \right|x\left)\right)=\int \mathit{log}{\displaystyle \frac{p\left(\theta \right|x)}{q\left(\theta \right|x)}}p\left(\theta \right|x)d\theta$$

$$=\int \mathit{log}{\displaystyle \frac{c_1{p}_1\left(\theta \right)L\left(\theta \right|x)}{c_2{p}_2\left(\theta \right)L\left(\theta \right|x)}}p\left(\theta \right|x)d\theta$$

$$=\int \mathit{log}{\displaystyle \frac{c_1{p}_1\left(\theta \right)}{c_2{p}_2\left(\theta \right)}}p\left(\theta \right|x)d\theta$$

$$=\mathit{log}(c_1/c_2)+\int \mathit{log}{\displaystyle \frac{p_1\left(\theta \right)}{p_2\left(\theta \right)}}p(\theta \left|x\right)d\theta$$

$$=C+E_{\theta |x}\left[\mathit{log}{p}_1\left(\theta \right)-\mathit{log}{p}_2\left(\theta \right)\right]$$

where $E_{\theta |x}[\cdot ]$ is taken with regard to $p\left(\theta \right|x)=c\cdot p_1(\theta \left)L\right(\theta \left|x\right)$. This implies the following theorem.

Theorem 3: Given two distinct prior densities and a common likelihood, the distance between the resulting posterior densities is a function of the difference in the posterior mean rate of change of the difference in the$log$prior mean values as a function of$\theta$, assuming regularity conditions support the interchange of differentiation and integration. For the case of two priors satisfying Definition 2, the distance is a function of the first two posterior moments.

Proof: Consider two information similar priors and a common likelihood function. The common likelihood function cancels out on the $log$ scale. Using the above result, the Kullback-Liebler distance between the posterior densities is given by:

$$D\left(p\right(\theta \left|x\right),q\left(\theta \right|x\left)\right)=C+E_{\theta |x}\left[\mathit{log}{p}_1\left(\theta \right)-\mathit{log}{p}_2\left(\theta \right)\right]$$

$$=C+E_{\theta |x}\left[\left(a_1{\theta }^2+b_1\theta +d_1\right)-\left(a_2{\theta }^2+b_2\theta +d_2\right)\right]$$

$$=C+E_{\theta |x}\left[(a_1-a_2){\theta }^2+(b_1-b_2)\theta +(d_1-d_2)\right]$$

$$=C^{\text{'}}+\left[(a_1-a_2)E_{\theta |x}\left({\theta }^2\right)+(b_1-b_2)E_{\theta |x}\left(\theta \right)\right]$$

11. Multiparameter Extensions

The considerations above can be extended to multivariate models and are briefly mentioned here. In multiparameter settings, where $\theta =({\theta }_1,...,{\theta }_p)$, the matching of second derivates can be expressed:

$${\displaystyle \frac{{\partial }^2\mathit{ln}p\left(\theta \right|x)}{\partial \underset{j}{\overset{2}{\theta }}}}{|}_{{\theta }_{mle}}={\displaystyle \frac{{\partial }^2\mathit{ln}L\left(\theta \right|x)}{\partial {\theta }_j^2}}{|}_{{\theta }_{mle}}+c,\hspace{0.33em}j=1,...,p$$

The related prior density restrictions can be generalized via partial derivatives:

$${\displaystyle \frac{\partial }{\partial {\theta }_1}}{\mathit{ln}p}^{\text{'}}\left(\theta \right)=k_1$$

$${\displaystyle \frac{\partial }{\partial {\theta }_2}}{\mathit{ln}p}^{\text{'}}\left(\theta \right)=k_2$$

$$⋮$$

$${\displaystyle \frac{\partial }{\partial {\theta }_p}}{\mathit{ln}p}^{\text{'}}\left(\theta \right)=k_p.$$

where $k_i$ are constants. This is a specific form of non-informativity in higher dimensions.

If symmetry or independence is useful, $p\left(\theta \right)$ $=\prod _{j=1}^p{p}_j\left({\theta }_i\right)$ can be assumed. If the cross derivatives can be set equal to zero, multiparameter priors can be taken with the general form:

$$\mathit{ln}p\left(\theta \right)=\sum a_j\underset{j}{\overset{2}{\theta }}+\sum b_j{\theta }_j+\sum d_j$$

This rules out multivariate prior distributions with factors of ${\theta }_j$ that are higher order polynomials ( $k>2)$ or transcendental functions such as:

$$p\left(\theta \right)\propto \left(\mathit{sin}({\theta }_1\right),...,\mathit{sin}({\theta }_p\left)\right)$$

or

$$p\left(\theta \right)\propto \left(\mathit{exp}(\mathit{exp}({\theta }_1\right)),...,\mathit{exp}(\mathit{exp}({\theta }_p)\left)\right).$$

The resulting posterior density in the large sample vector parameter setting is a multivariate distribution and can be expressed generally as:

$$p\left(\theta \right|x)=c\cdot \mathit{exp}\left(\sum a_i{\theta }_i^2+\sum b_i{\theta }_i+\sum d_i\right)\cdot L(\theta \left|x\right)$$

12. Discussion

Many applications of Bayesian methods view the selection of $p\left(\theta \right)$ as primarily subjective in nature. The parameter $\theta$ is typically defined within the context and constraints of the probability model, inferential approach and related likelihood function. The choice of prior however, requires justification within the scientific or medical context being studied.

When model-based restrictions on the prior are assumed, for example in relation to model symmetry or likelihood properties, the set of prior densities available for subjective choice reflects the context of the science and related statistical model that defines $\theta$.

In particular, if the posterior information, as characterized by the local curvature of the posterior density $p\left(\theta \right|x)$, is assumed proportional to the local curvature of the log-likelihood function $L\left(\theta \right|x)$, a type of matching condition, then exclusionary restrictions affect the form of the prior density $p\left(\theta \right)$ that is available to the researcher.

The class of prior densities available to the researcher within this matching restriction are non-informative in a unique manner, reflecting a structure similar to that found in the $ARA$ utility based measure related to the assessment of relative risk.

These considerations can be extended to multivariate statistical models, along with prior robustness, which can be examined applying the Kullback-Liebler measure of distance.