Bayesian estimation of the discrete coefficient of determination

where ε ₀ is the minimum error of predicting Y by a constant (i.e., in the absence of observations) and ε is the minimum error of predicting Y based on the observation of X. Since ε≤ε ₀ (all sensible error criteria satisfy this property), the CoD ranges from 0 to 1. The closer it is to one, the closer ε is to zero and the tighter the association between predictor and target variables, whereas the closer it is to zero, the closer ε is to ε ₀ and the weaker the association is. By convention, CoD=0 when ε ₀=0. The CoD is a function only of the distribution of (X,Y); in particular, it is not a function of sample data. This definition of the CoD reduces gracefully to the classical one in the case when (X,Y) is jointly Gaussian [7].

This formula gives the relationship between the CoD and the parameters of the distribution of (X,Y).

In practice, the distributional parameters are generally unknown, and one would like to estimate the CoD from sample data. We present in this section the derivation of two Bayesian estimators for the CoD in (4). One approach is analogous to that followed by [17] in defining the Bayesian MMSE prediction error estimator, whereas the other one makes use of the optimal Bayesian predictor (OBP), a straightforward generalization of the optimal Bayesian classifier (OBC), introduced in [20].

We will assume that an i.i.d. sample S _n ={(X ₁,Y ₁),…,(X _n ,Y _n )} from the distribution of (X,Y) is available. Given S _n , define U _i as the number of sample points such that X=x ⁱ and Y=0, and V _i as the number of sample points such that X=x ⁱ and Y=1, for \(i = 1, \dots,2^{d}\). Note that \(N_{0} = \sum _{i=1}^{2^{d}} U_{i}\) and \(N_{1} = \sum _{i=1}^{2^{d}} V_{i}\) are the (random) sample sizes corresponding to Y=0 and Y=1, respectively.

Let \(\phantom {\dot {i}\!}{\mathbf p} = (p_{1}, \ldots, p_{2^{d}})\), \(\phantom {\dot {i}\!}{\mathbf q} = (q_{1},\ldots,q_{2^{d}})\), and θ=(c,p,q), where 0≤c,p _i ,q _i ≤1, and \(\sum _{i=1}^{2^{d}} p_{i} = \sum _{i=1}^{2^{d}} q_{i} =1\). As shown in the previous section, the distribution of (X,Y) is completely specified by the parameter vector θ. The Bayesian approach treats θ as a random variable, the prior distribution of which can take advantage of a priori knowledge about the problem. We will assume that c, p, and q are independent, i.e., f(θ)=f(c)f(p)f(q). It is shown in [17] that this implies that the posterior distribution of θ also factors f(θ∣S _n )=f(c∣S _n )f(p∣S _n )f(q∣S _n ).

where the hyperparameters α, β, α _i , β _i , i=1,…,2 ^d , are positive numbers. These distributions have bounded supports; the Beta distribution is defined over the interval [0,1], while the Dirichlet distribution is defined over the simplex of 2 ^d nonnegative numbers that add up to one. The shapes of the distributions are controlled by the concentration parameters Δ _c =α+β, \(\Delta _{p} = \sum _{j=1}^{2^{d}} \alpha _{j}\), and \(\Delta _{q} = \sum _{j=1}^{2^{d}} \beta _{j}\), and the base measures c ₀=α/Δ _c , \({\mathbf p}_{0} = (\alpha _{1}/\Delta _{p}, \dots, \alpha _{2^{d}}/\Delta _{p})\), and \({\mathbf q}_{0} = (\beta _{1}/\Delta _{q}, \dots, \beta _{2^{d}}/\Delta _{q})\). Please refer to Appendices A and B for definitions and important facts about the Beta and Dirichlet distributions, which will be needed in the sequel.

where n ₀ and n ₁ are the observed sample sizes corresponding to Y=0 and Y=1, respectively, while U _i and V _i are the observed sample values of the random variables U _i and V _i , respectively.

In this section, we investigate the accuracy of the Bayesian CoD estimators proposed in the previous section. We distinguish between two types of accuracy metrics: global metrics concern the average performance over all samples and all distributions of (X,Y), weighted by the prior distribution of θ, whereas fixed-parameter metrics have to do with the average performance over all samples, but under a particular distribution of (X,Y), corresponding to a fixed value of the parameter θ. Fixed-parameter metrics thus evaluate the proposed Bayesian estimators from a purely frequentist perspective.

It becomes clear that the fixed-parameter bias, variance, and RMS of a Bayesian CoD estimator can be obtained with knowledge of the first and second moments \(E_{\mathbf {S}_{n} \mid \boldsymbol {\theta }}\!\left [ \frac {\hat {\varepsilon }}{\hat {\varepsilon }_{0}} \right ]\) and \(E_{\mathbf {S}_{n} \mid \boldsymbol {\theta }}\!\left [\frac {\hat {\varepsilon }^{2}}{\hat {\varepsilon }_{0}^{2}}\right ]\).

The corresponding global accuracy metrics are obtained by taking expectation of the previous quantities with respect to the marginal (i.e., prior) distribution of θ.

As mentioned previously, the global bias of the Bayesian MMSE CoD estimator is zero and its global RMS is minimal among all CoD estimators. However, this does not imply that its fixed-parameter bias is zero or that its fixed-parameter RMS is minimum for all values of the parameter.

In what follows, we give exact expressions for the computation of \(E_{\mathbf {S}_{n} \mid \boldsymbol {\theta }}\!\left [ \frac {\hat {\varepsilon }}{\hat {\varepsilon }_{0}} \right ]\) and \(E_{\mathbf {S}_{n} \mid \boldsymbol {\theta }}\!\left [\frac {\hat {\varepsilon }^{2}} {\hat {\varepsilon }_{0}^{2}}\right ]\)for the OBP CoD estimator. As argued previously, this allows the exact computation of the fixed-parameter bias, variance, and RMS of that CoD estimator. Via simple numerical integration, it is possible then to obtain the global bias, variance, and RMS. It turns out that similar expressions for the MMSE CoD estimator are much harder to obtain; the performance of that estimator are studied via a numerical approach in the next section.

All the expectations and probabilities below are with respect to S _n ∣ θ (the subscript will be omitted for convenience). In the expressions below, c, p _i , and q _i , for i=1,…,2 ^d refer to the (deterministic) parameters in θ.

We assume that α−⌊α⌋=β−⌊β⌋ but α≠β. Without loss of generality, we assume that α>β, and let α=β+δ, where δ is a positive integer. Notice that it is easy to show in this case that \(\lfloor \frac {n+\beta -\alpha }{2} \rfloor + \alpha = \lfloor \frac {n+\alpha -\beta }{2} \rfloor + \beta \) by considering the evenness and oddness of n and δ. Therefore, we have L ₀⊂L ₁. In the following, we discuss two cases when n+β−α is even and when n+β−α is odd.

(1) When n+β−α is even, the event [M=m] is equal to the union of the disjoint events [ n ₀=m−α], and [n ₁=m−β]=[n ₀=n−m+β], for \(m\in L_{0} \backslash \left \{\alpha + \frac {n+\beta -\alpha }{2} \right \}\), whereas \(\left [M=\alpha +\frac {n+\beta -\alpha }{2} \right ] = \left [n_{0} = m - \alpha = \frac {n+\beta -\alpha }{2} \right ]\), and [ M=m] = [ n ₀=n−m+β], for m∈L ₁∖L ₀.

with \(P(U_{i}=k,V_{i}=l \mid n_{0} = r) \,=\, \binom {r}{k}{p_{i}^{k}}(1-p_{i})^{r-k}\binom {n-r}{l}{q_{i}^{l}}(1-q_{i})^{n-r-l}\). The expression for \(E\left [\frac {\hat {\varepsilon }_{\text {OBP}}}{(n_{1}+\beta)/(n+\Delta _{c})} \;\big |\; n_{0} = n-r\right ]\) is obtained from (27), with r+α replaced by r+β.

(2) When n+β−α is odd, the event [M=m] is equal to the union of the disjoint events [n ₀=m−α], and [n ₁=m−β]=[n ₀=n−m+β], for m∈L ₀, whereas [M=m]=[n ₀=n−m+β], for m∈L ₁∖L ₀. By applying the same reasoning, we have the same expression as in (26). Note that \(I_{n_{1} \neq \frac {n+\alpha -\beta }{2}} \) is always equal to 1 in this case since \(\frac {n+\alpha -\beta }{2}\) is not an integer.

with \(P(U_{i}=k, V_{i}=l, U_{j}=r, V_{j}=s \mid n_{0}=t) = \binom {n_{0}}{k,r}{p_{i}^{k}}{p_{j}^{r}}(1-p_{i}-p_{j})^{n_{0}-k-r} \binom {n-n_{0}}{l,s}{q_{i}^{l}}{q_{j}^{s}}(1-q_{i}-q_{j})^{n-n_{0}-l-s}\). The expression for \(E\left [\frac {\hat {\varepsilon }_{\text {OBP}}^{2}}{(n_{1}+\beta)^{2}/(n+\Delta _{c})^{2}} \;\bigg |\; n_{0} = n - t\right ]\) is obtained from (30) with t+α replaced by t+β.

We discuss in this section the application of the Bayesian CoD estimation approach discussed previously to the inference of gene regulatory networks. We apply the proposed inference procedure on data collected in a study of metastatic melanoma [24], containing 31 binarized sample expression profiles, which have been binarized, with 0 indicate no significant expression whereas 1 represents significant expression (either over- or under-expression). It was found in [24] that the WNT5A gene is a major driver of processes that lead to metastatic melanoma. We derive the logic relationships and wiring of a 7-gene WNT5A network consisting of genes selected using data analysis and prior biological knowledge: WNT5A, pirin, S100P, RET1, MART1, HADHB, and STC2; for more details about the selection of these genes, see [25, 26].

where f:{0,1} ^d →{0,1} is a Boolean function, the symbol “ ⊕” indicates modulo-2 addition, and N∈{0,1} is a noise Bernoulli random variable, independent from X, such that P(N=0)=p. The modulo-2 addition behaves as a XOR operation, which flips the state of the target Y when N=1, and leaves it unaltered when N=0. Hence, p quantifies the predictive power of the model: if p=1, the system is noiseless and prediction is deterministic, while if p<1, there is a degree of indeterminacy in the state of the target given the state of the predictors. This model is studied in detail in [15], where an inference procedure, based on a maximum-likelihood CoD estimator, is proposed to select the unknown Boolean function f, assuming that f is a member of a candidate model set F containing Boolean functions that depend on the same number k of essential variables. Each f in F is specified by (1) a Boolean function g:{0,1} ^k →{0,1} and (2) the indices for the predicting variable set {i ₁,…,i _k }⊂{1,…,d}, or wiring, such that \(f(\mathbf {X}) \,=\, g\left (X_{i_{1}},\ldots,X_{i_{k}}\right)\). If the candidate boolean functions g belong to a model set G, then the total number of possible models is \(|G| \times \binom {d}{l}\).

Here, we modify the network inference in [15] to allow the use of the Bayesian CoD estimators described previously. For a given target Y and predictor set X, we assume Dirichlet prior distributions as in (5). Instead of adopting a non-informative choice of hyperparameters, we employ an “empirical Bayes” approach, where the hyperparameters are estimated in part from the sample data, as described next.

Recall from Section 3 that the shape of the Dirichlet prior distribution is determined by the hyperparameters through a location parameter, called the base measure, and a concentration parameter. Our strategy is to set up the estimates in (34) as the base measure, so that the Dirichlet priors are concentrated around them, to a degree specified by the concentration parameter. Formally, the hyperparameters are set to: \(\{\alpha _{1}, \dots, \alpha _{2^{d}}\} = \{ \lceil \hat {p}_{1} \Delta \rceil, \ldots, \lceil \hat {p}_{2^{d}} \Delta \rceil \}\), \(\{\beta _{1}, \dots, \beta _{2^{d}}\} = \{ \lceil \hat {q}_{1} \Delta \rceil, \ldots, \lceil \hat {q}_{2^{d}} \Delta \rceil \}\), \(\alpha = \lceil \hat {c} \Delta \rceil \) and \(\beta = \lceil (1-\hat {c}) \Delta \rceil \), where ⌈x⌉ gives the smallest integer larger or equal to x. The value of δ is tuned by the experimenter, either manually or using a data-driven procedure.

We are now ready to state the procedure to select a function f in F, consisting of a k-predictor Boolean function g and its wiring.

6.1 Bayesian model selection procedure

1. 1.

   For each of the Boolean functions g∈G, compute the prior hyperparameters as described earlier. Obtain the MMSE Bayesian CoD / OBP CoD estimate under each of the \(\binom {d}{k}\) possible wirings. Pick the wiring for g that produces the largest CoD estimate. Ties, if any, are broken randomly.

    
2. 2.

   Among the |G| pairs of Boolean function g and wiring obtained in the previous step, select the one that produces the largest predictive power estimate \(\hat {p}\). Ties, if any, are broken randomly.

    

In our experiment with the 7-gene WNT5A network, we consider in turn each gene as a target and the remaining six genes as predictors (so that a gene cannot be a predictor of itself). Hence, d=6. In addition, we assume that that each gene is predicted by three genes out of the six predictors. Therefore, k=3 and there are \(\binom {6}{3}=20\) possible wirings for each target gene. The set G contains all 218 Boolean functions of exactly three essential variables (this is less than the full set of \(2^{2^{3}} = 256\) 3-input Boolean functions since those that are reducible to 0-, 1-, and 2-input logics are not considered). We set Δ=1.0 and apply the proposed Bayesian model selection procedure to infer a gene regulatory network for the MMSE and OBP CoD estimators. We also obtain the gene regulatory network produced by employing the standard model selection procedure, which picks the predictor set (among all \(\binom {6}{3} = 20\) choices, in this case) with the largest estimated resubstitution CoD [25].

The results are presented in Fig. 6. The diagrams represent the predicted logic functions as binary strings (in the usual logic table order; e.g., AND = 00000001) and the predicted wirings as oriented edges, and, in addition, the estimated CoD in each case is displayed. We can see that the predicted logic functions and wirings for the three networks are similar, especially in the cases of the OBP and resubstitution CoD networks. If one considers only the three top predicted relationships according to CoD magnitude, one obtains the diagrams depicted in Fig. 7, which show that the same network is inferred by the OBP and resubstitution CoDs, which differ from the network obtained with the MMSE CoD by only a single arrow shift in the wirings (the inferred logics in all three cases are also very similar, differing by only a few bit shifts). The important difference between the Bayesian and standard approaches that can be observed from this experiment is in the estimated CoD magnitudes: those estimated with the standard resubstitution CoD tend to be much larger than the ones estimated with the Bayesian CoDs. This reflects the optimistic bias that tends to be displayed by resubstitution [27], a problem that is avoided by the Bayesian CoD estimators.

¹ A proof of this fact is given in the Appendix of [14].

Clearly, B(a,b)=B(b,a).

For example, E[ X]=B(a+1,b)/B(a,b)=a/(a+b) (the second equality can be proved using the definition of the Beta function in terms of the Gamma function and the properties of the latter [28]). Similarly, E[ 1/X]=B(a,b−1)/B(a,b)=(a+b−1)/(b−1), provided that b>1.

Notice that B(a,b)=IB(1;a,b).

which follows easily from the definitions of the Beta density and the incomplete Beta function, and the fact that X∈[ 0,1]. In particular, if x≥1, then E[X ^k (1−X) ^l I _X≤x ]=E[X ^k (1−X) ^l ].

Clearly, all the previous quantities can be computed in terms of the incomplete beta function, an expression of which is given by the next result.

Notice that \(B(a,b) = \sum _{i=0}^{P} r_{i}(a,b)\). Note also that the general case reduces to the special case if b is an integer. An equivalent expression can be derived where a appears in the binomial coefficient instead, which can then be used if a is an integer. If neither a nor b are integers, an approximation can be obtained by truncating the resulting infinite series, or by using a numerical software package.

If both a and b are integers, then IB(x;a,b) reduces to a polynomial in x. Otherwise, it is a simple matter to replace the finite summations by infinite series as specified in Theorem 1.

From the previous equations, one can see that, as δ approaches infinity, variances converge to zero and X becomes equal to the base measure with probability 1; in addition, covariances also go to zero, rendering the components of X uncorrelated. The special case a _i =1, for all i=1,…,K corresponds to a uniform over S _K−1. This corresponds to a uniform base measure and concentration parameter Δ=K. If the base measure is not uniform but Δ=K, the distribution is approximately uniform. For δ approaching zero, the distribution becomes concentrated at the boundary of the simplex.

Summing up, large δ implies large probability density around the base measure, Δ=K implies a nearly uniform distribution, whereas δ close to zero produces sparse sample vectors with most of the components close to zero.

The Dirichlet distribution is the multivariate generalization of the Beta distribution, in the sense that the components of a Dirichlet-distributed vector X=(X ₁,…,X _K ) are Beta distributed: X _i ∼Beta(a _i ,Δ−a _i ), for i=1,…,K. Notice that in the case K=2 the Dirichlet distribution essentially reduces to the Beta distribution.