 Research Article
 Open Access
A Hypothesis Test for Equality of Bayesian Network Models
 Anthony Almudevar^{1}Email author
https://doi.org/10.1155/2010/947564
© Anthony Almudevar. 2010
 Received: 26 March 2010
 Accepted: 5 August 2010
 Published: 9 August 2010
Abstract
Bayesian network models are commonly used to model gene expression data. Some applications require a comparison of the network structure of a set of genes between varying phenotypes. In principle, separately fit models can be directly compared, but it is difficult to assign statistical significance to any observed differences. There would therefore be an advantage to the development of a rigorous hypothesis test for homogeneity of network structure. In this paper, a generalized likelihood ratio test based on Bayesian network models is developed, with significance level estimated using permutation replications. In order to be computationally feasible, a number of algorithms are introduced. First, a method for approximating multivariate distributions due to Chow and Liu (1968) is adapted, permitting the polynomialtime calculation of a maximum likelihood Bayesian network with maximum indegree of one. Second, sequential testing principles are applied to the permutation test, allowing significant reduction of computation time while preserving reported error rates used in multiple testing. The method is applied to geneset analysis, using two sets of experimental data, and some advantage to a pathway modelling approach to this problem is reported.
Keywords
 False Discovery Rate
 Bayesian Network
 Minimum Span Tree
 Bayesian Network Model
 Sequential Probability Ratio Test
1. Introduction
Graphical models play a central role in modelling genomic data, largely because the pathway structure governing the interactions of cellular components induces statistical dependence naturally described by directed or undirected graphs [1–3]. These models vary in their formal structure. While a Boolean network can be interpreted as a set of state transition rules, Bayesian or Markov networks reduce to static multivariate densities on random vectors extracted from genomic data. Such densities are designed to model coexpression patterns resulting from functional cooperation. Our concern will be with this type of multivariate model. Although the ideas presented here extend naturally to various forms of genomic data, to fix ideas we will refer specifically to multivariate samples of microarray gene expression data.
In this paper, we consider the problem of comparing network models for a common set of genes under varying phenotypes. In principle, separately fit models can be directly compared. This approach is discussed in [3] and is based on distances definable on a space of graphs. Significance levels are estimated using replications of random graphs similar in structure to the estimated models.
The algorithm proposed below differs significantly from the direct graph approach. We will formulate the problem as a twosample test in which significance levels are estimated by randomly permuting phenotypes. This requires only the minimal assumption of independence with respect to subjects.
Our strategy will be to confine attention to Bayesian network models (Section 2). Fitting Bayesian networks is computationally difficult, so a simplified model is developed for which a polynomialtime algorithm exists for maximum likelihood calculations. A twosample hypotheses test based on the general likelihood ratio test statistic is introduced in Section 3. In Section 4, we discuss the application of sequential testing principles to permutation replications. This may be done in a way which permits the reporting of error rates commonly used in multiple testing procedures. In Section 5, the methodology is applied to the problem of gene set (GS) analysis, in which high dimensional arrays of gene expression data are screened for differential expression (DE) by comparing gene sets defined by known functional relationships, in place of individual gene expressions. This follows the paradigm originally proposed in gene set enrichment analysis (GSEA) [4–6]. The method will be applied to two wellknown microarray data sets.
An R library of source code implementing the algorithms proposed here may be downloaded at http://www.urmc.rochester.edu/biostat/people/faculty/almudevar.cfm.
2. Network Models
where is the set of parents of node . Intuitively, describes a causal relationship between node and nodes .
The advantage of (1) is the reduction in the degrees of freedom of the model while preserving coexpression structure. Also, some flexibility is available with respect to the choice of the conditional densities of (1), with Gaussian, multinomial, and Gamma forms commonly used [7]. We note that BNs are commonly used in many genomic applications [7–9].
2.1. Gaussian Bayesian Network Model
where is the mean squared error of a linear regression fit of the offspring expressions onto those of the parents.
2.2. Restricted Bayesian Networks
Fitting BNs involves optimization over the space of topologies and hence is computationally intensive [9]. While exact algorithms are available [11], they will generally require too great a computation time for the application described below. A recent application of exact techniques to the problem of pedigree reconstruction (a BN with maximum indegree of 2) was described in [12]. Using methods proposed in [13] the exact computation of the maximum likelihood of a pedigree with 29 individuals (nodes) required 8 minutes. The author of [12] agrees with the conclusion reported in [13], that the method is not viable for BNs with greater than 32 nodes.
It is possible to control the size of the computation by placing a cap on the permissable indegree of each node, though the problem remains difficult even for (see, e.g., [14]). On the other hand, a method for fitting BNs with constraint in polynomial time is available under certain assumptions satisfied in our application. This method is based on the equivalence of the approximation of multivariate probability models using treestructured dependence and the minimum spanning tree (MST) problem as described in [15]. The objective is the minimization of an information difference , where is the target density, and is selected from a class of treestructured approximating densities. Interest in [15] is restricted to discrete densities. We find, however, that the basic idea extends to general BNs in a natural way. See [16] for further discussion of this model.
Many heuristic or approximate methods exist for fitting Bayesian networks. See [17] for a recent survey. Such algorithms are usually based on MCMC techniques or heuristic algorithms such as TABU searches [18]. We note that the proposed hypothesis test will depend on the calculation of a maximum likelihood ratio, hence it is important to have reasonable guarantees that a maximum has been reached. Thus, given the choice between an exact solution of a restricted class of models or an approximate solution of a general class of models, the former seems preferable. Considering also that in the application described below a solution is required for cases number in "10 s or 100 s'' of thousands, a polynomial time exact solution to a restricted class of models appears to be the best choice.
Suppose we may construct estimators , . We then assume there is some selection rule for each . This will typically be the exact or approximate maximum likelihood estimate (MLE) on parameter space . We will need the following assumptions.
(A1) For each , , and .
(A2) For each we have .
A spanning tree on nodes is an acyclic connected undirected graph. Given edge weights , a minimum spanning tree (MST) is any spanning tree minimizing the sum of its edge weights among all spanning trees. A number of wellknown polynomial time algorithms exist to construct a MST. Two that are commonly described are Prim's and Kruskal's algorithms [19]. Kruskal's algorithm is described in [15]. In the following theorem, the problem of maximizing is expressed as a MST problem.
Theorem 1.
If assumptions hold, then maximizing over is equivalent to determining the MST for edge weights .
Proof.
Under assumption (A1), from definition (4) it follows that depends on only through the term . Then suppose maximizes . For any spanning tree define and suppose minimizes . Assume is not connected. There must be at least two nodes for which , and for which the respective subgraphs containing are unconnected. In this case, extend to by adding directed edge . We must have , and by (A2) we have . We may therefore assume is connected. The undirected graph of is a spanning tree, so .
Next, note that can be identified with an element of by defining any node as a root node, enumerating all paths from the root node to terminal nodes, then assigning edge directions to conform to these paths. This implies , which in turn implies , and that may be selected so that can be identified with .
Remark 1.
In general, the optimizing graph from will not be unique. First, the solution to the MST problem need not be unique. Second, there will always be at least two extensions of a spanning tree to a BN.
noting that, since , assumption (A2) holds.
3. General Maximum Likelihood Ratio Test
Identification of nonhomogeneity between two Bayesian networks will be based on a general maximum likelihood ratio test (MLRT). It is important to note the properties of the MLRT are well understood in parametric inference of limited dimension, and a sampling distribution can be accurately approximated with a large enough sample size. These known properties no longer apply in the type of problem considered here, primarily due to the small sample size, large number of parameters, and the fact that optimization over a discrete space is performed. In addition, the maximum likelihood principle itself favors spurious complexity when no model selection principles are used. While we cannot claim that the MLRT possesses any optimum properties in this application, the use of a permutation procedure will permit accurate estimates of the observed significance level while the use of the restricted model class will control to some degree the degrees of freedom of the model. See, for example, [20] for a general discussion of these issues.
Asymptotic distribution theory is not relevant here due to small sample size and the fact that optimization is performed in part over a discrete space of models, so a two sample permutation procedure will be used. Permutations will be approximately balanced to reduce spurious variability when a true difference in expression pattern exists (see, e. g., [21] for discussion). This can be done by changing group labels of randomly selecting sample vectors from each of and . This results in permutation replicate samples and . The balanced procedure ensures that each permutation replicate sample contains approximately equal proportions of the original samples.
We now define Algorithm 1.
 (1)
Determine by maximizing , , (MST algorithm).
( ) Set .
( ) Construct replications in the following way. For each replication , create random replicate samples and , then determine which maximize , . Set .
Note that the quantity is permutation invariant and hence need not be recalculated within the permutation procedure.
4. Permutation Tests with Stopping Rules
Permutation or bootstrap tests usually reduce to the estimation of a binomial probability by direct simulation. Since interest is usually in identifying small values, it would seem redundant to continue sampling when, for example, the first ten simulations lead to an estimate of 1/2. This suggests that a stopping rule may be applied to permutation sampling, resulting in significant reduction in computation time, provided it can be incorporated into a valid inference statement. A variety of such procedures have been described in the literature but do not seem to have been widely adopted in genomic discovery applications [22–24].
Formally, is a stopping time if the occurrence of event can be determined from . We may then design an algorithm which terminates after sampling a sequence of exactly length from , then outputs , from which the hypothesis decision is resolved. We refer to such a procedure as a stopped procedure. A fixed procedure (such as Algorithm 1) can be regarded as a special case of a stopped procedure in which .
An important distinction will have to be made between a single test and a multiple testing procedure (MTP), which is a collection of hypothesis tests with rejection rules that control for a global error rate such as false discovery rate (FDR), familywise error rate (FWER), or per family error rate (PFER) [25]. In the single test application, we may set a fixed significance level and continue replications until we conclude that the value is above or below . For an MTP, it will be important to be able to estimate small values, so a stopping rule which permits this is needed. Although the two cases have different structure, in our development they will both be based on the sequential probability ratio test (SPRT), first proposed in [26], which we now describe.
4.1. Sequential Probability Ratio Test (SPRT)
It can be shown that . If we conclude and conclude otherwise. We define errors and . It turns out that the SPRT is optimal under the given assumptions in the sense that it minimizes among all sequential tests (which includes fixed sample tests) with respective error probabilities no larger than . Approximate formulae for and are given in [27].
Hypothesis testing usually involves composite hypotheses, with distinct interpretations for the null and alternative hypothesis. One method of adapting the SPRT to this case is to select surrogate simple hypotheses. For example, to test versus , we could select simple hypotheses and . In this case, we would need to know the entire power function, which may be estimated using simulations.
An additional issue then arises in that the expected stopping time may be very large for . This can be accommodated using truncation. Suppose a reasonable choice for a fixed sample size is . We would then use truncated stopping time , with defined in (10). When , we could, for example, select hypothesis if . These modifications are discussed in [27].
4.2. Single Hypothesis Test
Suppose we adopt a fixed significance level for a single hypothesis test. If is the (unknown) true significance level, we are interested in resolving the hypothesis : . The properties of the test are summarized in a power curve, that is, the probability of deciding is true for each . An example of this procedure is given in [28], for , using a SPRT with parameters , , , , and truncation at . Hypothesis is concluded if when ; otherwise when .
4.3. Multiple Hypothesis Tests
where the quantity defines the particular MTP. It is assumed that is an increasing function of for all . The procedure is implemented by rejecting all null hypotheses for which . Depending on the MTP, various forms of error, usually either familywise error rate (FWER) or false discovery rate (FDR), are controlled at the level. For example, the BenjaminiHochberg (BH) procedure is a stepup procedure defined by and controls for FDR for independent hypothesis tests. A comprehensive treatment of this topic is given in, for example, [25].
Suppose we have probabilities ( values associated with tests). For each test , we may generate as the cumulative sum defined in (9). Now suppose we define any stopping time , bounded by , for each sequence (this may or may not be related to the SPRT). Then define estimates , with .
For a fixed MTP, the estimates would replace the true values in (11), yielding estimated adjusted values while for the stopped MTP adjusted values are produced in the same manner using . It is easily seen that while the rankings of (accounting for ties) are equal to the rankings of . Furthermore, the formulae in (11) are monotone in , so we must have . Thus, the stopped procedure may be seen as being embedded in the fixed procedure. It inherits whatever error control is given for the fixed MTP, with the advantage that the calculation of the adjusted values uses only the first replications for the th test.
The procedure will always be correct in that it is strictly more conservative than the fixed MTP in which it is embedded, no matter which stopping time is used. The remaining issue is the selection of which will equal for small enough values of but will also have for larger values of . It is a simple matter, then, to modify the SPRT described in Section 4.2 by eliminating the lower bound (equivalently ). We will adopt this design in this paper. This gives Algorithm 2.
 (1)
Same as Algorithm 1, step 1.
( ) Same as Algorithm 1, step 2.
( ) Simulate replicates in Algorithm 1, step 3, until the following stopping criterion is met. Set , and let , where . Stop sampling at the th replication if , where , or until , whichever occurs first.
otherwise set .
The values generated by Algorithm 2 can then be used in a stopped MTP as described in this section.
5. GeneSet Analysis
A recent trend in the analysis of microarray data has been to base the discovery of phenotypeinduced DE on gene sets rather than individual genes. The reasoning is that if genes in a given set are related by common pathway membership or other transcriptional process, then there should be an aggregate change in gene expression pattern. This should give increased statistical power, as well as enhanced interpretability, especially given the lack of reproducibility in univariate gene discovery due to the stringent requirements imposed by multiple testing adjustments. Thus, the discovery process reduces to a much smaller number of hypothesis tests with more direct biological meaning. Some objections may be raised concerning the selection of the gene sets when theses sets are themselves determined experimentally. Additionally, gene sets may overlap. While these problems need to be addressed, it is also true that such gene set methods have been shown to detect DE not uncovered by univariate screens.
A crucial problem in gene set analysis is the choice of test statistic. The problem of testing against equality of random vectors in , , is fundamentally different from the univariate case . The range of statistics one would consider for is reasonably limited, the choice being largely driven by distributional considerations. For , new structural or geometric considerations arise. For example, we may have differential expression between some but not all genes in the gene set, which makes selection of a single optimal test statistic impossible. Alternatively, the experimental random vectors may differ in their level of coexpression independently of their level of marginal DE.
In fact, almost all GS procedures directly measure aggregate DE, so an important question is whether or not phenotypic variation is almost completely expressible as DE. If so, then a DE based statistic will have fewer degrees of freedom, hence more power, than one based on a more complex model. Otherwise, a reasonable conjecture is that a compound GS analysis will work best, employing a DE statistic as well as one more sensitive to changes in coexpression patterns.
Correlations have been used in a number of gene discovery applications. They may be used to associate genes of unknown function with known pathways [29, 30]. Additionally, a number of GS procedures exist which incorporate correlation structure into the procedure [31–33]. However, a direct comparison of correlations is not practical due to the large number ( ) of distinct correlation parameters. Therefore, there is a considerable advantage to the statistic (7) based on the reduced BN model, in that the correlation structure can be summarized by the correlation parameters output by the MST algorithm, yielding a transitive dependence model similar to that effectively exploited in [29].
It is important to refer to a methodological characterization given in [34]. A distinction is made between two types of null hypotheses. Suppose we are given samples of expression levels from a gene set from two phenotypes. Suppose also that for each gene in and its complement , a statistical measure of differential expression is available. For a competitive test, the null hypothesis is that the prevalence of differential expression in is no greater than in . For a selfcontained test, the null hypothesis is that no genes in are differentially expressed. In the GSEA method of [4, 5] concern is with . In most subsequent methods, including the one proposed here, is used.
For general discussions of the issues raised here, see [35–37]. Comprehensive surveys of specific methods can be found in [38] or [39].
5.1. Experimental Data
We will demonstrate the algorithm proposed here on two data sets examined elsewhere in the literature. These were obtained from the GSEA website www.broad.mit.edu/gsea [6]. In [5], a data set p53 is extracted from the NCI60 collection of cancer cell lines, with 17 cell lines classified as normal, and 33 classified as carrying mutations of p53. We also examine the DIABETES data set introduced in [4], consisting of microaray profiles of skeletal muscle biopsies from 43 males. For the DIABETES data set used here, there were 17 normal glucose tolerance (NGT) subjects and 17 diabetes (DMT) subjects. For gene sets, we used one of the gene set lists compiled in [5], denoted , consisting of 472 gene sets with products collectively involved in various metabolic and signalling pathways, as well as 50 sets containing genes exhibiting coregulated response to various perturbations. In our analyses, FDR will be estimated using the BH procedure.
5.1.1. P53 Data
A test was performed on each of the 10,100 genes. Only 1 gene had an adjusted value less than FDR = 0.25 (bax, , ). Several GS analyses for this data set (using ) have been reported. We cite the GSEA analysis in [5] and a modification of the GSEA proposed in [40]. Also, in [38], this data set is used to test three procedures, each using various standardization procedures. Two are based on logistic regression (Global test [41] ANCOVA Global test [42]). The third is an extension of the Significance Analysis of Microarray (SAM) procedure [43] to gene sets proposed in [44] (SAMGS).
P53 pathways, with GS size ( ), unadjusted and FDR adjusted values ( )
Pathway 


 Sub  Efr  Liu 

SA_G1_AND_S_PHASES  14  .001  .08  n  y  n 
atmPathway  19  .001  .08  n  n  y 
g2Pathway  23  .001  .08  n  n  n 
p53Pathway  16  .001  .08  y  y  y 
cell_cycle_checkpointII  10  .001  .08  n  n  n 
SA_FAS_SIGNALLING  9  .002  .14  n  n*  n* 
cellcyclePathway  23  .002  .16  n  n*  n* 
DNA_  90  .003  .17  n  n*  n* 
SA_TRKA_RECEPTOR  16  .003  .17  n  n*  y* 
radiation_sensitivity  26  .003  .17  y  y*  y* 
ngfPathway  19  .004  .17  n  y*  n* 
GO_ROS  23  .004  .17  n  n*  n* 
etsPathway  16  .004  .17  n  n*  n* 
ck1Pathway  15  .006  .21  n  n*  n* 
erkPathway  29  .007  .23  n  n*  n* 
 18  .007  .23  n  n*  n* 
arfPathway  13  .007  .23  n  n*  n* 
To further clarify the procedure, we compare the BN model obtained from the data for the ten genes associated with the cell cycle checkpoint II pathway, separately for mutation and wildtype conditions. If there is interest in a posthoc analysis of any particular pathway, the rational for the MST algorithm no longer holds, since only one fit is required. It is therefore instructive to compare the MST model to a more commonly used method. In this case, we will use the Bayesian Information Criterion (BIC) (see, e.g., [7]), with a maximum indegree of 2. To fit the model we use a simulated annealing algorithm adapted from [45]. The resulting graphs are shown in Figures 2 (mutation) and 3 (wildtype). The MST and BIC fits are labelled (a) and (b) respectively. For the mutation fit, there is a very close correspondence between the topologies produced by the respective methods. For the wildtype data, some correspondence still exists, but less so then for the mutation data. The topologies between the conditions differ more significantly, as predicted by the hypothesis test.
5.1.2. Diabetes
Correlation analysis for DIABETES data
atr brca pathway  Alanine pathway  

NGT  cor  NGT  cor  
genes  ngt  dmt  p  genes  ngt  dmt  p 
fancc/rad17  83  69  349  crat/got1  81  30  031 
fancc/brca2  76  44  156  nars/dars  80  24  1 
rad9a/rad17  76  87  338  crat/gpt  75  15  028 
chek2/rad17  71  35  172  got2/adss  75  02  012 
brca1/hus1  69  29  148  got2/abat  73  34  001 
rad17/brca2  67  56  632  ddx3x/got1  72  17  004 
atr/mre11a  64  41  403  crat/ass  72  12  037 
chek1/nbs1  62  09  030  ddx3x/dars  71  12  043 
rad51/rad1  62  23  198  gpt/got1  70  33  175 
rad9a/fancc  59  76  388  ddx3x/abat  68  41  305 
DMT  cor  DMT  cor  
genes  dmt  ngt 
 genes  dmt  ngt 

rad9a/rad17  87  76  338  ddx3x/aars  76  55  325 
fanca/fance  81  14  009  crat/nars  74  26  074 
rad9a/fancc  76  59  388  ddx3x/nars  73  66  715 
fanca/hus1  72  27  002  asns/ddo  60  42  502 
brca1/mre11a  71  11  039  pc/aars  58  15  031 
fancc/rad17  69  83  349  crat/pc  58  53  862 
fancf/hus1  67  53  563  crat/ddx3x  58  51  813 
brca1/atr  67  16  011  got1/dars  56  40  006 
rad17/mre11a  64  11  086  pc/nars  55  18  244 
fancg/rad51  64  22  160  asns/gad2  54  44  723 
Examining the first table, differences in correlation appear to be explainable by sampling variation. In the second there are two gene pairs fanca/fance and fanca/hus1 with small values (.009, .002). We note that they share a common gene fanca and that they involve the only gene fance exhibiting differential expression. The correlation patterns within the two samples are otherwise similar, suggesting a specific alteration of the network model.
The situation differs for the pathway MAP00252 Alanine and aspartate metabolism, summarized in Table 2 using the same analysis. The change in correlation is more widespread. The 8 gene pairs with the highest correlation magnitudes within the NGT sample differ between NGT and DMT at a 0.05 significance level. Furthermore, the number of gene pairs with correlation magnitudes exceeding 0.7 is 9 in the NGT sample, but only 3 in the DMT sample.
5.1.3. Comparison of Fixed and Stopped Procedures
For stopped (St) and fixed (Fx) procedures, the table gives computation times; mean number of replications; % gene sets completely sampled; number of pathways with values ; 01; and number of such pathways in agreement.
Data  Time (hrs)  Mean rep  % comp  ^{#}  

St  Fx  St  Fx  St  Fx  St  Fx  Both  
diab  3.7  35.8  341.0  5000  5.4  100  6  6  6 
p53  2.1  30.0  612.3  5000  10.5  100  18  19  18 
6. Conclusion
We have introduced a twosample general likelihood ratio test for the equality of Bayesian network models. Significance levels are estimated using a permutation procedure. The algorithm was proposed as an alternative form of geneset analysis. It was noted that the fitting of Bayesian networks is computationally time consuming, hence a need for the efficient calculation of a model fit was identified, particularly for this application.
Two procedures were introduced to meet this requirement. First, we implemented a version of a minimum spanning tree algorithm first proposed in [15] which permits the polynomialtime calculation of the maximum likelihood Bayesian network among those with maximum indegree of one. Second, we introduced sequential testing principles to the problem of multiple testing, finding that a straightforward stopping rule could be developed which preserves group error rates for a wide range of procedures.
We may expect this form of test to be especially sensitive to changes in coexpression patterns, in contrast to most geneset procedures, which directly measure aggregate differential expression. In an application of the algorithm to two data sets considered in [5], a number of selected genesets exhibited clear differences in coexpression patterns while exhibiting very little differential expression. This leads to the conjecture that the optimal approach to geneset analysis is to couple a test which directly measures aggregate differential expression with one designed to detect differential coexpression.
Declarations
Acknowledgments
This paper was supported by NIH Grant no. R21HG004648. The Clinical Translational Science Institute of the University of Rochester Medical Center also provided funding for this research.
Authors’ Affiliations
References
 Dougherty ER, Shmulevich I, Chen J, Wang ZJ: Genomic Signal Processing and Statistics, EURASIP Book Series on Signal Processing and Communications. Volume 2. Hindawi Publishing Corporation, New York, NY, USA; 2005.View ArticleGoogle Scholar
 Shmulevich I, Dougherty ER: Genomic Signal Processing. Princeton University Press, Princeton, NJ, USA; 2007.View ArticleMATHGoogle Scholar
 EmmertStreib F, Dehmer M: Detecting pathological pathways of a complex disease by a comparitive analysis of networks. In Analysis of Microarray Data: A NetworkBased Approach. Edited by: EmmertStreib F, Dehmer M. WileyVCH, Weinheim, Germany; 2008:285305.View ArticleGoogle Scholar
 Mootha VK, Lindgren CM, Eriksson KF, Subramanian A, Sihag S, Lehar J, Puigserver P, Carlsson E, Ridderstråle M, Laurila E, Houstis N, Daly MJ, Patterson N, Mesirov JP, Golub TR, Tamayo P, Spiegelman B, Lander ES, Hirschhorn JN, Altshuler D, Groop LC: PGC1 α responsive genes involved in oxidative phosphorylation are coordinately downregulated in human diabetes. Nature Genetics 2003, 34(3):267273. 10.1038/ng1180View ArticleGoogle Scholar
 Subramanian A, Tamayo P, Mootha VK, Mukherjee S, Ebert BL, Gillette MA, Paulovich A, Pomeroy SL, Golub TR, Lander ES, Mesirov JP: Gene set enrichment analysis: a knowledgebased approach for interpreting genomewide expression profiles. Proceedings of the National Academy of Sciences of the United States of America 2005, 102(43):1554515550. 10.1073/pnas.0506580102View ArticleGoogle Scholar
 Subramanian A, Kuehn H, Gould J, Tamayo P, Mesirov JP: GSEAP: a desktop application for gene set enrichment analysis. Bioinformatics 2007, 23(23):32513253. 10.1093/bioinformatics/btm369View ArticleGoogle Scholar
 Sebastiani P, Abad M, Ramoni MF: Bayesian networks for genomic analysis. In Genomic Signal Processing and Statistics, EURASIP Book Series on Signal Processing and Communications. Edited by: Dougherty ER, Shmulevich I, Chen J, Wang ZJ. Hindawi Publishing Corporation, New York, NY, USA; 2005.Google Scholar
 Friedman N, Linial M, Nachman I, Pe'er D: Using Bayesian networks to analyze expression data. Journal of Computational Biology 2000, 7(34):601620. 10.1089/106652700750050961View ArticleGoogle Scholar
 Needham CJ, Bradford JR, Bulpitt AJ, Westhead DR: A primer on learning in Bayesian networks for computational biology. PLoS Computational Biology 2007, 3(8):e129. 10.1371/journal.pcbi.0030129View ArticleGoogle Scholar
 Chu T, Glymour C, Scheines R, Spirtes P: A statistical problem for inference to regulatory structure from associations of gene expression measurements with microarrays. Bioinformatics 2003, 19(9):11471152. 10.1093/bioinformatics/btg011View ArticleGoogle Scholar
 Cowell RG, Dawid P, Lauritzen SL, Spiegelhalter DJ: Probabilistic Networks and Expert Systems: Exact Computational Methods for Bayesian Networks, Information Science and Statistics. Spring, New York, NY, USA; 1999.MATHGoogle Scholar
 Cowell RG: Efficient maximum likelihood pedigree reconstruction. Theoretical Population Biology 2009, 76(4):285291. 10.1016/j.tpb.2009.09.002View ArticleGoogle Scholar
 Silander T, Myllymki P: A simple approach to finding the globally optimal bayesian network structure. In Proceedings of the 22nd Conference on Artificial intelligence (UAI '06), 2006. Edited by: Dechter R, Richardson T. AUAI Press; 445452.Google Scholar
 Chickering DM: Learning Bayesian net works is NPcomplete. In Learning from Data: Artificial Intelligence and Statistics V. Edited by: Fisher D, Lenz H. Springer, New York, NY, USA; 1996:121130.View ArticleGoogle Scholar
 Chow CK, Liu CN: Approximating discrete probability distributions with dependence trees. IEEE Transactions on Information Theory 1968, 14: 462467. 10.1109/TIT.1968.1054142View ArticleMATHGoogle Scholar
 Abbeel P, Koller D, Ng AY: Learning factor graphs in polynomial time and sample complexity. Journal of Machine Learning Research 2006, 7: 17431788.MathSciNetMATHGoogle Scholar
 Murphy K: Software packages for graphical models bayesian networks. Bulletin of the International Society for Bayesian Analysis 2007, 14: 1315.Google Scholar
 Teyssier M, Koller D: Orderingbased search: a simple and effective algorithm for learning bayesian networks. Proceedings of the 21st Conference on Uncertainty in AI (UAI '05), 2005 584590.Google Scholar
 Papadimitriou CH, Steiglitz K: Combinatorial Optimization: Algorithms and Complexity. PrenticeHall, Englewood Cliffs, NJ, USA; 1982.MATHGoogle Scholar
 Walsh AH: Aspects of Statistical Inference. John Wiley & Sons, New York, NY, USA; 1996.View ArticleGoogle Scholar
 Efron B: Robbins, empirical Bayes and microarrays. Annals of Statistics 2003, 31(2):366378. 10.1214/aos/1051027871MathSciNetView ArticleMATHGoogle Scholar
 Besag J, Clifford P: Sequential monte carlo p values. Biometrika 1991, 78: 301304.MathSciNetView ArticleGoogle Scholar
 Lock RH: A sequential approximation to a permutation test. Communications in Statistics. Simulation and Computation 1991, 20(1):341363. 10.1080/03610919108812956MathSciNetView ArticleMATHGoogle Scholar
 Fay MP, Follmann DA: Designing Monte Carlo implementations of permutation or bootstrap hypothesis tests. American Statistician 2002, 56(1):6370. 10.1198/000313002753631385MathSciNetView ArticleMATHGoogle Scholar
 Dudoit S, van der Laan MJ: Multiple Testing Procedures with Applications to Genomics. Springer, New York, NY, USA; 2008.View ArticleMATHGoogle Scholar
 Wald A: Sequential Analysis. John Wiley & Sons, New York, NY, USA; 1947.MATHGoogle Scholar
 Siegmund D: Sequential Analysis: Tests and Confidence Intervals. Springer, New York, NY, USA; 1985.View ArticleMATHGoogle Scholar
 Almudevar A: Exact confidence regions for species assignment based on DNA markers. Canadian Journal of Statistics 2000, 28(1):8195.MathSciNetView ArticleMATHGoogle Scholar
 Zhou X, Kao MCJ, Wong WH: Transitive functional annotation by shortestpath analysis of gene expression data. Proceedings of the National Academy of Sciences of the United States of America 2002, 99(20):1278312788. 10.1073/pnas.192159399View ArticleGoogle Scholar
 Braun R, Cope L, Parmigiani G: Identifying differential correlation in gene/pathway combinations. BMC Bioinformatics 2008., 9: article no. 488Google Scholar
 Barry WT, Nobel AB, Wright FA: Significance analysis of functional categories in gene expression studies: a structured permutation approach. Bioinformatics 2005, 21(9):19431949. 10.1093/bioinformatics/bti260View ArticleGoogle Scholar
 Jiang Z, Gentleman R: Extensions to gene set enrichment. Bioinformatics 2007, 23(3):306313. 10.1093/bioinformatics/btl599View ArticleGoogle Scholar
 Klebanov L, Glazko G, Salzman P, Yakovlev A, Xiao Y: A multivariate extension of the gene set enrichment analysis. Journal of Bioinformatics and Computational Biology 2007, 5(5):11391153. 10.1142/S0219720007003041View ArticleGoogle Scholar
 Goeman JJ, Bühlmann P: Analyzing gene expression data in terms of gene sets: methodological issues. Bioinformatics 2007, 23(8):980987. 10.1093/bioinformatics/btm051View ArticleGoogle Scholar
 Allison DB, Cui X, Page GP, Sabripour M: Microarray data analysis: from disarray to consolidation and consensus. Nature Reviews Genetics 2006, 7(1):5565. 10.1038/nrg1749View ArticleGoogle Scholar
 Bild A, Febbo PG: Application of a priori established gene sets to discover biologically important differential expression in microarray data. Proceedings of the National Academy of Sciences of the United States of America 2005, 102(43):1527815279. 10.1073/pnas.0507477102View ArticleGoogle Scholar
 Manoli T, Gretz N, Gröne HJ, Kenzelmann M, Eils R, Brors B: Group testing for pathway analysis improves comparability of different microarray datasets. Bioinformatics 2006, 22(20):25002506. 10.1093/bioinformatics/btl424View ArticleGoogle Scholar
 Liu Q, Dinu I, Adewale AJ, Potter JD, Yasui Y: Comparative evaluation of geneset analysis methods. BMC Bioinformatics 2007., 8: article no. 431Google Scholar
 Ackermann M, Strimmer K: A general modular framework for gene set enrichment analysis. BMC Bioinformatics 2009., 10: article no. 47Google Scholar
 Efron B, Tibshirani R: On testing the significance of sets of genes. Annals of Applied Statistics 2007, 1: 107129. 10.1214/07AOAS101MathSciNetView ArticleMATHGoogle Scholar
 Goeman JJ, van de Geer S, de Kort F, van Houwellingen HC: A global test for groups fo genes: testing association with a clinical outcome. Bioinformatics 2004, 20(1):9399. 10.1093/bioinformatics/btg382View ArticleGoogle Scholar
 Mansmann U, Meister R: Testing differential gene expression in functional groups: goeman's global test versus an ANCOVA approach. Methods of Information in Medicine 2005, 44(3):449453.Google Scholar
 Tusher VG, Tibshirani R, Chu G: Significance analysis of microarrays applied to the ionizing radiation response. Proceedings of the National Academy of Sciences of the United States of America 2001, 98(9):51165121. 10.1073/pnas.091062498View ArticleMATHGoogle Scholar
 Dinu I, Potter JD, Mueller T, Liu Q, Adewale AJ, Jhangri GS, Einecke G, Famulski KS, Halloran P, Yasui Y: Improving gene set analysis of microarray data by SAMGS. BMC Bioinformatics 2007., 8: article 242Google Scholar
 Almudevar A: A simulated annealing algorithm for maximum likelihood pedigree reconstruction. Theoretical Population Biology 2003, 63(2):6375. 10.1016/S00405809(02)000485View ArticleMATHGoogle Scholar
Copyright
This article is published under license to BioMed Central Ltd. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.