A Hypothesis Test for Equality of Bayesian Network Models
© Anthony Almudevar. 2010
Received: 26 March 2010
Accepted: 5 August 2010
Published: 9 August 2010
Bayesian network models are commonly used to model gene expression data. Some applications require a comparison of the network structure of a set of genes between varying phenotypes. In principle, separately fit models can be directly compared, but it is difficult to assign statistical significance to any observed differences. There would therefore be an advantage to the development of a rigorous hypothesis test for homogeneity of network structure. In this paper, a generalized likelihood ratio test based on Bayesian network models is developed, with significance level estimated using permutation replications. In order to be computationally feasible, a number of algorithms are introduced. First, a method for approximating multivariate distributions due to Chow and Liu (1968) is adapted, permitting the polynomial-time calculation of a maximum likelihood Bayesian network with maximum indegree of one. Second, sequential testing principles are applied to the permutation test, allowing significant reduction of computation time while preserving reported error rates used in multiple testing. The method is applied to gene-set analysis, using two sets of experimental data, and some advantage to a pathway modelling approach to this problem is reported.
Graphical models play a central role in modelling genomic data, largely because the pathway structure governing the interactions of cellular components induces statistical dependence naturally described by directed or undirected graphs [1–3]. These models vary in their formal structure. While a Boolean network can be interpreted as a set of state transition rules, Bayesian or Markov networks reduce to static multivariate densities on random vectors extracted from genomic data. Such densities are designed to model coexpression patterns resulting from functional cooperation. Our concern will be with this type of multivariate model. Although the ideas presented here extend naturally to various forms of genomic data, to fix ideas we will refer specifically to multivariate samples of microarray gene expression data.
In this paper, we consider the problem of comparing network models for a common set of genes under varying phenotypes. In principle, separately fit models can be directly compared. This approach is discussed in  and is based on distances definable on a space of graphs. Significance levels are estimated using replications of random graphs similar in structure to the estimated models.
The algorithm proposed below differs significantly from the direct graph approach. We will formulate the problem as a two-sample test in which significance levels are estimated by randomly permuting phenotypes. This requires only the minimal assumption of independence with respect to subjects.
Our strategy will be to confine attention to Bayesian network models (Section 2). Fitting Bayesian networks is computationally difficult, so a simplified model is developed for which a polynomial-time algorithm exists for maximum likelihood calculations. A two-sample hypotheses test based on the general likelihood ratio test statistic is introduced in Section 3. In Section 4, we discuss the application of sequential testing principles to permutation replications. This may be done in a way which permits the reporting of error rates commonly used in multiple testing procedures. In Section 5, the methodology is applied to the problem of gene set (GS) analysis, in which high dimensional arrays of gene expression data are screened for differential expression (DE) by comparing gene sets defined by known functional relationships, in place of individual gene expressions. This follows the paradigm originally proposed in gene set enrichment analysis (GSEA) [4–6]. The method will be applied to two well-known microarray data sets.
An R library of source code implementing the algorithms proposed here may be downloaded at http://www.urmc.rochester.edu/biostat/people/faculty/almudevar.cfm.
2. Network Models
The advantage of (1) is the reduction in the degrees of freedom of the model while preserving coexpression structure. Also, some flexibility is available with respect to the choice of the conditional densities of (1), with Gaussian, multinomial, and Gamma forms commonly used . We note that BNs are commonly used in many genomic applications [7–9].
2.1. Gaussian Bayesian Network Model
2.2. Restricted Bayesian Networks
Fitting BNs involves optimization over the space of topologies and hence is computationally intensive . While exact algorithms are available , they will generally require too great a computation time for the application described below. A recent application of exact techniques to the problem of pedigree reconstruction (a BN with maximum indegree of 2) was described in . Using methods proposed in  the exact computation of the maximum likelihood of a pedigree with 29 individuals (nodes) required 8 minutes. The author of  agrees with the conclusion reported in , that the method is not viable for BNs with greater than 32 nodes.
It is possible to control the size of the computation by placing a cap on the permissable indegree of each node, though the problem remains difficult even for (see, e.g., ). On the other hand, a method for fitting BNs with constraint in polynomial time is available under certain assumptions satisfied in our application. This method is based on the equivalence of the approximation of multivariate probability models using tree-structured dependence and the minimum spanning tree (MST) problem as described in . The objective is the minimization of an information difference , where is the target density, and is selected from a class of tree-structured approximating densities. Interest in  is restricted to discrete densities. We find, however, that the basic idea extends to general BNs in a natural way. See  for further discussion of this model.
Many heuristic or approximate methods exist for fitting Bayesian networks. See  for a recent survey. Such algorithms are usually based on MCMC techniques or heuristic algorithms such as TABU searches . We note that the proposed hypothesis test will depend on the calculation of a maximum likelihood ratio, hence it is important to have reasonable guarantees that a maximum has been reached. Thus, given the choice between an exact solution of a restricted class of models or an approximate solution of a general class of models, the former seems preferable. Considering also that in the application described below a solution is required for cases number in "10 s or 100 s'' of thousands, a polynomial time exact solution to a restricted class of models appears to be the best choice.
Suppose we may construct estimators , . We then assume there is some selection rule for each . This will typically be the exact or approximate maximum likelihood estimate (MLE) on parameter space . We will need the following assumptions.
A spanning tree on nodes is an acyclic connected undirected graph. Given edge weights , a minimum spanning tree (MST) is any spanning tree minimizing the sum of its edge weights among all spanning trees. A number of well-known polynomial time algorithms exist to construct a MST. Two that are commonly described are Prim's and Kruskal's algorithms . Kruskal's algorithm is described in . In the following theorem, the problem of maximizing is expressed as a MST problem.
Under assumption (A1), from definition (4) it follows that depends on only through the term . Then suppose maximizes . For any spanning tree define and suppose minimizes . Assume is not connected. There must be at least two nodes for which , and for which the respective subgraphs containing are unconnected. In this case, extend to by adding directed edge . We must have , and by (A2) we have . We may therefore assume is connected. The undirected graph of is a spanning tree, so .
Next, note that can be identified with an element of by defining any node as a root node, enumerating all paths from the root node to terminal nodes, then assigning edge directions to conform to these paths. This implies , which in turn implies , and that may be selected so that can be identified with .
3. General Maximum Likelihood Ratio Test
Identification of nonhomogeneity between two Bayesian networks will be based on a general maximum likelihood ratio test (MLRT). It is important to note the properties of the MLRT are well understood in parametric inference of limited dimension, and a sampling distribution can be accurately approximated with a large enough sample size. These known properties no longer apply in the type of problem considered here, primarily due to the small sample size, large number of parameters, and the fact that optimization over a discrete space is performed. In addition, the maximum likelihood principle itself favors spurious complexity when no model selection principles are used. While we cannot claim that the MLRT possesses any optimum properties in this application, the use of a permutation procedure will permit accurate estimates of the observed significance level while the use of the restricted model class will control to some degree the degrees of freedom of the model. See, for example,  for a general discussion of these issues.
Asymptotic distribution theory is not relevant here due to small sample size and the fact that optimization is performed in part over a discrete space of models, so a two sample permutation procedure will be used. Permutations will be approximately balanced to reduce spurious variability when a true difference in expression pattern exists (see, e. g.,  for discussion). This can be done by changing group labels of randomly selecting sample vectors from each of and . This results in permutation replicate samples and . The balanced procedure ensures that each permutation replicate sample contains approximately equal proportions of the original samples.
We now define Algorithm 1.
4. Permutation Tests with Stopping Rules
Permutation or bootstrap tests usually reduce to the estimation of a binomial probability by direct simulation. Since interest is usually in identifying small values, it would seem redundant to continue sampling when, for example, the first ten simulations lead to an estimate of 1/2. This suggests that a stopping rule may be applied to permutation sampling, resulting in significant reduction in computation time, provided it can be incorporated into a valid inference statement. A variety of such procedures have been described in the literature but do not seem to have been widely adopted in genomic discovery applications [22–24].
Formally, is a stopping time if the occurrence of event can be determined from . We may then design an algorithm which terminates after sampling a sequence of exactly length from , then outputs , from which the hypothesis decision is resolved. We refer to such a procedure as a stopped procedure. A fixed procedure (such as Algorithm 1) can be regarded as a special case of a stopped procedure in which .
An important distinction will have to be made between a single test and a multiple testing procedure (MTP), which is a collection of hypothesis tests with rejection rules that control for a global error rate such as false discovery rate (FDR), family-wise error rate (FWER), or per family error rate (PFER) . In the single test application, we may set a fixed significance level and continue replications until we conclude that the -value is above or below . For an MTP, it will be important to be able to estimate small -values, so a stopping rule which permits this is needed. Although the two cases have different structure, in our development they will both be based on the sequential probability ratio test (SPRT), first proposed in , which we now describe.
4.1. Sequential Probability Ratio Test (SPRT)
It can be shown that . If we conclude and conclude otherwise. We define errors and . It turns out that the SPRT is optimal under the given assumptions in the sense that it minimizes among all sequential tests (which includes fixed sample tests) with respective error probabilities no larger than . Approximate formulae for and are given in .
Hypothesis testing usually involves composite hypotheses, with distinct interpretations for the null and alternative hypothesis. One method of adapting the SPRT to this case is to select surrogate simple hypotheses. For example, to test versus , we could select simple hypotheses and . In this case, we would need to know the entire power function, which may be estimated using simulations.
An additional issue then arises in that the expected stopping time may be very large for . This can be accommodated using truncation. Suppose a reasonable choice for a fixed sample size is . We would then use truncated stopping time , with defined in (10). When , we could, for example, select hypothesis if . These modifications are discussed in .
4.2. Single Hypothesis Test
Suppose we adopt a fixed significance level for a single hypothesis test. If is the (unknown) true significance level, we are interested in resolving the hypothesis : . The properties of the test are summarized in a power curve, that is, the probability of deciding is true for each . An example of this procedure is given in , for , using a SPRT with parameters , , , , and truncation at . Hypothesis is concluded if when ; otherwise when .
4.3. Multiple Hypothesis Tests
where the quantity defines the particular MTP. It is assumed that is an increasing function of for all . The procedure is implemented by rejecting all null hypotheses for which . Depending on the MTP, various forms of error, usually either family-wise error rate (FWER) or false discovery rate (FDR), are controlled at the level. For example, the Benjamini-Hochberg (BH) procedure is a step-up procedure defined by and controls for FDR for independent hypothesis tests. A comprehensive treatment of this topic is given in, for example, .
Suppose we have probabilities ( -values associated with tests). For each test , we may generate as the cumulative sum defined in (9). Now suppose we define any stopping time , bounded by , for each sequence (this may or may not be related to the SPRT). Then define estimates , with .
For a fixed MTP, the estimates would replace the true values in (11), yielding estimated adjusted -values while for the stopped MTP adjusted -values are produced in the same manner using . It is easily seen that while the rankings of (accounting for ties) are equal to the rankings of . Furthermore, the formulae in (11) are monotone in , so we must have . Thus, the stopped procedure may be seen as being embedded in the fixed procedure. It inherits whatever error control is given for the fixed MTP, with the advantage that the calculation of the adjusted -values uses only the first replications for the th test.
The procedure will always be correct in that it is strictly more conservative than the fixed MTP in which it is embedded, no matter which stopping time is used. The remaining issue is the selection of which will equal for small enough values of but will also have for larger values of . It is a simple matter, then, to modify the SPRT described in Section 4.2 by eliminating the lower bound (equivalently ). We will adopt this design in this paper. This gives Algorithm 2.
Same as Algorithm 1, step 1.
( ) Simulate replicates in Algorithm 1, step 3, until the following stopping criterion is met. Set , and let , where . Stop sampling at the th replication if , where , or until , whichever occurs first.
5. Gene-Set Analysis
A recent trend in the analysis of microarray data has been to base the discovery of phenotype-induced DE on gene sets rather than individual genes. The reasoning is that if genes in a given set are related by common pathway membership or other transcriptional process, then there should be an aggregate change in gene expression pattern. This should give increased statistical power, as well as enhanced interpretability, especially given the lack of reproducibility in univariate gene discovery due to the stringent requirements imposed by multiple testing adjustments. Thus, the discovery process reduces to a much smaller number of hypothesis tests with more direct biological meaning. Some objections may be raised concerning the selection of the gene sets when theses sets are themselves determined experimentally. Additionally, gene sets may overlap. While these problems need to be addressed, it is also true that such gene set methods have been shown to detect DE not uncovered by univariate screens.
A crucial problem in gene set analysis is the choice of test statistic. The problem of testing against equality of random vectors in , , is fundamentally different from the univariate case . The range of statistics one would consider for is reasonably limited, the choice being largely driven by distributional considerations. For , new structural or geometric considerations arise. For example, we may have differential expression between some but not all genes in the gene set, which makes selection of a single optimal test statistic impossible. Alternatively, the experimental random vectors may differ in their level of coexpression independently of their level of marginal DE.
In fact, almost all GS procedures directly measure aggregate DE, so an important question is whether or not phenotypic variation is almost completely expressible as DE. If so, then a DE based statistic will have fewer degrees of freedom, hence more power, than one based on a more complex model. Otherwise, a reasonable conjecture is that a compound GS analysis will work best, employing a DE statistic as well as one more sensitive to changes in coexpression patterns.
Correlations have been used in a number of gene discovery applications. They may be used to associate genes of unknown function with known pathways [29, 30]. Additionally, a number of GS procedures exist which incorporate correlation structure into the procedure [31–33]. However, a direct comparison of correlations is not practical due to the large number ( ) of distinct correlation parameters. Therefore, there is a considerable advantage to the statistic (7) based on the reduced BN model, in that the correlation structure can be summarized by the correlation parameters output by the MST algorithm, yielding a transitive dependence model similar to that effectively exploited in .
It is important to refer to a methodological characterization given in . A distinction is made between two types of null hypotheses. Suppose we are given samples of expression levels from a gene set from two phenotypes. Suppose also that for each gene in and its complement , a statistical measure of differential expression is available. For a competitive test, the null hypothesis is that the prevalence of differential expression in is no greater than in . For a self-contained test, the null hypothesis is that no genes in are differentially expressed. In the GSEA method of [4, 5] concern is with . In most subsequent methods, including the one proposed here, is used.
5.1. Experimental Data
We will demonstrate the algorithm proposed here on two data sets examined elsewhere in the literature. These were obtained from the GSEA website www.broad.mit.edu/gsea . In , a data set p53 is extracted from the NCI-60 collection of cancer cell lines, with 17 cell lines classified as normal, and 33 classified as carrying mutations of p53. We also examine the DIABETES data set introduced in , consisting of microaray profiles of skeletal muscle biopsies from 43 males. For the DIABETES data set used here, there were 17 normal glucose tolerance (NGT) subjects and 17 diabetes (DMT) subjects. For gene sets, we used one of the gene set lists compiled in , denoted , consisting of 472 gene sets with products collectively involved in various metabolic and signalling pathways, as well as 50 sets containing genes exhibiting coregulated response to various perturbations. In our analyses, FDR will be estimated using the BH procedure.
5.1.1. P53 Data
A -test was performed on each of the 10,100 genes. Only 1 gene had an adjusted -value less than FDR = 0.25 (bax, , ). Several GS analyses for this data set (using ) have been reported. We cite the GSEA analysis in  and a modification of the GSEA proposed in . Also, in , this data set is used to test three procedures, each using various standardization procedures. Two are based on logistic regression (Global test  ANCOVA Global test ). The third is an extension of the Significance Analysis of Microarray (SAM) procedure  to gene sets proposed in  (SAM-GS).
To further clarify the procedure, we compare the BN model obtained from the data for the ten genes associated with the cell cycle checkpoint II pathway, separately for mutation and wildtype conditions. If there is interest in a post-hoc analysis of any particular pathway, the rational for the MST algorithm no longer holds, since only one fit is required. It is therefore instructive to compare the MST model to a more commonly used method. In this case, we will use the Bayesian Information Criterion (BIC) (see, e.g., ), with a maximum indegree of 2. To fit the model we use a simulated annealing algorithm adapted from . The resulting graphs are shown in Figures 2 (mutation) and 3 (wildtype). The MST and BIC fits are labelled (a) and (b) respectively. For the mutation fit, there is a very close correspondence between the topologies produced by the respective methods. For the wildtype data, some correspondence still exists, but less so then for the mutation data. The topologies between the conditions differ more significantly, as predicted by the hypothesis test.
Correlation analysis for DIABETES data
atr brca pathway
Examining the first table, differences in correlation appear to be explainable by sampling variation. In the second there are two gene pairs fanca/fance and fanca/hus1 with small -values (.009, .002). We note that they share a common gene fanca and that they involve the only gene fance exhibiting differential expression. The correlation patterns within the two samples are otherwise similar, suggesting a specific alteration of the network model.
The situation differs for the pathway MAP00252 Alanine and aspartate metabolism, summarized in Table 2 using the same analysis. The change in correlation is more widespread. The 8 gene pairs with the highest correlation magnitudes within the NGT sample differ between NGT and DMT at a 0.05 significance level. Furthermore, the number of gene pairs with correlation magnitudes exceeding 0.7 is 9 in the NGT sample, but only 3 in the DMT sample.
5.1.3. Comparison of Fixed and Stopped Procedures
We have introduced a two-sample general likelihood ratio test for the equality of Bayesian network models. Significance levels are estimated using a permutation procedure. The algorithm was proposed as an alternative form of gene-set analysis. It was noted that the fitting of Bayesian networks is computationally time consuming, hence a need for the efficient calculation of a model fit was identified, particularly for this application.
Two procedures were introduced to meet this requirement. First, we implemented a version of a minimum spanning tree algorithm first proposed in  which permits the polynomial-time calculation of the maximum likelihood Bayesian network among those with maximum indegree of one. Second, we introduced sequential testing principles to the problem of multiple testing, finding that a straightforward stopping rule could be developed which preserves group error rates for a wide range of procedures.
We may expect this form of test to be especially sensitive to changes in coexpression patterns, in contrast to most gene-set procedures, which directly measure aggregate differential expression. In an application of the algorithm to two data sets considered in , a number of selected gene-sets exhibited clear differences in coexpression patterns while exhibiting very little differential expression. This leads to the conjecture that the optimal approach to gene-set analysis is to couple a test which directly measures aggregate differential expression with one designed to detect differential coexpression.
This paper was supported by NIH Grant no. R21HG004648. The Clinical Translational Science Institute of the University of Rochester Medical Center also provided funding for this research.
- Dougherty ER, Shmulevich I, Chen J, Wang ZJ: Genomic Signal Processing and Statistics, EURASIP Book Series on Signal Processing and Communications. Volume 2. Hindawi Publishing Corporation, New York, NY, USA; 2005.View ArticleGoogle Scholar
- Shmulevich I, Dougherty ER: Genomic Signal Processing. Princeton University Press, Princeton, NJ, USA; 2007.View ArticleMATHGoogle Scholar
- Emmert-Streib F, Dehmer M: Detecting pathological pathways of a complex disease by a comparitive analysis of networks. In Analysis of Microarray Data: A Network-Based Approach. Edited by: Emmert-Streib F, Dehmer M. Wiley-VCH, Weinheim, Germany; 2008:285-305.View ArticleGoogle Scholar
- Mootha VK, Lindgren CM, Eriksson K-F, Subramanian A, Sihag S, Lehar J, Puigserver P, Carlsson E, Ridderstråle M, Laurila E, Houstis N, Daly MJ, Patterson N, Mesirov JP, Golub TR, Tamayo P, Spiegelman B, Lander ES, Hirschhorn JN, Altshuler D, Groop LC: PGC-1 α -responsive genes involved in oxidative phosphorylation are coordinately downregulated in human diabetes. Nature Genetics 2003, 34(3):267-273. 10.1038/ng1180View ArticleGoogle Scholar
- Subramanian A, Tamayo P, Mootha VK, Mukherjee S, Ebert BL, Gillette MA, Paulovich A, Pomeroy SL, Golub TR, Lander ES, Mesirov JP: Gene set enrichment analysis: a knowledge-based approach for interpreting genome-wide expression profiles. Proceedings of the National Academy of Sciences of the United States of America 2005, 102(43):15545-15550. 10.1073/pnas.0506580102View ArticleGoogle Scholar
- Subramanian A, Kuehn H, Gould J, Tamayo P, Mesirov JP: GSEA-P: a desktop application for gene set enrichment analysis. Bioinformatics 2007, 23(23):3251-3253. 10.1093/bioinformatics/btm369View ArticleGoogle Scholar
- Sebastiani P, Abad M, Ramoni MF: Bayesian networks for genomic analysis. In Genomic Signal Processing and Statistics, EURASIP Book Series on Signal Processing and Communications. Edited by: Dougherty ER, Shmulevich I, Chen J, Wang ZJ. Hindawi Publishing Corporation, New York, NY, USA; 2005.Google Scholar
- Friedman N, Linial M, Nachman I, Pe'er D: Using Bayesian networks to analyze expression data. Journal of Computational Biology 2000, 7(3-4):601-620. 10.1089/106652700750050961View ArticleGoogle Scholar
- Needham CJ, Bradford JR, Bulpitt AJ, Westhead DR: A primer on learning in Bayesian networks for computational biology. PLoS Computational Biology 2007, 3(8):e129. 10.1371/journal.pcbi.0030129View ArticleGoogle Scholar
- Chu T, Glymour C, Scheines R, Spirtes P: A statistical problem for inference to regulatory structure from associations of gene expression measurements with microarrays. Bioinformatics 2003, 19(9):1147-1152. 10.1093/bioinformatics/btg011View ArticleGoogle Scholar
- Cowell RG, Dawid P, Lauritzen SL, Spiegelhalter DJ: Probabilistic Networks and Expert Systems: Exact Computational Methods for Bayesian Networks, Information Science and Statistics. Spring, New York, NY, USA; 1999.MATHGoogle Scholar
- Cowell RG: Efficient maximum likelihood pedigree reconstruction. Theoretical Population Biology 2009, 76(4):285-291. 10.1016/j.tpb.2009.09.002View ArticleGoogle Scholar
- Silander T, Myllymki P: A simple approach to finding the globally optimal bayesian network structure. In Proceedings of the 22nd Conference on Artificial intelligence (UAI '06), 2006. Edited by: Dechter R, Richardson T. AUAI Press; 445-452.Google Scholar
- Chickering DM: Learning Bayesian net- works is NP-complete. In Learning from Data: Artificial Intelligence and Statistics V. Edited by: Fisher D, Lenz H. Springer, New York, NY, USA; 1996:121-130.View ArticleGoogle Scholar
- Chow CK, Liu CN: Approximating discrete probability distributions with dependence trees. IEEE Transactions on Information Theory 1968, 14: 462-467. 10.1109/TIT.1968.1054142View ArticleMATHGoogle Scholar
- Abbeel P, Koller D, Ng AY: Learning factor graphs in polynomial time and sample complexity. Journal of Machine Learning Research 2006, 7: 1743-1788.MathSciNetMATHGoogle Scholar
- Murphy K: Software packages for graphical models bayesian networks. Bulletin of the International Society for Bayesian Analysis 2007, 14: 13-15.Google Scholar
- Teyssier M, Koller D: Ordering-based search: a simple and effective algorithm for learning bayesian networks. Proceedings of the 21st Conference on Uncertainty in AI (UAI '05), 2005 584-590.Google Scholar
- Papadimitriou CH, Steiglitz K: Combinatorial Optimization: Algorithms and Complexity. Prentice-Hall, Englewood Cliffs, NJ, USA; 1982.MATHGoogle Scholar
- Walsh AH: Aspects of Statistical Inference. John Wiley & Sons, New York, NY, USA; 1996.View ArticleGoogle Scholar
- Efron B: Robbins, empirical Bayes and microarrays. Annals of Statistics 2003, 31(2):366-378. 10.1214/aos/1051027871MathSciNetView ArticleMATHGoogle Scholar
- Besag J, Clifford P: Sequential monte carlo p -values. Biometrika 1991, 78: 301-304.MathSciNetView ArticleGoogle Scholar
- Lock RH: A sequential approximation to a permutation test. Communications in Statistics. Simulation and Computation 1991, 20(1):341-363. 10.1080/03610919108812956MathSciNetView ArticleMATHGoogle Scholar
- Fay MP, Follmann DA: Designing Monte Carlo implementations of permutation or bootstrap hypothesis tests. American Statistician 2002, 56(1):63-70. 10.1198/000313002753631385MathSciNetView ArticleMATHGoogle Scholar
- Dudoit S, van der Laan MJ: Multiple Testing Procedures with Applications to Genomics. Springer, New York, NY, USA; 2008.View ArticleMATHGoogle Scholar
- Wald A: Sequential Analysis. John Wiley & Sons, New York, NY, USA; 1947.MATHGoogle Scholar
- Siegmund D: Sequential Analysis: Tests and Confidence Intervals. Springer, New York, NY, USA; 1985.View ArticleMATHGoogle Scholar
- Almudevar A: Exact confidence regions for species assignment based on DNA markers. Canadian Journal of Statistics 2000, 28(1):81-95.MathSciNetView ArticleMATHGoogle Scholar
- Zhou X, Kao M-CJ, Wong WH: Transitive functional annotation by shortest-path analysis of gene expression data. Proceedings of the National Academy of Sciences of the United States of America 2002, 99(20):12783-12788. 10.1073/pnas.192159399View ArticleGoogle Scholar
- Braun R, Cope L, Parmigiani G: Identifying differential correlation in gene/pathway combinations. BMC Bioinformatics 2008., 9: article no. 488Google Scholar
- Barry WT, Nobel AB, Wright FA: Significance analysis of functional categories in gene expression studies: a structured permutation approach. Bioinformatics 2005, 21(9):1943-1949. 10.1093/bioinformatics/bti260View ArticleGoogle Scholar
- Jiang Z, Gentleman R: Extensions to gene set enrichment. Bioinformatics 2007, 23(3):306-313. 10.1093/bioinformatics/btl599View ArticleGoogle Scholar
- Klebanov L, Glazko G, Salzman P, Yakovlev A, Xiao Y: A multivariate extension of the gene set enrichment analysis. Journal of Bioinformatics and Computational Biology 2007, 5(5):1139-1153. 10.1142/S0219720007003041View ArticleGoogle Scholar
- Goeman JJ, Bühlmann P: Analyzing gene expression data in terms of gene sets: methodological issues. Bioinformatics 2007, 23(8):980-987. 10.1093/bioinformatics/btm051View ArticleGoogle Scholar
- Allison DB, Cui X, Page GP, Sabripour M: Microarray data analysis: from disarray to consolidation and consensus. Nature Reviews Genetics 2006, 7(1):55-65. 10.1038/nrg1749View ArticleGoogle Scholar
- Bild A, Febbo PG: Application of a priori established gene sets to discover biologically important differential expression in microarray data. Proceedings of the National Academy of Sciences of the United States of America 2005, 102(43):15278-15279. 10.1073/pnas.0507477102View ArticleGoogle Scholar
- Manoli T, Gretz N, Gröne H-J, Kenzelmann M, Eils R, Brors B: Group testing for pathway analysis improves comparability of different microarray datasets. Bioinformatics 2006, 22(20):2500-2506. 10.1093/bioinformatics/btl424View ArticleGoogle Scholar
- Liu Q, Dinu I, Adewale AJ, Potter JD, Yasui Y: Comparative evaluation of gene-set analysis methods. BMC Bioinformatics 2007., 8: article no. 431Google Scholar
- Ackermann M, Strimmer K: A general modular framework for gene set enrichment analysis. BMC Bioinformatics 2009., 10: article no. 47Google Scholar
- Efron B, Tibshirani R: On testing the significance of sets of genes. Annals of Applied Statistics 2007, 1: 107-129. 10.1214/07-AOAS101MathSciNetView ArticleMATHGoogle Scholar
- Goeman JJ, van de Geer S, de Kort F, van Houwellingen HC: A global test for groups fo genes: testing association with a clinical outcome. Bioinformatics 2004, 20(1):93-99. 10.1093/bioinformatics/btg382View ArticleGoogle Scholar
- Mansmann U, Meister R: Testing differential gene expression in functional groups: goeman's global test versus an ANCOVA approach. Methods of Information in Medicine 2005, 44(3):449-453.Google Scholar
- Tusher VG, Tibshirani R, Chu G: Significance analysis of microarrays applied to the ionizing radiation response. Proceedings of the National Academy of Sciences of the United States of America 2001, 98(9):5116-5121. 10.1073/pnas.091062498View ArticleMATHGoogle Scholar
- Dinu I, Potter JD, Mueller T, Liu Q, Adewale AJ, Jhangri GS, Einecke G, Famulski KS, Halloran P, Yasui Y: Improving gene set analysis of microarray data by SAM-GS. BMC Bioinformatics 2007., 8: article 242Google Scholar
- Almudevar A: A simulated annealing algorithm for maximum likelihood pedigree reconstruction. Theoretical Population Biology 2003, 63(2):63-75. 10.1016/S0040-5809(02)00048-5View ArticleMATHGoogle Scholar
This article is published under license to BioMed Central Ltd. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.