 Research Article
 Open Access
Which Is Better: Holdout or FullSample Classifier Design?
 Marcel Brun^{1},
 Qian Xu^{2} and
 Edward R Dougherty^{1, 2}Email author
https://doi.org/10.1155/2008/297945
© Marcel Brun et al. 2008
 Received: 26 March 2007
 Accepted: 2 December 2007
 Published: 12 December 2007
Abstract
Is it better to design a classifier and estimate its error on the full sample or to design a classifier on a training subset and estimate its error on the holdout test subset? Fullsample design provides the better classifier; nevertheless, one might choose holdout with the hope of better error estimation. A conservative criterion to decide the best course is to aim at a classifier whose error is less than a given bound. Then the choice between fullsample and holdout designs depends on which possesses the smaller expected bound. Using this criterion, we examine the choice between holdout and several fullsample error estimators using covariance models and a patientdata model. Fullsample design consistently outperforms holdout design. The relation between the two designs is revealed via a decomposition of the expected bound into the sum of the expected true error and the expected conditional standard deviation of the true error.
Keywords
 Feature Selection
 Linear Discriminant Analysis
 Classification Rule
 True Error
 Standard Normal Variable
1. Introduction
In most microarraybased classification studies, the number of data points (microarrays) is very small (under 100) and one has no choice but to use the full cohort of data for both training and testing (error estimation). One must choose among error estimators for which the full sample is used for training. In smallsample situations, these estimators usually suffer from either low bias (resubstitution) or high variance (crossvalidation) [12]. Studies indicate that either bootstrap [3] or bolstering [4] tend to provide better estimation. But what happens when samples sizes are not so small, a situation that will become more common as technology improves? Then, rather than using fullsample design and estimation, one has the option of holding out data from the design and using the holdout data for estimating the error of the classifier designed on the data not held out.
Based upon colloquial discussions, it appears that some people prefer to hold out data except for very small samples, thereby splitting the sample into training and testing data; however, these discussions usually lack any precise statistical justification. On the other hand, when discussing holding out test data to estimate the error of a designed classifier, Devroye et al. state [5], "A serious problem concerning the practical applicability of the [holdout] estimate introduced above is that it requires a large, independent testing sequence. In practice, however, an additional sample is rarely available. One usually wants to incorporate all available [sample points] pairs in the decision function." When made by premier patternrecognition researchers such as L. Devroye, L. Gyorfi, and G. Lugosi, such a statement should give pause to anyone taking a counter position. The holdout issue arises because, even though we are assured of a smaller true error using fullsample design, we desire a satisfactory estimate of the error. The salient word in the Devroye et al. quote [5] is "rarely." Reasoning in a hyperbolic extreme, if there were an infinite amount of data, it could be split into infinite training and test data sets and this would constitute one of the rare cases. But why do so? For many popular fullsample error estimators, the meansquare error between the estimated and true errors goes to as the sample size tends to infinity. For instance, for the histogram rule with cells, the resubstitution estimator is low biased; nevertheless, it satisfies the bound , where and are the estimated and true errors, respectively [5]. In the other direction, if one has only sample points, then clearly one does not want to hold out data from training. But what is the preferred course of action in moderate cases. Since these are not rare, are we to conclude from the Devroye et al. statement that even in these one should not hold out data for error estimation?
Let us motivate the issue with an illustration of the kind of pathology that can afflict holdout error estimation. Suppose that one randomly splits the available data in the sample, , into training and test data samples, say and , respectively. Let and be the classifiers trained on and , respectively. Now suppose that provides a faithful sampling of the featurelabel distribution, at least to the extent possible given the size of the sample; however, owing to chance in the splitting process, and represent different parts of the featurelabel distribution. Since provides a representative sample, should provide good classification and this will likely be reflected in its estimated error based on . On the other hand, may or may not provide good classification, depending on how well reflects the featurelabel distribution, but in either event, its estimated error will likely indicate poor performance because the estimate will be done on data significantly different from the training data. Splitting the data has had two undesirable effects: poorer design and poorer error estimation. The latter effect is pernicious: one has the data to design a good classifier, and indeed may even do so, but gets a high testdata error and mistakenly walks away with nothing.
One might argue that, owing to the high variance associated with many fullsample error estimators, it is more conservative, and thus safer, to split the data. But even if we desire conservativeness, this argument requires refinement. The empirical testdata error estimator also has variance, which is substantial for small testdata sets. Hence, to be meaningful, the conservative holdout argument requires a specification of the proportion of data to be held out.
Stating the matter quantitatively, given a sample of size , is it better to design a classifier and estimate its error on the full sample or take a holdout approach by designing on a training subset of size and testing on a disjoint subset of size , where Letting and denote the classifiers designed using fullsample and holdout, respectively, then the expected error of on the full featurelabel distribution is less than the expected error of on the full featurelabel distribution: , where denotes classifier error. Were we able to compute the true error of a designed classifier, there would be no issue: design on the full sample. In practice, this error must be estimated and therefore we must concern ourselves with the relation between the error estimates and for and , respectively, where is obtained by some fullsample method and is the error rate of on the test data. If is approximately unbiased, meaning that , then since is unbiased, on average the fullsampleand testsamplebased estimators agree with the true errors of the classifiers they are estimating; however, if one of the estimators has a much greater variance than the other, say, the variance of is large in comparison to , then we have greater confidence in the estimated error of a particular trainingdata designed classifier than the error of the corresponding particular fullsample designed classifier. Since holding out a significant amount of data usually means that , it is common to trust the holdout estimate over the fullsample estimate. This conservative approach has a price, that being poorer performing classifiers.
To get at the key practical dilemma facing holdout design, consider a situation in which one has data points and wishes to split the data into training and test sets. With given, how is one to choose ? Unless this question is to be answered in an ad hoc manner, there needs to be a criterion. A very conservative way to proceed is to take a minimax approach and choose so as to minimize the maximum variance of the estimator. While certainly rigorous, this minimax criterion leads to the decision : the training data consists of one point from each class and the resulting classifier is tested on the points held out. No one would opt for this minimax criterion on the variance because the expected error of the designed classifier would be very large. One would have an excellent error estimate for a useless classifier.
But this relation alone does not provide a good criterion for making the choice since, in analogy with the minimax approach to holdout, the inequality can best be achieved by letting . We are in the conundrum because the criterion of the choice, either better classifier design or better error estimation, is wrong. We want good classifier design and good error estimation, so the choice should be based on a criterion that takes the full process, design and error estimation, into account, not just one or the other.
Whereas the estimates themselves contain no information regarding their imprecision, the confidence intervals do. Since we have equal confidence in both intervals, and , the better classifier is the one possessing the smaller confidence bound. Under this criterion, the choice between fullsample and holdout design becomes a choice as to which is smaller, or .
where and are the densities for the estimation values and , respectively, and we use in both integrals because in this context it is a dummy variable. M is used to denote a mean because and are the means of the bounds and , respectively.
Given that a fullsample error estimator is close to being unbiased, the criterion is to choose fullsample design if and only if , where the decision depends on , , and the fullsample estimator (as well as the classification rule and featurelabel distribution). As we will see in the examples, it does not appear that the relation is sensitive to the choice of . We emphasize that we only apply the confidencebound criterion when the error estimator is not strongly biased. In particular, we will not apply it when using resubstitution because we wish to avoid situations in which we expect that the error estimate is low; indeed, the criterion is reasonable precisely because it incorporates variance information to discriminate between approximately unbiased estimators.
2. Systems and Mthods
Using simulations we will compare and for several data models and classification rules. The classification rules used are 3nearest neighbor ( NN), linear discriminant analysis (LDA), quadratic discriminant analysis (QDA), and Gaussian Kernel (Kernel).
The estimators considered are leaveoneout cross validation (Loo), 5fold crossvalidation with 20 replications (CV), 0.632bootstrap (B632), bolstered resubstitution (Bolster), and semibolstered resubstitution (SBolster) [4]. For the computation of CV we use stratified crossvalidation, whereby the classes are represented in each fold by the same proportion as in the original data. For the computation of the B632 estimator we use a technique called balanced bootstrap resampling [6], where each sample point is made to appear times in the computation. For bolstering estimators, 10 Monte Carlo samples are used for each bolstering kernel.
2.1. ModelBased Simulation
Simulated data consists in points of dimension , generated randomly from three different twoclasses models:
Linear Model (0)
The Bayes classifier is linear and its decision boundary is a hyperplane.
Nonlinear Model (1)
This is similar to the previous model, but the covariance matrices differ by a scaling factor such that . Throughout the study we use . The Bayes classifier is nonlinear and its decision boundary is quadratic.
Bimodal Model (2)
where and are defined by (6), with means at and , respectively. All of the Gaussians possess identical covariance matrices, .
As in a number of other studies [7–10], we use a block structure for the covariance matrices that models a feature set partitioned so that the features in a partition are correlated and features in different partitions are uncorrelated. All features have common variance, so that the diagonal elements have identical value . To set the correlations between features, the features are equally divided into groups, with each group having features. Possible values of are . Features from different groups are uncorrelated and features from the same group possess the same correlation . When , all the features are uncorrelated. Values of , and are used in the simulations, varying the amount of confusion and redundancy between the variables.
An special case is considered when using feature selection, being the number of features the classifier will use. The values used are or . When there is no feature selection. Otherwise, there is feature selection, and the error is estimated using the design described in [11] to avoid bias introduced by the feature selection process. In each case, the best features were obtained by applying statistical ttest and selecting the features with the lowest pvalue.
Rather than considering a covariance matrix with a fixed value , for which the Bayes error will also be fixed, we can let vary, thereby letting the Bayes error vary, thereby emulating the practical situation in which methods are applied to classification problems of varying difficulty. To do this, we assume that the Bayes error can be any value between and and that it obeys a Beta distribution . The expected Bayes error is . In our simulation, we use the values and . These generate six pairs and the corresponding expected Bayes errors : .
List of experiments and their parameters: and are the parameters of the Beta distribution used for the Bayes error, is the number of groups, is the classification algorithm, is the twoclasses model, is the correlation for features in the same group, is the number of training samples, is the number of features, and is the number of features used by the classifier.










 1  1  2  NN  1  0.125  100  (10,10) 
LDA  
QDA  
Kernel  
 1  1  2  NN 

 100  (10,10) 


 2  NN  1  0.125  100  (10,10) 
 1  1 
 NN 
 0.125  100  (10,10) 
 1  1  2  NN  1  0.125 
 (10,10) 
 1  1  5  LDA  1  0.125  100  (10,10) 
(10,5)  
(25,5)  
(50,5)  
(100,5) 
In all cases we use a fixed sample size . Additional results and experiments are available at http://www.ece.tamu.edu/~edward/holdout.
2.2. Patient Data
In addition to the covariance models, we consider a model based on a microarray classification study. The microarrays were prepared with RNA from 295 breast cancer patients [12]. Using a previously established 70gene prognosis profile [13], a prognosis signature based on geneexpression was proposed that correlates well with patient survival data and other existing clinical measures. Of the 295 microarrays, 115 belong to the "good prognosis" class (label 1) and the remaining 180 belong to the "poor prognosis" class (label 0). Each data point is a 70expression vector corresponding to a single microarray, with expression values being log intensity. The best 2gene sets for linear classification (LDA) were obtained using a full search [14] and have been selected for this analysis. The data are available at the supplementary data web page cited in [14].
2.3. Estimation
The true error is estimated using a large additional dataset (above 2000 samples) sampled from the distribution .
The simulation first generates the Bayes error given the Beta distribution and the value of the variance is taken from a table of Bayes error versus variance. A set of size is drawn from the featurelabel distribution and split in two sets and for the holdout analysis. Each classification rule (and the feature selection algorithm, when needed) is applied to both and to obtain the classifiers and (and the list of selected features when FS is applied). These classifiers are applied to test points independently sampled from and the average error rates are used as the true errors and . Holdout error estimation is accomplished by applying the classifier to the holdout sample to obtain the holdout estimated error as the proportion of errors makes on . Fullsample error estimation for each method is evaluated using the whole set to obtain the estimated error . When feature selection is used, each classifier design involves feature selection. For resampling techniques it involves an additional cost for the process, since FS is applied to each iteration.
This procedure is repeated times ( times for experiment ) to obtain pairs and , which provide tight approximations to the joint distributions and . From these we compute the upperconfidence bounds and , and from these the expected upperconfidence bounds and , where the expectations are relative to the distributions of the estimated errors and , respectively.
3. Results and Discussion
3.1. Quantitative Results
Tables available at http://www.ece.tamu.edu/~edward/holdout. provide the actual numerical values. In all cases, holdout error estimation has the highest expected bound, meaning that holdout error estimator is outperformed by the fullsample error estimators. Among the latter, leaveoneout crossvalidation generally performs the worst.
3.2. Analysis
Equation (14) quantifies the dichotomy between opting for better error estimation or better actual performance.
According to (16), a large holdout reduces at the cost of increasing . Indeed, large decreases at the cost of increasing and small decreases at the cost of increasing . The combined effect is seen in Figure 3(e), where for increasing , first decreases and then increases. This effect can also be seen for QDA in similar graphs available at http://www.ece.tamu.edu/~edward/holdout. None of this should make us lose sight of the main observation: in all cases, both for 3NN and QDA, holdout performs worse than the fullsample estimators.
The appearance of the expected conditional standard deviation of the true error in the partition of in (14) is not counterintuitive. If we assume that the error estimator is unbiased, then . If we now assume that is small, then is small relative to the distributional mass of , which in turn means that relative to the mass of , which then implies that is small; that is, the error estimator is performing well.
3.3. Concluding Remarks
We propose a confidencebased criterion to decide between experimental designs, our particular interest being between fullsample and holdout classifier designs. One is free to propose other criteria, but reasonable probabilistic criteria upon which to ground a decision are certainly needed. Given the importance of the applications being considered, to leave matters in an ad hoc state of affairs is unacceptable. A critical point of the experiments is that the decision for fullsample design holds across various models and parametric settings, and the decision is generally clear cut. This consistency is important for practical application, where one does not know the featurelabel models.
Authors’ Affiliations
References
 BragaNeto UM, Dougherty ER: Is crossvalidation valid for smallsample microarray classification? Bioinformatics 2004,20(3):374380. 10.1093/bioinformatics/btg419View ArticleGoogle Scholar
 Molinaro AM, Simon R, Pfeiffer RM: Prediction error estimation: a comparison of resampling methods. Bioinformatics 2005,21(15):33013307. 10.1093/bioinformatics/bti499View ArticleGoogle Scholar
 Efron B: Bootstrap methods: another look at the jacknife. Annals of Statistics 1979, 7: 126. 10.1214/aos/1176344552MathSciNetView ArticleMATHGoogle Scholar
 BragaNeto UM, Dougherty ER: Bolstered error estimation. Pattern Recognition 2004,37(6):12671281. 10.1016/j.patcog.2003.08.017View ArticleMATHGoogle Scholar
 Devroye L, Gyorfi L, Lugosi G: A Probabilistic Theory of Pattern Recognition. Springer, New York, NY, USA; 1996.View ArticleMATHGoogle Scholar
 Chernick M: Bootstrap Methods: A Practitioners Guide. John Wiley & Sons, New York, NY, USA; 1999.MATHGoogle Scholar
 Sima C, Attoor S, BragaNeto UM, Lowey J, Suh E, Dougherty ER: Impact of error estimation on feature selection. Pattern Recognition 2005,38(12):24722482. 10.1016/j.patcog.2005.03.026View ArticleGoogle Scholar
 Sima C, Dougherty ER: Optimal convex error estimators for classification. Pattern Recognition 2006,39(9):17631780. 10.1016/j.patcog.2006.03.020View ArticleMATHGoogle Scholar
 Xu Q, Hua J, BragaNeto UM, Xiong Z, Suh E, Dougherty ER: Confidence intervals for the true classification error conditioned on the estimated error. Technology in Cancer Research and Treatment 2006,5(6):579590.View ArticleGoogle Scholar
 Xiao Y, Hua J, Dougherty ER: Quantification of the impact of feature selection on the variance of crossvalidation error estimation. EURASIP Journal on Bioinformatics and Systems Biology 2007, 2007:11.Google Scholar
 Tabus I, Astola J: Gene feature selection. In Genomic Signal Processing and Statistics. Hindawi, New York, NY, USA; 2005:6792.Google Scholar
 van't Veer LJ, Dai H, van de Vijver MJ, et al.: Gene expression profiling predicts clinical outcome of breast cancer. Nature 2002,415(6871):530536. 10.1038/415530aView ArticleGoogle Scholar
 van de Vijver MJ, He YD, van't Veer LJ, et al.: A geneexpression signature as a predictor of survival in breast cancer. New England Journal of Medicine 2002,347(25):19992009. 10.1056/NEJMoa021967View ArticleGoogle Scholar
 Choudhary A, Brun M, Hua J, Lowey J, Suh E, Dougherty ER: Genetic test bed for feature selection. Bioinformatics 2006,22(7):837842. [http://www.ece.tamu.edu/~edward/fstestbed] 10.1093/bioinformatics/btl008View ArticleGoogle Scholar
Copyright
This article is published under license to BioMed Central Ltd. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.