Performance-Weighted Resampling Consensus Variable Selection for High-Dimensional Cancer Classification

Abstract

The colon and prostate gene-expression matrices contain, respectively, 2,000 variables for 62 samples and 12,600 variables for 102 samples. In such small-sample settings, a useful selector must be judged by more than the accuracy of one fitted model.We developed Performance-Weighted Resampling Consensus Variable Selection (PWRCVS) to examine whether a short gene list can remain competitive after repeated perturbation of the training data. In each outer-training sample, Fisher Score and Information Gain were crossed with Random Forest, Gradient Boosting, and Extreme Gradient Boosting. Thirty stratified resamples produced six filter–learner paths per resample. A path contributed in proportion to its internally validated Matthews correlation coefficient (MCC), and variables were scored from their repeated positive permutation importance. Correlation components were then reduced to one representative before LASSO, Adaptive LASSO, Elastic Net, SCAD, MCP, RF, GBOOST, and XGBOOST were refitted. Evaluation used five repeats of outer stratified five-fold cross-validation; no test-fold observation entered filtering, tuning, or threshold selection. PWRCVS retained 7.76 ± 2.24 variables for colon cancer and 14.12 ± 4.00 for prostate cancer, reductions of 99.612% and 99.888%. On colon data, PWRCVS-XGBOOST gave MCC= 0.663 versus 0.666 for FS+IG-ENET, while the proposed list was 56.5% smaller and won eight of eleven matched MCC comparisons. On prostate data, PWRCVS-RF gave MCC = 0.853 versus 0.864 for IG-RF with 21.6% fewer variables. None of the 11 prespecified classifier-matched MCC contrasts was significant after Holm adjustment in either dataset. Stability selection, bootstrap selection and recursive feature elimination were then run inside the identical consensus machinery; none of the 12 resulting contrasts was significant either, and the proposed selector cost about four to six times more per fold. Thus, the empirical gain is parsimony with little loss of discrimination in these two datasets, not uniform predictive superiority.