Development of Penalized Estimation for Conway-Maxwell-Poisson Regression under Multicollinearity and High Dimensionality

Abstract

Modern count data applications are increasingly characterized by dispersion, high dimensionality, and strong predictor dependence, conditions under which maximum likelihood estimation for Conway--Maxwell--Poisson (COM-Poisson) regression becomes unstable and prone to severe overfitting. The purpose of this study is to develop an estimation framework for COM-Poisson regression that remains stable when the predictor dimension is large relative to the sample size and when predictors are strongly collinear, and that selects predictors rather than shrinking all of them. Such a framework is needed because the shrinkage estimators currently available for this model retain the full predictor set, so that a sparse solution cannot be obtained precisely in the settings where sparsity is most valuable. We propose an L1-regularized framework for COM-Poisson regression, to our knowledge the first of its kind, by introducing COMP-LASSO and COMP-ALASSO estimators based on the penalized COM-Poisson likelihood. The proposed estimators are computed using an iteratively reweighted coordinate descent algorithm that explicitly incorporates the dispersion structure of the COM-Poisson model. Simulation experiments across a wide range of dimensionality, multicollinearity, and dispersion settings show that penalized estimation consistently outperforms maximum likelihood in both prediction accuracy and variable selection, with the largest gains occurring in severely ill-conditioned problems. Empirical applications in clinical research, consumer finance, and community ecology demonstrate that the proposed methods produce substantially sparser and more accurate models while avoiding the instability and overfitting behaviour of maximum likelihood estimation. These results establish sparse penalized estimation as an effective and practical framework for high-dimensional dispersed count regression.