On Shrinkage Estimation: Non-orthogonal Case

  • A.K.Md. Ehsanes Saleh School of Mathematics and Statistics, Carleton University, Ottawa, Canada
  • Mina Norouzirad Department of Statistics, Faculty of Mathematical Sciences, Shahrood University of Technology, Shahrood, Iran
Keywords: Dominance, Efficiency, LASSO, PTE and Stein-type estimators, Penalty estimator, Risk function, Sparsity.


In this paper, we consider the estimation of the parameters of the non-orthogonal regression model, when we suspect a sparsity condition. We provide with a comparative performance characteristics of the primary penalty estimators, namely, the ridge and the LASSO, with the least square estimator, restricted LSE, preliminary test  and Stein-type of  estimators, when the dimension of the parameter  space is less than the dimension  of the sample space. Using the principle of marginal distribution theory, the analysis of risks leads to the following conclusions: (i) ridge estimator outperforms least squares, preliminary test and Stein-type estimators uniformly, (ii) The restricted least squares estimator and LASSO are competitive, although LASSO lags behind the restricted least squares estimator uniformly. Both estimators outperform the least squares, preliminary test, and Stein-type estimators in a subspace, respectively.  (iii) The lower bound risk expression of LASSO does not depend on the threshold parameter. (iv) Performance of the estimators depends upon the size of numbers of active coefficients, non-active coefficients, and the divergence parameter. In support of our conclusion, we prepare some tables and graphs relevant to the properties of the estimators.


A. Belloni, and V. Chernozhukov, Least squares after model selection in high-dimensional sparse models, Bernoulli, vol. 2, pp. 521–547, 2013.

L. Breiman, Heuristics of instability and stabilization in model selection, The Annals of Statistics, vol. 24, no. 6, pp. 2350–2383, 1996.

N. R. V. Draper, and R. Craig, Ridge Regression and James- Stein Estimation: Review and Comments, Technometrics, vol. 21, pp. 451–466., 1979.

D. L. Donoho, and I. M. Johnstone, Ideal Spatial Adaptation by Wavelet Shrinkage, Biometrika, vol. 81, no. 3, pp. 425–455, 1994.

J. Fan, and R. Li, Variable selection via nonconcave penalized likelihood and its oracle properties, Journal of the American Statistical Association, vol. 96, no. 456, pp. 1348–1360, 2001.

I. E. Frank, and J. H. Friedman, A Statistical View of Some Chemometrics Regression Tools, Technometrics, vol. 35, no. 2, pp. 109–135, 1993.

B. E. Hansen, The risk of James-Stein and LASSO Shrinkage, Econometric Reviews, vol. 35, no. 8-10, pp. 1456–1470, 2013.

A. E. Hoerl and R. W. Kennard, Ridge regression biased estimation for non-orthogonal problems, Thechnometrics, vol. 12, pp. 69–89, 1970.

G. C. McDonald, and D. I. Galarneau, A Monte Carol evaluation of some ridge-type estimators, J. Amer. Statist. Assoc, vol. 70, pp. 407–416, 1975.

A. K. Md. Ehsanes. Saleh, Theory of Preliminary Test and Stein-Type Estimation with Applications, Wiley; United Stated of America., 2006.

Stamey, T.A., Kabalin, J.N., McNeal, J.E., Johnstone, I.M., Freiha, F., Redwine, E.A. and Yang, N. Prostate specific antigen in the diagnosis and treatment of adenocarcinoma of the prostate: II. radical prostatectomy treated patients, Journal of Urology, 141(5), 1076-1083.

R. Tibshirani, Regression shrinkage and selection via the LASSO, J. Royal. Statist. Soc. B., vol. 58, no. 1, pp. 267-288, 1996.

A. N. Tikhonov, Solution of incorrectly formulated problems and the regularization method, Doklady Academia Nauk SSSR, Translated in Soviet Mathematics 4, vol. 151, pp. 501-504, 1963.

H. Zou, The adaptive Lasso and its oracle properties, Journal of the American Statistical Association, vol.101, no. 476, pp. 1418– 1429, 2006.

H. Zou, and T. Hastie, Regularization and variable selection via the elastic net, J. R. Stat. Soc. Ser. B Stat. Methodol, vol. 67, no. 2, pp. 301-320, 2005.

How to Cite
Ehsanes Saleh, A., & Norouzirad, M. (2018). On Shrinkage Estimation: Non-orthogonal Case. Statistics, Optimization & Information Computing, 6(3), 427-451. https://doi.org/10.19139/soic.v6i3.582
Research Articles