A Software Engineering Perspective on Documentation Quality: Predicting Newcomer Attraction in Open-Source Projects

Abstract

Attracting new contributors is fundamental to the long-term sustainability of Open-Source Software (OSS) projects, yet the project-level documentation and community-health characteristics that predict newcomer attraction remain understudied. We present a large-scale empirical study of 717 GitHub repositories, extracting 21 documentation and community-health features (17 documentation/community/issue-communication features plus 4 control variables) and measuring whether each project attracted at least 10 new contributors within six months. Two gradient-boosting models --- LightGBM and CatBoost --- are evaluated as primary candidates using stratified 5-fold cross-validation, with TabNet included as a deep-learning robustness check rather than a competing candidate, consistent with prior tabular-data benchmarks. CatBoost achieves the highest predictive performance (AUC-ROC = 0.8730 $\pm$ 0.0143, F1-score = 0.8006 $\pm$ 0.0179), a statistically consistent though not significant improvement over LightGBM at the five-fold sample size (paired $t$-test $p$ = 0.029; Wilcoxon signed-rank $p$ = 0.0625). SHAP analysis shows that issue responsiveness --- Issue Close Rate (average SHAP = 0.722) and Average First Response Time (average SHAP = 0.430) --- are the strongest predictors of newcomer attraction, with README Length (average SHAP = 0.420) the strongest static documentation predictor; directional analysis further reveals that this association is non-monotonic, peaking for moderate-length READMEs rather than increasing indefinitely with length. An ablation study further confirms that documentation features contribute an incremental AUC gain of +0.0315 above control variables alone and achieve an out-of-fold AUC of 0.7124 in isolation, showing that the documentation signal is not merely a proxy for project size or maturity. A complete replication package is publicly available to support reproducibility.