Comparison of XGBoost and Neural Networks for Quality Factor Prediction in Optical Communication

Abstract

In this paper, we evaluate two regression approaches for predicting the Q-factor in optical communications model: completely interconnected neural networks and XGBoost, utilizing a synthetic dataset that replicates real fiber imperfections. Ten thousand optical connection possibilities were compiled into a dataset. Prior to being divided into 80% training and 20% test sets, inputs and target values were standardized. Three layers that are hidden (128, 64, and 32 neurones) with premature halting, dropout normalization (rate 0.2), and ReLU activation processes are present in the neural network. It was trained for an optimal number of 500 epochs using the Adam optimizer. 100 trees (maximum depth = 6), subsampling (0.8), a learning level of 0.1, and a premature terminate after 10 rounds were all utilized in the XGBoost model. Performance was assessed employing the determination coefficient (R²) and mean absolute error (MAE). On the held-out testing set, XGBoost fared marginally better than the neural network, with an MAE of 0.38 and an R² of 0.92