An Empirical Study of Label Smoothing in Transformer-Based Text Classification

Authors

  • ALI ALSHAHRANI Department of Computer Science and Artificial Intelligence, College of Computing and Information Technology, University of Bisha, Saudi Arabia
  • XIUWEN LIU Department of Computer Science, Florida State University, Tallahassee, USA
  • KHALID ALHARTHI Department of Computer Science and Artificial Intelligence, College of Computing and Information Technology, University of Bisha, Saudi Arabia
  • AHMED ABDELGHANY Department of Computer Science and Artificial Intelligence, College of Computing and Information Technology, University of Bisha, Saudi Arabia
  • SAAD ALSHAHRANI Department of English Language and Literature, University of Bisha, Saudi Arabia

DOI:

https://doi.org/10.19139/soic-2310-5070-3645

Keywords:

BERT, transformer, fine-tuning, soft label, regularization, attention

Abstract

Regularization methods play an important role in improving the generalization performance of deep neural networks. Label smoothing is an output-level regularization technique that replaces one-hot targets with softened class distributions and can reduce excessive confidence in model predictions. This study examines label smoothing in Transformer-based text classification across the Optimism/Pessimism Twitter (OPT), TweetEval, and FinancialPhraseBank benchmarks. BERT, RoBERTa, and ELECTRA are evaluated under hard-label and label-smoothing training conditions. The study also examines the sensitivity of performance to the smoothing coefficient and examines representation geometry and self-attention behavior as complementary analyses. A validation-based screening of α ∈ {0.00, 0.05, 0.10, 0.15} was conducted, and α = 0.05 was selected for the main experimental protocol based on aggregate validation performance.

Downloads

Published

2026-10-05

How to Cite

ALI ALSHAHRANI, XIUWEN LIU, KHALID ALHARTHI, ABDELGHANY, A., & SAAD ALSHAHRANI. (2026). An Empirical Study of Label Smoothing in Transformer-Based Text Classification. Statistics, Optimization & Information Computing. https://doi.org/10.19139/soic-2310-5070-3645

Issue

Online First

Section

Research Articles

License

Copyright (c) 2026 ALI ALSHAHRANI, XIUWEN LIU, KHALID ALHARTHI, AHMED ABDELGHANY, SAAD ALSHAHRANI

Creative Commons License

This work is licensed under a Creative Commons Attribution 4.0 International License.

Authors who publish with this journal agree to the following terms:

    1. Authors retain copyright and grant the journal right of first publication with the work simultaneously licensed under a Creative Commons Attribution License that allows others to share the work with an acknowledgement of the work's authorship and initial publication in this journal.
    2. Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgement of its initial publication in this journal.
    3. Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work (See The Effect of Open Access).