A Comprehensive Evaluation of ViLT and CLIP Transformers for Multimodal Fake News Detection with Metadata Integration - Abd Alhamid, Ahmed; Fakhr, Mohamed Waleed; Ashry, Mahmoud M; Abd Elhafeez, Ahmed (Statistics, Optimization & Information Computing, 2026)
The swift expansion of social media has fueled the prevalence of multimodal misinformation wherein textual information, visual material, and context combine to influence perception. In this research, we compare two vision-language transformer-based systems, namely, ViLT and CLIP, in the task of detecting multimodal fake news using two benchmark datasets Fakeddit and FakeNewsNet. These evaluations include binary as well as three-class classifications in both scenarios of metadata available and metadata-free. Furthermore, Late Fusion and Gated Fusion approaches are considered in order to explore the effectiveness of complementation through transformers. The results indicate that the effectiveness of metadata integration depends on the architecture and the dataset. ViLT can gain from metadata integration due to the presence of the unified transformer architecture enabling the interaction between textual, visual, and contextual features inside the attention mechanism. On the contrary, CLIP exhibits better behavior in semantic image-text alignment, especially when there is no metadata. ViLT has an accuracy score of 0.906 for the FakeNewsNet data set and 0.831 for the binary version of Fakeddit task, while CLIP obtains competitive accuracy scores for the three-class version of the same task without metadata. However, the improvements introduced by metadata were small for several cases, which suggests that metadata should not be taken as universally effective