Performance Evaluation of Distributed LSTM-Based Sentiment Analysis on Hadoop MapReduce and Apache Spark

Abstract

The release of massive text data volumes has led to an increased demand for distributed processing frameworks that are able to support deep learning applications. However, despite the widespread use of LSTM for sentiment analysis due to its ability to capture long-term contextual dependencies, limited studies have systematically evaluated Hadoop MapReduce and Apache Spark under identical experimental conditions for large-scale LSTM-based sentiment analysis. Data from 200 MB to 1 TB in size were used in experimentation in the same cluster configuration. The findings show that Spark provides systematically lower execution time and better scalability on all workloads due to its in-memory processing framework, whereas MapReduce is slower because disk writes and file merges are time-consuming. Spark consistently achieved up to a 63% decrease in execution time when processing the 1 TB dataset on a three-node cluster, demonstrating its efficiency for large-scale deep learning workloads. Furthermore, this study develops empirical execution-time models with R^2 > 0.95 to estimate the scalability of both frameworks across different dataset sizes, providing practical guidance for selecting appropriate distributed platforms for large-scale sentiment analysis. Our results provide partial insight into trade-offs between the two frameworks and assist the practitioners in selecting appropriate architectures of distributed NLP applications. Our future work will build on the analysis of transformer-based architecture along with streaming real-time data.