WASPO: Workload-Aware Spark Performance Optimization Using NSGA-II
Conference paper
Karami, A. and Amirhosseini, M. 2025. WASPO: Workload-Aware Spark Performance Optimization Using NSGA-II. Cognitive Models and Artificial Intelligence Conference. Prague-Czech Republic 13 - 14 Jun 2025 IEEE.
Authors | Karami, A. and Amirhosseini, M. |
---|---|
Type | Conference paper |
Abstract | The rapid growth of data-intensive applications has heightened the need for efficient big data processing frameworks like Apache Spark. However, optimizing Spark cluster configurations remains a complex challenge due to the diverse workload characteristics, varying data sizes, and conflicting resource demands. This paper introduces WASPO (Workload-Aware Spark Performance Optimization), a novel framework using NSGA-II for multi-objective optimization of Spark configurations. WASPO dynamically balances performance, resource efficiency, and scalability by incorporating workload-specific characteristics and adaptive scaling strategies. The proposed framework addresses the limitations of existing approaches, including static configurations, single-objective optimization, and neglect of workload heterogeneity. Experimental results demonstrate significant improvements in resource utilization and processing performance for both Machine Learning and Mixed workloads across data sizes ranging from 0.1TB to 1,000,000TB (1000PB). |
Year | 2025 |
Conference | Cognitive Models and Artificial Intelligence Conference |
Publisher | IEEE |
Accepted author manuscript | License File Access Level Anyone |
Publication process dates | |
Accepted | 03 May 2025 |
Deposited | 14 May 2025 |
Journal citation | p. In press |
ISBN | 979-8-3315-0969-9 |
Copyright holder | © 2025 IEEE |
Copyright information | Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works. |
https://repository.uel.ac.uk/item/8z750
Download files
25
total views18
total downloads12
views this month6
downloads this month