WASPO: Workload-Aware Spark Performance Optimization Using NSGA-II

Conference paper

Karami, A. and Amirhosseini, M. 2025. WASPO: Workload-Aware Spark Performance Optimization Using NSGA-II. Cognitive Models and Artificial Intelligence Conference. Prague-Czech Republic 13 - 14 Jun 2025 IEEE.
AuthorsKarami, A. and Amirhosseini, M.
TypeConference paper
Abstract

The rapid growth of data-intensive applications has heightened the need for efficient big data processing frameworks like Apache Spark. However, optimizing Spark cluster configurations remains a complex challenge due to the diverse workload characteristics, varying data sizes, and conflicting resource demands. This paper introduces WASPO (Workload-Aware Spark Performance Optimization), a novel framework using NSGA-II for multi-objective optimization of Spark configurations. WASPO dynamically balances performance, resource efficiency, and scalability by incorporating workload-specific characteristics and adaptive scaling strategies. The proposed framework addresses the limitations of existing approaches, including static configurations, single-objective optimization, and neglect of workload heterogeneity. Experimental results demonstrate significant improvements in resource utilization and processing performance for both Machine Learning and Mixed workloads across data sizes ranging from 0.1TB to 1,000,000TB (1000PB).

Year2025
ConferenceCognitive Models and Artificial Intelligence Conference
PublisherIEEE
Accepted author manuscript
License
File Access Level
Anyone
Publication process dates
Accepted03 May 2025
Deposited14 May 2025
Journal citationp. In press
ISBN979-8-3315-0969-9
Copyright holder© 2025 IEEE
Copyright informationPersonal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.
Permalink -

https://repository.uel.ac.uk/item/8z750

Log in to edit

Download files

Accepted author manuscript
Manuscript-IEEE.pdf
License: All rights reserved
File access level: Anyone

  • 25
    total views
  • 18
    total downloads
  • 12
    views this month
  • 6
    downloads this month

Export as

Related outputs

Integrated Sentiment and Emotion Analysis of the Ukraine-Russia Conflict Using Machine Learning and Transformer Models
Amirhosseini, M., Berardinelli, N., Gaikwad, K., Iwuchukwu, C. E. and Ahmed, M. 2025. Integrated Sentiment and Emotion Analysis of the Ukraine-Russia Conflict Using Machine Learning and Transformer Models. International Conference on Data Science, Technology and Applications. Bilbao - Spain 10 - 12 Jun 2025 SciTePress.
Advancing Personality Type Prediction: Utilizing Enhanced Machine and Deep Learning Models with the Myers-Briggs Type Indicator
Amirhosseini, M., Karami, A. and Kalabi, F. 2025. Advancing Personality Type Prediction: Utilizing Enhanced Machine and Deep Learning Models with the Myers-Briggs Type Indicator. Cognitive Models and Artificial Intelligence Conference. Prague-Czech Republic 13 - 14 Jun 2025 IEEE.
Harnessing Social Media Sentiment for Predictive Insights into the Nigerian Presidential Election
Alao, J. O., Amirhosseini, M., Karami, A. and Ghorashi, S. A. 2025. Harnessing Social Media Sentiment for Predictive Insights into the Nigerian Presidential Election. Cognitive Models and Artificial Intelligence Conference. Prague-Czech Republic 13 - 14 Jun 2025 IEEE.
AI-Driven Mortality Prediction in COVID-19 Patients Using Advanced Feature Selection
Rajakaruna, I., Amirhosseini, M., Li, Y., Karami, A. and Arachchillage, D. J. 2025. AI-Driven Mortality Prediction in COVID-19 Patients Using Advanced Feature Selection. Cognitive Models and Artificial Intelligence Conference. Prague-Czech Republic 13 - 14 Jun 2025 IEEE.
Prediction of assistance dog training outcomes using machine learning and deep learning models
Amirhosseini, M., Serpell, J. A., Bray, E. E., Block, T. A., Douglas, L. E. L. C., Kennedy, B. S., Evans, K. M., Freeberg, K. and Pettigrew, P. 2025. Prediction of assistance dog training outcomes using machine learning and deep learning models. Applied Animal Behaviour Science. 287 (Art. 106632). https://doi.org/10.1016/j.applanim.2025.106632
Comparison of 7 Artificial Intelligence models in Predicting Venous Thromboembolism in COVID-19 Patients
Rajakaruna, I., Amirhosseini, M., Makris, M., Laffan, M., Li, Y. and Arachchillage, D. J. 2025. Comparison of 7 Artificial Intelligence models in Predicting Venous Thromboembolism in COVID-19 Patients. Research and Practice in Thrombosis and Haemostasis. 9 (2), p. Art. 102711. https://doi.org/10.1016/j.rpth.2025.102711
Harmony in Federated Learning: A Comprehensive Review of Techniques to Tackle Heterogeneity and Non-IID Data
Karami, M. and Karami, A. 2025. Harmony in Federated Learning: A Comprehensive Review of Techniques to Tackle Heterogeneity and Non-IID Data. Cluster Computing. p. In press.
The impact of big data characteristics on credit risk assessment
Karami, A. and Igbokwe, C. 2025. The impact of big data characteristics on credit risk assessment. International Journal of Data Science and Analytics. p. In press. https://doi.org/10.1007/s41060-025-00753-8
Ethereum Smart Contracts: A Hierarchical Analysis of Vulnerability Challenges and Mitigation Strategies
Soofiyan, S. and Karami, A. 2025. Ethereum Smart Contracts: A Hierarchical Analysis of Vulnerability Challenges and Mitigation Strategies. Cluster Computing. p. In press.
Leveraging Big Data Characteristics for Enhanced Healthcare Fraud Detection
Karami, A. and Jafari, F. 2025. Leveraging Big Data Characteristics for Enhanced Healthcare Fraud Detection. Cluster Computing. p. In press.
Breaking Down SEO Complexity: Bridging PCA and Bayesian-Optimized t-SNE
Karami, A., Ghasemabadi, S. F. and Amirhosseini, M. 2024. Breaking Down SEO Complexity: Bridging PCA and Bayesian-Optimized t-SNE. 2024 IEEE International Conference on Big Knowledge (ICBK). IEEE. https://doi.org/10.1109/ICKG63256.2024.00028
Exploring the Ethical Implications of AI-Powered Personalization in Digital Marketing
Karami, A., Shemshaki, M. and Ghazanfar, M. 2024. Exploring the Ethical Implications of AI-Powered Personalization in Digital Marketing. Data Intelligence. p. In Press. https://doi.org/10.3724/2096-7004.di.2024.0055
Utilizing machine Learning Techniques to Predict State-of-Charge in Li-ion Batteries
Khatri, A., Lota, J., Nepal, P. and Amirhosseini, M. H. 2024. Utilizing machine Learning Techniques to Predict State-of-Charge in Li-ion Batteries. IS'24: 12th IEEE International Conference on Intelligent Systems. Varna, Bulgaria 29 - 31 Aug 2024 IEEE. https://doi.org/10.1109/IS61756.2024.10705192
AI-Enhanced Prediction of Multi Organ Failure in COVID-19 Patients
Rajakaruna, I., Amirhosseini, M. H., Li, Y. and Arachcillage, D. J. 2024. AI-Enhanced Prediction of Multi Organ Failure in COVID-19 Patients. IS'24: 12th IEEE International Conference on Intelligent Systems. Varna, Bulgaria 29 - 31 Aug 2024 IEEE. https://doi.org/10.1109/IS61756.2024.10705181
Prediction of Depression Severity and Personalised Risk Factors Using Machine Learning on Multimodal Data
Amirhosseini, M. H., Ayodele, A. L. and Karami, A. 2024. Prediction of Depression Severity and Personalised Risk Factors Using Machine Learning on Multimodal Data. IS'24: 12th IEEE International Conference on Intelligent Systems. Varna, Bulgaria 29 - 31 Aug 2024 IEEE. https://doi.org/10.1109/IS61756.2024.10705185
An AI Powered System to Detect Autism Spectrum Disorder in Toddlers
Amirhosseini, M. H., Alam, N., Kalabi, F. and Virdee, B. 2024. An AI Powered System to Detect Autism Spectrum Disorder in Toddlers. ICDAM-2024: 5th International Conference on Data Analytics and Management. London, UK 14 - 15 Jun 2024
Machine Learning in Lithium-Ion Battery: Applications, Challenges, and Future Trends
Valizadeh, A. and Amirhosseini, M. 2024. Machine Learning in Lithium-Ion Battery: Applications, Challenges, and Future Trends. SN Computer Science. 5 (Art. 717). https://doi.org/10.1007/s42979-024-03046-2
A Graph-Based Method for Identity Resolution to Assist Police Force Investigative Process
Amirhosseini, M., Kazemian, H. and Phillips, M. 2024. A Graph-Based Method for Identity Resolution to Assist Police Force Investigative Process. Journal of Cyber Security and Technology. In Press. https://doi.org/10.1080/23742917.2024.2354555
Predictive precision in battery recycling: unveiling lithium battery recycling potential through machine learning
Valizadeh, A., Amirhosseini, M. H. and Ghorbani, Y. 2024. Predictive precision in battery recycling: unveiling lithium battery recycling potential through machine learning. Computers and Chemical Engineering. 183 (Art. 108623). https://doi.org/10.1016/j.compchemeng.2024.108623
An artificial intelligence approach to predicting personality types in dogs
Amirhosseini, M. H., Yadav, V., Serpell, J. A., Pettigrew, P. and Kain, P. 2024. An artificial intelligence approach to predicting personality types in dogs. Scientific Reports. 14 (Art. 2404). https://doi.org/10.1038/s41598-024-52920-9
Forecasting Bitcoin Prices in the Context of the COVID-19 Pandemic Using Machine Learning Approaches
Sontakke, P., Jafari, F., Saeedi, M. and Amirhosseini, M. 2024. Forecasting Bitcoin Prices in the Context of the COVID-19 Pandemic Using Machine Learning Approaches. ICDAM-2023: 4th International Conference on Data Analytics & Management. London, UK 23 - 24 Jun 2023 Springer. https://doi.org/10.1007/978-981-99-6544-1_7
An AI powered system to enhance self-reflection practice in coaching
Jelodari, M., Amirhosseini, M. H. and Giraldez Hayes, A. 2023. An AI powered system to enhance self-reflection practice in coaching. Cognitive Computation and Systems. 5 (4), pp. 243-254. https://doi.org/10.1049/ccs2.12087
Sentiment-Driven Cryptocurrency Price Prediction: A Machine Learning Approach Utilizing Historical Data and Social Media Sentiment Analysis
Bhatt, S., Ghazanfar, M. and Amirhosseini, M. 2023. Sentiment-Driven Cryptocurrency Price Prediction: A Machine Learning Approach Utilizing Historical Data and Social Media Sentiment Analysis. Machine Learning and Applications: An International Journal (MLAIJ). 10 (2/3), pp. 1-15. https://doi.org/10.5121/mlaij.2023.10301
Machine Learning based Cryptocurrency Price Prediction using historical data and Social Media Sentiment
Bhatt, S., Ghazanfar, M. and Amirhosseini, M. 2023. Machine Learning based Cryptocurrency Price Prediction using historical data and Social Media Sentiment . 5th International Conference on Machine Learning & Applications (CMLA 2023). Sydney, Australia 17 - 18 Jun 2023 AIRCC Publishing Corporation.
Large-Scale Music Genre Analysis and Classification Using Machine Learning with Apache Spark
Chaudhury, M., Karami, A. and Ghazanfar, M. A. 2022. Large-Scale Music Genre Analysis and Classification Using Machine Learning with Apache Spark. Electronics. 11 (16), p. 2567. https://doi.org/10.3390/electronics11162567
A Machine Learning Approach to Identify the Preferred Representational System of a Person
Amirhosseini, M. and Wall, J. 2022. A Machine Learning Approach to Identify the Preferred Representational System of a Person. Multimodal Technologies and Interaction. 6 (12), p. 112. https://doi.org/10.3390/mti6120112
Application of Graph-Based Technique to Identity Resolution
Kazemian, H., Amirhosseini, M. H. and Phillips, M. 2022. Application of Graph-Based Technique to Identity Resolution. AIAI 2022: 18th International Conference on Artificial Intelligence Applications and Innovations. Crete, Greece 17 - 20 Jun 2022 Springer. https://doi.org/10.1007/978-3-031-08333-4_38
Designing a Cost-Efficient Network for a Small Enterprise
Jafari, F., Karami, A. and Osemwengie, L. 2021. Designing a Cost-Efficient Network for a Small Enterprise. SAI Computing Conference 2021. Online 15 - 16 Jul 2021 Springer, Cham. https://doi.org/10.1007/978-3-030-80119-9_14
Stock market prediction using machine learning classifiers and social media, news
Khan, W., Ghazanfar, M., Azam, M. A., Karami, A., Alyoubi, K. H. and Alfakeeh, A. S. 2020. Stock market prediction using machine learning classifiers and social media, news. Journal of Ambient Intelligence and Humanized Computing. 13, pp. 3433-3456. https://doi.org/10.1007/s12652-020-01839-w
A novel centroids initialisation for K-means clustering in the presence of benign outliers
Karami, A., Ur Rehman, S. and Ghazanfar, M. 2020. A novel centroids initialisation for K-means clustering in the presence of benign outliers. International Journal of Data Analysis Techniques and Strategies. 12 (4), pp. 287-298. https://doi.org/10.1504/IJDATS.2020.111498
A Rule and Graph-Based Approach for Targeted Identity Resolution on Policing Data
Phillips, M., Amirhosseini, M. and Kazemian, H. 2020. A Rule and Graph-Based Approach for Targeted Identity Resolution on Policing Data. 2020 IEEE Symposium Series on Computational Intelligence. Online 01 - 04 Dec 2020 IEEE. https://doi.org/10.1109/SSCI47803.2020.9308182
Machine Learning Approach to Personality Type Prediction Based on the Myers–Briggs Type Indicator®
Amirhosseini, M.H. and Kazemian, H. 2020. Machine Learning Approach to Personality Type Prediction Based on the Myers–Briggs Type Indicator®. Multimodal Technologies and Interaction. 4 (Art. 9). https://doi.org/10.3390/mti4010009
Automating the process of identifying the preferred representational system in Neuro Linguistic Programming using Natural Language Processing
Amirhosseini, M.H. and Kazemian, H. 2019. Automating the process of identifying the preferred representational system in Neuro Linguistic Programming using Natural Language Processing. Cognitive Processing. 20 (2), p. 175–193. https://doi.org/10.1007/s10339-019-00912-3
An Anomaly-based Intrusion Detection System in Presence of Benign Outliers with Visualization Capabilities
Karami, A. 2018. An Anomaly-based Intrusion Detection System in Presence of Benign Outliers with Visualization Capabilities. Expert Systems with Applications. 108, pp. 36-60. https://doi.org/10.1016/j.eswa.2018.04.038
Natural Language Processing approach to NLP Meta model automation
Amirhosseini, M.H., Kazemian, H., Ouazzane, K. and Chandler, C. 2018. Natural Language Processing approach to NLP Meta model automation. 2018 International Joint Conference on Neural Networks (IJCNN). Rio de Janeiro, Brazil 08 - 13 Jul 2018 IEEE. https://doi.org/10.1109/IJCNN.2018.8489609
Functional Connectivity Evaluation for Infant EEG Signals based on Artificial Neural Network
Sharif, M., Naeem, U., Islam, S. and Karami, A. 2018. Functional Connectivity Evaluation for Infant EEG Signals based on Artificial Neural Network. Arai, Kohei, Kapoor, Supriya and Bhatia, Rahul (ed.) Intelligent Systems Conference (IntelliSys) 2018. London, UK 06 - 07 Sep 2018 Springer, Cham. https://doi.org/10.1007/978-3-030-01057-7_34
The Application of a Semantic-Based Process Mining Framework on a Learning Process Domain
Okoye, Kingsley, Islam, S., Naeem, U., Sharif, M., Azam, Muhammad Awais and Karami, A. 2018. The Application of a Semantic-Based Process Mining Framework on a Learning Process Domain. Arai, Kohei, Kapoor, Supriya and Bhatia, Rahul (ed.) Intelligent Systems Conference (IntelliSys) 2018. London, UK 06 - 07 Sep 2018 Springer, Cham. https://doi.org/10.1007/978-3-030-01054-6_96
A Framework for Uncertainty-Aware Visual Analytics in Big Data
Karami, A. 2015. A Framework for Uncertainty-Aware Visual Analytics in Big Data. CEUR Workshop Proceedings. 1510, pp. 146-155.
Utilization of multi attribute decision making techniques to integrate automatic and manual ranking of options
Karami, A. and Johansson, Ronnie 2013. Utilization of multi attribute decision making techniques to integrate automatic and manual ranking of options. Journal of Information Science and Engineering. 30 (2), pp. 519-534.
Choosing DBSCAN parameters automatically using differential evolution
Karami, A. and Johansson, Ronnie 2014. Choosing DBSCAN parameters automatically using differential evolution. International Journal of Computer Applications. 91 (7), pp. 1-11. https://doi.org/10.5120/15890-5059
A fuzzy anomaly detection system based on hybrid PSO-Kmeans algorithm in content-centric networks
Karami, A. and Guerrero-Zapata, Manel 2014. A fuzzy anomaly detection system based on hybrid PSO-Kmeans algorithm in content-centric networks. Neurocomputing. 149 (Part C), pp. 1253-1269. https://doi.org/10.1016/j.neucom.2014.08.070
A hybrid multiobjective RBF-PSO method for mitigating DoS attacks in Named Data Networking
Karami, A. and Guerrero-Zapata, Manel 2014. A hybrid multiobjective RBF-PSO method for mitigating DoS attacks in Named Data Networking. Neurocomputing. 151 (3), pp. 1262-1282. https://doi.org/10.1016/j.neucom.2014.11.003
An ANFIS-based cache replacement method for mitigating cache pollution attacks in Named Data Networking
Karami, A. and Guerrero-Zapata, Manel 2015. An ANFIS-based cache replacement method for mitigating cache pollution attacks in Named Data Networking. Computer Networks. 80 (April), pp. 51-65. https://doi.org/10.1016/j.comnet.2015.01.020
ACCPndn: Adaptive Congestion Control Protocol in Named Data Networking by learning capacities using optimized Time-Lagged Feedforward Neural Network
Karami, A. 2015. ACCPndn: Adaptive Congestion Control Protocol in Named Data Networking by learning capacities using optimized Time-Lagged Feedforward Neural Network. Journal of Network and Computer Applications. 56 (Oct.), pp. 1-18. https://doi.org/10.1016/j.jnca.2015.05.017
A Wormhole Attack Detection and Prevention Technique in Wireless Sensor Networks
Siddiqui, A., Karami, A. and Johnson, M. O. 2017. A Wormhole Attack Detection and Prevention Technique in Wireless Sensor Networks. International Journal of Computer Applications. 174 (Art. 4). https://doi.org/10.5120/ijca2017915376