PSM: Proactive Spill Mitigation in PySpark

Apache Spark's performance is critically dependent on efficient in-memory computation; however, insufficient executor memory frequently results in costly disk spills, severely degrading performance. Traditional spill mitigation techniques, including static configuration tuning, reactive Adaptive Query Execution (AQE), and internal spill handling mechanisms, exhibit limitations in addressing the dynamic and fine-grained nature of memory pressure in complex and large-scale workloads. This paper introduces Proactive Spill Mitigation (PSM), a novel methodology designed to anticipate and mitigate memory spills in PySpark applications before they occur. PSM leverages real-time task and executor metrics, accessible via the Spark Listener API, as input to a machine learning model trained to predict the likelihood and potential severity of impending spills. Upon detecting a high spill risk, a control mechanism triggers preemptive actions, such as checkpointing intermediate results or dynamically adjusting partition counts, to alleviate memory pressure. Experimental evaluation using the TPC-DS and HiBench Sort benchmarks demonstrates that PSM significantly reduces both execution time and the volume of data spilled to disk across various cluster sizes, outperforming default Spark configurations, statically tuned setups, and configurations with AQE enabled. This work establishes the efficacy of predictive control for enhancing Spark's memory management and improving application performance in dynamic environments.