Harmony in Federated Learning: A Comprehensive Review of Techniques to Tackle Heterogeneity and Non-IID Data

This comprehensive review paper provides a critical analysis of the significant challenges posed by heterogeneity and non-independent, non-identically distributed (Non-IID) data within Federated Learning environments. The review explores different types of heterogeneity challenges, including data space, system and device, statistical, and model heterogeneity as well as diverse Non-IID data challenges, including feature distribution skew, label distribution skew, different features with the same label, same features with different labels, quantity skew, and temporal skew. Practical examples show the distinct advantages and disadvantages of each problem, highlighting the necessity of strong, flexible approaches to deal with these complexity and realise Federated Learning's full potential in mission-critical applications. By critically examining state-of-the-art solutions, this review hopes to support continued efforts to remove obstacles that have prevented Federated Learning from being widely adopted, creating a future where organizations can utilize AI while respecting user privacy.