Divide2Conquer (D2C): a comprehensive study on decentralized overfitting remediation in deep learning

Abstract

Overfitting remains a persistent challenge in deep learning, primarily attributed to data outliers, noise, and limited training set sizes. This thesis presents Divide2Conquer (D2C), a novel technique designed to address this issue. D2C proposes partitioning the training data into multiple subsets and training separate identical models on them. To avoid overfitting on any specific subset, the trained parameters from these models are aggregated and averaged periodically throughout the training phase, enabling the model to learn from the entire dataset while mitigating the impact of individual outliers or noise. Empirical evaluations on multiple benchmark datasets across various deep learning tasks from different domains demonstrate that D2C effectively improves generalization performance, particularly for larger datasets. This study verifies D2C’s ability to achieve significant performance gains as a standalone technique and also when used in conjunction with other overfitting reduction methods through a series of experiments, including analysis of decision boundaries, loss curves, and other performance metrics. Additionally, we provide a rigorous mathematical justification for our hypothesis and analyze the applicability of the D2C method through extensive experimentation on various datasets covering multiple domains. We also delve into the trade-offs associated with D2C and explore strategies to mitigate these challenges, providing a comprehensive understanding of D2C’s strengths and weaknesses.