Peer-guided optimization: Incorporating collaborative learning into stochastic optimization in machine learning

Abstract

A single optimizer is usually used to minimize the cost function in conventional machine learning. This paper introduces Peer-Guided Optimization (PGO), where two optimizers collaborate by sharing gradient information to help each other explore the cost surface and accelerate convergence. Unlike traditional optimization methods, PGO alternates steps between two optimizers across iterations. Based on the proximity to the optimal solution, each optimizer dynamically calculates a guidance parameter that influences the contribution of the peer optimizer during updates. PGO has two variants: Homogeneous PGO (using identical optimizers with different learning rates) and Heterogeneous PGO (combining different types of optimizers, such as Adam and SGD). Experimental results demonstrate that PGO consistently outperforms conventional single-optimizer methods in terms of accuracy and convergence speed, making it a promising strategy for optimizing deep learning models in various applications.