BE-KNN: an efficient extension of k-nearest neighbors via stable and deterministic ensemble learning

Abstract

The k-Nearest Neighbors (KNN) algorithm is widely applied in machine learning due to its simplicity and versatility. However, its reliance on distance calculations leads to scalability challenges, sensitivity to noise, and high computational costs when applied to large datasets. Ensemble approaches have been proposed to enhance robustness, yet they typically introduce randomness and increase runtime complexity. To address these limitations, we propose Batch Ensemble k-Nearest Neighbors (BE-KNN), a deterministic variant that partitions training data into batches, applies KNN within each batch, and aggregates predictions via majority voting. This design preserves reproducibility, reduces computational burden, and produces an ensemble-like effect without requiring multiple KNN models. We evaluated BE-KNN against standard KNN and stochastic ensemble KNN using 30 synthetic datasets and four real-world benchmarks (Wine, Iris, Breast Cancer, and Digits), with the F1-score as the primary evaluation metric. Results demonstrate that while standard KNN and stochastic ensembles achieve slightly higher accuracy on synthetic datasets, BE-KNN provides a significant reduction in execution time, up to 12 times faster than stochastic ensembles while maintaining competitive predictive performance. On real-world datasets, BE-KNN outperformed standard KNN in mean F1-score and demonstrated more stable performance across diverse tasks. These findings position BE-KNN as an efficient and robust alternative for real-time and resource-constrained applications.