TRI-FED-RKD: integrating forward-reverse distillation with SNN and CNN within federated learning using tri layer hierarchical aggregation based architecture

Abstract

Federated Learning (FL) is a decentralized machine learning paradigm that enables training a global model across numerous edge devices while preserving data privacy. However, FL faces significant challenges, particularly in environments with heterogeneous hardware capabilities, communication burdens, and constrained resources. In this paper, we introduce a novel framework, TRI-FED-RKD, which incorporates forward and reverse knowledge distillation (RKD) along with FedAvg using a hybrid architecture of convolutional neural networks (CNNs) and spiking neural networks (SNNs). Our approach employs a tri-layer hierarchical aggregation-based architecture consisting of client devices, intermediate (middle) servers, and a global server. We compared two federated architectures: standard federated learning and federated learning with forward and reverse distillation in a hierarchical setting (TRI-FED-RKD). The same model is used across several datasets to evaluate the architectures, not the model performance. Depending on the use case, the network administrator can pick their own teacher and student models. The teacher model can also be different for each client if needed. This means that our architecture can deal with model heterogeneity when it comes to teacher models. We evaluate TRIFED- RKD on neuromorphic datasets such as DVS Gesture and NMNIST. We also tested it using non-neuromorphic datasets such as MNIST, EMNIST, and CIFAR10. Furthermore, we have shown that using forward and reverse knowledge distillation in federated learning can lead to much better performance than federated learning without knowledge distillation for non-neuromorphic datasets.