A comparative analysis of optimizers in recurrent neural networks for text classification

Abstract

The performance of any deep learning model depends heavily on the choice of optimizers and their corresponding hyper-parameters. For any given problem researchers struggle to select the best possible optimizer from a myriad of optimizers proposed in existing literature. Currently the process of optimizer selection in practice is anecdotal at best whereby practitioners either randomly select an optimizer or rely on best practices or online recommendations not grounded on empirical evidence base. In our paper, we delve deep into this problem of picking the right optimizer for text based datasets and linguistic classification problems, by bench-marking ten optimizers on three different RNN models (Bi-GRU, Bi-LSTM and BRNN) on three spam email based benchmark datasets. We analyse the performance of models employing these optimizers using train accuracy, train loss, validation accuracy, validation loss, test accuracy, test loss and RO-AUC score as metrics. The results show that Adaptive Optimization methods (RMSprop, Adam, Adam weight decay and Nadam) with default hyper-parameters outperform other optimizers in all three datasets and RNN model variations.