State-space vs. transformer: a comparative analysis of architectures and optimization techniques on mathematical reasoning benchmark

Abstract

Large Language Models (LLMs) are incredible and sophisticated models that have remarkable capabilities that towers over a wide range of language processing tasks. However, that capability comes with a substantial computational and memory cost which is often too great. This is why it is harder for the general population to access this technology. This paper explores other alternative architectures such as State- Space models in order to understand which models are lightweight but also retain strong performance. The aim is to compare the SSM based Llamba model family performances on mathematical reasoning tasks which remains unexplored. A deep investigation into the accuracy of these models is performed along with other computational metrics such as decode speed, memory usage etc. An exploration has been made on how different architectural choices impact both efficiency and effectiveness of these models. The results provide a clear picture of the existing landscape that Transformers are the obvious architectural choice for the best reasoning accuracy while SSMs can be the viable and efficient alternative in the situations when the computational resources are the main constraint.