Evaluating LLMs in higher education: a SEM based comparison of ChatGPT and DeepSeek using the IS success model

Abstract

Deploying Large Language Models (LLMs) necessitates tight evaluation models to measure the operational efficiency and learning skills. The research evaluates two of the greatest generative AI systems- ChatGPT and DeepSeek utilizing a widely established DeLone and McLean Information Systems Success Model. Quantitative research was applied to 365 university respondents in Bangladesh and was carried out using a repeatable 28-item Likert scale questionnaire. Structural Equation Modeling (SEM) using SmartPLS was utilized to calculate the system quality, information quality, service quality, intention to use, user happiness, and individual effect. Findings reveal that ChatGPT is more successful in developing the user's intention based on its phrasal flexibility and conversational communication. Nonetheless, DeepSeek is more successful in terms of user happiness and structural stability, particularly task-oriented features and individualized academic aid. The quality of service did not play a significant role in the satisfaction of the model of ChatGPT, unlike DeepSeek, where it turned out to be the most critical feature. The study further critiques the applicability of the traditional IS Success Model in the LLM domain, highlighting the need for augmented constructs such as trust, cognitive load, and emotional resonance. These findings contribute to the evolving discourse on AI adoption in education and provide actionable insights for stakeholders seeking pedagogically aligned AI integration.