Analyzing MOOC reviews: a comparative study of learner feedback and sentiment

Abstract

MOOCs (Massive Open Online Courses) offer broad and diverse access to education, but summarizing learner feedback remains difficult because of the large volume and wide-ranging content of the reviews. Extracting meaningful insights from learner feedback saves the time of the learner and helps to make a quick decision about the course. This study focuses on analyzing and summarizing MOOC reviews across multiple courses to understand learner impressions towards the course employing advanced Natural Language Processing (NLP) models including BART, PEGASUS, T5 and DISTILBART with an aim of generating brief yet coherent summaries. Since MOOC reivews lacked human-written summaries, our experiment demonstrate that, even in the absence of supervised data from MOOCs, our method greatly enhances summary quality. Finally, our evaluation framework incorporated semantic similarity, coherence, and human evaluation. T5 achieved the highest semantic similarity score (0.60), while PEGASUS demonstrated the best coherence (0.40). In human evaluations, PEGASUS received the highest ratings in fluency (mean score: 4.75) and maintained strong performance across relevance (4.35) and factual accuracy (4.40). T5 closely followed with the highest relevance (4.40) and factual accuracy (4.45) scores. However, Cohen’s Kappa scores revealed low inter-rater agreement, with most values indicating slight or even negative agreement—highlighting subjectivity and inconsistency among raters. Ultimately, this study intended to help prospective learners quickly perceive the overall sentiment and key takeaways of a course that saves their time and effort of reading through thousands of individual comments.