Detecting developer emotions in GitHub commits using large language models: implications for project health

Abstract

Acknowledging the emotions of developers is crucial for maintaining productivity, team morale, and project success. Developers’ emotions are often reflected in their commit messages which can reveal subtle cues of satisfaction, frustration, or caution. Monitoring these signals helps project managers recognize early signs of stress and respond proactively, leading to healthier team dynamics and better project outcomes. This thesis investigates emotions in formal GitHub commit messages and their connection to long-term project health. A Project Health Dataset was built by sampling 20,000 commits from 34 repositories and aggregating them quarterly across four dimensions: bug activity, productivity, emotions, and code churn. From this, a gold-standard set of 2,000 commits was manually annotated into four categories: Satisfaction, Frustration, Caution, and Neutral. Several zero-shot models were evaluated, and a fine-tuned CodeBERT model named CommiTune was developed which was trained on both human-labeled and LLaMA-augmented data. CommiTune achieved a Macro-F1 of 0.88 and Accuracy of 86% on a held-out test set which significantly outperformed all baselines. When applied at scale, the model revealed that while short-term emotion trends show weak correlations with project metrics however, long-term patterns are strong. Frustration aligns with higher bugfix rates, satisfaction correlates with lower churn and caution signals often precede rollback activity. This work contributes a novel dataset, a high-performing finetuned model and empirical evidence that developer emotions can serve as meaningful indicators of software quality and stability over time.