Developing an empathetic conversational system using fine-tuned language models and psychometric evaluation

Abstract

Developing an empathetic conversational system remains challenging, particularly when models must generate contextually relevant, accurate, and sensitive responses from limited data. Recent advances in large language models (LLMs) have demonstrated that architectural innovation strategies can enhance dialogue performance and adaptability. In this work, we experimented with multiple LLM architectures: Gemma 2 9B IT, Zephyr 7B Beta, Llama 2 7B and Phi-3 Mini 4K Instruct for dialogue generation and structured question–answer tasks, and selected the bestperforming model based on response quality and accuracy. The models were evaluated on a dataset of 839 samples, derived from validated psychometric scales including MSPSS (Multidimensional Scale of Perceived Social Support), PHQ-4 (Patient Health Questionnaire-4), and PSS-4 (Perceived Stress Scale-4), curated and reviewed by a psychiatrist to ensure clinical relevance. MSPSS measures perceived social support from family, friends, and significant others; PHQ- 4 screens for anxiety and depression severity; and PSS-4 assesses perceived stress levels, with reverse scoring for certain items. Responses to these scales were converted into numeric scores and integrated as structured input, allowing the selected model to generate dialogue that adapts to each user’s emotional state, stress level, and social support network. Parameter-efficient fine-tuning techniques such as LoRA and PEFT were employed to maximize learning from the small dataset. Overall, this study demonstrates that instruction-tuning a carefully chosen LLM with psychometric-informed inputs enables empathetic, context-aware, and clinically informed dialogue, offering a promising approach for personalized mental health assessment and conversational support.