An LLM-based framework for automated Python test generation and mutation testing evaluation

Abstract

The reliability of software systems lies in effective test case design, especially in mutation testing, where the objective is to detect subtle faults introduced as mutants. Manually writing unit tests is time-consuming and increases development costs. While traditional test generation tools like Pynguin and Klara provide baseline automation, recent advancements in Large Language Models (LLMs) offer a promising alternative. In this research, we first conducted a comparative analysis between four state-of-the-art LLMs: ChatGPT, Gemini, Claude, and Llama and traditional test generation tools for Python, using mutation score as the primary metric. Motivated by the superior performance of LLMs, we developed a fully automated test generation tool that leverages prompt-based LLMs to create unit tests for Python functions. This tool not only generates syntactically valid and executable tests but also evaluates their quality using both mutation score (via MutPy) and test coverage metrics. The tool employs prompt-engineering strategies, repair loops, and error-driven feedback to iteratively refine failing test cases. The results show that our tool can be reliably harnessed for automated test generation and can outperform traditional tools in both mutation detection and coverage on Python programs.