Automating radiology report generation with CDGPT3.5: a deep learning approach for enhancing medical image interpretation

Abstract

The process of generating radiology reports at healthcare facilities is time-consuming and needs vast experience from the Radiologists. Here I am introducing a deep learning approach for the automatic generation of radiological reports using chest X-ray and mammogram images as input. This technique consists of three major stages: (1) tuning a pre-trained CheXNet model to label the relevant tags from images during training time, with each tag representing visual signal(s); (2) extracting weighted high-level semantic features out of these embeddings; and then set up the retrieved visual/semantic signals as conditioning signals towards a GPT-2 medical report generator. It was trained using the publicly available IU-XRay dataset (with reports for chest X-rays) and a new Categorized Digital Database for Low energy and Subtracted Contrast Enhanced Spectral Mammography images (CDD-CESM) dataset that included mammograms with full-text reports. There is a new hierarchy mammogram dataset because of the few public and open-source resources. I used this word-overlap metric to assess the reports generated as well as some new semantic similarity measures. The researchers showed that the proposed model, (CDGPT3.5) achieved very competitive quantitative results compared to several non-hierarchical recurrent and transformer-based models while being trained much faster. Significantly, it can be used with any programming language and easily modified to support different datasets without changing the architecture. This contribution is one of the prior attempts at to publicly available dataset for contrast-enhanced mammogram samples along with both semantic and visual report modalities, utilizing pre-trained transformer models to generate reports that encourage further research in multimodal medical imaging.