Hierarchical transformer-based semantic segmentation of intraoperative anatomical structures

Abstract

We propose a novel hierarchical Transformer-based model that significantly advances the accuracy of semantic segmentation for intraoperative abdominal organs, outperforming existing methods in both precision and generalizability. Despite the growing prominence of surgical data science, current semantic segmentation models for intraoperative abdominal organ segmentation remain severely limited, both in terms of the number and capabilities. Only one prior approach exists, which fails to generalise beyond dominant anatomical structures due to large-organ bias and poor spatial contextualization. Addressing these critical gaps, this study introduces a hierarchical Transformer-based architecture explicitly tailored for the semantic segmentation of intraoperative imagery. Anchored by a DINOv2 backbone and a custom multi-scale cross-attention decoder, our model captures both boundary-level granularity and global anatomical consistency. The architecture is further supported by a hybrid loss function that combines class-weighted cross-entropy with Dice loss to enhance performance on structurally complex, low-pixel regions. The research objective focuses on enabling high-precision segmentation for surgical environments, overcoming occlusions, spatial overlap, and class imbalance that are endemic to existing datasets. On the Dresden Surgical Anatomy Dataset, our novel architecture achieves a Dice score of 0.9098 and mIoU of 0.8654, outperforming existing benchmarks by up to 28% in challenging classes such as the pancreas and colon. By addressing both the architectural and data-centric limitations of existing literature, this work establishes a new frontier in surgical precision.