PAUSE: Post-hoc sequential instance unlearning in panoptic segmentation via asymmetric conflict aware gradient projection

Abstract

Panoptic segmentation models face increasing pressure to comply with privacy regulations and data-subject withdrawals, necessitating the post-hoc removal of specific training data. While machine unlearning bypasses the prohibitive cost of full retraining, it remains severely underexplored in dense prediction tasks. This challenge is particularly acute in transformer-based decoders, which couple individual object instances through shared query representations; consequently, suppressing a single target risks catastrophic degradation of its entire semantic class. This bottleneck intensifies under sequential deletion, where continuous updates often accumulate collateral damage or restore previously forgotten targets. To address this, we propose PAUSE (Post-Hoc Asymmetric Unlearning for Sequential Erasure), a post-hoc teacher-student framework tailored for query-structured architectures like Mask2Former. By integrating localized instance erasure with asymmetric gradient projection, damage-aware dynamic weighting, and a priority-ordered replay memory, PAUSE safely untangles the unlearning objective from shared features. Extensive evaluations on the Cityscapes dataset against NegGrad, NegGrad+, and SCRUB baselines demonstrate that PAUSE successfully processes sequential deletion queues, cleanly erasing targeted instances while bounding global utility degradation to a marginal ΔPQ of −0.4 percentage points. Finally, we formalize the fundamental tension between class-level retention and instance-level erasure, establishing a critical trade-off baseline for future dense-prediction unlearning systems.