Towards automated nursery management: instance segmentation of 55 Bangladeshi plant species using mask R-CNN and YOLO architectures

Abstract

The world’s horticultural sector struggles to scale nursery operations because man- ual plant identification and inventory control are labour-intensive, especially under

severe class imbalance and challenging visual conditions. We present an end-to- end deep-learning pipeline that automatically recognises plant species by instance- segmenting 55 ornamental and bedding species. The system is trained on a new

in-situ dataset of 1 716 images collected from Bangladeshi nurseries. Extensive ex- ploratory data analysis revealed a heavily skewed class distribution (imbalance ratio

ρ = 38.7) and guided subsequent oversampling strategies. The proposed pipeline surpasses both YOLOv8 and YOLOv11 variants under short (15-epoch) and extended (50-epoch) training schedules, reaching a maximum mask mAP50 of 0.849 with ≤10 ms inference on edge hardware. Two-stage Mask R-CNN

models achieve superior boundary accuracy (up to 69.4 AP50) but their latency ex- ceeds acceptable limits for online use; they are therefore reserved for offline auditing.

Custom bio-inspired architectures, DNA-HelixNets and RNA-PinNets, embed rotational- invariant and attention modules, yielding 2–3 % mAP gains without extra computa- tional cost. Overall, the pipeline is expected to cut identification errors and labour

by 40 %, supporting sustainable nursery management and biodiversity tracking. Re- maining challenges include dataset localisation, reliance on static images, and the

need for future work on video segmentation and quantisation to broaden ecological applicability.