Multimodal approach to human detection in unconstrained environments using YOLOV7 for conventional, infrared & thermal cameras

Abstract

Search and rescue operations in disaster-stricken areas are often hindered by chal lenging environmental conditions, such as poor visibility, limited lighting, and high levels of noise and clutter. These conditions can make it difficult to locate and res cue survivors in a timely manner, which can have significant implications for their survival and recovery. Traditional methods of human detection, such as visual obser vation, can be ineffective in these environments, and new and innovative approaches are needed to address these challenges. This research presents a novel multimodal approach to human detection in unconstrained environments using YOLOv7 for con ventional, infrared and thermal cameras. The proposed approach aims to improve human detection performance in challenging environments, such as post-disaster situations, where traditional methods may fail. A unique dataset of 7,087 images was created for this research, including both conventional and thermal images, which were collected to capture the realistic scenario of disaster environments. The dataset was used to train various CNN models for human life detection, and the results were evaluated using standard metrics. Additionally, to further enhance the search and rescue operations in post-disaster situations, a Bangla speech recognition model was integrated into the system. The results of this research demonstrate the effectiveness of the proposed approach in detecting humans in challenging environments, such as low-light and obscured conditions. The use of thermal imaging in particular, has the potential to significantly improve human detection in disaster scenarios where visibility is limited. This research provides a valuable contribution to the field of human detection in unconstrained environments and has the potential to improve search and rescue operations in the future.