Automated hazard detection for AR/VR Mars terrain navigation using computer vision

Abstract

The exploration of Mars brings about a series of challenges that are occasioned by the risky topographical features, random weather patterns, and the fundamental need to have self-driving equipment. The study builds a combined computer vision and immersive technology system to improve the safety of the human astronauts and robot rovers in their navigation on the surfaces of the Martian environment. Our solution is a multi-modal deep-learning system consisting of object detection, semantic segmentation, and monocular depth estimation to generate complete hazard awareness in simulated Mars environments. We use datasets to train terrain classification models that are able to detect important surface features such as rocks, boulders, and potholes as well as other geological features. The system combines a number of deep-learning networks to detect hazards in real-time and locate bounding-boxes, semantic-segmentation, and pixel-level terrain-classification as well as a depth-estimation architecture to give the system spatial information of the Martian terrain. These models are synergistically used to produce an environmental cognition that drives into an AR/VR interface that provides users with visual cues in safe path planning. The AR/VR element converts raw computer-vision data into usable navigation data, and deciphers warnings of hazards and terrain complexity data to the Martian landscape. The initial studies have shown strong detection of varied terrain conditions, and the multi-modal strategy has a great benefit on improving the safety of navigation in comparison to the single-modality systems. The study has been applied to the development of autonomous planetary exploration technologies and created a scalable model of pre-mission astronaut training and rover operation plan.