Neuro-symbolic image manipulation through natural language commands

Abstract

The task of image manipulation via natural prompts is very useful across multiple AI domains. Our research proposes a novel neuro-symbolic approach to address the challenge of detailed and context-aware image manipulations. Even though traditional neural network models are proficient in handling raw image data, they usually lack interpretability and reasoning abilities. We propose a model combining symbolic concepts with neural networks, drawing inspiration from existing neuro-symbolic models NeuroSIM [1] and SIM-SG [2]. While most of these models work with relatively simpler datasets, we wish to train our model to handle more complex real world data. Our methodology involves training our model with image data along with natural language prompts. The system interprets and manipulates the images based on user descriptions and manipulation commands. Using Scene Graph generation and manipulation techniques, our proposed model aims to outperform existing models in terms of complexity and interpretability. This approach has many potential applications in improving content generation. By advancing the capabilities of neurosymbolic AI in the context of image manipulation, we wish to set a new benchmark for accuracy and clarity in the field.