An efficient text preprocessing and classification technique for multilingual and transliterated data

Abstract

Individuals express their opinions on a daily basis through various online platforms in the form of text, comments, and reviews, which is becoming a popular arena for text classification researchers. For a better cause, we aimed to ease the researchers' workload by uncovering the true meaning of their perspective using a text preprocessing method and a text classification algorithm. Our goal was to discover a connection that could work well on multilingual and cross-language data. In this paper, we worked with five different datasets from different languages - English, Bangla, and Banglish (transliterated dataset) - on which we ran our experiment and discovered that byte-mLSTM outperforms with XGB classification model. We achieved more than 90% accuracy on English datasets, 93.43% on Bangla datasets, and 78% on transliterated datasets using mLSTM preprocessing, outperforming the existing performance of traditional approaches on English datasets. For preprocessing, mLSTM performs well across all languages.