A framework for disease identification from unstructured data using text classification and disease knowledge base

Abstract

With the increasing number of internet user, online searching for health advice has gone through a rapid popularization. In today's world, people tend to search online for health-related advice before consulting a doctor whenever they face health problems initially instead of consulting with a health professional. With the rapid proliferation of online symptom checker sites and health forums, it has become handy to acquire information regarding health condition supported by a number of symptoms. Though these existing symptom checkers afford an instant sense of disease diagnosis, these question-answering and selection based systems lack in interactivity. Online health forum sites can also be disappointing because of their time demanding nature and reliability issues. In this work, we propose a webbased automated disease identification framework which will take unstructured textual data like health forum posts as input and provide a ranking of probable diseases based on symptom-disease correlation considering all important factors. A lexicographic and semantic feature-based two-phase text classification system and a disease knowledge base-based similarity measurement module to identify probable disease have been incorporated in the proposed framework. We have evaluated the framework by varying the number of feature components and got the result that, significant accuracy and reliability is obtained over baseline systems by effective feature engineering at the same time of keeping up with increased user interactivity.