Complex and Composite human activity recognition utilizing wearable sensor data focusing on low-power low-cost low-compute IoT devices

Abstract

Human activity recognition is a foundational technology for applications like remote health monitoring, though accurately classifying complex and composite activities often necessitates cloud-based processing. This conventional approach, while powerful, introduces significant latency, power consumption, and privacy concerns, driving a trend toward on-device solutions. However, current on-device systems frequently overlook highly informative sensor modalities. Plantar pressure sensors, in particular, offer a distinct advantage by capturing rich kinetic data related to gait and balance unlike the kinematic data from standard inertial sensors. However, their application within knowledge distillation frameworks remains a significant, unexplored research gap. Hence, this study presents a cost-effective, low-computation system for complex and composite human activity recognition that leverages knowledgedistilled neural networks on a microcontroller unit to minimize reliance on cloud processing. A key contribution of this work is the investigation of plantar pressure sensor data within a knowledge distillation framework, addressing a notable gap in the existing literature. The proposed solution centers around the ESP32-S3 DevKit C1, equipped with a dual-core 240 MHz Tensilica chip, 320 KiB of usable static random access memory, and built-in Wi-Fi and Bluetooth. Significantly, both the teacher and the student models surpass existing state-of-the-art methods, achieving F1 Scores of 99.33%, 98.36%, and 97.68% respectively, in classifying a comprehensive set of 21 activities (15 composite and 6 simple) in CAPPIMU dataset. The distilled student models demonstrate remarkable efficiency, with execution times of 1.83 and 0.64 seconds, memory footprints of only 62 KB and 82 KB, and flash memory usage of approximately 209 KB and 127 KB, while maintaining low power consumption of 210 mW and 215 mW, respectively. Furthermore, validation on external public datasets confirms the generalizability of the proposed models. They not only match the performance of state-of-the-art methods but also exhibit superior computational efficiency, achieving an outstanding balance between accuracy and resource consumption. Lastly, we have developed an end-to-end prototype that integrates the ESP32-S3 with a WitMotion inertial measurement unit sensor. This system autonomously manages data acquisition, feature extraction, and inference in under 7 seconds with a total power consumption of approximately 295 mW. By enabling on-device data processing and inference, our approach significantly reduces dependency on cloud resources, making it highly suitable for power and computationally constrained applications such as remote health monitoring, fitness tracking, and smart home automation.