Mohammad Bhanu Setyawan, Didik Dwi Prasetya, Triyanna Widyaningtyas, Busro Akramul Umam, Sahlan M. Saleh
Accurate assessment of student engagement plays a crucial role in optimizing teaching methods and improving learning performance in an online environment. This study proposes and evaluates a multimodal deep learning framework to assess the level of student engagement by conducting a comparative study between unimodal approaches and multimodal fusion. Specifically, we analyzed data from two main modalities: visual (video) data to capture emotional engagement, and log data (quiz scores) to reflect behavioral and cognitive engagement. We implemented four deep learning architectures for the classification of time-series data - Encoder, FCN, Time-CNN, and MCNN - on three scenarios: video-only, quiz-only, and multimodal fusion (video+quiz). The results of the evaluation showed that the multimodal approach consistently delivered the best performance across the model. The MCNN model achieved the highest accuracy of 72.68% in multimodal configurations, showing a significant improvement over the unimodal approach. These findings confirm that the synergy between emotional data and objective performance data results in more comprehensive and reliable predictions of engagement and highlights the great potential of multimodal systems to create more personalized and effective learning interventions. © 2025 IEEE.
Universitas Negeri, Department of Electrical Engineering and Informatics, Malang, Indonesia
