Herdianti Darwis, Adam Adnan, Purnawansyah, Abdul Rachman Manga, Aji Prasetya Wibawa, Irawati
The proliferation of online learning platforms has necessitated automated systems for monitoring students' emotional states, given that variations in facial expressions significantly influence engagement and learning outcomes. This study proposes a spatio-temporal classification framework for recognizing emotion intensity levels in the DAiSEE dataset, utilizing Vision Transformer as a spatial feature extractor alongside various temporal models, including LSTM, BiLSTM, TimeSformer, and their hybrid variants. Embeddings are extracted via ViT, after which temporal dependencies are captured by each classifier, incorporating feature-level oversampling to mitigate severe class imbalance. Experimental findings reveal that, despite ViT's ability to generate robust spatial representations, all temporal models struggle to identify minority classes, resulting in predictions biased toward the majority class as evidenced by low balanced accuracy scores and overlapping clusters in t-SNE visualizations. Among all configurations, the ViT + LSTM model delivered the most reliable performance, attaining 59 % accuracy and a 0.59 weighted F1-score on engagement labels, while remaining competitive with prior methods. In essence, integrating spatial and temporal features enhances classification efficacy, yet its effectiveness is substantially constrained by imbalanced data distributions. These results offer a thorough examination of representational challenges in imbalanced affective datasets, along with recommendations for mitigation techniques, crossdataset assessments, and multimodal integrations. © 2026 IEEE.
Universitas Muslim Indonesia, Faculty of Computer Science, Makassar, Indonesia; Universitas Negeri Malang, Department of Electrical and Informatics Engineering, Malang, Indonesia
