Supriyono, Aji Prasetya Wibawa, Suyono, Fachrul Kurniawan
Transcribing Indonesian stand-up comedy poses significant challenges due to its dynamic nature, including rapid speech, colloquialisms, regional dialects, overlapping dialogue, and non-verbal humor cues such as gestures and facial expressions. Traditional unimodal transcription methods often struggle to capture the full contextual richness of such content. This study introduces a novel approach by employing multimodal deep learning, combining both auditory and visual data to improve transcription accuracy. The model processes audio streams using LSTM and Wav2Vec2 to extract linguistic features and visual streams through CNN s to analyze non-verbal indicators like facial expressions and body language. The integration of these modalities at the feature fusion stage enables the model to better understand comedic timing, cultural nuances, and humor-related context, yielding a more comprehensive transcription. The model's performance was evaluated using standard metrics Word Error Rate (WER), Character Error Rate (CER), F1 Score, and BLEU Score and demonstrated substantial improvements compared to audio-only and visual-only models. The multimodal approach reduced WER by 37.45%, CER by 41.67%, and improved the F1 Score by 13.58%. These results highlight the model's ability to effectively handle the complexities of stand-up comedy, capturing both verbal and non-verbal elements essential to its humor. This research underscores the potential of multimodal deep learning in addressing transcription challenges in content with intricate linguistic and contextual features, offering a promising solution for more accurate and contextually aware transcription in multimedia applications. © 2025 IEEE.
Faculty of Engineering, Universitas Negeri Malang, Department of Electrical Engineering and Informatics, Malang, 65145, Indonesia; Informatics Engineering, Faculty of Science and Technology, Universitas Islam Negeri Maulana Malik Ibrahim, Malang, 65144, Indonesia; Faculty of Letters, Universitas Negeri Malang, Department of Indonesian Literature, Malang, 65145, Indonesia
