Irwan Darmawan, Hakkun Elmunsyah, Didik Dwi Prasetya
This study introduces Al-Qur’an BERT for Indonesian Religious Texts (ALBERTIR), a domain-adaptive Bidirectional Encoder Representations from Transformers (BERT) model pretrained on Indonesian religious texts, including official Qur’an and Hadith translations. The corpus comprises over 1.2 million tokens sourced from verified government publications and optimized for Masked Language Modeling (MLM). ALBERTIR features weighted MLM, sacred term preservation, and factorized embeddings to enhance understanding of religious semantics and maintain doctrinal integrity. Training was conducted on Google Colab Pro with TPU v3-8, where ALBERTIR outperformed BERT-base and A Lite BERT for Indonesian (ALBERT-ID), improving religious term prediction by 10.9% and reducing training time by more than 40%. Across downstream tasks such as religious question answering, sentiment analysis, and text classification, it achieved up to 8% higher F1-scores. Ablation studies confirmed the effectiveness of its core components, demonstrating advantages in semantic accuracy, contextual sensitivity, and reliability in religious Natural Language Processing (NLP) applications. Unlike general-purpose models like Indonesian BERT (IndoBERT) and multilingual BERT (mBERT), the proposed model is specifically optimized for theological language, thereby reducing vague or contextually inappropriate outputs. This makes it especially suitable for applications such as fatwa retrieval, Islamic education tools, and religious chatbot systems. Cross-lingual evaluations further showed that ALBERTIR surpasses mBERT by +13.3 Bilingual Evaluation Understudy (BLEU)-4 points in religious Questioning-Answering (QA) tasks, while maintaining competitive performance in general benchmarks. Ablation results identified sacred term preservation as the most critical contributor to accuracy gains, underscoring the importance of domain specific features. Overall, ALBERTIR demonstrates strong capabilities in capturing linguistic precision and theological nuance, establishing a robust foundation for future religious NLP research and applications. © (2025), (Engineering). All rights reserved.
Department of Electrical Engineering and Informatics, Universitas Negeri Malang, Indonesia
