Tri Lathif Mardi Suryanto, Aji Prasetya Wibawa, Hariyono, Andrew Nafalski
Although Transformer-based technologies have pioneered the advancement of Natural Language Processing (NLP), the evaluation of such technologies in specific niche fields is notably lacking. The purpose of this research is to evaluate the BERT-base, DistilBERT, RoBERTa-base, RoBerta-large, and ALBERT-BASE V2 models in relation to the peculiarities of the language and the context of the cultural and historical texts. F1, recall, precision, BERTScore, and F1, BERTScore metrics proved the highest results to be BERT-base (F1 0.915, BERTScore 0.897), confirming statistical significance with Bootstrap Resampling and McNemar's Test. In the relation of domain specific interpretative weaknesses, it was found that RoBERT did the best, with sarcasm proving to be problematic for BERT and long narratives for DistilBERT. There is little previous research on benchmarking Transformers for cultural heritage, and this research sheds light on the area, providing a balance between the accuracy and the needed computational resources. This research aims to deepen the understanding and to implement the storing and the explanation of the historical and cultural knowledge. © 2025 IEEE.
Universitas Pembangunan Nasional Veteran Jawa Timur, Department of Information System, Surabaya, Indonesia; Universitas Negeri Malang, Department Electrical and Informatics Engineering, Malang, Indonesia; Universitas Negeri Malang, Department History of Science, Malang, Indonesia; University of South Australia, Deparment Electrical Engineering, Adelaide, Australia
