Evaluation of Instagram's neural machine translation for literary texts: an MQM-Based Analysis

Abstract

Addressing the global increase in social media users, platforms such as Instagram introduced automatic translation to broaden information dissemination and improve cross-cultural communication. Yet, the accuracy of these platforms' machine translation systems is still a concern. Therefore, this paper aims to explore the potential of Neural Machine Translation utilized by Instagram in producing high-quality translations. In doing so, this study attempts to scrutinize the reliability of Instagram's "See Translation" feature in the translation of literary texts from Arabic to English. A selection of auto-translated Instagram captions is analyzed through the identification, classification, and assignment of error types and penalty points, utilizing the MQM core typology. Subsequently, the Overall Quality Score of the error-based analysis is calculated automatically using the ContentQuo platform. Furthermore, the study investigates whether Instagram Neural Machine Translation can effectively convey the intended message within literary texts. From 30 purposively selected Instagram captions with literary content, the study found Instagram's machine translation lacking in 90% of cases, particularly in accuracy, fluency, and style. Among these, 61 errors were identified: 26 in fluency, 25 in accuracy, and 10 in style, adversely affecting the quality and failing to convey the original message. The findings suggest a need for enhanced algorithms and linguistic architecture in Neural Machine Translation systems to better recognize linguistic variants and text genres for more accurate and fluent translations.