An exploratory case study on translating performance: ChatGPT vs. NMT vs. Human Translators

Abstract

This exploratory case study compares the translation output of a Japanese-language academic paper to English from four sources: the original author, Large Language Model (LLM, namely ChatGPT), Neural Machine Translation (NMT) and a professional, focusing on the differences in the output of ChatGPT and NMT. The recent development of these systems as a means of generating human-like translations has received increased scholarly attention comparing their strengths and weaknesses. However, there is a lack of research with respect to their performance with the Japanese-English language pair in the context of academic research publications. Two researchers in the field of Applied Linguistics were provided four samples to quantitatively rate in the categories of Accuracy, Fluency, Terminology and Style, and gave justifications for each score. The results showed that the author’s own translation rated highest, followed by NMT, then ChatGPT and, finally, the professional translation. Raters felt the author’s translation captured the intent behind the original text whilst the latter samples took a direct approach that more strictly adhered to the structure and literal meaning of the original text, which hurt their style and terminology but gave them relatively high accuracy scores. Moreover, raters considered the ChatGPT and NMT samples ill-fitting for the intended context: an academic publication. Regarding ChatGPT and NMT, specifically, both raters considered ChatGPT to provide a stricter, word-by-word translation of the text than that generated by NMT. However, given the influence of prompt design in ChatGPT’s output, it may be possible to provide a more successful translation with a revised input.