Ever since its debut last November, ChatGPT has repeatedly made the headlines and gone viral on social media for being a general-purpose tool that performs impressively on many tasks demonstrating knowledge in various domains with fluent and human-like results. Meanwhile, concerns are raised over whether this advanced AI chatbot will replace the job roles like translators. Does ChatGPT translation have a better quality than neural machine translation models or human translation? This article provides an unbiased opinion on this state-of-the-art technology and probes its impact on the language industry.
What is ChatGPT, and Why it’s So Viral?
We are just a few months removed from the jaw-dropping AI art created by the almighty DALL·E 2, a deep-learning model that generates realistic digital images from prompts (natural language descriptions). In the past few weeks, all eyes have been on its developer OpenAI again with the launch of ChatGPT.
What exactly is ChatGPT? ChatGPT is an AI-driven natural language processing bot that produces coherent and relevant detailed answers to the instructions written by users. According to OpenAI, this intelligent chatbot can answer follow-up questions, challenge incorrect premises, admit its errors, and reject improper requests. Its excellence in performing natural language processing tasks and its impressive interaction in conversations with humans have made a big splash in social media. Another reason for ChatGPT’s viral is its ability to generate detailed responses and articulate answers across various subject domains in real-time, with which people can complete tasks like composing essays, content, and code in a jiff. However, the value of its output remains a hotly debated topic since over 80% of students are using this technology for homework assignments, increasing the challenge of anti-plagiarism and anti-cheating.
Recent ChatGPT Translation Review
Dreaded by ChatGPT’s powerful ability to create, many workers relying on content creation are worried about losing their jobs in the immediate future. Translation professionals are among them. Is ChatGPT ready to take over the translation industry? Maybe not. At least not from what we’ve concluded in the latest ChatGPT translation review, a preliminary study on ChatGPT’s translation ability published by Tencent AI Lab. The study evaluates the translation prompt, multilingual translation, and translation robustness of ChatGPT by comparing its natural language processing performance with online machine translation systems, including Google Translate, DeepLTranslator, and Tencent’s TranSmart. Read on if you’re interested in the gap in the performance between ChatGPT and the commercial translation products mentioned above.
ChatGPT Translation Prompt
You may wonder what a translation prompt is. In this context, prompts are natural language processing descriptions or instructions input by the users. ChatGPT is generally a large generative language model that uses prompts as guidance to trigger its ability to translate. The better the prompt, the more accurate the translation output.
Suppose we need ChatGPT to translate a sentence in a source language (denoted “[SRC]”) into a target language (denoted “[TGT]”). We will require ChatGPT to provide multiple prompts to choose the good ones. That’s what the Tencent AI team did. The team directly asked ChatGPT to provide 10 prompts based on this prompt:
“Provide ten concise prompts or templates that can make you translate.”
As expected, it gave 10 prompts with slight differences between them. Among these prompts, Tencent AI evaluated the most representative 3 prompts on a Chinese-English translation task.
Prompt 1: Translate these sentences from [SRC] to [TGT]:
Prompt 2: Answer with no quotes. What do these sentences mean in [TGT]?
Prompt 3: Please provide the [TGT] translation for these sentences:
The results below reveal the scores obtained with three automatic metrics for evaluating machine translation quality: BLEU, chrF++, and TRE. For BLEU and chrF++, the higher the scores are, the better the translation quality is. For TER, it’s the opposite – a lower score means better quality.
System | BLEU ↑ | chrF++↑ | TER↓ |
ChatGPT w/Prompt 1 | 23.25 | 53.07 | 66.03 |
ChatGPT w/Prompt 2 | 24.54 | 53.05 | 63.79 |
ChatGPT w/Prompt 3 | 24.73 | 53.71 | 62.84 |
From what the team found in the results, Prompt 3 outperforms Prompt 2 and Prompt 1 and hence can trigger its ability to translate better. Prompt 2 also performs better than Prompt 1 despite the slight difference in their chrF++ scores. Interestingly, the prompt that mentions the source language (Prompt 1) surprisingly underperforms those that don’t (Prompt 2 & Prompt 3), indicating that ChatGPT can translate without knowing the source text’s language.
Although it is impressive, it meanwhile goes against the laws of logic. Normally, ChatGPT is expected to generate more accurate translations with the source language precisely indicated in its prompts. The reason why ChatGPT obtained lower scores when the source language is mentioned remains unsolved. Suppose it can infer the source language from the user’s instructive input, indicating the source language should not have any (negative) impact. This is why we believe that the quality of the translations by ChatGPT can be tricky to assess and thus inherently unreliable, at least at this stage.
ChatGPT Translation Accuracy - Higher or Not
Compared with predictive machine translation systems, ChatGPT is trained with the data crawled from the world wide web. It seems to produce impressively fluent results, but not necessarily the more accurate ones. The Tencent AI Lab evaluated ChatGPT against the three abovementioned commercial translation products with Prompt 3. According to the results, three online machine translation systems obtain similar scores in three different metrics, and they all seem to outperform ChatGPT. It makes sense because online machine translation models are specifically trained for the translation task by consuming millions of subject-specific text corpora in the designated language combinations and directions.
The scores of the testing also vary with different language pairs and directions. Maybe it’s because ChatGPT is trained with English data. And for languages distant from English or with a limited amount of text/sentence resources on the Web to be crawled from, the ChatGPT inevitably provides translations of a lower quality.
System | BLEU ↑ | chrF++↑ | TER↓ |
Google Translate | 31.66 | 57.09 | 56.21 |
DeepLTranslator | 31.22 | 56.74 | 57.84 |
Tencent TranSmart | 29.69 | 56.24 | 57.16 |
ChatGPT w/Prompt 3 | 24.73 | 53.71 | 62.84 |
Need Professional Human Translation?
What are the impacts of ChatGPT on the translation industry?
Is ChatGPT good for the translation industry? Yes and no. Every coin has two sides. On the one hand, when handling simple tasks, translators can use ChatGPT as a translation tool to improve their work efficiency. Individuals and small businesses with linguistic needs can reduce translation costs. On the other hand, the revenue of the translation service providers will be reduced with the advent of ChatGPT translation since more individuals and small businesses will turn to it as a cheaper alternative.
Although ChatGPT translation appears to be more affordable and accessible, it is not recommended to apply it to the translations for official/publication purposes. In the short term, ChatGPT is unlikely to beat machine translation engines in terms of accuracy simply because MTs are trained with domain-specific data and terminology, and ChatGPT is not. Not to mention the human touch such as the handling of cultural nuances and the adaptations to meet the requirements of specific style guides or industry regulations. There are a bunch of value-added services that a generic multi-purpose model can’t provide. Even data privacy is not available with ChatGPT since the training data is shared by everyone with access to the Web.
Conclusion
With its limitation in quality, ChatGPT translation has not replaced machine translation as yet. Let alone human translation. However, this generative AI model is here to stay, and it’s bound to impact the language industry by creating more needs for services to revise, adapt, and certify the AI output. MTPE services will focus on linguistic validation, tone adjustment, cultural adaptation, bias removal, and fact-checking. That’s also what we’ve excelled in.
If you need coherently reliable human or AI-powered translation services that will add value to your business, please contact Wordspath to learn more about how you can get the most out of content translation.
Wordspath can help
Wordspath provides highly reliable human translation and state-of-the-art machine translation services for content related to diverse subject matters with accuracy and consistency.
We take pride in our quality-driven workflow that combines the excellent work of our linguists, desktop publishers, project managers, customer service, and technical team. Their endless support allows Wordspath to provide first-rate language solutions in 150+ languages for thousands of customers who need to connect with the world.
Wordspath also offers machine translation post-editing services translating the content with our proprietary MT engine and having our in-house/contracted linguists review, edit, polish, and proofread the results.
Meanwhile, we are highly experienced in delivering tailor-made localization-related solutions such as desktop publishing, transcription, subtitling, and voiceover. Our ability to quickly handle a wide range of content types between nearly all language combinations sets us apart from our competitors. Should you need to consult on your best-fit language solution, you can contact us through live chat or email to info@wordspath.com. Or simply request a free quote.
