Rethinking Round-Trip Translation for Machine Translation Evaluation
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.
Tom: We're looking at the paper: "Rethinking Round-Trip Translation for Machine Translation Evaluation".
Jane: Here's the gist of the paper. Automatic evaluation on low-resource language translation suffers from a deficiency of parallel corpora Round-trip translation could be served as a clever and straightforward technique to alleviate the requirement of the parallel evaluation corpus However, there was an observation of obscure correlations between the evaluation scores by forward and roundtrip translations in the era of statistical machine translation (SMT).
Tom: In this paper, we report the surprising finding that round-trip translation can be used for automatic evaluation without the references.
Jane: Pruning redundant reasoning tokens The authors revisit the long-standing misunderstanding regarding round-trip translation in SMT evaluation, which is essentially caused by copying mechanism After removing copying mechanism in SMT, round-trip translation scores can appropriately reflect the forward translation performance.
Tom: The rectification is deemed overdue because round-trip translation could benefit multiple machine translation evaluation tasks.
Title and authors: Tom: We're starting with the title and authors of this paper, "Rethinking Round-Trip Translation for Machine Translation Evaluation," and who did the work.
Jane: They were Terry Yue Zhuo, Qiongkai Xu, Xuanli He, and Trevor Cohn from Monash University, Melbourne University, and UCL.
Lu: I'm interested in how they approach the core problem described in the title—rethinking round-trip translation for evaluation itself.
Meng: I just wonder if this is just a clever trick to avoid needing parallel data without actually solving the underlying quality measurement issue.
Tom: The paper sets out to show that round-trip translation can be used for automatic evaluation even without those parallel references, which is the main goal here.
Jane: So, what's the main takeaway from this revisit of round-trip translation in SMT evaluation?
The paper's summary: Tom: Basically, they look at how forward translation and backward translation work together to form a round-trip process on monolingual data.
Jane: They explain that the forward translation takes a sentence from language A to language B, and then the output from that first step gets translated back to language A using a backward system.
Lu: The core of their argument is that this round-trip translation process can be used to define new evaluation scores, which they call RT-SCORE.
Meng: They contrast this with the traditional forward translation score you get when you have a parallel corpus, which is what we're used to seeing in most SMT evaluations.
Tom: The paper shows that these two types of scores, the FT-SCORE and the RT-SCORE, can actually be strongly correlated across different machine translation systems.
Jane: That's a big deal because it means we might be able to predict the forward translation score just by looking at the round-trip translation score.
The paper's improvements: Lu: One major improvement they point out is that by removing the copying mechanism, which was an issue in SMT evaluations, the round-trip translation scores can correctly reflect how good the forward translation actually is.
Tom: That addresses a long-standing misunderstanding where copying tokens sometimes got penalized in round-trip but rewarded in forward evaluation.
Jane: So what are the practical applications they suggest for using these new RT-SCOREs?
Meng: They see three main things: first, you can use these scores to predict the forward translation scores by training a simple linear regression model.
Tom: That means we don't need a parallel corpus to get an idea of how good a system is, we just need some training data for that predictor.
Lu: Second, they introduce something called an X-Check mechanism for real-world shared tasks, which can help identify adversarial competitors in the field.
Jane: And the third improvement is that these round-trip translation features can actually improve the performance of recently advanced quality estimation models.
Conclusion: Tom: So to wrap up, this paper "Rethinking Round-Trip Translation for Machine Translation Evaluation" shows that round-trip translation offers a way to evaluate machine translation without relying on parallel corpora.
Jane: They demonstrated strong correlations between forward and round-trip scores across various systems, and they gave us tools like a regression predictor and an X-Check mechanism to use those scores.
Lu: It opens up possibilities for evaluating transferred machine translation systems and even low-resource languages, because you don't need those huge parallel datasets anymore.
Meng: From my side, the practical implication is that these features can make our quality estimation models much more robust when we deploy them on actual traffic without running into unexpected issues.
Lalam: I'm quite confident in this direction because the regression predictors they build perform well even on transferred systems and unseen language pairs, showing a Pearson correlation above zero point eight eight on MBART50-M2M.
Tom: So they suggest that RT-SCORE BA is generally more correlated to the forward score than RT-SCORE AB, which makes sense because the last step in RT-SCORE BA is the forward translation itself.
Jane: It’s a solid foundation for moving toward reference-free evaluation methods, and we should definitely keep an eye on how these features affect other quality estimation models.
Terry Yue Zhuo, Qiongkai Xu, Xuanli He, Trevor Cohn
Monash University · The University of Melbourne · University College London
cs.CL
Submitted: 2022-09-15
Updated: 2026-11-13
Comments: Accepted to Findings of ACL 2023
Code: https://github.com/UKPLab/EasyNMT.8https:
License: http://creativecommons.org/licenses/by-nc-nd/4.0/
Importance score: 88/100
The gist: Automatic evaluation on low-resource language translation suffers from a deficiency of parallel corpora Round-trip translation could be served as a clever and straightforward technique to alleviate…
Key concepts
- Round-Trip Translation (RTT)
- This involves translating a sentence from language A to B (Forward Translation, FT), and then translating that output back from B to A using a backward translation system. The resulting RTT score measures the consistency between these two steps, contrasting with traditional parallel corpus scores.
- FT-SCORE vs. RT-SCORE
- FT-SCORE measures how well a translation goes from source language A to target language B directly. RT-SCORE measures the performance of the round trip (A->B->A). The paper shows these two types of scores are strongly correlated, suggesting RTT can substitute for parallel corpora in evaluation.
- X-Check Mechanism
- This is a cross-system verification technique used during RTT evaluation on shared tasks. It compares the outputs of multiple translation systems to identify adversarial competitors—systems that rely heavily on copying strategies—thereby making the evaluation more robust.
Terminology
Summary
Automatic evaluation on low-resource language translation suffers from a deficiency of parallel corpora Round-trip translation could be served as a clever and straightforward technique to alleviate the requirement of the parallel evaluation corpus However, there was an observation of obscure correlations between the evaluation scores by forward and roundtrip translations in the era of statistical machine translation (SMT). p. 0 Round-trip translation could
be served as a clever and straightforward technique to alleviate the requirement of the parallel evaluation corpus. However, there was an
observation of obscure correlations between
the evaluation scores by forward and roundtrip translations in the era of statistical machine translation (SMT). However, there was an
observation of obscure correlations between
the evaluation scores by forward and roundtrip translations in the era of statistical machine translation (SMT). In this paper, we report the surprising finding that round-trip translation can be used for automatic evaluation
without the references.Where the paper says it
Where the paper says it
Pruning redundant reasoning tokens
The authors revisit the long-standing misunderstanding regarding round-trip translation in SMT evaluation, which is essentially caused by copying mechanism After removing copying mechanism in SMT, round-trip translation scores can appropriately reflect the forward translation performance. The rectification is deemed overdue because round-trip translation could benefit multiple machine translation evaluation tasks. p. 0 After removing copying mechanism in
SMT, round-trip translation scores can appropriately reflect the forward translation performance. Then, we demonstrate the rectification
is overdue as round-trip translation could benefit multiple machine translation evaluation
tasks. Then, we demonstrate the rectification
is overdue as round-trip translation could benefit multiple machine translation evaluation
tasks. To be more specific, round-trip translation could be used i) to predict corresponding forward translation scores; ii) to improve
the performance of the recently advanced quality estimation model; and iii) to identify adversarial competitors in shared tasks via crosssystem verification.Where the paper says it
Where the paper says it
How the training signal is built
The round-trip translation (RTT) entails two components, one forward translation (FT), and the other backward translation (BT). FT translates a given sentence in source language A to a sentence in target language B, then the output sentence from FT is translated back to language A via a back translation system. p. 1 As illustrated in Figure 1, RTT entails two
components, one forward translation (FT), and the
other backward translation (BT). FT translates a
given sentence in source language A to a sentence
in target language B, then the output sentence from
FT is translated back to language A via a back
translation system. FT and BT constitute
a round-trip translation (RTT). The evaluation scores on round-trip translation
(RT-SCORE) with regard to an automatic evaluation metric M is
RT-SCOREM
AB =
1
N
X
N
i=1
M(TB→A(TA→B(ai)), ai)
(1)
where BLEU, spBLEU, chrF and BERTScore
are target metrics M in our discussion. The evaluation scores on round-trip translation
(RT-SCORE) with regard to an automatic evaluation metric M is
RT-SCOREM
AB =
1
N
X
N
i=1
M(TB→A(TA→B(ai)), ai)
(1)
where BLEU, spBLEU, chrF and BERTScore
are target metrics M in our discussion. On the other hand, traditional MT evaluation on
parallel corpus is
FT-SCOREM
A→B =
1
N
X
N
i=1
M(TA→B(ai), bi) (2)
given a (virtual) parallel corpus DA||B =
{(ai
, bi)}
N
i=1.Where the paper says it
Where the paper says it
Where the paper says it
Where the gains come from
The authors demonstrate that there are strong correlations between FT-SCOREs and RT-SCOREs on various MT systems, including NMT and SMT without the copying mechanism. Three application scenarios for RT-SCORE are investigated:
-
RT-SCOREs can be used to predict FT-SCOREs via training a simple but effective linear regression model on several hundred language pairs. p. 1
Where the paper says it
Three application scenarios in MT evaluation have been investigated to show the effectiveness of RT-SCORE. Firstly, RT-SCOREs can be used to predict FT-SCOREs via training a simple but effective linear regression model on several hundred language pairs.
-
A cross-system check (X-Check) mechanism is introduced to RTT evaluation for real-world MT shared tasks, which manages to identify those adversarial competitors. p. 1
Where the paper says it
Then, a cross-system check (X-Check) mechanism is introduced to RTT evaluation for real-world MT shared tasks. By leveraging the estimation from multiple translation systems, X-Check manages to identify those adversarial competitors, which relies heavily on the copy strategy.
-
RT-SCOREs are proved effective in improving the performance of a recently advanced quality estimation model. p. 1
Where the paper says it
By leveraging the estimation from multiple translation systems, X-Check manages to identify those adversarial competitors, which relies heavily on the copy strategy. Finally, RT-SCOREs are proved effective in improving the performance of a recently advanced quality estimation model.
The negative observations between RTT and FT scores in SMT were attributed to reversible transformation rules like word copy, where copying unrecognized tokens forward to target languages is sometimes penalized by FT evaluation while it is usually awarded by RTT evaluation. Extensive experiments show that removing the copying mechanism can almost perfectly resolve the negation in their experiments. p. 2 Then, we conduct
experiments to replicate the negative conclusion. Interestingly, removing the copying mechanism
can almost perfectly resolve the negation in our
experiments. Exactly matched words in outputs from
the input words are considered copying, although
the system may not intrinsically intend to copy
them. In Table 2, we observe that copying frequency is about two times in SMT than in NMT.Where the paper says it
Where the paper says it
The authors validate whether FT-SCOREs could be predicted by RT-SCOREs by constructing a linear regressor f to predict FT-SCORE M A->B using corresponding RT-SCORE features. The prediction model is defined as f M(S) = W1 · SM AB + W2 · SM BA + β. p. 1 Firstly, RT-SCOREs
can be used to predict FT-SCOREs via training
a simple but effective linear regression model on
several hundred language pairs. The prediction performance is robust in evaluating transferred MT
systems and unseen language pairs including lowresource languages. Firstly, RT-SCOREs
can be used to predict FT-SCOREs via training
a simple but effective linear regression model on
several hundred language pairs. The prediction performance is robust in evaluating transferred MT
systems and unseen language pairs including lowresource languages.Where the paper says it
Where the paper says it
The prediction performance of the regressor is evaluated across various challenging settings, including transferred MT systems and unseen language pairs. The results show that the rankings of predicted scores are rational, with all Pearson’s r above or equal to 0. p. 5 Meanwhile,
the correlation between prediction and ground truth
is strong, with all Pearson’s r above or equal to
0. This indicates that the rankings of predicted
scores are rational. liance of copying mechanism in NMT and SMT. For NMT, we choose MBART50-M2M (Tang et al. Meanwhile, our model obtains adequate language
transferability results, as demonstrated in Table 4. In Table 5, we provide detailed performance
of our regressor on language pairs of different
resources categories on FLORES-AE33, with
RT-SCOREs of MBART50-M2M and M2M-100-
BASE respectively.Where the paper says it
Where the paper says it
Where the paper says it
The X-Check mechanism successfully identifies adversarial MT systems among benign competitors, and this management of adversaries improves correlation scores significantly. The selection of benign systems for X-Check includes OPUS-MT, M2M-100-LARGE and MBART50-M2M. p. 4 2 Neural Machine Translation Systems
We experiment with five MT systems which
support most of the languages appearing in
FLORES-AE33 and WMT. Except MBART50-
M2M, we adopt M2M-100-BASE and M2M-100-
LARGE (Fan et al. X-Check evidently successfully identifies these adversarial systems in all
our experiments and manages to improve the correlation scores significantly. With the empirical
study, we find that X-Check is able to make RTT
evaluation more robust.Where the paper says it
Where the paper says it
The features acquired by round-trip translation benefit quality estimation (QE) models, and a combination of COMET-QE-DA scores and RT-SCOREs is investigated to acquire a more competitive QE scorer. Both Kendall’s τ and Pearson’s r provide consistent results in Table 7. p. 6 Discussion. Both Kendall’s τ and Pearson’s r
provide consistent results in Table 7. We also observe that RT-SCOREs can
further boost the performance of COMET-QE-DA. We believe RT-SCOREs advances QE research and
urge more investigation in this direction.Where the paper says it
Where the paper says it
The X-Check mechanism can be used to estimate the translation quality of an FT system on a BT system by calculating Si,j = fM(Bj (Fi(x)), x). The overall system ranking could be severely affected by adversarial systems, according to Pearson’s r and Kendall’s τ. p. 7 Discussion. From Table 8, we observe that the
overall system ranking could be severely affected
by the adversarial systems, according to Pearson’s
r and Kendall’s τ . The adversarial systems are
stealthy among normal competitors, according to
Hit@K and Avg. X-Check evidently successfully identifies these adversarial systems in all
our experiments and manages to improve the correlation scores significantly. The adversarial systems are
stealthy among normal competitors, according to
Hit@K and Avg. X-Check evidently successfully identifies these adversarial systems in all
our experiments and manages to improve the correlation scores significantly.Where the paper says it
Where the paper says it
Where the paper says it
The authors demonstrate that the predicted automatic evaluation scores could be further improved via X-Check when adversaries are included. The overall system ranking could be severely affected by the adversarial systems, according to Pearson’s r and Kendall’s τ. p. 7 Discussion. From Table 8, we observe that the
overall system ranking could be severely affected
by the adversarial systems, according to Pearson’s
r and Kendall’s τ . From Table 8, we observe that the
overall system ranking could be severely affected
by the adversarial systems, according to Pearson’s
r and Kendall’s τ . The adversarial systems are
stealthy among normal competitors, according to
Hit@K and Avg.Where the paper says it
Where the paper says it
The overall correlation scores are reported in Table 10, indicating at least moderately positive correlations between all pairs of RT-SCOREs and FT-SCOREs. p. 6 In Table 6, we display the results
on WMT2020-News. Although MAE and RMSE
vary among experiments for different language
pairs, the overall correlation scores are favorable. Limitations
Although we have observed positive correlation
between FT-SCOREs and RT-SCOREs and conducted experiments to predict FT-SCOREs using
RT-SCOREs, their relations could be complicated
and non-linear. We encourage future research to
investigate various RT-SCORE features and more
complex machine learning models for better prediction models. In this work, we revisit the dispute on the
usefulness of RT-SCORE in the era of SMT versus NMT. The main reason is due to the fact that
SMT (and RMT) usually incorporate implicitly reversible rules in their translation. In this work, we revisit the dispute on the
usefulness of RT-SCORE in the era of SMT versus NMT. The main reason is due to the fact that
SMT (and RMT) usually incorporate implicitly reversible rules in their translation.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
The results of their predictors on ranking WMT2020-News synthetic data demonstrate that overall ranking errors on 17 MT systems are small on all three selected language pairs. The overall correlation scores are favorable, with Pearson’s r values on all language pairs above 0. p. 6 Although MAE and RMSE
vary among experiments for different language
pairs, the overall correlation scores are favorable. Pearson’s r values on all language pairs are above
0. Pearson’s r values on all language pairs are above
0. 5, showing strong ranking correlations. 5, showing strong ranking correlations. While
prediction performances on A B have some
variances among different language pairs, the results of the experiments using B A are competitive to those using both A B and B A
features, showing the feasibility of predicting FTSCORE using monolingual data.Where the paper says it
Where the paper says it
Where the paper says it
The results on WMT2020-Bio show that overall performance is positive, while it is relatively worse than the results of WMT2020-News reported in Table 6. This difference is attributed to the fact that the M used on WMT2020-Bio are calculated on document, while their regression models rely on sentence-level translation metrics in training.
The authors conclude that their regression-based predictors can be practical in ranking MT systems in WMT-style shared tasks. The overall correlation scores are reported in Table 10, indicating at least moderately positive correlations between all pairs of RT-SCOREs and FT-SCOREs. Moreover, they observe that RT-SCORE BA is generally more correlated to FT-SCORE than RT-SCORE AB. p. 1 While these automatic evaluation metrics are widely applied in MT evaluation, they are
generally not applicable to low-resource language
translation or new translation domains (Mathur
et al. Our work demonstrates that referencefree MT metrics (RT-SCORE) could be used to estimate traditional reference-based automatic metrics. While these automatic evaluation metrics are widely applied in MT evaluation, they are
generally not applicable to low-resource language
translation or new translation domains (Mathur
et al. Our work demonstrates that referencefree MT metrics (RT-SCORE) could be used to estimate traditional reference-based automatic metrics. relevant MT evaluation tasks, such as predicting
FT metrics using RTT scores, enhancing state-ofthe-art QE systems, and filtering out unreliable
MT competitors for WMT shared tasks. We believe
our work will inspire research on reference-free
evaluation on low-resource machine translation. relevant MT evaluation tasks, such as predicting
FT metrics using RTT scores, enhancing state-ofthe-art QE systems, and filtering out unreliable
MT competitors for WMT shared tasks. We believe
our work will inspire research on reference-free
evaluation on low-resource machine translation. relevant MT evaluation tasks, such as predicting
FT metrics using RTT scores, enhancing state-ofthe-art QE systems, and filtering out unreliable
MT competitors for WMT shared tasks. We believe
our work will inspire research on reference-free
evaluation on low-resource machine translation.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Although they have observed positive correlation between FT-SCOREs and RT-SCOREs and conducted experiments to predict FT-SCOREs using RT-SCOREs, their relations could be complicated and non-linear. They encourage future research to investigate various RT-SCORE features and more complex machine learning models for better prediction models. p. 8 Limitations
Although we have observed positive correlation
between FT-SCOREs and RT-SCOREs and conducted experiments to predict FT-SCOREs using
RT-SCOREs, their relations could be complicated
and non-linear. We encourage future research to
investigate various RT-SCORE features and more
complex machine learning models for better prediction models. Although we have examined the
prediction models on low-resource languages in
FLORES-101, we have not tested those very lowresource languages out of these 101 languages. We
suggest auditing FT-SCORE prediction models on
a small validation dataset for any new low-resource
languages in future applications.Where the paper says it
Where the paper says it
The results of their predictors on ranking WMT2020-Bio shared tasks show that overall performance is positive, while it is relatively worse than the results of WMT2020-News reported in Table 6. This difference is attributed to the fact that the M used on WMT2020-Bio are calculated on document, while their regression models rely on sentence-level translation metrics in training. The large granularity difference of text may result in a distribution shift. p. 4 We select these languages based on two criteria: i)
We rank languages given the scale of their bi-text
corpora; ii) We prioritize the languages covered
by WMT2020-News and WMT2020-Bio. As a
result, FLORES-AE33 includes 7 high-resource
languages, 16 medium-resource languages and 10
low-resource languages.Where the paper says it
The authors have shown that RT-SCORE BA is generally more correlated to FT-SCORE than RT-SCORE AB, leading to strongly positive correlation scores. This advantage is attributed to the fact that TA->B serves as the last translation step in RT-SCORE BA. The authors believe their work will inspire research on reference-free evaluation on low-resource machine translation.
The overall correlation scores are reported in Table 10, indicating at least moderately positive correlations between all pairs of RT-SCOREs and FT-SCOREs. Moreover, they observe that RT-SCORE BA is generally more correlated to FT-SCORE than RT-SCORE AB. The authors believe their work will inspire research on reference-free evaluation on low-resource machine translation.
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Predict forward translation scores using round-trip translation scores by training a linear regression model fM(S) = W1 · SMAB + W2 · SMBA + β, which allows for robust prediction on transferred MT systems and unseen language pairs including low-resource languages.
-
Improve quality estimation performance by using features acquired from round-trip translation, specifically RTTBLEU, RTTspBLEU, RTTchrF, and RTTBERTScore. This combination of features can be used to acquire a more competitive QE scorer compared to models relying on only direct assessment scores (DA).
-
Identify adversarial competitors in shared tasks via crosssystem verification using an X-Check mechanism. By leveraging the estimation from multiple translation systems, this mechanism manages to identify those adversarial competitors which rely heavily on the copy strategy.
Abstract
Automatic evaluation on low-resource language translation suffers from a deficiency of parallel corpora. Round-trip translation could be served as a clever and straightforward technique to alleviate the requirement of the parallel evaluation corpus. However, there was an observation of obscure correlations between the evaluation scores by forward and round-trip translations in the era of statistical machine translation (SMT). In this paper, we report the surprising finding that round-trip translation can be used for automatic evaluation without the references. Firstly, our revisit on the round-trip translation in SMT evaluation unveils that its long-standing misunderstanding is essentially caused by copying mechanism. After removing copying mechanism in SMT, round-trip translation scores can appropriately reflect the forward translation performance. Then, we demonstrate the rectification is overdue as round-trip translation could benefit multiple machine translation evaluation tasks. To be more specific, round-trip translation could be used i) to predict corresponding forward translation scores; ii) to improve the performance of the recently advanced quality estimation model; and iii) to identify adversarial competitors in shared tasks via cross-system verification.
Sources
- Building Machine Translation Systems for the Next Thousand Languages
- Quality Estimation Using Round-trip Translation with Sentence Embeddings
- Towards the Next 1000 Languages in Multilingual Machine Translation: Exploring the Synergy Between Supervised and Self-Supervised Learning
- Multilingual Translation with Extensible Multilingual Pretraining and Finetuning
- Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
- Generalized Data Augmentation for Low-Resource Translation
Related papers
- ALARM: Audio-Language Alignment for Reasoning Models
- ThinkSum: Probabilistic reasoning over sets using large language models
- Coherence boosting: When your pretrained language model is not paying enough attention
- From Facts to Insights: A Persona-Driven Dual Memory Framework and Dataset for Role-Playing Agents
- Improving General Role-Playing Agents via Psychology-Grounded Reasoning and Role-Aware Policy Optimization
- The Art of Asking: Multilingual Prompt Optimization for Synthetic Data