Rethinking Round-Trip Translation for Machine Translation Evaluation

arXiv:2209.07351 · cs.CL · Submitted 2022-09-15 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "Rethinking Round-Trip Translation for Machine Translation Evaluation".

Jane: Here's the gist of the paper. Automatic evaluation on low-resource language translation suffers from a deficiency of parallel corpora Round-trip translation could be served as a clever and straightforward technique to alleviate the requirement of the parallel evaluation corpus However, there was an observation of obscure correlations between the evaluation scores by forward and roundtrip translations in the era of statistical machine translation (SMT).

Tom: In this paper, we report the surprising finding that round-trip translation can be used for automatic evaluation without the references.

Jane: Pruning redundant reasoning tokens The authors revisit the long-standing misunderstanding regarding round-trip translation in SMT evaluation, which is essentially caused by copying mechanism After removing copying mechanism in SMT, round-trip translation scores can appropriately reflect the forward translation performance.

Tom: The rectification is deemed overdue because round-trip translation could benefit multiple machine translation evaluation tasks.

Title and authors: Tom: We're starting with the title and authors of this paper, "Rethinking Round-Trip Translation for Machine Translation Evaluation," and who did the work.

Jane: They were Terry Yue Zhuo, Qiongkai Xu, Xuanli He, and Trevor Cohn from Monash University, Melbourne University, and UCL.

Lu: I'm interested in how they approach the core problem described in the title—rethinking round-trip translation for evaluation itself.

Meng: I just wonder if this is just a clever trick to avoid needing parallel data without actually solving the underlying quality measurement issue.

Tom: The paper sets out to show that round-trip translation can be used for automatic evaluation even without those parallel references, which is the main goal here.

Jane: So, what's the main takeaway from this revisit of round-trip translation in SMT evaluation?

The paper's summary: Tom: Basically, they look at how forward translation and backward translation work together to form a round-trip process on monolingual data.

Jane: They explain that the forward translation takes a sentence from language A to language B, and then the output from that first step gets translated back to language A using a backward system.

Lu: The core of their argument is that this round-trip translation process can be used to define new evaluation scores, which they call RT-SCORE.

Meng: They contrast this with the traditional forward translation score you get when you have a parallel corpus, which is what we're used to seeing in most SMT evaluations.

Tom: The paper shows that these two types of scores, the FT-SCORE and the RT-SCORE, can actually be strongly correlated across different machine translation systems.

Jane: That's a big deal because it means we might be able to predict the forward translation score just by looking at the round-trip translation score.

The paper's improvements: Lu: One major improvement they point out is that by removing the copying mechanism, which was an issue in SMT evaluations, the round-trip translation scores can correctly reflect how good the forward translation actually is.

Tom: That addresses a long-standing misunderstanding where copying tokens sometimes got penalized in round-trip but rewarded in forward evaluation.

Jane: So what are the practical applications they suggest for using these new RT-SCOREs?

Meng: They see three main things: first, you can use these scores to predict the forward translation scores by training a simple linear regression model.

Tom: That means we don't need a parallel corpus to get an idea of how good a system is, we just need some training data for that predictor.

Lu: Second, they introduce something called an X-Check mechanism for real-world shared tasks, which can help identify adversarial competitors in the field.

Jane: And the third improvement is that these round-trip translation features can actually improve the performance of recently advanced quality estimation models.

Conclusion: Tom: So to wrap up, this paper "Rethinking Round-Trip Translation for Machine Translation Evaluation" shows that round-trip translation offers a way to evaluate machine translation without relying on parallel corpora.

Jane: They demonstrated strong correlations between forward and round-trip scores across various systems, and they gave us tools like a regression predictor and an X-Check mechanism to use those scores.

Lu: It opens up possibilities for evaluating transferred machine translation systems and even low-resource languages, because you don't need those huge parallel datasets anymore.

Meng: From my side, the practical implication is that these features can make our quality estimation models much more robust when we deploy them on actual traffic without running into unexpected issues.

Lalam: I'm quite confident in this direction because the regression predictors they build perform well even on transferred systems and unseen language pairs, showing a Pearson correlation above zero point eight eight on MBART50-M2M.

Tom: So they suggest that RT-SCORE BA is generally more correlated to the forward score than RT-SCORE AB, which makes sense because the last step in RT-SCORE BA is the forward translation itself.

Jane: It’s a solid foundation for moving toward reference-free evaluation methods, and we should definitely keep an eye on how these features affect other quality estimation models.

Terry Yue Zhuo, Qiongkai Xu, Xuanli He, Trevor Cohn

Monash University · The University of Melbourne · University College London

cs.CL

Submitted: 2022-09-15

Updated: 2026-11-13

Comments: Accepted to Findings of ACL 2023

Code: https://github.com/UKPLab/EasyNMT.8https:

License: http://creativecommons.org/licenses/by-nc-nd/4.0/

Importance score: 88/100

The gist: Automatic evaluation on low-resource language translation suffers from a deficiency of parallel corpora Round-trip translation could be served as a clever and straightforward technique to alleviate…

Key concepts

Round-Trip Translation (RTT)
This involves translating a sentence from language A to B (Forward Translation, FT), and then translating that output back from B to A using a backward translation system. The resulting RTT score measures the consistency between these two steps, contrasting with traditional parallel corpus scores.
FT-SCORE vs. RT-SCORE
FT-SCORE measures how well a translation goes from source language A to target language B directly. RT-SCORE measures the performance of the round trip (A->B->A). The paper shows these two types of scores are strongly correlated, suggesting RTT can substitute for parallel corpora in evaluation.
X-Check Mechanism
This is a cross-system verification technique used during RTT evaluation on shared tasks. It compares the outputs of multiple translation systems to identify adversarial competitors—systems that rely heavily on copying strategies—thereby making the evaluation more robust.

Terminology

Summary

Automatic evaluation on low-resource language translation suffers from a deficiency of parallel corpora Round-trip translation could be served as a clever and straightforward technique to alleviate the requirement of the parallel evaluation corpus However, there was an observation of obscure correlations between the evaluation scores by forward and roundtrip translations in the era of statistical machine translation (SMT). p. 0

Where the paper says it

Round-trip translation could be served as a clever and straightforward technique to alleviate the requirement of the parallel evaluation corpus. However, there was an observation of obscure correlations between the evaluation scores by forward and roundtrip translations in the era of statistical machine translation (SMT).

In this paper, we report the surprising finding that round-trip translation can be used for automatic evaluation without the references. p. 0
Where the paper says it

However, there was an observation of obscure correlations between the evaluation scores by forward and roundtrip translations in the era of statistical machine translation (SMT). In this paper, we report the surprising finding that round-trip translation can be used for automatic evaluation without the references.

Pruning redundant reasoning tokens

The authors revisit the long-standing misunderstanding regarding round-trip translation in SMT evaluation, which is essentially caused by copying mechanism After removing copying mechanism in SMT, round-trip translation scores can appropriately reflect the forward translation performance. The rectification is deemed overdue because round-trip translation could benefit multiple machine translation evaluation tasks. p. 0

Where the paper says it

After removing copying mechanism in SMT, round-trip translation scores can appropriately reflect the forward translation performance. Then, we demonstrate the rectification is overdue as round-trip translation could benefit multiple machine translation evaluation tasks.

Specifically, it could be used to i) predict corresponding forward translation scores; ii) improve the performance of the recently advanced quality estimation model; and iii) identify adversarial competitors in shared tasks via crosssystem verification. p. 0
Where the paper says it

Then, we demonstrate the rectification is overdue as round-trip translation could benefit multiple machine translation evaluation tasks. To be more specific, round-trip translation could be used i) to predict corresponding forward translation scores; ii) to improve the performance of the recently advanced quality estimation model; and iii) to identify adversarial competitors in shared tasks via crosssystem verification.

How the training signal is built

The round-trip translation (RTT) entails two components, one forward translation (FT), and the other backward translation (BT). FT translates a given sentence in source language A to a sentence in target language B, then the output sentence from FT is translated back to language A via a back translation system. p. 1

Where the paper says it

As illustrated in Figure 1, RTT entails two components, one forward translation (FT), and the other backward translation (BT). FT translates a given sentence in source language A to a sentence in target language B, then the output sentence from FT is translated back to language A via a back translation system.

The evaluation scores on round-trip translation (RT-SCORE) with regard to an automatic evaluation metric M are defined as RT-SCORE M AB = 1/N Sum i M(TB->A(TA->B(ai)), ai). p. 2
Where the paper says it

FT and BT constitute a round-trip translation (RTT). The evaluation scores on round-trip translation (RT-SCORE) with regard to an automatic evaluation metric M is RT-SCOREM AB = 1 N X N i=1 M(TB→A(TA→B(ai)), ai) (1) where BLEU, spBLEU, chrF and BERTScore are target metrics M in our discussion.

This contrasts with the traditional FT-SCORE M A->B = 1/N Sum i M(TA->B(ai), bi) given a (virtual) parallel corpus DAB = 1/N Sum i (ai, bi). p. 2
Where the paper says it

The evaluation scores on round-trip translation (RT-SCORE) with regard to an automatic evaluation metric M is RT-SCOREM AB = 1 N X N i=1 M(TB→A(TA→B(ai)), ai) (1) where BLEU, spBLEU, chrF and BERTScore are target metrics M in our discussion. On the other hand, traditional MT evaluation on parallel corpus is FT-SCOREM A→B = 1 N X N i=1 M(TA→B(ai), bi) (2) given a (virtual) parallel corpus DA||B = {(ai , bi)} N i=1.

Where the gains come from

The authors demonstrate that there are strong correlations between FT-SCOREs and RT-SCOREs on various MT systems, including NMT and SMT without the copying mechanism. Three application scenarios for RT-SCORE are investigated:

  1. RT-SCOREs can be used to predict FT-SCOREs via training a simple but effective linear regression model on several hundred language pairs. p. 1

    Where the paper says it

    Three application scenarios in MT evaluation have been investigated to show the effectiveness of RT-SCORE. Firstly, RT-SCOREs can be used to predict FT-SCOREs via training a simple but effective linear regression model on several hundred language pairs.

  2. A cross-system check (X-Check) mechanism is introduced to RTT evaluation for real-world MT shared tasks, which manages to identify those adversarial competitors. p. 1

    Where the paper says it

    Then, a cross-system check (X-Check) mechanism is introduced to RTT evaluation for real-world MT shared tasks. By leveraging the estimation from multiple translation systems, X-Check manages to identify those adversarial competitors, which relies heavily on the copy strategy.

  3. RT-SCOREs are proved effective in improving the performance of a recently advanced quality estimation model. p. 1

    Where the paper says it

    By leveraging the estimation from multiple translation systems, X-Check manages to identify those adversarial competitors, which relies heavily on the copy strategy. Finally, RT-SCOREs are proved effective in improving the performance of a recently advanced quality estimation model.

The negative observations between RTT and FT scores in SMT were attributed to reversible transformation rules like word copy, where copying unrecognized tokens forward to target languages is sometimes penalized by FT evaluation while it is usually awarded by RTT evaluation. Extensive experiments show that removing the copying mechanism can almost perfectly resolve the negation in their experiments. p. 2

Where the paper says it

Then, we conduct experiments to replicate the negative conclusion. Interestingly, removing the copying mechanism can almost perfectly resolve the negation in our experiments.

For NMT, copying frequency is about two times in SMT than in NMT. p. 3
Where the paper says it

Exactly matched words in outputs from the input words are considered copying, although the system may not intrinsically intend to copy them. In Table 2, we observe that copying frequency is about two times in SMT than in NMT.

The authors validate whether FT-SCOREs could be predicted by RT-SCOREs by constructing a linear regressor f to predict FT-SCORE M A->B using corresponding RT-SCORE features. The prediction model is defined as f M(S) = W1 · SM AB + W2 · SM BA + β. p. 1

Where the paper says it

Firstly, RT-SCOREs can be used to predict FT-SCOREs via training a simple but effective linear regression model on several hundred language pairs. The prediction performance is robust in evaluating transferred MT systems and unseen language pairs including lowresource languages.

This approach is robust in evaluating transferred MT systems and unseen language pairs including low-resource languages. p. 1
Where the paper says it

Firstly, RT-SCOREs can be used to predict FT-SCOREs via training a simple but effective linear regression model on several hundred language pairs. The prediction performance is robust in evaluating transferred MT systems and unseen language pairs including lowresource languages.

The prediction performance of the regressor is evaluated across various challenging settings, including transferred MT systems and unseen language pairs. The results show that the rankings of predicted scores are rational, with all Pearson’s r above or equal to 0. p. 5

Where the paper says it

Meanwhile, the correlation between prediction and ground truth is strong, with all Pearson’s r above or equal to 0. This indicates that the rankings of predicted scores are rational.

88 when tested on MBART50-M2M. p. 3
Where the paper says it

liance of copying mechanism in NMT and SMT. For NMT, we choose MBART50-M2M (Tang et al.

Furthermore, the regressor shows adequate language transferability results across different resource scales in FLORES-AE33. p. 5
Where the paper says it

Meanwhile, our model obtains adequate language transferability results, as demonstrated in Table 4. In Table 5, we provide detailed performance of our regressor on language pairs of different resources categories on FLORES-AE33, with RT-SCOREs of MBART50-M2M and M2M-100- BASE respectively.

The X-Check mechanism successfully identifies adversarial MT systems among benign competitors, and this management of adversaries improves correlation scores significantly. The selection of benign systems for X-Check includes OPUS-MT, M2M-100-LARGE and MBART50-M2M. p. 4

Where the paper says it

2 Neural Machine Translation Systems We experiment with five MT systems which support most of the languages appearing in FLORES-AE33 and WMT. Except MBART50- M2M, we adopt M2M-100-BASE and M2M-100- LARGE (Fan et al.

This suggests that X-Check can make RTT evaluation more robust. p. 7
Where the paper says it

X-Check evidently successfully identifies these adversarial systems in all our experiments and manages to improve the correlation scores significantly. With the empirical study, we find that X-Check is able to make RTT evaluation more robust.

The features acquired by round-trip translation benefit quality estimation (QE) models, and a combination of COMET-QE-DA scores and RT-SCOREs is investigated to acquire a more competitive QE scorer. Both Kendall’s τ and Pearson’s r provide consistent results in Table 7. p. 6

Where the paper says it

Discussion. Both Kendall’s τ and Pearson’s r provide consistent results in Table 7.

The authors believe that RT-SCOREs advance QE research and urge more investigation in this direction. p. 7
Where the paper says it

We also observe that RT-SCOREs can further boost the performance of COMET-QE-DA. We believe RT-SCOREs advances QE research and urge more investigation in this direction.

The X-Check mechanism can be used to estimate the translation quality of an FT system on a BT system by calculating Si,j = fM(Bj (Fi(x)), x). The overall system ranking could be severely affected by adversarial systems, according to Pearson’s r and Kendall’s τ. p. 7

Where the paper says it

Discussion. From Table 8, we observe that the overall system ranking could be severely affected by the adversarial systems, according to Pearson’s r and Kendall’s τ .

X-Check evidently successfully identifies these adversarial systems in all their experiments and manages to improve the correlation scores significantly. p. 7
Where the paper says it

The adversarial systems are stealthy among normal competitors, according to Hit@K and Avg. X-Check evidently successfully identifies these adversarial systems in all our experiments and manages to improve the correlation scores significantly.

p. 7
Where the paper says it

The adversarial systems are stealthy among normal competitors, according to Hit@K and Avg. X-Check evidently successfully identifies these adversarial systems in all our experiments and manages to improve the correlation scores significantly.

The authors demonstrate that the predicted automatic evaluation scores could be further improved via X-Check when adversaries are included. The overall system ranking could be severely affected by the adversarial systems, according to Pearson’s r and Kendall’s τ. p. 7

Where the paper says it

Discussion. From Table 8, we observe that the overall system ranking could be severely affected by the adversarial systems, according to Pearson’s r and Kendall’s τ .

The adversarial systems are stealthy among normal competitors, according to Hit@K and Avg. p. 7
Where the paper says it

From Table 8, we observe that the overall system ranking could be severely affected by the adversarial systems, according to Pearson’s r and Kendall’s τ . The adversarial systems are stealthy among normal competitors, according to Hit@K and Avg.

Rank. X-Check evidently successfully identifies these adversarial systems in all their experiments and manages to improve the correlation scores significantly.

The overall correlation scores are reported in Table 10, indicating at least moderately positive correlations between all pairs of RT-SCOREs and FT-SCOREs. p. 6

Where the paper says it

In Table 6, we display the results on WMT2020-News. Although MAE and RMSE vary among experiments for different language pairs, the overall correlation scores are favorable.

Moreover, they observe that RT-SCORE BA is generally more correlated to FT-SCORE than RT-SCORE AB, leading to strongly positive correlation scores. p. 8
Where the paper says it

Limitations Although we have observed positive correlation between FT-SCOREs and RT-SCOREs and conducted experiments to predict FT-SCOREs using RT-SCOREs, their relations could be complicated and non-linear. We encourage future research to investigate various RT-SCORE features and more complex machine learning models for better prediction models.

This advantage is attributed to the fact that TA->B serves as the last translation step in RT-SCORE BA. p. 1
Where the paper says it

In this work, we revisit the dispute on the usefulness of RT-SCORE in the era of SMT versus NMT. The main reason is due to the fact that SMT (and RMT) usually incorporate implicitly reversible rules in their translation.

p. 1
Where the paper says it

In this work, we revisit the dispute on the usefulness of RT-SCORE in the era of SMT versus NMT. The main reason is due to the fact that SMT (and RMT) usually incorporate implicitly reversible rules in their translation.

The results of their predictors on ranking WMT2020-News synthetic data demonstrate that overall ranking errors on 17 MT systems are small on all three selected language pairs. The overall correlation scores are favorable, with Pearson’s r values on all language pairs above 0. p. 6

Where the paper says it

Although MAE and RMSE vary among experiments for different language pairs, the overall correlation scores are favorable. Pearson’s r values on all language pairs are above 0.

5, showing strong ranking correlations. p. 6
Where the paper says it

Pearson’s r values on all language pairs are above 0. 5, showing strong ranking correlations.

This shows the feasibility of predicting FTSCORE using monolingual data. p. 6
Where the paper says it

5, showing strong ranking correlations. While prediction performances on A B have some variances among different language pairs, the results of the experiments using B A are competitive to those using both A B and B A features, showing the feasibility of predicting FTSCORE using monolingual data.

The results on WMT2020-Bio show that overall performance is positive, while it is relatively worse than the results of WMT2020-News reported in Table 6. This difference is attributed to the fact that the M used on WMT2020-Bio are calculated on document, while their regression models rely on sentence-level translation metrics in training.

The authors conclude that their regression-based predictors can be practical in ranking MT systems in WMT-style shared tasks. The overall correlation scores are reported in Table 10, indicating at least moderately positive correlations between all pairs of RT-SCOREs and FT-SCOREs. Moreover, they observe that RT-SCORE BA is generally more correlated to FT-SCORE than RT-SCORE AB. p. 1

Where the paper says it

While these automatic evaluation metrics are widely applied in MT evaluation, they are generally not applicable to low-resource language translation or new translation domains (Mathur et al. Our work demonstrates that referencefree MT metrics (RT-SCORE) could be used to estimate traditional reference-based automatic metrics.

p. 1
Where the paper says it

While these automatic evaluation metrics are widely applied in MT evaluation, they are generally not applicable to low-resource language translation or new translation domains (Mathur et al. Our work demonstrates that referencefree MT metrics (RT-SCORE) could be used to estimate traditional reference-based automatic metrics.

The authors believe their work will inspire research on reference-free evaluation on low-resource machine translation. p. 8
Where the paper says it

relevant MT evaluation tasks, such as predicting FT metrics using RTT scores, enhancing state-ofthe-art QE systems, and filtering out unreliable MT competitors for WMT shared tasks. We believe our work will inspire research on reference-free evaluation on low-resource machine translation.

p. 8
Where the paper says it

relevant MT evaluation tasks, such as predicting FT metrics using RTT scores, enhancing state-ofthe-art QE systems, and filtering out unreliable MT competitors for WMT shared tasks. We believe our work will inspire research on reference-free evaluation on low-resource machine translation.

p. 8
Where the paper says it

relevant MT evaluation tasks, such as predicting FT metrics using RTT scores, enhancing state-ofthe-art QE systems, and filtering out unreliable MT competitors for WMT shared tasks. We believe our work will inspire research on reference-free evaluation on low-resource machine translation.

Although they have observed positive correlation between FT-SCOREs and RT-SCOREs and conducted experiments to predict FT-SCOREs using RT-SCOREs, their relations could be complicated and non-linear. They encourage future research to investigate various RT-SCORE features and more complex machine learning models for better prediction models. p. 8

Where the paper says it

Limitations Although we have observed positive correlation between FT-SCOREs and RT-SCOREs and conducted experiments to predict FT-SCOREs using RT-SCOREs, their relations could be complicated and non-linear. We encourage future research to investigate various RT-SCORE features and more complex machine learning models for better prediction models.

The authors suggest auditing FT-SCORE prediction models on a small validation dataset for any new low-resource languages in future applications. p. 8
Where the paper says it

Although we have examined the prediction models on low-resource languages in FLORES-101, we have not tested those very lowresource languages out of these 101 languages. We suggest auditing FT-SCORE prediction models on a small validation dataset for any new low-resource languages in future applications.

The results of their predictors on ranking WMT2020-Bio shared tasks show that overall performance is positive, while it is relatively worse than the results of WMT2020-News reported in Table 6. This difference is attributed to the fact that the M used on WMT2020-Bio are calculated on document, while their regression models rely on sentence-level translation metrics in training. The large granularity difference of text may result in a distribution shift. p. 4

Where the paper says it

We select these languages based on two criteria: i) We rank languages given the scale of their bi-text corpora; ii) We prioritize the languages covered by WMT2020-News and WMT2020-Bio. As a result, FLORES-AE33 includes 7 high-resource languages, 16 medium-resource languages and 10 low-resource languages.

The authors have shown that RT-SCORE BA is generally more correlated to FT-SCORE than RT-SCORE AB, leading to strongly positive correlation scores. This advantage is attributed to the fact that TA->B serves as the last translation step in RT-SCORE BA. The authors believe their work will inspire research on reference-free evaluation on low-resource machine translation.

The overall correlation scores are reported in Table 10, indicating at least moderately positive correlations between all pairs of RT-SCOREs and FT-SCOREs. Moreover, they observe that RT-SCORE BA is generally more correlated to FT-SCORE than RT-SCORE AB. The authors believe their work will inspire research on reference-free evaluation on low-resource machine translation.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Predict forward translation scores using round-trip translation scores by training a linear regression model fM(S) = W1 · SMAB + W2 · SMBA + β, which allows for robust prediction on transferred MT systems and unseen language pairs including low-resource languages.

  2. Improve quality estimation performance by using features acquired from round-trip translation, specifically RTTBLEU, RTTspBLEU, RTTchrF, and RTTBERTScore. This combination of features can be used to acquire a more competitive QE scorer compared to models relying on only direct assessment scores (DA).

  3. Identify adversarial competitors in shared tasks via crosssystem verification using an X-Check mechanism. By leveraging the estimation from multiple translation systems, this mechanism manages to identify those adversarial competitors which rely heavily on the copy strategy.

Abstract

Automatic evaluation on low-resource language translation suffers from a deficiency of parallel corpora. Round-trip translation could be served as a clever and straightforward technique to alleviate the requirement of the parallel evaluation corpus. However, there was an observation of obscure correlations between the evaluation scores by forward and round-trip translations in the era of statistical machine translation (SMT). In this paper, we report the surprising finding that round-trip translation can be used for automatic evaluation without the references. Firstly, our revisit on the round-trip translation in SMT evaluation unveils that its long-standing misunderstanding is essentially caused by copying mechanism. After removing copying mechanism in SMT, round-trip translation scores can appropriately reflect the forward translation performance. Then, we demonstrate the rectification is overdue as round-trip translation could benefit multiple machine translation evaluation tasks. To be more specific, round-trip translation could be used i) to predict corresponding forward translation scores; ii) to improve the performance of the recently advanced quality estimation model; and iii) to identify adversarial competitors in shared tasks via cross-system verification.

Sources

Related papers