BLEU Score: Bilingual Evaluation Understudy
leimao.github.io
BLEU Score: Bilingual Evaluation Understudy
1–10 of 14 posts
Re: BLEU Score: Bilingual Evaluation Understudy
#2BLEU and ROUGE scores are almost useless for what they are trying to do (measure how good translation or summarization systems are). The author shows a great example of an excellent translation getting a 0 score.
I tried submitting a NLP paper where I explicitly laid out my reasons for avoiding evaluating my system with ROGUE scores and I learned really quickly that despite having a terrible scoring metric, the NLP community would rather reject all papers without ROGUE scores reported rather than admit that there is an incredible lack of methods for automatically evaluating summaries or translations.
How many good translation or summarization ideas are not published or utilized just because they don't get high BLEU scores? I bet it's a lot of them...
Re: BLEU Score: Bilingual Evaluation Understudy
#3BLEU and ROUGE scores are almost useless for what they are trying to do (measure how good translation or summarization systems are). The author shows a great example of an excellent translation getting a 0 score. I tried submitting a NLP paper where I explicitly laid out my reasons for avoiding evaluating my system with ROGUE scores and I learned really quickly that despite having a terrible scoring metric, the NLP c…
Which metric did you use instead? Human evaluation?
Re: BLEU Score: Bilingual Evaluation Understudy
#4[deleted]
Re: BLEU Score: Bilingual Evaluation Understudy
#5BLEU and ROUGE scores are almost useless for what they are trying to do (measure how good translation or summarization systems are). The author shows a great example of an excellent translation getting a 0 score. I tried submitting a NLP paper where I explicitly laid out my reasons for avoiding evaluating my system with ROGUE scores and I learned really quickly that despite having a terrible scoring metric, the NLP c…
The point of BLEU (and ROUGE and METEOR) is to correlate with human grading. It's unusual to find cases where a MT model change increases BLEU but hurts human ratings.
Re: BLEU Score: Bilingual Evaluation Understudy
#6BLEU and ROUGE scores are almost useless for what they are trying to do (measure how good translation or summarization systems are). The author shows a great example of an excellent translation getting a 0 score. I tried submitting a NLP paper where I explicitly laid out my reasons for avoiding evaluating my system with ROGUE scores and I learned really quickly that despite having a terrible scoring metric, the NLP c…
Which metric did you use instead? Human evaluation?
Ultimately there are no good automated metrics with regards to summarization in my opinion, human evaluation alone is also no silver bullet since it is very labor intensive and what constitutes a good summary is highly dependent on the domain.
Re: BLEU Score: Bilingual Evaluation Understudy
#7BLEU and ROUGE scores are almost useless for what they are trying to do (measure how good translation or summarization systems are). The author shows a great example of an excellent translation getting a 0 score. I tried submitting a NLP paper where I explicitly laid out my reasons for avoiding evaluating my system with ROGUE scores and I learned really quickly that despite having a terrible scoring metric, the NLP c…
The point of BLEU (and ROUGE and METEOR) is to correlate with human grading. It's unusual to find cases where a MT model change increases BLEU but hurts human ratings.
Maybe that was the case when we were using DUC datasets with several gold standard summaries or translations. It certainly is not the case on classic datasets like CNN/Dailymail where there is only one gold standard summary.
Re: BLEU Score: Bilingual Evaluation Understudy
#8BLEU and ROUGE scores are almost useless for what they are trying to do (measure how good translation or summarization systems are). The author shows a great example of an excellent translation getting a 0 score. I tried submitting a NLP paper where I explicitly laid out my reasons for avoiding evaluating my system with ROGUE scores and I learned really quickly that despite having a terrible scoring metric, the NLP c…
Computing a perfect metrics is as hard as the task itself, so lf course scores like BLEU and ROUGE use and easy approach instead that can be gamed. Every researcher reading this kind of paper normally understands that.
In this post, the 0 example for a good translation arise because the question is not formulated in the standard English way which use sujet/verb inversion. This problem is easily solved by extending the corpus with more diverse way to formulate questions. It’s also notable than the way a question would be translated in Chinese would also similarly be affected by the construct used by the MT system and the constructs present in the corpus. Instead of ditching the scores completely, improving the corpus seems to be a more productive approach, which would also benefit other researchers and the field in the long term.
Re: BLEU Score: Bilingual Evaluation Understudy
#9BLEU and ROUGE scores are almost useless for what they are trying to do (measure how good translation or summarization systems are). The author shows a great example of an excellent translation getting a 0 score. I tried submitting a NLP paper where I explicitly laid out my reasons for avoiding evaluating my system with ROGUE scores and I learned really quickly that despite having a terrible scoring metric, the NLP c…
Which metric did you use instead? Human evaluation?
Frechet Inception Distance using language model is a solution (commonly used in text GAN). I think the core idea is that you should never rely on one metrics for evaluation but rather a mix of them ( even statistic ones: unique word count, word distribution similarity etc )
Re: BLEU Score: Bilingual Evaluation Understudy
#10BLEU and ROUGE scores are almost useless for what they are trying to do (measure how good translation or summarization systems are). The author shows a great example of an excellent translation getting a 0 score. I tried submitting a NLP paper where I explicitly laid out my reasons for avoiding evaluating my system with ROGUE scores and I learned really quickly that despite having a terrible scoring metric, the NLP c…
The point of BLEU (and ROUGE and METEOR) is to correlate with human grading. It's unusual to find cases where a MT model change increases BLEU but hurts human ratings.
It's not so unusual in my experience. I used to do research in this area and we ran side-by-side human evals with our experiments. Yes, huge jumps in BLEU typically imply better human ratings, but the variance is big. Many times, significant jumps in BLEU had no significant effect on human ratings. ROUGE is even more useless.
You just don't see these things reported in the literature because 1. human evals are a pain to run and can be expensive 2. researchers need to publish or perish.