BLEU Score: Bilingual Evaluation Understudy • Hacker News Board

Re: BLEU Score: Bilingual Evaluation Understudy

#11

Post by bitforger » Thu, Jun 25, 2020, 5:43 PM UTC

Der_Einzige wrote: post #2

BLEU and ROUGE scores are almost useless for what they are trying to do (measure how good translation or summarization systems are). The author shows a great example of an excellent translation getting a 0 score. I tried submitting a NLP paper where I explicitly laid out my reasons for avoiding evaluating my system with ROGUE scores and I learned really quickly that despite having a terrible scoring metric, the NLP c…

A recent trend has been to use a language model to do the scoring: https://arxiv.org/abs/1904.09675

Re: BLEU Score: Bilingual Evaluation Understudy

#12

Post by thaumasiotes » Thu, Jun 25, 2020, 7:50 PM UTC

Der_Einzige wrote: post #2

BLEU and ROUGE scores are almost useless for what they are trying to do (measure how good translation or summarization systems are). The author shows a great example of an excellent translation getting a 0 score. I tried submitting a NLP paper where I explicitly laid out my reasons for avoiding evaluating my system with ROGUE scores and I learned really quickly that despite having a terrible scoring metric, the NLP c…

> BLEU and ROUGE scores are almost useless for what they are trying to do (measure how good translation or summarization systems are). The author shows a great example of an excellent translation getting a 0 score.

The author also (perhaps unintentionally) shows a great example of being unable to tell whether a translation is excellent or awful.

猫坐在垫子上 is not a great correspondence to either reference sentence. Going purely by grammar, the equivalent of "the cat is on the mat" would be 猫在垫子上 (note the missing 坐, which is the verb "to sit", not present in the english references), and "there is a cat on the mat" would be 垫子上有猫 [literally "the top of the mat has a cat"].

猫坐在垫子上 could be a good translation of "the cat is on the mat". But it could also be a good translation of "cats sit on mats". Those two sentences are radically different in English; to judge the translation, you need to be aware of whether e.g. someone just asked "where's the cat?" or "what do cats do?"

Re: BLEU Score: Bilingual Evaluation Understudy

#13

Post by arugulum » Thu, Jun 25, 2020, 8:08 PM UTC

theblackcat1002 wrote: post #9

probably_wrong wrote: post #3

Earlier quoted context omitted.

Which metric did you use instead? Human evaluation?

Frechet Inception Distance using language model is a solution (commonly used in text GAN). I think the core idea is that you should never rely on one metrics for evaluation but rather a mix of them ( even statistic ones: unique word count, word distribution similarity etc )

"You have a model evaluation problem. You decide to use Frechet Inception Distance. Now you have two model evaluation problems."

Re: BLEU Score: Bilingual Evaluation Understudy

#14

Post by mintrain » Thu, Jun 25, 2020, 10:00 PM UTC

There are many ways of computing BLEU. Researchers may use different tokenization, byte pair encoding or raw text processing methods which will affect the final score.

sacrebleu package helps to standardize BLEU computation to make the comparison of model performance easier. https://github.com/mjpost/sacrebleu