It's obvious that there are limits to how well machine translation can work unless the models have sensory grounding. I wonder if the problem is that people haven't figured out how to do sensory grounding or that the hardware is still too slow for it to work.
Using AI to match human performance in translating news from Chinese to English
11–20 of 62 posts
Re: Using AI to match human performance in translating news from Chinese to English
#12Translate "sentences of news" is very different to translating an entire article, which is obviously what's interesting. Is anybody in MT or text comprehension/generation really working on systems that construct a model/"understanding" of the bigger narrative in a longer-running text? Even just to be able to do correct anaphora resolution across sentence and paragraph boundaries, but intuitively also WSD seems easier…
I think Google translate already has this. I was translating some text into German a few days ago, and after a few sentences I used a word that made it clear that I was talking about a specific type of contract appointment, and it went back and adjusted earlier sentences to use more precise terminology. You only notice this when you a) speak the language you're translating into somewhat; b) actually type/compose the…
I've found that google translate works decently (as far as these automated translations can be expected to work) from/to English, however translating from a different pair of languages the results are often very off in my experience. In particular it seems it has some sort of internal bias, as if it always an english-like intermediary representation.
For instance, if you ask google to translate the Portuguese "báculo" into French it gives you "personnel". It's nonsense as far as I can tell, a báculo is a "crosier of a bishop"[1]. So what's going on here? Well if you translate it from PT to EN it gives you "staff" and suddenly it starts making sense, because while staff means "A long, straight, thick wooden rod or stick, especially one used to assist in walking" (which fits báculo) it can also mean "The employees of a business" which is an accurate definition for french "personnel". And I believe that's how you end up with the nonsensical PT -> FR translation.
Similarly Google used to be confused by the tu/vous (informal/formal) distinction that exists in many languages but not in English. At some point the portuguese "tu és" would be translated in french by the formal "vous êtes" instead of the informal "tu es". This appears to have been fixed however, I can't reproduce it at the moment.
Conjugations don't fare so well however, for instance imperfect past french "je chantais" is translated into portuguese preterite "eu cantei" even though "eu cantava" would make more sense I think. Obviously with such small phrases I can't really be too harsh on google's bad grammar, they're probably not optimizing for that case.
Re: Using AI to match human performance in translating news from Chinese to English
#13The most impressive ML translation tool I've seen so far is DeepL[0]. Sometimes, it manages to translate whole articles without errors. [0]: https://www.deepl.com/translator
I can also confirm, it's very impressive. Not sure how their setup differs from Google Translate but I've read articles translated with it and only knew it was translated when it was written at the end of the article that it was using DeepL.
Re: Using AI to match human performance in translating news from Chinese to English
#14As impressive as it may be, these people should refrain from claiming 'human-like' translation from a system that has no way of 'knowing' anything about context, other than statistical occurrences. It is certain that, on occasion, the system will make such mistakes as stating the opposite of what is being said in the first place, or attribute one action to the wrong person, and what not. Perhaps on average it's as go…
This was much the point that John Searle made in his criticism of AI prognostications. Ultimately I think he'll be shown to be wrong but the time frame for this will be (I suggest) much longer than is currently touted.
Re: Using AI to match human performance in translating news from Chinese to English
#15The most impressive ML translation tool I've seen so far is DeepL[0]. Sometimes, it manages to translate whole articles without errors. [0]: https://www.deepl.com/translator
DeepL is what I use for European language translations. I wish they added more languages and maybe a nice app like Google Translate. It blows every other translation service out of the water.
Re: Using AI to match human performance in translating news from Chinese to English
#16Compare this to one of Google's blog post promoting their MT research: https://research.googleblog.com/2016/09/a-neural-network-for... It is: 1) More accurate, compared to hyperbole like e.g. "Bridging the Gap between Human and Machine Translation" we have right there in the title the domain: news. 2) A more impressive result. This result is on an independently set up evaluation framework, compared to Google's which…
The outputs of Microsoft Research are really good. At least in my field, it is one of the few places where if they published something you can be sure of being able to reproduce the results using only what is described in the paper, no secret sauce required.
Re: Using AI to match human performance in translating news from Chinese to English
#17Translate "sentences of news" is very different to translating an entire article, which is obviously what's interesting. Is anybody in MT or text comprehension/generation really working on systems that construct a model/"understanding" of the bigger narrative in a longer-running text? Even just to be able to do correct anaphora resolution across sentence and paragraph boundaries, but intuitively also WSD seems easier…
I think Google translate already has this. I was translating some text into German a few days ago, and after a few sentences I used a word that made it clear that I was talking about a specific type of contract appointment, and it went back and adjusted earlier sentences to use more precise terminology. You only notice this when you a) speak the language you're translating into somewhat; b) actually type/compose the…
That sounds extremely interesting. I had not noticed that feature before. Do you happen to have some example input at hand that triggers such an adjustment?
Re: Using AI to match human performance in translating news from Chinese to English
#18I find these types of "match human performance" claims to be ridiculous, especially when it comes to Chinese -> English translations. Translation is both an art and a science, requiring nuanced understanding of the languages, cultures, and context. It also demands quite a bit of creativity. No translation tool I've tried has come even close to matching human performance of a good human translator, including microsoft's tools. AI will need to reach the point where its understanding of language, culture, context, and creative ability matches that of humans to truly be capable of "human performance" in translation.
Re: Using AI to match human performance in translating news from Chinese to English
#19Be careful when reading such claims: https://www.theatlantic.com/technology/archive/2018/01/the-s...
[deleted]
Re: Using AI to match human performance in translating news from Chinese to English
#20I find these types of "match human performance" claims to be ridiculous, especially when it comes to Chinese -> English translations. Translation is both an art and a science, requiring nuanced understanding of the languages, cultures, and context. It also demands quite a bit of creativity. No translation tool I've tried has come even close to matching human performance of a good human translator, including microsoft…
After reading the paper, my takeaway is that humans aren't really very good at translation either. None of the methods scores higher than 70% in the evaluation and that includes several different human translations (whose performance varies greatly depending on how they were sourced). So while matching the quality of the average human translator is a great milestone, there's still lots of room to improve.