Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Finnish breaks natural language processors

twitter.com

Post reply on HN
Print view View original

121–130 of 131 posts

Re: Finnish breaks natural language processors

#121

Post by chousuke »

Griceraae50100 wrote: post #54

Really interesting! I was wondering a few months ago how autocompletion on phones works for agglutinating languages like Finnish, but I didn't have a Finn to ask =/ If you wouldn't mind enlightening me, does it even exist for Finnish? If so, how does it work?

Writing Finnish on a phone is terrible until it learns all of your most commonly used words and their most common forms, at which point out becomes merely bad. Whenever I write Finnish using my phone I basically have to work around the keyboard not doing what I want. I'll still swipe to form words, but I have to go back to "fix" them because the system usually can't even handle simple compounds.

Re: Finnish breaks natural language processors

#122

Post by qbaqbaqba »

frosted-flakes wrote: post #97

It's interesting that Nokia's auto-completion worked well for Finnish, but modern solutions don't work. > Joose Rajamäki 🇫🇮🇪🇺 @joose_rajamaeki Feb 20 > Yes, autocompletion [on phones] exists. But I hardly ever manage to compose a message where it wouldn't encounter new words. Also, it doesn't know the inflections, so it needs to encounter each word in each possible form before suggesting them. > Joose Rajamäki 🇫…

Same goes for Polish!

Re: Finnish breaks natural language processors

#123

Post by pndy »

smcl wrote: post #98
pndy wrote: post #23

Earlier quoted context omitted.

> The words mom (appearing in your contact list), mom (as in call mom) and mom (as in send money to mom) are different. Each time I tried WP8.1 voice assistant to "zadzwoń do mama" it felt me extremely awkward due to nominative case of the word "mama". It should be "zadzwoń do mamy" which is genitive case, if I'm not mistaken, that would make whole phrase sound natural.

Interesting, Polish and Czech are so alike and so different :D I guess the word Polish "mama" is pretty much the Czech "mamka" which similarly declines to "mamky" in the genitive case, but "do mamky" here is " into mum" which is ... rather odd (I think Czech would use the Dative "mamce" - so “zavolej mamce” or “zavolejte mamce” depending on your relationship with your phone!) I get tripped up when, for example, talki…

It's the same West Slavic language group along with Slovak, thus those are similar to the certain degree but sometimes differ way too much like for example, infamous "szukać" (look for, seek; hledat) and "šukat" (to have sexual intercourse, to put it mildly) or "czerstwy" (stale) and "čerstvy" (fresh). I'm partially able to understand my Czech friend when he does speak in his language and same goes for him.

As for "mama", there's a quite good explanation why it does looks similar (if not same) in many languages: https://www.mother.ly/parenting/mama-is-most-universal-word

Re: Finnish breaks natural language processors

#124

Post by yorwba »

Grue3 wrote: post #115
yorwba wrote: post #111

Earlier quoted context omitted.

How do you test the accuracy of your tokenizer? Is there a database of ground-truth tokenized sentences that you evaluate on? It would be nice to see some kind of benchmark results comparing with MeCab.

It's not a matter of benchmarks really. It's more of a matter of MeCab simply not knowing that certain grammar exists. For example 〜ちゃった for 〜てしまった is a pretty common form. MeCab doesn't understand the word 忘れちゃった because it only knows 忘れて and しまった as completely separate words and can't combine them in any way, but Ichiran can because my word-form generator is much more sophisticated. This is also why I cannot compar…

If I do echo "忘れちゃった" | mecab, I get

  忘れちゃ 動詞,*,母音動詞,タ系連用チャ形,忘れる,わすれちゃ,代表表記:忘れる/わすれる 付属動詞候補（基本） 反義:動詞:覚える/おぼえる
  った 接尾辞,動詞性接尾辞,子音動詞ワ行,タ形,う,った,代表表記:う/う
  EOS
So MeCab most definitely knows about that grammar. It's special-cased even! Admittedly, the way った is split off as a separate suffix is a bit ugly.

Re: Finnish breaks natural language processors

#125

Post by MereInterest »

woliveirajr wrote: post #91
MereInterest wrote: post #68

Earlier quoted context omitted.

The murderer came to my dinner party, and I had it all planned. In one of the ice cubes, I had frozen arsenic. The murderer would eat the same food, drink the same drink, and nobody would guess that they would die on leaving. When the evening was over, I knew what I would tell people. Justicehadbeenserved.

Please, share this with the world on tweeter.

If you would like to, feel free. For myself, I think that the comment's context of showing how ambiguity may not be resolved merely be contextual information is important, and that it would not stand as strongly without it.

Re: Finnish breaks natural language processors

#126

Post by Grue3 »

yorwba wrote: post #124
Grue3 wrote: post #115

Earlier quoted context omitted.

It's not a matter of benchmarks really. It's more of a matter of MeCab simply not knowing that certain grammar exists. For example 〜ちゃった for 〜てしまった is a pretty common form. MeCab doesn't understand the word 忘れちゃった because it only knows 忘れて and しまった as completely separate words and can't combine them in any way, but Ichiran can because my word-form generator is much more sophisticated. This is also why I cannot compar…

If I do echo "忘れちゃった" | mecab , I get 忘れちゃ 動詞,*,母音動詞,タ系連用チャ形,忘れる,わすれちゃ,代表表記:忘れる/わすれる 付属動詞候補（基本） 反義:動詞:覚える/おぼえる った 接尾辞,動詞性接尾辞,子音動詞ワ行,タ形,う,った,代表表記:う/う EOS So MeCab most definitely knows about that grammar. It's special-cased even! Admittedly, the way った is split off as a separate suffix is a bit ugly.

That's the thing though, 忘れちゃ is a completely different, unrelated form that means 忘れて+は. So yes, this split is in fact incorrect and if it wasn't past tense, but dictionary form 忘れちゃう, the result would be even worse with う being split off as "rain"? or something.

Re: Finnish breaks natural language processors

#128

Post by wodenokoto »

Seems like they are splitting their tokens wrong. Spaces are just a suggestion.

But seriously, the problems outlined reminded me very much of Japanese and korean, just turned up to 11.

Japanese has endless amounts of homonyms and at least in theory words can go on and on and on with added conjugations. They have a lot of compound nouns, often build from abbreviations of the words compounded.

The author mentions that these compounds are a problem in Finnish NLP because the explode the size of the vocabulary.

Written Japanese does not contain any word boundaries at all. They split on morphemes for NLP tasks, which helps against exploding vocabularies, but also disseminate the parts into their own meaning-unit.

For written Japanese you have characters which can guide you in meaning for a lot of compound words, but they blow up your character space. That is not the case for Finnish, but I can't really decide if that is an advantage or disadvantage.

Also, a spellchecker and a chatbot would rely on completely different techniques, so you can have one without the other. Japanese doesn't even have spellcheckers, but they have a ton of other tools that millions of people rely on everyday for writing faster and better text, like kana-kanji conversion and word suggestions.

Re: Finnish breaks natural language processors

#129

Post by fakghy »

kbouck wrote: post #118
arka2147483647 wrote: post #70

Earlier quoted context omitted.

This is a specificially constructed corner case, not a general finnish thing: - kuu = moon - kuusi = spruce - palaa = returns - pala = piece From those you can see how one can intrepent the compound word (with or without suffix) however you wish. Eg: - kuu+si = your moon (where 'si' is the suffix which means 'your') - pala+a = pieces (where 'a' suffix is the plural)

also: - kuusi = six

- palaa = burning

Re: Finnish breaks natural language processors

#130

Post by mda »

Lasokki wrote: post #48
stonewhite wrote: post #27

Turkish being a member of ural-altaic language family, just like Finnish, suffers from similar problems. Google Translate improved much over the years yet it still generates laughable text at best. Stemmers used to create show-stopper word stems (don't really know the current situation). Although, it imported a lot of technical terms from various european languages, making technical texts seem to be more legible due…

Ural-Altaic language family is today considered an obsolete concept [1] and the families are considered unrelated [1] https://en.wikipedia.org/wiki/Ural%E2%80%93Altaic_languages

Yes,but both languages share some important structural similarities. Rich morphology, extreme agglutination and vowel harmony.
Post reply on HN