Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Finnish breaks natural language processors

twitter.com

Post reply on HN
Print view View original

31–40 of 131 posts

Re: Finnish breaks natural language processors

#31

Post by fluffycat »

szatkus wrote: post #19
plq wrote: post #10

These are common problems for languages with complex morphology. Finnish, Japanese, Turkish and Hungarian are closely related in this sense and share a lot of NLP research. While implementing spell checkers for these languages need a bit more effort than just compiling a list of words, it's far from an unsolved problem. AFAIK Zemberek[1] is a Turkish spellchecker that implements two-level finite state morphology. Als…

What's the problem with Japanese? It's highly regular language, so it should be easy to tokenize. AFAIR the whole language has maybe few irregular verbs, compared to few hundreds in English.

Turkish is also highly regular and has only one irregular verb. The main problem arises from morphological complexity. Because of these well defined rules that change the rich set of suffixes (>100 different), analysis of a lot of words ends up with many possible parses, it is hard to resolve this ambiguity.

I can't speak Japanese, but if it is also morphologically rich, it should face similar problems.

Re: Finnish breaks natural language processors

#32

Post by zorked »

aasasd wrote: post #25

I still haven't figured out how Germans use swipe keyboards on smartphones with the monster compound sentence-words. If they do.

It just means phones ship with larger dictionaries.

And I would guess this isn't even the proper way to handle German. The long words are just sequences of words not split by spaces when writing. In a parallel universe the German writing system would use spaces and the language wouldn't need to change a bit.

Words are features of writing systems, not of languages.

Re: Finnish breaks natural language processors

#33

Post by babuskov »

ajuc wrote: post #14
miki123211 wrote: post #5

Polish has some of those problems too, at least the ones with conjugations. The words mom (appearing in your contact list), mom (as in call mom) and mom (as in send money to mom) are different. That applies to proper names too, and it isn't regular. It's not easy to figure oujt who the user refers to if you just have the contacts list and the command. Polish had problems way before nlp, for example "Annie has sent yo…

All Slavic languages are quite quirky in this regard. Most native speakers don't realize it, but when you try to write correct message parametrized with numbers in Polish, you need 3 or 4 special cases: - 1 - 2-4, 22-24, 32-34, ... - 5-21, 25-31, 35-41, ... If you say "out of X" you also need to handle numbers 100-199, 100 000-199 000, 100 000 000-199 000 000, ... separately (because 100 = sto, and if number begins w…

> Almost no software handles this correctly

Gettext solved it 20 years ago. The problem is that people try to reinvent the wheel instead of looking at existing solutions. So, they make their own inferior versions.

The rule for plural has 3 cases.

1. When n mod 10 = 1

2. When n mod 10 is 2,3,4

3. Everything else

And there's a special rule that the first 2 cases are not used when n mod 100 is between 11 and 29. For such numbers, the 3rd case is used instead.

This is actually not that complex. Compare to Arabic, where you have 9 cases IIRC.

Re: Finnish breaks natural language processors

#34

Post by perlgeek »

aasasd wrote: post #25

I still haven't figured out how Germans use swipe keyboards on smartphones with the monster compound sentence-words. If they do.

I often just write separate words. Not correct, strictly speaking, but understandable.

I still prefer regular keyboards so (desktop and laptop) -- not sure if that's related.

Re: Finnish breaks natural language processors

#35

Post by ajuc »

babuskov wrote: post #33
ajuc wrote: post #14

Earlier quoted context omitted.

All Slavic languages are quite quirky in this regard. Most native speakers don't realize it, but when you try to write correct message parametrized with numbers in Polish, you need 3 or 4 special cases: - 1 - 2-4, 22-24, 32-34, ... - 5-21, 25-31, 35-41, ... If you say "out of X" you also need to handle numbers 100-199, 100 000-199 000, 100 000 000-199 000 000, ... separately (because 100 = sto, and if number begins w…

> Almost no software handles this correctly Gettext solved it 20 years ago. The problem is that people try to reinvent the wheel instead of looking at existing solutions. So, they make their own inferior versions. The rule for plural has 3 cases. 1. When n mod 10 = 1 2. When n mod 10 is 2,3,4 3. Everything else And there's a special rule that the first 2 cases are not used when n mod 100 is between 11 and 29. For suc…

Doesn't solve the problem with "X out of Y".

Re: Finnish breaks natural language processors

#36

Post by a_imho »

DocG wrote: post #9

There was a joke that Estonian is more closer to Japanese than any other European language. As to not say its related to Japanese, but how much it differs from European ones. No future tense, no genders, but 14 cases. This illustrates nicely where finno-ugric languages(Finnish, Hungarian, Estonian) reside compared to other European languages. https://images.mentalfloss.com/sites/default/files/196.jpg Benefit is that…

What is the benefit of genders anyway? Especially when gender neutral alternatives are available. It never made much sense to me, they introduce a lot of complexity for very little. Worse, nowadays you have all the problems of misaddressing and offending folks.

Re: Finnish breaks natural language processors

#37

Post by szatkus »

fluffycat wrote: post #31
szatkus wrote: post #19

Earlier quoted context omitted.

What's the problem with Japanese? It's highly regular language, so it should be easy to tokenize. AFAIR the whole language has maybe few irregular verbs, compared to few hundreds in English.

Turkish is also highly regular and has only one irregular verb. The main problem arises from morphological complexity. Because of these well defined rules that change the rich set of suffixes (>100 different), analysis of a lot of words ends up with many possible parses, it is hard to resolve this ambiguity. I can't speak Japanese, but if it is also morphologically rich, it should face similar problems.

In Japanese I think it's pretty straightforward to trace the original word. But there is a fair share of homophones, so if your input is speech-to-text you already have many possibilities without much grammar.

Re: Finnish breaks natural language processors

#38

Post by Geimfari »

aasasd wrote: post #25

I still haven't figured out how Germans use swipe keyboards on smartphones with the monster compound sentence-words. If they do.

I just write the words separately and remove the spaces afterwards. An "I'm about to write a compound word"-button would be an easy solution, but it shouldn't be hard for the keyboard to automatically join the words after you've finished your sentence, it's mostly rule based.

Re: Finnish breaks natural language processors

#39

Post by iainmerrick »

a_imho wrote: post #36
DocG wrote: post #9

There was a joke that Estonian is more closer to Japanese than any other European language. As to not say its related to Japanese, but how much it differs from European ones. No future tense, no genders, but 14 cases. This illustrates nicely where finno-ugric languages(Finnish, Hungarian, Estonian) reside compared to other European languages. https://images.mentalfloss.com/sites/default/files/196.jpg Benefit is that…

What is the benefit of genders anyway? Especially when gender neutral alternatives are available. It never made much sense to me, they introduce a lot of complexity for very little. Worse, nowadays you have all the problems of misaddressing and offending folks.

There was an interesting/amusing post here the other day about learning German and how it relates to programming concepts. They suggested that gendered words work as a kind of parity bit for error correction.

Re: Finnish breaks natural language processors

#40

Post by isani »

szatkus wrote: post #19
plq wrote: post #10

These are common problems for languages with complex morphology. Finnish, Japanese, Turkish and Hungarian are closely related in this sense and share a lot of NLP research. While implementing spell checkers for these languages need a bit more effort than just compiling a list of words, it's far from an unsolved problem. AFAIK Zemberek[1] is a Turkish spellchecker that implements two-level finite state morphology. Als…

What's the problem with Japanese? It's highly regular language, so it should be easy to tokenize. AFAIR the whole language has maybe few irregular verbs, compared to few hundreds in English.

Japanese is usually written without spaces. Words and sentences just run into each other. When writing in hiragana (syllabic characters), word boundaries are often ambiguous.

Englishwouldbemuchhardertoparseifwrittenlikethis.

Post reply on HN