Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Finnish breaks natural language processors

twitter.com

Post reply on HN
Print view View original

11–20 of 131 posts

Re: Finnish breaks natural language processors

#11

Post by BerislavLopac »

stevekemp wrote: post #8
hpaavola wrote: post #3

Finn here. There are countless memes about oddities of our language. The one I like the most is "kuusi palaa". It can mean; the spruce is on fire the spruce returns the number six is on fire the number six returns six of them are on fire six of them return your moon is on fire your moon returns six pieces Good luck for all implementing NLP :)

I like the "dog example" too, which highlights the different forms/cases: https://linustechtips.com/main/topic/72936-english-swedish-g... I'm a recent immigrant to Finland, and while the regular grammar makes some things simple, the cases and suffixes are enough to drive me crazy! The simple things are easy to handle if you're reading a textbook, but remembering them in real-time when speaking is very hard.

No wonder Tolkien used it as a basis for Elvish.

Re: Finnish breaks natural language processors

#12

Post by szatkus »

miki123211 wrote: post #5

Polish has some of those problems too, at least the ones with conjugations. The words mom (appearing in your contact list), mom (as in call mom) and mom (as in send money to mom) are different. That applies to proper names too, and it isn't regular. It's not easy to figure oujt who the user refers to if you just have the contacts list and the command. Polish had problems way before nlp, for example "Annie has sent yo…

I started learning Finnish and it looks much worse. Twice as much cases, consonant gradation and puhekieli. Everything else seems on similar difficulty level as in Polish (maybe except genders, but it's pretty regular).

But I may not have a proper perspective as Polish is my native language and I don't work with NLP (although I created a small tool to determine a word based on its inflected form with data from Wiktionary).

Re: Finnish breaks natural language processors

#13

Post by htns »

hibbelig wrote: post #2

Why can't the NLP split the compounds? I think this problem of splitting happens in multiple languages -- the submission mentions German, and then you have Chinese which doesn't have spaces, so you have to split words. (I understand that splitting Japanese into words is simpler.)

Another complication not mentioned in the tweets is consonant gradation, in which the stem changes in dozens of ways when suffixed (examples: http://users.jyu.fi/~pamakine/kieli/suomi/vaihtelu/astevaiht...), which makes Finnish a lot less cleanly agglutinative than most other languages. It's not like it can't be done, but it's yet another step (find the likely split points, then try to undo the consonant gradation) and I'd imagine casual foreign researchers aren't going to bother.

Re: Finnish breaks natural language processors

#14

Post by ajuc »

miki123211 wrote: post #5

Polish has some of those problems too, at least the ones with conjugations. The words mom (appearing in your contact list), mom (as in call mom) and mom (as in send money to mom) are different. That applies to proper names too, and it isn't regular. It's not easy to figure oujt who the user refers to if you just have the contacts list and the command. Polish had problems way before nlp, for example "Annie has sent yo…

All Slavic languages are quite quirky in this regard. Most native speakers don't realize it, but when you try to write correct message parametrized with numbers in Polish, you need 3 or 4 special cases:

     - 1
     - 2-4, 22-24, 32-34, ...
     - 5-21, 25-31, 35-41, ...
If you say "out of X" you also need to handle numbers 100-199, 100 000-199 000, 100 000 000-199 000 000, ... separately (because 100 = sto, and if number begins with "s" - "out of X" is "ze" instead if "z") - this is in combination with previous 3 cases, so "199 ouf of 199" is different from "99 out of 99", and different from "122 out of 122".

So "Copied X file(s) out of Y" would be:

     - "Skopiowano 1 plik"
     - "Skopiowano 2 pliki z 3"
     - "Skopiowano 5 plików z 5"
     - "Skopiowano 1 plik ze 100"
     - "Skopiowano 2 pliki ze 100"
     - "Skopiowano 5 plików ze 100".
Almost no software handles this correctly :) But spell checkers work OK (they just ignore relationships between words usually).

Re: Finnish breaks natural language processors

#16

Post by balfirevic »

miki123211 wrote: post #5

Polish has some of those problems too, at least the ones with conjugations. The words mom (appearing in your contact list), mom (as in call mom) and mom (as in send money to mom) are different. That applies to proper names too, and it isn't regular. It's not easy to figure oujt who the user refers to if you just have the contacts list and the command. Polish had problems way before nlp, for example "Annie has sent yo…

Croatian is the same. Noun declination makes some sense, as it sometimes allows you to omit prepositions (although I sure am glad English doesn't do that). But gendered verbs are just stupid.

Re: Finnish breaks natural language processors

#18

Post by jacobush »

Whatitat90 wrote: post #6
hpaavola wrote: post #3

Finn here. There are countless memes about oddities of our language. The one I like the most is "kuusi palaa". It can mean; the spruce is on fire the spruce returns the number six is on fire the number six returns six of them are on fire six of them return your moon is on fire your moon returns six pieces Good luck for all implementing NLP :)

I wonder what oddities lie in Linus Torvalds' "perkeleen vittupää" ( https://lkml.org/lkml/2013/7/13/132 ).

I'm afraid it very unambiguously just means fucking cunts.

Re: Finnish breaks natural language processors

#19

Post by szatkus »

plq wrote: post #10

These are common problems for languages with complex morphology. Finnish, Japanese, Turkish and Hungarian are closely related in this sense and share a lot of NLP research. While implementing spell checkers for these languages need a bit more effort than just compiling a list of words, it's far from an unsolved problem. AFAIK Zemberek[1] is a Turkish spellchecker that implements two-level finite state morphology. Als…

What's the problem with Japanese? It's highly regular language, so it should be easy to tokenize.

AFAIR the whole language has maybe few irregular verbs, compared to few hundreds in English.

Re: Finnish breaks natural language processors

#20

Post by breakingcups »

pakitan wrote: post #7

So, does that mean that if OpenAI's text generator turns out to be as "dangerous" as it's being marketed, we can all just switch to Finnish as lingua franca and Skynet won't have a chance? :)

That implies we can all "just" switch. I have a feeling Skynet will have a better chance than I do!
Post reply on HN