Live data from Hacker News

Finnish breaks natural language processors

twitter.com

21–30 of 131 posts

Re: Finnish breaks natural language processors

#21
My suspicion is that most NLP techniques were invented by native speakers of "easy" languages

Though romance languages/germanic languages (and even English ancestors) have their quirks they're not at Finno-Ugric levels

(Not that English has no quirks - especially in pronunciation - but it is "easy" to deal with most of the weird exceptions)

Re: Finnish breaks natural language processors

#22
post #14

Polish has some of those problems too, at least the ones with conjugations. The words mom (appearing in your contact list), mom (as in call mom) and mom (as in send money to mom) are different. That applies to proper names too, and it isn't regular. It's not easy to figure oujt who the user refers to if you just have the contacts list and the command. Polish had problems way before nlp, for example "Annie has sent yo…

All Slavic languages are quite quirky in this regard. Most native speakers don't realize it, but when you try to write correct message parametrized with numbers in Polish, you need 3 or 4 special cases: - 1 - 2-4, 22-24, 32-34, ... - 5-21, 25-31, 35-41, ... If you say "out of X" you also need to handle numbers 100-199, 100 000-199 000, 100 000 000-199 000 000, ... separately (because 100 = sto, and if number begins w…

Almost no software handles this correctly because almost no software gives us the tools to handle it correctly. I've translated some software to polish once or twice and they've been using a simple templating system, like message from {name}: {content}, {time} minutes ago. I didn't have access to the source code. As a consequence of that design, some messages were very malformed. Also there were some overused strings, like "liked" for "You've just liked this", "liked" for "things you've liked" and "liked" as in "someone liked your post". Also untranslatable.

Re: Finnish breaks natural language processors

#23

Polish has some of those problems too, at least the ones with conjugations. The words mom (appearing in your contact list), mom (as in call mom) and mom (as in send money to mom) are different. That applies to proper names too, and it isn't regular. It's not easy to figure oujt who the user refers to if you just have the contacts list and the command. Polish had problems way before nlp, for example "Annie has sent yo…

> The words mom (appearing in your contact list), mom (as in call mom) and mom (as in send money to mom) are different.

Each time I tried WP8.1 voice assistant to "zadzwoń do mama" it felt me extremely awkward due to nominative case of the word "mama". It should be "zadzwoń do mamy" which is genitive case, if I'm not mistaken, that would make whole phrase sound natural.

Re: Finnish breaks natural language processors

#24

Earlier quoted context omitted.

I wonder what oddities lie in Linus Torvalds' "perkeleen vittupää" ( https://lkml.org/lkml/2013/7/13/132 ).

I'm afraid it very unambiguously just means fucking cunts.

In singular genetive and nominative cases no less, so, absolutely no grammatical fancy going on there.

Re: Finnish breaks natural language processors

#26
post #14

Earlier quoted context omitted.

All Slavic languages are quite quirky in this regard. Most native speakers don't realize it, but when you try to write correct message parametrized with numbers in Polish, you need 3 or 4 special cases: - 1 - 2-4, 22-24, 32-34, ... - 5-21, 25-31, 35-41, ... If you say "out of X" you also need to handle numbers 100-199, 100 000-199 000, 100 000 000-199 000 000, ... separately (because 100 = sto, and if number begins w…

Almost no software handles this correctly because almost no software gives us the tools to handle it correctly. I've translated some software to polish once or twice and they've been using a simple templating system, like message from {name}: {content}, {time} minutes ago. I didn't have access to the source code. As a consequence of that design, some messages were very malformed. Also there were some overused strings…

Qt framework has system in place for several plural forms, so plik/pliki/plików problem is solved, and quite elegantly. If the application programmer gives a damn, that is.

There's also a way to add context to the strings for translation, so that you can see this "like" is used in this meaning, and that other "like" is used in another meaning.

But there's no generic solution to "z/ze" problem as far as I know, you need to do it by hand if you want correct messages. And most of the time programmers don't even realize they need to parametrize such stuff, so the systems don't do any good :)

Re: Finnish breaks natural language processors

#27
Turkish being a member of ural-altaic language family, just like Finnish, suffers from similar problems. Google Translate improved much over the years yet it still generates laughable text at best. Stemmers used to create show-stopper word stems (don't really know the current situation).

Although, it imported a lot of technical terms from various european languages, making technical texts seem to be more legible due to lack of composited words in these contexts.

similarly from tweets: yiyecek = food yiyecek miydi = will he/she eat that (2nd word is not a separate word, just a conjunction that is written separately)

göz = eye gözcü = scout gözlük = glasses gözlükçü = glass salesman gözcülük = scouting gözlükçüler = glass salesmen gözlükçüydüler = they were glass salesmen gözlükçü müydüler = were they glass salesmen?

also an all time classic: çekoslavakyalılaştıramadıklarımızdan mısınız = are you one of those people whom we tried unsuccessfully to assimilate into a Czechoslovakian citizen?

Re: Finnish breaks natural language processors

#29
post #25

I still haven't figured out how Germans use swipe keyboards on smartphones with the monster compound sentence-words. If they do.

On iOS, the keyboard just suggests compounds. This works for those you use often. Another option is to choose a prefix, then backspace (over the space), then add to the prefix.

Example compound word: Schleifmaschinenverleih (word boundaries: Schleif'maschinen'verleih)

You begin with "Schl" which might suggest "Schleifer", then you delete "er", add "m", and if you're lucky, you get "Schleifmaschine".

"Schleifen" means to grind. "Maschine" means machine. So "Schleifmaschine" is grinding machine, and the disappearance of "en" is a grammatical artifact. "Verleih" means rental (as in a company that does rentals).

Re: Finnish breaks natural language processors

#30
> tietokone = knowledge machine (literally) = computer

This is somewhat beside the point, but although in everyday speech "tieto" usually means knowledge in the sense of knowing a fact or skill, it can also be taken as "information", which seems more likely to have been the original intent of the term.

Nevertheless, pretty much every elementary computing course in Finnish begins by stating that "the computer doesn't actually know anything". :)

Post reply on HN