Live data from Hacker News

Finnish breaks natural language processors

twitter.com

41–50 of 131 posts

Re: Finnish breaks natural language processors

#41

Polish has some of those problems too, at least the ones with conjugations. The words mom (appearing in your contact list), mom (as in call mom) and mom (as in send money to mom) are different. That applies to proper names too, and it isn't regular. It's not easy to figure oujt who the user refers to if you just have the contacts list and the command. Polish had problems way before nlp, for example "Annie has sent yo…

Croatian is the same. Noun declination makes some sense, as it sometimes allows you to omit prepositions (although I sure am glad English doesn't do that). But gendered verbs are just stupid.

"gendered" verbs are a thing in Italian too: "I fell" becomes "sono caduto" or "sono caduta" depending on the gender of the speaker. Transitive verbs reflect the gender of the objects when using clitics, e.g. "I ate the apple, I ate it really" -> "Ho mangiato la mela, l'ho mangiata davvero".

What's odd in Croatian is that the all the verbs behave as intransitive Italian verbs, reflecting the gender of the speaker: "I fell" -> "pao sam" or "pala sam" but also "pojeo sam jabuku" vs "pojela sam jabuku" if the speaker is a woman.

I'm not a linguist, but as a native speaker of both, I always thought it had something to do with the fact that in Croatian the verb "to be" is the only auxiliary verb used to built past tenses while in Italian transitive verbs use the verb "to have".

The past participles in many languages behave very similarly to adjectives and thus I thought it might not be surprising that you'd decline them according to number and gender as you'd do with any adjective.

So if when you fall you "are fallen", you decline the "fallen" participle/adjective according to the subject, namely you in this case.

Thus instead of "have eaten" Croatian says "am eaten" (not meaning the passive voice but just using the verb to be as axillary instead of have) as a consequence you notice the "gendered verbs" more often than in other languages.

As to why Italian declines participles when clitics are involved, I assume it adds some redundancy so you can more easily guess what the clitic refers to.

Re: Finnish breaks natural language processors

#42
post #36
post #9

There was a joke that Estonian is more closer to Japanese than any other European language. As to not say its related to Japanese, but how much it differs from European ones. No future tense, no genders, but 14 cases. This illustrates nicely where finno-ugric languages(Finnish, Hungarian, Estonian) reside compared to other European languages. https://images.mentalfloss.com/sites/default/files/196.jpg Benefit is that…

What is the benefit of genders anyway? Especially when gender neutral alternatives are available. It never made much sense to me, they introduce a lot of complexity for very little. Worse, nowadays you have all the problems of misaddressing and offending folks.

I don't know what you mean exactly, but genders can be something intrinsic to a language and I don't think they have been introduced as such. If you wanted to call a table in Italian, you could choose the masculine or feminine noun for it (tavolo - tavola), no neuter form though. Ask the table first.

Re: Finnish breaks natural language processors

#43
post #10

These are common problems for languages with complex morphology. Finnish, Japanese, Turkish and Hungarian are closely related in this sense and share a lot of NLP research. While implementing spell checkers for these languages need a bit more effort than just compiling a list of words, it's far from an unsolved problem. AFAIK Zemberek[1] is a Turkish spellchecker that implements two-level finite state morphology. Als…

But a spell checker is just a tiny part of the NLP domain.

When people talk about NLP, they are usually referring to understanding the meaning of the entire text being analyzed, rather than just providing meanings for individual words. A spell checker can't tell you the intent or subtleties of what is being conveyed.

Re: Finnish breaks natural language processors

#45
post #9

There was a joke that Estonian is more closer to Japanese than any other European language. As to not say its related to Japanese, but how much it differs from European ones. No future tense, no genders, but 14 cases. This illustrates nicely where finno-ugric languages(Finnish, Hungarian, Estonian) reside compared to other European languages. https://images.mentalfloss.com/sites/default/files/196.jpg Benefit is that…

>No future tense

Many European languages also lack a future tense. English, for example. (https://www.quora.com/Is-it-true-that-English-has-no-future-...)

Re: Finnish breaks natural language processors

#46
post #42
post #36

Earlier quoted context omitted.

What is the benefit of genders anyway? Especially when gender neutral alternatives are available. It never made much sense to me, they introduce a lot of complexity for very little. Worse, nowadays you have all the problems of misaddressing and offending folks.

I don't know what you mean exactly, but genders can be something intrinsic to a language and I don't think they have been introduced as such. If you wanted to call a table in Italian, you could choose the masculine or feminine noun for it (tavolo - tavola), no neuter form though. Ask the table first.

Why would it be intrinsic? Somewhere in their evolution they must have been introduced for some reason I would guess. Also, they could be removed the same way if there is no benefit distinguishing between a masculine and feminine table.

Re: Finnish breaks natural language processors

#47
post #36
post #9

There was a joke that Estonian is more closer to Japanese than any other European language. As to not say its related to Japanese, but how much it differs from European ones. No future tense, no genders, but 14 cases. This illustrates nicely where finno-ugric languages(Finnish, Hungarian, Estonian) reside compared to other European languages. https://images.mentalfloss.com/sites/default/files/196.jpg Benefit is that…

What is the benefit of genders anyway? Especially when gender neutral alternatives are available. It never made much sense to me, they introduce a lot of complexity for very little. Worse, nowadays you have all the problems of misaddressing and offending folks.

To make things 100x complicated in our "modern" world :)

For example you can select a gender pronoun in English to match your gender. https://uwm.edu/lgbtrc/support/gender-pronouns/

In Hungarian we just say "Ő" to refer to someone, privacy is baked into the language :)

Re: Finnish breaks natural language processors

#48

Turkish being a member of ural-altaic language family, just like Finnish, suffers from similar problems. Google Translate improved much over the years yet it still generates laughable text at best. Stemmers used to create show-stopper word stems (don't really know the current situation). Although, it imported a lot of technical terms from various european languages, making technical texts seem to be more legible due…

Ural-Altaic language family is today considered an obsolete concept [1] and the families are considered unrelated

[1] https://en.wikipedia.org/wiki/Ural%E2%80%93Altaic_languages

Re: Finnish breaks natural language processors

#49
post #7

So, does that mean that if OpenAI's text generator turns out to be as "dangerous" as it's being marketed, we can all just switch to Finnish as lingua franca and Skynet won't have a chance? :)

That implies we can all "just" switch. I have a feeling Skynet will have a better chance than I do!

I've been trying to learn and "switch" to Finnish for 5 years. I can confirm this.

Re: Finnish breaks natural language processors

#50
post #40
post #19

Earlier quoted context omitted.

What's the problem with Japanese? It's highly regular language, so it should be easy to tokenize. AFAIR the whole language has maybe few irregular verbs, compared to few hundreds in English.

Japanese is usually written without spaces. Words and sentences just run into each other. When writing in hiragana (syllabic characters), word boundaries are often ambiguous. Englishwouldbemuchhardertoparseifwrittenlikethis.

I have no stake in natural language processing, but it looks to me like a computer might be able to do a pretty good job at splitting that given a dictionary.
Post reply on HN