Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Finnish breaks natural language processors

twitter.com

Post reply on HN
Print view View original

111–120 of 131 posts

Re: Finnish breaks natural language processors

#111

Post by yorwba »

Grue3 wrote: post #95
szatkus wrote: post #19

Earlier quoted context omitted.

What's the problem with Japanese? It's highly regular language, so it should be easy to tokenize. AFAIR the whole language has maybe few irregular verbs, compared to few hundreds in English.

You can do it for dictionary words with a relatively high degree of accuracy, but it's very hard. I have developed the most accurate tokenizer that currently exists [1], but there are still sentences where it fails. Words written in hiragana create an inherent ambiguity because almost any combination of hiragana can be split into existing words in several ways. Another problem is proper names. This one is inherently…

How do you test the accuracy of your tokenizer? Is there a database of ground-truth tokenized sentences that you evaluate on? It would be nice to see some kind of benchmark results comparing with MeCab.

Re: Finnish breaks natural language processors

#112

Post by checkyoursudo »

wolfgke wrote: post #57
iainmerrick wrote: post #39

Earlier quoted context omitted.

There was an interesting/amusing post here the other day about learning German and how it relates to programming concepts. They suggested that gendered words work as a kind of parity bit for error correction.

> There was an interesting/amusing post here the other day about learning German and how it relates to programming concepts. They suggested that gendered words work as a kind of parity bit for error correction. Slightly off topic (not about grammatical genus, but about German's case system): I often compare German's case system to a type system in a programming language: the verb expects the sentence's object in the…

I don't think you're wrong, but my experience has some nuanced differences maybe.

I have known plenty of German speakers, including natives and often myself, who make case/gender mistakes with no real impact on comprehension.

There certainly can be some ambiguity, but context and word order can often mitigate that, no? I feel like if you used "die" for everything and just a neuter nominative case, I could still figure most things out. My experience is that many German speakers (especially when speaking a dialect) just mush articles and suffixes together and it's not entirely clear that they use specific genders and cases most of the time anyway. For example, my relatives near Frankfurt pretty much say "d- Milch", "d- Hund", "d- Mauer", etc. SOMETIMES, you can hear a bit of an "s" when it's supposed to be das, des. YMMV of course.

I don't really find German to be less forgiving of mistakes than English or the handful of other languages I've studied.

At least, this what I take away from your comment about compile errors. Maybe -Wcase or -Wgender is a closer analogy?

Anyway, I like your analogy overall. :)

Re: Finnish breaks natural language processors

#113

Post by foldr »

DFHippie wrote: post #104
foldr wrote: post #81

Earlier quoted context omitted.

It is indeed the case that (spoken) French and German have no past tense, strictly speaking. (I don't speak German, so I don't know to exactly what extent the German imperfect is unused in everyday speech. I think it may not be as unused as the passé simple in French.) Tense is a piece of technical terminology with a particular definition. There's nothing very peculiar about the definition that I can see. The definit…

It's not a question of taking linguists' word for it. It's a question of taking particular linguists' word for it. The discussion in that language log is interesting. You will notice that it isn't simply people calling "amen". Certain things are settled: "will" in English behaves like other English auxiliary verb (setting aside those cases where it doesn't, as in contractions). Saying "English has no future tense", t…

>is settled only within the confines of a particular linguistic model with a particular technical definition of the term "tense", of which there are many.

Could you then provide a reference to a linguist arguing that English has a future tense (in the usual syntactic/morphological sense of the term)?

I'm baffled by the reference to the "confines of a particular linguistic model". Linguists who can agree on hardly anything else (and linguists are pretty good at disagreeing!) can agree that English lacks a future tense.

Re: Finnish breaks natural language processors

#114

Post by wolfgke »

bravura wrote: post #107
wolfgke wrote: post #57

Earlier quoted context omitted.

> There was an interesting/amusing post here the other day about learning German and how it relates to programming concepts. They suggested that gendered words work as a kind of parity bit for error correction. Slightly off topic (not about grammatical genus, but about German's case system): I often compare German's case system to a type system in a programming language: the verb expects the sentence's object in the…

I am trying to learn German and gender is infuriating there. Why? Because you can't simply scan an existing sentence and color-code the words based upon simple heuristics to guess the gender. The first thing you learn is that three genders are indicated in the nominative case using der (masculine) / die (feminine) / das (neuter). So you see "der" in a sentence and that thing must be masculine, right? Nope, because "d…

> The first thing you learn is that three genders are indicated in the nominative case using der (masculine) / die (feminine) / das (neuter).

I personally would never teach it that way. To me the four cases are like different faces of, say, a cube (I know a cube has six faces - sorry), where the cube represents the noun. Just like looking at a cube from different perspectives (in the direction of some face) gives you a different view on it, each case gives a different "perspective" of a way that the word relates to some different clause in a sentence.

How the articles and endings "rotate" when the case changes ("as the cube rotates") is something that I developed a really smart scheme for (which can be found in no book on German grammar that I am aware of) that every "mathematical-minded person" immediatelly grasped, told me it perfectly made sense and wished that someone had taught it this way in school. On the other hand, nearly every "humanities-minded person" told me that they cannot understand what I am even talking about and what advantage this perspective is supposed to have.

Typical language teachers are "humanities-minded". :-(

Unluckily, I have far too less time to write this all down in sufficient detail that other people can learn from it; I have far too many other things to do. :-(

> Imagine if in your type system "float" could mean two different types, and you needed more information to figure out which type it is (like, for example, memorizing the type of the object its referring to).

You are describing declarations and definitions in C++ and the problems of parsing C++? ;-)

Re: Finnish breaks natural language processors

#115

Post by Grue3 »

yorwba wrote: post #111
Grue3 wrote: post #95

Earlier quoted context omitted.

You can do it for dictionary words with a relatively high degree of accuracy, but it's very hard. I have developed the most accurate tokenizer that currently exists [1], but there are still sentences where it fails. Words written in hiragana create an inherent ambiguity because almost any combination of hiragana can be split into existing words in several ways. Another problem is proper names. This one is inherently…

How do you test the accuracy of your tokenizer? Is there a database of ground-truth tokenized sentences that you evaluate on? It would be nice to see some kind of benchmark results comparing with MeCab.

It's not a matter of benchmarks really. It's more of a matter of MeCab simply not knowing that certain grammar exists. For example 〜ちゃった for 〜てしまった is a pretty common form. MeCab doesn't understand the word 忘れちゃった because it only knows 忘れて and しまった as completely separate words and can't combine them in any way, but Ichiran can because my word-form generator is much more sophisticated. This is also why I cannot compare the results of segmentation directly: I combine words and their suffixes into a single "compound word" which has a tree-like recursive structure [1]. MeCab and other tokenizers can only detect basic forms and leave suffixes as a completely separate "word". Which makes them automatically inferior.

[1] https://ichi.moe/cl/qr/?q=%E5%BF%98%E3%82%8C%E3%81%A1%E3%82%...

Re: Finnish breaks natural language processors

#116

Post by atombender »

peterwwillis wrote: post #100
a_imho wrote: post #36

Earlier quoted context omitted.

What is the benefit of genders anyway? Especially when gender neutral alternatives are available. It never made much sense to me, they introduce a lot of complexity for very little. Worse, nowadays you have all the problems of misaddressing and offending folks.

"Tommy, Frank, and Susan burned their homework and fed it to the dog." "Who fed it to the dog?" "That person." Overhearing this conversation, you don't know who they were pointing at. If they said "She did." , you know who they were pointing at. There are many other situations where gender provides extra data, which is convenient. Our languages keep having gender, so it's clearly been advantageous from an evolutionar…

But that's human gender, not noun gender. In grammar, gender is a class of noun and generally unrelated to human gender.

In German, words like "library" (Bibliothek) and "street" (Strasse/Straße) are feminine, for example. It's true that some feminine words, such as "woman" (Frau) and "mother" (Mutter), are feminine. But then you have words like "girl" (Mädchen), which are not.

Your example ("she did") could actually have ambiguity in languages that extends the noun gender to pronouns, such as French. "She" doesn't need to refer to a woman, just a feminine noun. For example, "car" (voiture) is feminine, and you can refer to "it" as "elle" (she).

Re: Finnish breaks natural language processors

#117

Post by Balgair »

a_imho wrote: post #36
DocG wrote: post #9

There was a joke that Estonian is more closer to Japanese than any other European language. As to not say its related to Japanese, but how much it differs from European ones. No future tense, no genders, but 14 cases. This illustrates nicely where finno-ugric languages(Finnish, Hungarian, Estonian) reside compared to other European languages. https://images.mentalfloss.com/sites/default/files/196.jpg Benefit is that…

What is the benefit of genders anyway? Especially when gender neutral alternatives are available. It never made much sense to me, they introduce a lot of complexity for very little. Worse, nowadays you have all the problems of misaddressing and offending folks.

> Worse, nowadays you have all the problems of misaddressing and offending folks.

Languages are tightly inherited, even more so than genes. How languages are structured varies highly. Some have gender-like tenses for the cardinal directions, where things are not feminine but 'northy'. Some have no genders, but do have time-like tenses, where things are 'future-y' or 'present-y'. Just look at sign-language, with it's use of height similar to an exclamation point in English at times. Human communication is weird.

Even in English, our sense of gender has changed over time. It used to be that we cared more about the number of items/people over their gender, but now we care about the gender. IE: Thee/Thou/You/ became Him/Her/You.

https://en.wikipedia.org/wiki/Gender_in_English

English is, herself, a pidgin language. The irregularity of the verbs and the reliance on context make it very accommodating to new terms and nouns, but VERY difficult to learn fluently. It is standardizing though, the Afro/Urban-American vernacular use of 'be' is a perfect example (I be going to the store, not, I am going to the store). This change of 'be' is very similar to the romance languages. English is very slowly being wrenched into standardization. It will be very interesting to see how Chinese and Computer Programming languages affect English.

The Story of English is a great look into this language and it's story:

https://www.youtube.com/watch?v=Mj9jJiPwsp0

Re: Finnish breaks natural language processors

#118

Post by kbouck »

arka2147483647 wrote: post #70
hpaavola wrote: post #3

Finn here. There are countless memes about oddities of our language. The one I like the most is "kuusi palaa". It can mean; the spruce is on fire the spruce returns the number six is on fire the number six returns six of them are on fire six of them return your moon is on fire your moon returns six pieces Good luck for all implementing NLP :)

This is a specificially constructed corner case, not a general finnish thing: - kuu = moon - kuusi = spruce - palaa = returns - pala = piece From those you can see how one can intrepent the compound word (with or without suffix) however you wish. Eg: - kuu+si = your moon (where 'si' is the suffix which means 'your') - pala+a = pieces (where 'a' suffix is the plural)

also:

- kuusi = six

Re: Finnish breaks natural language processors

#119

Post by gtrfreewifi »

szatkus wrote: post #12
miki123211 wrote: post #5

Polish has some of those problems too, at least the ones with conjugations. The words mom (appearing in your contact list), mom (as in call mom) and mom (as in send money to mom) are different. That applies to proper names too, and it isn't regular. It's not easy to figure oujt who the user refers to if you just have the contacts list and the command. Polish had problems way before nlp, for example "Annie has sent yo…

I started learning Finnish and it looks much worse. Twice as much cases, consonant gradation and puhekieli. Everything else seems on similar difficulty level as in Polish (maybe except genders, but it's pretty regular). But I may not have a proper perspective as Polish is my native language and I don't work with NLP (although I created a small tool to determine a word based on its inflected form with data from Wiktio…

Puhekiäli voi olla vähä vaikiaa, mutta sitten on viälä eriksensä murtehet.

Re: Finnish breaks natural language processors

#120

Post by ubermonkey »

DocG wrote: post #9

There was a joke that Estonian is more closer to Japanese than any other European language. As to not say its related to Japanese, but how much it differs from European ones. No future tense, no genders, but 14 cases. This illustrates nicely where finno-ugric languages(Finnish, Hungarian, Estonian) reside compared to other European languages. https://images.mentalfloss.com/sites/default/files/196.jpg Benefit is that…

That's a great graphic, but where are Arabic and Georgian?
Post reply on HN