Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Finnish breaks natural language processors

twitter.com

Post reply on HN
Print view View original

91–100 of 131 posts

Re: Finnish breaks natural language processors

#91

Post by woliveirajr »

MereInterest wrote: post #68
saagarjha wrote: post #55

Earlier quoted context omitted.

I guess that’s where context will have to be considered. Those two are valid sentences, so presumably humans are using context to distinguish between them, right?

The murderer came to my dinner party, and I had it all planned. In one of the ice cubes, I had frozen arsenic. The murderer would eat the same food, drink the same drink, and nobody would guess that they would die on leaving. When the evening was over, I knew what I would tell people. Justicehadbeenserved.

Please, share this with the world on tweeter.

Re: Finnish breaks natural language processors

#92

Post by Sharlin »

a_imho wrote: post #46
nyagaga wrote: post #42

Earlier quoted context omitted.

I don't know what you mean exactly, but genders can be something intrinsic to a language and I don't think they have been introduced as such. If you wanted to call a table in Italian, you could choose the masculine or feminine noun for it (tavolo - tavola), no neuter form though. Ask the table first.

Why would it be intrinsic? Somewhere in their evolution they must have been introduced for some reason I would guess. Also, they could be removed the same way if there is no benefit distinguishing between a masculine and feminine table.

PIE (Proto-Indo-European) was most likely a gendered language. So the concept goes deep into prehistory. English lost its gender system fairly recently, during the Middle English period.

Re: Finnish breaks natural language processors

#93

Post by xg15 »

usrusr wrote: post #69
xg15 wrote: post #51

Earlier quoted context omitted.

A pragmatic advantage is that you use different pronouns to refer to multiple different persons (or things, with grammatical gender) in the same sentence without ambiguity. In programming terms, you have a "local variable" slot for each pronoun, instead of just a single one with gender neutrality.

Like having separate registers for even and for odd numbers. Now try building a good compiler for that... (no bitshift trickery allowed)

I don't think that analogy works. You're not forced to use the gendered forms - you can just say "the person", "them", etc - and (at least as far as the language is concerned), it's only about communication, not about reasoning. So your "calculations" are unaffected.

Re: Finnish breaks natural language processors

#94

Post by gizmo686 »

foldr wrote: post #89
gizmo686 wrote: post #85

Earlier quoted context omitted.

Are you getting at the distinction between "will" being the T and "will" being the daughter of T? Regardless, under either analysis "will" would have just as much claim to being a tense as "run" has to being a verb. You can still argue over wheather there is a syntactic feature of future tense (which is a different question) which I have never seen a solid arguement for either way.

>Regardless, under either analysis "will" would have just as much claim to being a tense as "run" has to being a verb. If you're punning on "tense" to refer to a lexical category, then sure. But we're talking about tense in the sense of past, present and future. In many textbook analyses, lots of things go in T. For example, "can" and "might" go in T. It doesn't follow that, say, "I might go to the store" and "I can…

I would say that "I can/might go to the store" go to the store having different tenses is what we mean when we say "can" and "might" are lexixally T.

Even restricting ourselves to the syntactic features, I have still never seen an arguement for or against English having a syntactic future tense feature that did not come down to UG, or a disagreement about which model is simpler, and neither side is convoluted enough for me to confidently invoke Occam's razor.

Re: Finnish breaks natural language processors

#95

Post by Grue3 »

szatkus wrote: post #19
plq wrote: post #10

These are common problems for languages with complex morphology. Finnish, Japanese, Turkish and Hungarian are closely related in this sense and share a lot of NLP research. While implementing spell checkers for these languages need a bit more effort than just compiling a list of words, it's far from an unsolved problem. AFAIK Zemberek[1] is a Turkish spellchecker that implements two-level finite state morphology. Als…

What's the problem with Japanese? It's highly regular language, so it should be easy to tokenize. AFAIR the whole language has maybe few irregular verbs, compared to few hundreds in English.

You can do it for dictionary words with a relatively high degree of accuracy, but it's very hard. I have developed the most accurate tokenizer that currently exists [1], but there are still sentences where it fails. Words written in hiragana create an inherent ambiguity because almost any combination of hiragana can be split into existing words in several ways.

Another problem is proper names. This one is inherently unsolvable, because anything can be a word. You can have a dictionary of proper names, but there will still be people, companies, fictional characters who aren't in the dictionary.

[1] https://github.com/tshatrov/ichiran

Re: Finnish breaks natural language processors

#96

Post by babuskov »

ajuc wrote: post #35
babuskov wrote: post #33

Earlier quoted context omitted.

> Almost no software handles this correctly Gettext solved it 20 years ago. The problem is that people try to reinvent the wheel instead of looking at existing solutions. So, they make their own inferior versions. The rule for plural has 3 cases. 1. When n mod 10 = 1 2. When n mod 10 is 2,3,4 3. Everything else And there's a special rule that the first 2 cases are not used when n mod 100 is between 11 and 29. For suc…

Doesn't solve the problem with "X out of Y".

It does. You have two numbers, so you have to split it into X and Y part. When I look at your example, 1 plik is always 1 plik regardless of the Y quantity and 100 always go with "ze" regardless of X quantity. It's a little bit tricky to set up, but it works. "Copied X of Y files" needs to split into two translation units: "Copied X" and "of Y files".

Re: Finnish breaks natural language processors

#97

Post by frosted-flakes »

It's interesting that Nokia's auto-completion worked well for Finnish, but modern solutions don't work.

> Joose Rajamäki 🇫🇮🇪🇺 @joose_rajamaeki Feb 20 > Yes, autocompletion [on phones] exists. But I hardly ever manage to compose a message where it wouldn't encounter new words. Also, it doesn't know the inflections, so it needs to encounter each word in each possible form before suggesting them.

> Joose Rajamäki 🇫🇮🇪🇺 @joose_rajamaeki Feb 20 > Old Nokia phones were good. They let you indicate that the word root was finished and you wanted to add agglutination and/or inflections.

Re: Finnish breaks natural language processors

#98

Post by smcl »

pndy wrote: post #23
miki123211 wrote: post #5

Polish has some of those problems too, at least the ones with conjugations. The words mom (appearing in your contact list), mom (as in call mom) and mom (as in send money to mom) are different. That applies to proper names too, and it isn't regular. It's not easy to figure oujt who the user refers to if you just have the contacts list and the command. Polish had problems way before nlp, for example "Annie has sent yo…

> The words mom (appearing in your contact list), mom (as in call mom) and mom (as in send money to mom) are different. Each time I tried WP8.1 voice assistant to "zadzwoń do mama" it felt me extremely awkward due to nominative case of the word "mama". It should be "zadzwoń do mamy" which is genitive case, if I'm not mistaken, that would make whole phrase sound natural.

Interesting, Polish and Czech are so alike and so different :D I guess the word Polish "mama" is pretty much the Czech "mamka" which similarly declines to "mamky" in the genitive case, but "do mamky" here is "into mum" which is ... rather odd (I think Czech would use the Dative "mamce" - so “zavolej mamce” or “zavolejte mamce” depending on your relationship with your phone!)

I get tripped up when, for example, talking about getting presents from people and never seem to use the correct preposition - "od" or "z". I seem to use the one that would implied "came out of" not "came from"

Re: Finnish breaks natural language processors

#100

Post by peterwwillis »

a_imho wrote: post #36
DocG wrote: post #9

There was a joke that Estonian is more closer to Japanese than any other European language. As to not say its related to Japanese, but how much it differs from European ones. No future tense, no genders, but 14 cases. This illustrates nicely where finno-ugric languages(Finnish, Hungarian, Estonian) reside compared to other European languages. https://images.mentalfloss.com/sites/default/files/196.jpg Benefit is that…

What is the benefit of genders anyway? Especially when gender neutral alternatives are available. It never made much sense to me, they introduce a lot of complexity for very little. Worse, nowadays you have all the problems of misaddressing and offending folks.

  "Tommy, Frank, and Susan burned their homework and fed it to the dog."
  "Who fed it to the dog?"
  "That person."
Overhearing this conversation, you don't know who they were pointing at. If they said "She did.", you know who they were pointing at. There are many other situations where gender provides extra data, which is convenient.

Our languages keep having gender, so it's clearly been advantageous from an evolutionary perspective. The problem is when we let gender become more important than the data it's trying to express, like when a society lets gender dictate your social class. That shit needs to stop.

Post reply on HN