Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Finnish breaks natural language processors

twitter.com

Post reply on HN
Print view View original

71–80 of 131 posts

Re: Finnish breaks natural language processors

#71

Post by jeromebaek »

This is interesting. Especially the bit about compounding taking the meaning from figurative to literal. Modern NLP embeddings still assume a canonical meaning for each word and therefore cannot (immediately, anyway) distinguish between a figurative or literal meaning. Not that such efforts don't exist, but they all seem to be missing something fundamental - namely, is the "canonical" meaning, the vector, supposed to be literal, or figurative? This is a question with potentially no right answer, yet we all assume that the answer is "literal".

At the same time, I do wonder if part of the problem with NLP algorithms and Finnish isn't so much its complexity but the fact that there id very little Finnish data to train on.

Re: Finnish breaks natural language processors

#72

Post by dullgiulio »

ajuc wrote: post #26
miki123211 wrote: post #22

Earlier quoted context omitted.

Almost no software handles this correctly because almost no software gives us the tools to handle it correctly. I've translated some software to polish once or twice and they've been using a simple templating system, like message from {name}: {content}, {time} minutes ago. I didn't have access to the source code. As a consequence of that design, some messages were very malformed. Also there were some overused strings…

Qt framework has system in place for several plural forms, so plik/pliki/plików problem is solved, and quite elegantly. If the application programmer gives a damn, that is. There's also a way to add context to the strings for translation, so that you can see this "like" is used in this meaning, and that other "like" is used in another meaning. But there's no generic solution to "z/ze" problem as far as I know, you ne…

I imagine that before we have a technical solution that works well enough to be used, natural language (guided by the technological limits) will have dropped such rule in ordinary use.

Re: Finnish breaks natural language processors

#73

Post by plq »

tallanvor wrote: post #43
plq wrote: post #10

These are common problems for languages with complex morphology. Finnish, Japanese, Turkish and Hungarian are closely related in this sense and share a lot of NLP research. While implementing spell checkers for these languages need a bit more effort than just compiling a list of words, it's far from an unsolved problem. AFAIK Zemberek[1] is a Turkish spellchecker that implements two-level finite state morphology. Als…

But a spell checker is just a tiny part of the NLP domain. When people talk about NLP, they are usually referring to understanding the meaning of the entire text being analyzed, rather than just providing meanings for individual words. A spell checker can't tell you the intent or subtleties of what is being conveyed.

> But a spell checker is just a tiny part of the NLP domain.

Sure, but the linked tweet claims that not even a working spellchecker exists for Finnish. I'm sure Joakim Nivre would think otherwise !

Re: Finnish breaks natural language processors

#74

Post by gizmo686 »

foldr wrote: post #63
vorg wrote: post #60

Earlier quoted context omitted.

> English has no inflected future tense like Latin and Romance languages. In linguistics, “tense” refers generally to how a verb is inflected English has no inflected future tense because of the way words are usually defined. Verbs in English only have 4 types of inflections because words are defined as whatever is surrounded by spaces in writing. If we define words as the unit that carries a single beat in speaking,…

No, it isn't just a matter of orthography. English has no future tense on any reasonable analysis. There is no unit of English syntax which (i) behaves like a verb and (ii) has a distinct future form. This is a completely uncontroversial point. See e.g. http://languagelog.ldc.upenn.edu/nll/?p=897 for further discussion. > If we define words as the unit that carries a single beat in speaking Note that this definition…

There is a standard analysis of English in generative grammar where verbs and tense are are seperate nodes at deep structure [0], and we get inflected verbs through some form of V-T movement [0]. Under this analysis, English clearly has a future tense. It just gets pronounced as its own word (will) instead instead of being involved in V-T movement. A more popular analysis is to replace "T" with "I", and do everything I talked about above. (The question here is really if other features share a d structure node with tense or if they all have their own functional nodes. I haven't seen a clear argument either way)

Edit: not clearly. There are still some additional assumptions to make with no clear evidence either way (notably, the category of "will"), and the definition of "tense" is a amnigous.

[0] English's rules for V-T movemement are relativly complicated compared to most languages

Re: Finnish breaks natural language processors

#75

Post by ajuc »

dullgiulio wrote: post #72
ajuc wrote: post #26

Earlier quoted context omitted.

Qt framework has system in place for several plural forms, so plik/pliki/plików problem is solved, and quite elegantly. If the application programmer gives a damn, that is. There's also a way to add context to the strings for translation, so that you can see this "like" is used in this meaning, and that other "like" is used in another meaning. But there's no generic solution to "z/ze" problem as far as I know, you ne…

I imagine that before we have a technical solution that works well enough to be used, natural language (guided by the technological limits) will have dropped such rule in ordinary use.

Unlikely. The rule is dictated by ease of pronunciation. It's hard to say "z stu", much easier to say "ze stu".

The rule isn't hard to implement, it's just not expected to be a thing by framework/library creators.

Re: Finnish breaks natural language processors

#76

Post by foldr »

gizmo686 wrote: post #74
foldr wrote: post #63

Earlier quoted context omitted.

No, it isn't just a matter of orthography. English has no future tense on any reasonable analysis. There is no unit of English syntax which (i) behaves like a verb and (ii) has a distinct future form. This is a completely uncontroversial point. See e.g. http://languagelog.ldc.upenn.edu/nll/?p=897 for further discussion. > If we define words as the unit that carries a single beat in speaking Note that this definition…

There is a standard analysis of English in generative grammar where verbs and tense are are seperate nodes at deep structure [0], and we get inflected verbs through some form of V-T movement [0]. Under this analysis, English clearly has a future tense. It just gets pronounced as its own word (will) instead instead of being involved in V-T movement. A more popular analysis is to replace "T" with "I", and do everything…

>There is a standard analysis of English in generative grammar where verbs and tense are are seperate nodes at deep structure

Yes, there is.

>Under this analysis, English clearly has a future tense.

No, this does not follow. You can just as easily (and more correctly) analyze "will" as a modal auxiliary like "can" or "would". Supposing it's true that "will" is the daughter of I/T/whatever, it doesn't follow that the I/T/whatever head has to have a future tense feature.

You could argue, and some people have, that at some level, the interpretation of "will" involves future tense in some semantic sense, which conceivably might be realized as a feature on a head somewhere in the inflectional spine of the tree. But that tense feature would not have the same kind of straightforward syntactic and morphological motivation as e.g. the +past feature does in English. In the context of a general audience, this isn't a notion of "tense" that's relevant. After all, in this highly abstract sense, even a language such as Chinese might have tense.

FWIW I have a PhD in (generative) syntax. I'm don't think that any generative syntactician addressing a general audience would describe English as having a future tense.

Re: Finnish breaks natural language processors

#77

Post by DFHippie »

foldr wrote: post #63
vorg wrote: post #60

Earlier quoted context omitted.

> English has no inflected future tense like Latin and Romance languages. In linguistics, “tense” refers generally to how a verb is inflected English has no inflected future tense because of the way words are usually defined. Verbs in English only have 4 types of inflections because words are defined as whatever is surrounded by spaces in writing. If we define words as the unit that carries a single beat in speaking,…

No, it isn't just a matter of orthography. English has no future tense on any reasonable analysis. There is no unit of English syntax which (i) behaves like a verb and (ii) has a distinct future form. This is a completely uncontroversial point. See e.g. http://languagelog.ldc.upenn.edu/nll/?p=897 for further discussion. > If we define words as the unit that carries a single beat in speaking Note that this definition…

> There is no unit of English syntax which (i) behaves like a verb and (ii) has a distinct future form.

This seems like an arbitrary and peculiar way to define tense specifically to exclude periphrastic languages like English. If these are the rules by which we are arguing, the argument doesn't seem that interesting. By this argument colloquial French and German have no past tense because they use periphrastic constructions to expression this distinction.

Re: Finnish breaks natural language processors

#78

Post by gizmo686 »

isani wrote: post #40
szatkus wrote: post #19

Earlier quoted context omitted.

What's the problem with Japanese? It's highly regular language, so it should be easy to tokenize. AFAIR the whole language has maybe few irregular verbs, compared to few hundreds in English.

Japanese is usually written without spaces. Words and sentences just run into each other. When writing in hiragana (syllabic characters), word boundaries are often ambiguous. Englishwouldbemuchhardertoparseifwrittenlikethis.

But Japanese is not written as character soup. It mixes two (actually 3) types of characters, with the "grammatical" sounds being written in hiragana and most content sounds being written in kanji. Since the grammatical sounds are a closed class, and tend to occur at word boundries, it turns out to be relativly simple to seperate words.

Re: Finnish breaks natural language processors

#79

Post by plq »

saagarjha wrote: post #50
isani wrote: post #40

Earlier quoted context omitted.

Japanese is usually written without spaces. Words and sentences just run into each other. When writing in hiragana (syllabic characters), word boundaries are often ambiguous. Englishwouldbemuchhardertoparseifwrittenlikethis.

I have no stake in natural language processing, but it looks to me like a computer might be able to do a pretty good job at splitting that given a dictionary.

The stochastic strategy is to 1. enumerate every possible tag combination 2. assign a probability to each one 3. choose the parse with highest probability.

1. can be done either deterministically or stochastically.

2. requires you to have a language model trained with either human-tagged or semi-human-tagged corpus

3. was just the Viterbi algorithm last time I looked.

Implementing 1 and 2 are require broad domain knowledge in two very different domains (linguistics and machine learning respectively)

So while nowadays sentence segmentation can be considered a solved problem, it's far from trivial to implement one that can compete with the state of the art against real-world data.

There is also a nice body of deterministic (rule-based) literature that is practically ignored nowadays.

Re: Finnish breaks natural language processors

#80

Post by simonh »

gorbachev wrote: post #58
uasm wrote: post #52

Earlier quoted context omitted.

> "kuusi palaa" Can we get a bit more detail here? Are those variations on pronunciation? How does a native Finn able to infer the correct meaning?

No, there is no variation on pronunciation. They sound exactly the same when spoken. The only way to infer what's meant is from context.

My favourite homophone example is a 92 character poem in Chinese[0], where every syllable is a slight variation on Shi.

[0]https://en.wikipedia.org/wiki/Lion-Eating_Poet_in_the_Stone_...

Interestingly when I showed the actual text to my wife, who is Chinese, she didn't understand what was funny about it until I asked her to read it to me, when she burst out laughing. That's when I found out she doesn't sub-vocalise, where I do all the time.

Post reply on HN