Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A computational linguistic farce in three acts

earningmyturns.org

Post reply on HN
Print view View original

31–40 of 56 posts

Re: A computational linguistic farce in three acts

#31

Post by mchaver »

throwawuyar3231 wrote: post #8

I have to wonder if English is really the best language for NLP research. Things like the Winograd schemas which have attracted a lot of attention simply aren't possibilities in other languages. Why not start working with more structured agglutinative * languages like Japanese/Korean and Indic family (Sanskrit esp.) . How about other European languages ? Are they better structured empirically ? I hear German is very…

> Things like the Winograd schemas which have attracted a lot of attention simply aren't possibilities in other languages.

I do not think that is correct. Anaphora exists in many languages. Check out the Anaphora article on wikipedia and click on different language versions. There are example sentences for many languages.

https://en.wikipedia.org/wiki/Anaphora_(linguistics)

There are translation for the Winograd Schemas into a couple of languages. Granted I found some of the translations a little unnatural in some cases but they are still understandable and expose the problem.

http://www.cs.nyu.edu/faculty/davise/papers/WinogradSchemas/...

http://arakilab.media.eng.hokudai.ac.jp/~kabura/collection_j...

http://www.llf.cnrs.fr/winograd-fr

Re: A computational linguistic farce in three acts

#32

Post by paulsutter »

YeGoblynQueenne wrote: post #26
paulsutter wrote: post #4

This relates to the big Twitter uproar over this blog post: https://medium.com/@yoav.goldberg/an-adversarial-review-of-a... And here's the meat of his response: > Idea! Let's go back to toy problems where we can create the test conditions easily, like the rationalists did back then (even if we don't realize we are imitating them). After all, Atari is not real life, but it still demonstrates remarkable RL progress. Le…

My reading is that Pereira doesn't think that deep learning has quite conquered language, and in this he's in complete disagreement both with Goldberg and Le Cunn's side (who both champion deep learning for NLP and claim that it has led to great advances in the field). For me the problem with NLP and deep learning, or indeeed any empirical method, is that the evaluation metrics we have are imperfect. Take BLEU scores…

Google translate is now based on a neural network and you can be sure they have solid metrics. By analogy Google search has a large panel of humans whose subjective feedback is used to test the quality of search algorithm variations.

Re: A computational linguistic farce in three acts

#33

Post by gizmo686 »

SomeStupidPoint wrote: post #24
hcs wrote: post #12

Earlier quoted context omitted.

What do you mean?

That we artificially decompose the process of accepting a sentence as, eg, proper English into two phases: syntactic correctness and semantic correctness. However, that distinction is arbitrary -- there is only the question of if the sentence is accepted by an agent (eg, person) as a well-formed sentence. Any full accounting of the class of well-formed sentences must embed the semantic concerns; violating semantics i…

1) Colorless green ideas sleep furiously.

2) Me gizmo.

One of the above sentences is valid English, the other one is meaningful. That is the syntax/semantics distinction.

The distinction is a bit fuzzy in places (for instance, inflectional morphology), but does exist.

Re: A computational linguistic farce in three acts

#34

Post by SomeStupidPoint »

gizmo686 wrote: post #33
SomeStupidPoint wrote: post #24

Earlier quoted context omitted.

That we artificially decompose the process of accepting a sentence as, eg, proper English into two phases: syntactic correctness and semantic correctness. However, that distinction is arbitrary -- there is only the question of if the sentence is accepted by an agent (eg, person) as a well-formed sentence. Any full accounting of the class of well-formed sentences must embed the semantic concerns; violating semantics i…

1) Colorless green ideas sleep furiously. 2) Me gizmo. One of the above sentences is valid English, the other one is meaningful. That is the syntax/semantics distinction. The distinction is a bit fuzzy in places (for instance, inflectional morphology), but does exist.

I actually disagree with your assessment.

You contend that the top sentence is valid English; I disagree with that. It subscribes to the "first order" rules (or an overly simplistic model), but isn't a sentence that an English speaker would use. Not being one that an English speaker would use makes it invalid English -- it's just a case where the first order approximation is wrong.

Similarly, if your syntax rules reject the second sentence, they're wrong -- since it's a sentence that English speakers can parse: the conclusion can only be that your syntax rules don't actually match the language you're trying to model.

I get the distinction that you're trying to point out with syntax/semantics, but you're ignoring my point: that divide is artificial and 'semantics' as you mean it is merely higher order syntax.

You haven't shown there's an inherent meaning to the difference (ie, that you haven't just drawn an arbitrary line in the sand), just that you can find examples that (naively) fall on different sides of it.

Re: A computational linguistic farce in three acts

#35

Post by gizmo686 »

SomeStupidPoint wrote: post #34
gizmo686 wrote: post #33

Earlier quoted context omitted.

1) Colorless green ideas sleep furiously. 2) Me gizmo. One of the above sentences is valid English, the other one is meaningful. That is the syntax/semantics distinction. The distinction is a bit fuzzy in places (for instance, inflectional morphology), but does exist.

I actually disagree with your assessment. You contend that the top sentence is valid English; I disagree with that. It subscribes to the "first order" rules (or an overly simplistic model), but isn't a sentence that an English speaker would use. Not being one that an English speaker would use makes it invalid English -- it's just a case where the first order approximation is wrong. Similarly, if your syntax rules rej…

Language is not some inherent property of the Universe; it is an evolved behavior in humans. We can study how humans perform language; and when we do, we find the syntax/semantics distinction to be naturally occurring in humans. For instance, in my example, a native English speaker will find the second sentence "awkward" in a way that they do not for the first sentence. Similarly, a native English speaker will extract a clear meaning from the second sentence in a way that they would not from the first.

It is conceivable that there is some other language (eg. not natural human language) which does not have a syntax/semantics distinction, but that hypothetical language is not what linguistics studies.

Re: A computational linguistic farce in three acts

#36

Post by yorwba »

gizmo686 wrote: post #33
SomeStupidPoint wrote: post #24

Earlier quoted context omitted.

That we artificially decompose the process of accepting a sentence as, eg, proper English into two phases: syntactic correctness and semantic correctness. However, that distinction is arbitrary -- there is only the question of if the sentence is accepted by an agent (eg, person) as a well-formed sentence. Any full accounting of the class of well-formed sentences must embed the semantic concerns; violating semantics i…

1) Colorless green ideas sleep furiously. 2) Me gizmo. One of the above sentences is valid English, the other one is meaningful. That is the syntax/semantics distinction. The distinction is a bit fuzzy in places (for instance, inflectional morphology), but does exist.

My favorite example where you need semantics to get the syntax is "See the a are of I."

   +---+---+---+---+
   | x |   |   |   | I
   +---+---+---+---+
   |   |   |   |   | K
   +---+---+---+---+
     a   b   c   d
The map shows 8 ares, the a are of I is marked with a cross.

Re: A computational linguistic farce in three acts

#37

Post by mark_edward »

lgessler wrote: post #18
throwawuyar3231 wrote: post #8

I have to wonder if English is really the best language for NLP research. Things like the Winograd schemas which have attracted a lot of attention simply aren't possibilities in other languages. Why not start working with more structured agglutinative * languages like Japanese/Korean and Indic family (Sanskrit esp.) . How about other European languages ? Are they better structured empirically ? I hear German is very…

> I hear German is very grammatical, and that Hungarian is ... erm odd ? Just want to point out that "grammatical" probably isn't the word you want here. Every language is grammatical by definition in the sense that there are rules that govern its sound system, word formation system, syntax, etc. The concept you're getting at, though--that some languages are easier for computer programs and/or speakers of Indo-Europe…

do you think analytic would be a good term here? i heard mandarin is very analytic language, maybe that could be a good choice

Re: A computational linguistic farce in three acts

#38

Post by SomeStupidPoint »

gizmo686 wrote: post #35
SomeStupidPoint wrote: post #34

Earlier quoted context omitted.

I actually disagree with your assessment. You contend that the top sentence is valid English; I disagree with that. It subscribes to the "first order" rules (or an overly simplistic model), but isn't a sentence that an English speaker would use. Not being one that an English speaker would use makes it invalid English -- it's just a case where the first order approximation is wrong. Similarly, if your syntax rules rej…

Language is not some inherent property of the Universe; it is an evolved behavior in humans. We can study how humans perform language; and when we do, we find the syntax/semantics distinction to be naturally occurring in humans. For instance, in my example, a native English speaker will find the second sentence "awkward" in a way that they do not for the first sentence. Similarly, a native English speaker will extrac…

.....Which is an effect of the first satisfying first order approximations while failing higher order rules, while the latter is merely an unusual sentence and so requires more effort to parse because it falls off the "fast path". (It also arguably fails to encode embedded cultural messages present in word choice -- a second consideration for why it feels "awkward": it's valid English, but not my tribe's English.)

You haven't pointed out how semantics is anything but higher order syntax -- merely outlined the way in which higher order syntax interacts with our perception.

I agree that there's a difference between the two sentences -- I disagree that it's because they're different fields of study instead of different edge cases of the same underlying notion of parsing syntax. (I especially disagree that the way forward on teaching machines language involves that distinction.)

I would appreciate you referring me to references on the semantic/syntax divide being "natural", though.

Re: A computational linguistic farce in three acts

#39

Post by SomeStupidPoint »

yorwba wrote: post #36
gizmo686 wrote: post #33

Earlier quoted context omitted.

1) Colorless green ideas sleep furiously. 2) Me gizmo. One of the above sentences is valid English, the other one is meaningful. That is the syntax/semantics distinction. The distinction is a bit fuzzy in places (for instance, inflectional morphology), but does exist.

My favorite example where you need semantics to get the syntax is "See the a are of I." +---+---+---+---+ | x | | | | I +---+---+---+---+ | | | | | K +---+---+---+---+ a b c d The map shows 8 ares, the a are of I is marked with a cross.

You guys seem confused: I'm not claiming that what you're calling semantics doesn't exist; I'm saying that it's merely convoluted syntactical rules, and calling it a different name is misleading.

It would be correct to say that the sentence in isolation is a parse error, but with the diagram, it's merely elaborate syntax.

My point isn't that there aren't higher order rules (and approximations) -- just that the division of those rules into a separate area of study is artificial.

It's not contentious to point out that chemistry is just an approximation of physics because the actual higher order rules are too complex to study directly -- but it seems to be to point the same out in semantics and syntax.

Re: A computational linguistic farce in three acts

#40

Post by YeGoblynQueenne »

paulsutter wrote: post #32
YeGoblynQueenne wrote: post #26

Earlier quoted context omitted.

My reading is that Pereira doesn't think that deep learning has quite conquered language, and in this he's in complete disagreement both with Goldberg and Le Cunn's side (who both champion deep learning for NLP and claim that it has led to great advances in the field). For me the problem with NLP and deep learning, or indeeed any empirical method, is that the evaluation metrics we have are imperfect. Take BLEU scores…

Google translate is now based on a neural network and you can be sure they have solid metrics. By analogy Google search has a large panel of humans whose subjective feedback is used to test the quality of search algorithm variations.

This is something that needs to be repeated until everyone internalises it: for language pairs other than the "easy" ones Google translate sucks.

I am Greek and translations from and to my language are utterly ridiculous, on the level of Bozo the clown doing the translation with his underpants on his head back to front.

Typical example: I put in the Greek word for "swallow", the bird, and ask for the French translation. I get back the word "avaler" - the French word for "to swallow", the verb.

That's my little benchmark there, useful because Google translate has been doing this consistently, for a good few years, before it used neural networks, before it started claiming its setup essentially constitutes an "interlingua" etc etc.

Note that the bird and the verb sound nothing like each other in Greek, or French. They sound the same only in English, so GT goes from Greek to French through English. Because it doesn't have enough parallel texts to go directly to French. And so it sucks, because it doesn't have enough data. You can ask native users of other languages-that-are not-English or have few ish speakers, perhaps Turkish or Hungarian etc. I'm pretty sure you'll find out they have similar experiences.

So I don't know what metric they use to evaluate their results, it doesn't seem to be a particularly good metric of translation quality. Maybe they just care more about how many people use their system and try to optimise for that, rather than going for the much harder to know quality.

Post reply on HN