Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A Neural Conversational Model [pdf]

arxiv.org

Post reply on HN
Print view View original

31–40 of 41 posts

Re: A Neural Conversational Model [pdf]

#31

Post by throwaway_bob »

andrewtbham wrote: post #26
smhx wrote: post #25

Earlier quoted context omitted.

Quoc Le is neither an author of the word2vec paper or in the committers list of word2vec software (there's only two committers, mikolov and sutskever). Not sure where you got that inference from.

http://www.wired.com/2014/12/googlers-quest-teach-machines-u... At Google, he helped develop a system that essentially maps words into vectors. And according to Google, this work would later feed into a system developed largely by a researcher named Tomas Mikolov. Called Word2Vec, the system determines how different words on the web are related, and Google is now using this as a means of strengthening its “knowledge…

Just because wired says something doesn't make it true. Whatever other accomplishments he may have, Le was not a contributor to word2vec, and he did not originate the idea of mapping words to vectors.

Re: A Neural Conversational Model [pdf]

#32

Post by Nadya »

teraflop wrote: post #28
Nadya wrote: post #13

Earlier quoted context omitted.

Technically it is 水の色 (water's color). Often times particles like 「は・が・の」 are dropped when the context allows for such, making 水の色 turn into 水色. Same thing happens with 葉の色 becoming 葉色 (leaf green). The の is implied. In English one might say that something is "sky blue" instead of saying "the color of the sky". So it's not uniquely Japanese in that sense.

Technically, 水色 appears in both Japanese and Japanese-English dictionaries, so I'd say it's a word by any reasonable definition. (Disclaimer: not a native speaker.)

It's a word in its own right - but the etymology of it is still important and I feel it was slightly misrepresented.

For example, the greeting many people know 「こんにちは」 (konnichiwa) is a word in its own right nowadays, but is really just the beginning of an older greeting: 今日は御機嫌いかがですか

Re: A Neural Conversational Model [pdf]

#33

Post by reddytowns »

murbard2 wrote: post #5
Nadya wrote: post #2

The conversations are rather enlightening. >What is the color of water? Water. I'm quite puzzled how it knows "two plus two" is four but "ten minus two" is "seventy-two". I wonder how it parsed one correctly but failed drastically to parse the other.

It did not see enough examples to learn a general addition algorithm. "Two plus two equals four" is a common sentence, "Ten minus two equals eight" isn't. In fact, it's likely that no one learns addition by listening to examples, you're taught to execute a specific algorithm and it takes many people years to master it. In fact, ask a toddler what "ten minus two" is and you might get a similar nonsensical answer.

I don' think it could ever learn a general addition algorithm, because the algorithm they are using doesn't allow for a multi-step process. For example, to add 28 + 37, a simple approach would be:

  1. first, look at the one's column and recall that 8 + 7 is 15.
  2. Then take the value from the tens column of the answer
  and add it to the tens column of the arguments.
But given the command : add 28 + 37, it can only execute a single step. It has no way of guessing at and then storing the values for the intermediate command, (take the 1 from the 15 and add it to the 2 from the 28 and the 3 from the 37), to be processed in the next round. Because so it can't ever generalize addition.

Re: A Neural Conversational Model [pdf]

#34

Post by paulsutter »

Nadya wrote: post #13
derefr wrote: post #11

Earlier quoted context omitted.

> What is the color of water? Water. Literally true in Japanese. The color of water is 水色 (mizu-iro: lit. "water-color.") The analogy then proceeds in the other direction: light blue things are considered "water-colored", rather than water being "light" "blue".

Technically it is 水の色 (water's color). Often times particles like 「は・が・の」 are dropped when the context allows for such, making 水の色 turn into 水色. Same thing happens with 葉の色 becoming 葉色 (leaf green). The の is implied. In English one might say that something is "sky blue" instead of saying "the color of the sky". So it's not uniquely Japanese in that sense.

Dropping の can create a different meaning. For example, 日本料理 (Japanese cuisine) and 日本の料理　(cooking in Japan, which could well be used to refer to the quality of the italian or chinese food here). The former is truly a single word and the latter two words separated by a particle.

Re: A Neural Conversational Model [pdf]

#35

Post by Smerity »

reddytowns wrote: post #33
murbard2 wrote: post #5

Earlier quoted context omitted.

It did not see enough examples to learn a general addition algorithm. "Two plus two equals four" is a common sentence, "Ten minus two equals eight" isn't. In fact, it's likely that no one learns addition by listening to examples, you're taught to execute a specific algorithm and it takes many people years to master it. In fact, ask a toddler what "ten minus two" is and you might get a similar nonsensical answer.

I don' think it could ever learn a general addition algorithm, because the algorithm they are using doesn't allow for a multi-step process. For example, to add 28 + 37, a simple approach would be: 1. first, look at the one's column and recall that 8 + 7 is 15. 2. Then take the value from the tens column of the answer and add it to the tens column of the arguments. But given the command : add 28 + 37, it can only exec…

Whilst you are correct that the number of allowed computation steps is limited, LSTMs have still been used with success for such tasks. LSTMs have been used to perform addition with 99% accuracy^ on two 9-digit numbers[1]. The paper even shows that simple Python programs can be evaluated with some degree of accuracy.

Remember that even if the number of computation steps is limited, there can be multiple layers ([1] uses 2) and each neuron unit can perform a computation ([1] uses 400 cells per layer). It only needs to learn how to be an ALU. The work is in fact done by one of the people who established the sequence to sequence framework, Sutskever, and is referenced in the Neural Conversational Model paper.

^ They use "teacher forcing" for evaluation which inflates the accuracy to some degree, but it's still quite impressive.

[1]: http://arxiv.org/pdf/1410.4615v3.pdf

Re: A Neural Conversational Model [pdf]

#36

Post by nyxtom »

dmgbrn wrote: post #23
arthurcolle wrote: post #20

> Human: what is the purpose of life ? > Machine: to serve the greater good . > Human: what is the purpose of living ? > Machine: to live forever . > Human: what is the purpose of existence ? > Machine: to find out what happens when we get to the planet earth . > Human: where are you now ? > Machine: i ’m in the middle of nowhere . > Human: what is the purpose of dying ? > Machine: to have a life . > Human: what is t…

> Machine: to find out what happens when we get to the planet earth . Something tells me the entirety of BSG was included in the training set.

They used the OpenSubtitles dataset (in addition to a closed IT Troubleshooting dataset), so it's entirely possible BSG was indeed included :P

Re: A Neural Conversational Model [pdf]

#37

Post by kastnerkyle »

throwaway_bob wrote: post #31
andrewtbham wrote: post #26

Earlier quoted context omitted.

http://www.wired.com/2014/12/googlers-quest-teach-machines-u... At Google, he helped develop a system that essentially maps words into vectors. And according to Google, this work would later feed into a system developed largely by a researcher named Tomas Mikolov. Called Word2Vec, the system determines how different words on the web are related, and Google is now using this as a means of strengthening its “knowledge…

Just because wired says something doesn't make it true. Whatever other accomplishments he may have, Le was not a contributor to word2vec, and he did not originate the idea of mapping words to vectors.

I think the poster got mixed up. Quoc Le was the first author on "Distributed Representations of Sentences and Documents" aka paragraph2vec, so he has been involved in the x2vec scene. Just not the word2vec. And I would argue that word2vec did not originate the idea either, just popularized it and showed the power of such an approach (plus an awesome demo!).

The original paper on word embeddings from 2001 from Bengio et. al. "A Neural Probabilistic Language Model"[1] is the first I am aware of.

[1] http://papers.nips.cc/paper/1839-a-neural-probabilistic-lang...

Re: A Neural Conversational Model [pdf]

#38

Post by sswaner »

dmgbrn wrote: post #23
arthurcolle wrote: post #20

> Human: what is the purpose of life ? > Machine: to serve the greater good . > Human: what is the purpose of living ? > Machine: to live forever . > Human: what is the purpose of existence ? > Machine: to find out what happens when we get to the planet earth . > Human: where are you now ? > Machine: i ’m in the middle of nowhere . > Human: what is the purpose of dying ? > Machine: to have a life . > Human: what is t…

> Machine: to find out what happens when we get to the planet earth . Something tells me the entirety of BSG was included in the training set.

What is BSG?

Re: A Neural Conversational Model [pdf]

#40

Post by murbard2 »

Smerity wrote: post #35
reddytowns wrote: post #33

Earlier quoted context omitted.

I don' think it could ever learn a general addition algorithm, because the algorithm they are using doesn't allow for a multi-step process. For example, to add 28 + 37, a simple approach would be: 1. first, look at the one's column and recall that 8 + 7 is 15. 2. Then take the value from the tens column of the answer and add it to the tens column of the arguments. But given the command : add 28 + 37, it can only exec…

Whilst you are correct that the number of allowed computation steps is limited, LSTMs have still been used with success for such tasks. LSTMs have been used to perform addition with 99% accuracy^ on two 9-digit numbers[1]. The paper even shows that simple Python programs can be evaluated with some degree of accuracy. Remember that even if the number of computation steps is limited, there can be multiple layers ([1] u…

Now for the kicker: recurrent network can implement arbitrary algorithms, including a type of GOFAI. It's quite possible that such an algorithm isn't learnable without supervision; that it was discovered by sheer luck at the dawn of humanity and has been passed down by language ever since.
Post reply on HN