Differentiable Neural Computers • Hacker News Board

Re: Differentiable Neural Computers

#21

Post by svantana » Thu, Oct 13, 2016, 10:18 AM UTC

modeless wrote: post #12

rkaplan wrote: post #5

This paper builds off of DeepMind's previous work on differentiable computation: Neural Turing Machines. That paper generated a lot of enthusiasm when it came out in 2014, but not many researchers use NTMs today. The feeling among researchers I've spoken to is not that NTMs aren't useful. DeepMind is simply operating on another level. Other researchers don't understand the intuitions behind the architecture well enou…

The reason other researchers haven't jumped on NTMs may be that, unlike commonly-researched types of neural nets such as CNNs or RNNs, NTMs are not currently the best way to solve any real-world problem. The problems they have solved so far are relatively trivial, and they are very inefficient, inaccurate, and complex relative to traditional CS methods (e.g. Dijkstra's algorithm coded in C). That's not to say that NT…

They sure put a lot of focus on "toy" problems such as sorting and path planning in their papers - perhaps because they are easy to understand and show a major improvement over other ML approaches. IMHO they should focus more on "real" problems - e.g. in Table 1 of this paper it seems to be state of the art on the bAbl tasks, which is amazing.

Re: Differentiable Neural Computers

#22

Post by wiz21c » Thu, Oct 13, 2016, 10:22 AM UTC

outsideline wrote: post #15

wiz21c wrote: post #7

Earlier quoted context omitted.

>> DeepMind is simply operating on another level. Would you be so kind as to to explain what you mean here ? Thanks !

They're taking features that are present in the brain that aren't modeled and are making computational models for them. They're not a gold standard. You can create your own in under an hour. It's not another level. It's bio-inspired computing. Here.. take the 'Axon Hillock' https://en.wikipedia.org/wiki/Axon_hillock code up a function for it, attach it to present day neuron models, make it do something fancy, write a…

ok I get it :) Nice little sarcasm, I'm loving it :-)

Re: Differentiable Neural Computers

#23

Post by sherjilozair » Thu, Oct 13, 2016, 10:45 AM UTC

svantana wrote: post #21

modeless wrote: post #12

Earlier quoted context omitted.

The reason other researchers haven't jumped on NTMs may be that, unlike commonly-researched types of neural nets such as CNNs or RNNs, NTMs are not currently the best way to solve any real-world problem. The problems they have solved so far are relatively trivial, and they are very inefficient, inaccurate, and complex relative to traditional CS methods (e.g. Dijkstra's algorithm coded in C). That's not to say that NT…

They sure put a lot of focus on "toy" problems such as sorting and path planning in their papers - perhaps because they are easy to understand and show a major improvement over other ML approaches. IMHO they should focus more on "real" problems - e.g. in Table 1 of this paper it seems to be state of the art on the bAbl tasks, which is amazing.

bAbI isn't really a "real" problem either, although somewhat better than sorting and the like. bAbI works with extremely restrictive worlds and grammar. In contrast, current speech recognition, language modeling, and object detection do quite well with actual audio, text, and pictures.

I think the strength of NTMs will be best demonstrated by putting it to work on a long-range language modeling task where you need to organize what you read so that you can use it to predict better a paragraph or two later. Current language models based on LSTM are not really able to do this.

Re: Differentiable Neural Computers

#24

Post by gjm11 » Thu, Oct 13, 2016, 10:53 AM UTC

outsideline wrote: post #19

https://en.wikipedia.org/wiki/Bio-inspired_computing Present day Neuron models lack an incredible number of functional features that are clearly present in the human brain. NTMs = representing memory that is stored in neurons https://en.wikipedia.org/wiki/Neuronal_memory_allocation Decoupled Neural Interfaces using Synthetic Gradients = https://en.wikipedia.org/wiki/Electrochemical_gradient Differentiable Neural Comp…

You're not getting downvoted for being mean about DeepMind, you're getting downvoted for making overconfident pronouncements about things you don't understand.

"Neural Turing machines" are not the same thing as neuronal memory allocation: NTMs' memory is external and neuronal memory allocation is all about how memory is stored in neurons in the brain.

The "synthetic gradients" in that paper have nothing to do with the electrochemical gradients you mention other than the name.

No one is claiming that the DeepMind guys are "operating on another level" because they do bio-inspired things. They are claiming that because they are getting more impressive results than anyone else.

Now: Are they really? If so, is that enough justification for such a grand-sounding claim. I don't know. That would be an interesting discussion to have. But "Boooo, these people are just copying things present in the brain, there's nothing impressive about that" is not, especially when the parallels between the brain-things and the DeepMind-things are as feeble as in your examples.

Re: Differentiable Neural Computers

#25

Post by znah » Thu, Oct 13, 2016, 11:03 AM UTC

outsideline wrote: post #19

https://en.wikipedia.org/wiki/Bio-inspired_computing Present day Neuron models lack an incredible number of functional features that are clearly present in the human brain. NTMs = representing memory that is stored in neurons https://en.wikipedia.org/wiki/Neuronal_memory_allocation Decoupled Neural Interfaces using Synthetic Gradients = https://en.wikipedia.org/wiki/Electrochemical_gradient Differentiable Neural Comp…

Ideas are cheap, making them work is hard.

Re: Differentiable Neural Computers

#26

Post by [deleted] » Thu, Oct 13, 2016, 11:03 AM UTC

outsideline wrote: post #19

https://en.wikipedia.org/wiki/Bio-inspired_computing Present day Neuron models lack an incredible number of functional features that are clearly present in the human brain. NTMs = representing memory that is stored in neurons https://en.wikipedia.org/wiki/Neuronal_memory_allocation Decoupled Neural Interfaces using Synthetic Gradients = https://en.wikipedia.org/wiki/Electrochemical_gradient Differentiable Neural Comp…

[deleted]

Re: Differentiable Neural Computers

#27

Post by vintermann » Thu, Oct 13, 2016, 11:08 AM UTC

outsideline wrote: post #14

visarga wrote: post #8

Earlier quoted context omitted.

Deep mind is breaking new ground in number of directions. For example, "Decoupled Neural Interfaces using Synthetic Gradients" is simply amazing - they can make training a net async and run individual layers on separate machines by approximating the gradients with a local net. It's the kind of thing that sounds crazy on paper, but they proved it works. Another amazing thing they did was to generate audio by direct sy…

Decoupled Neural Interfaces using Synthetic Gradients is a fancy name for the electro-chemical gradient that lies outside the cell wall of neurons : https://en.wikipedia.org/wiki/Electrochemical_gradient It's decoupled yet stores transient local information regarding previous neuron activity. Another bio-inspired copy-pasta.

You should absolutely get a job doing it, if you think bio-inspired copy-pasta is all it takes. May I recommend Numenta?

Re: Differentiable Neural Computers

#28

Post by vintermann » Thu, Oct 13, 2016, 11:11 AM UTC

outsideline wrote: post #13

rkaplan wrote: post #5

This paper builds off of DeepMind's previous work on differentiable computation: Neural Turing Machines. That paper generated a lot of enthusiasm when it came out in 2014, but not many researchers use NTMs today. The feeling among researchers I've spoken to is not that NTMs aren't useful. DeepMind is simply operating on another level. Other researchers don't understand the intuitions behind the architecture well enou…

Alan Turing's tape machine + neuron model. In the human brain, Neurons store an incredible amount of information. Neuron models in neural networks only did so with weights. There is still a lack of understanding on how the human brain does it. Deep Mind grabbed a proven memory model from Alan Turing's work and applied it to the feature barren neuron models in use. Sprinkle magic ... They are not operating on another…

In order to use a turing machine in a neural network - or at least to train it, in any way that isn't impractical and/or cheating - you need to make it differentiable somehow.

Graves and co. have been really creative in overcoming problems in their ongoing program to differentiate ALL the things.

Re: Differentiable Neural Computers

#29

Post by vintermann » Thu, Oct 13, 2016, 11:17 AM UTC

orthoganol wrote: post #18

modeless wrote: post #17

Earlier quoted context omitted.

The "memory" in a typical RNN is akin to a human's short term working memory. It only holds a few things and forgets old things quickly as new things come in. This new memory can hold a large number of things and stores them for an unlimited amount of time, more like a human's long term memory or a computer's RAM. It's a big difference, and the implementation is completely different too.

I was not referring to typical RNNs, but LSTMs or RNNs with 'attention'. They are designed to overcome vanishing/ exploding gradient problems and hold arbitrary memory lengths.

They can technically be as long as you want them, but in practice there are still severe constraints. LSTMs alleviate the gradient problems, but you still get real trouble with long-term dependencies.

Alex Graves and some others in DeepMind have focused a lot in the past year or so on developing practical differentiable data structures, so that the LSTM can read and write to an external memory (and save its precious internal state for more immediate needs) yet still be trainable via backpropagation.

Re: Differentiable Neural Computers

#30

Post by TeeWEE » Thu, Oct 13, 2016, 11:30 AM UTC

svantana wrote: post #21

modeless wrote: post #12

Earlier quoted context omitted.

The reason other researchers haven't jumped on NTMs may be that, unlike commonly-researched types of neural nets such as CNNs or RNNs, NTMs are not currently the best way to solve any real-world problem. The problems they have solved so far are relatively trivial, and they are very inefficient, inaccurate, and complex relative to traditional CS methods (e.g. Dijkstra's algorithm coded in C). That's not to say that NT…

They sure put a lot of focus on "toy" problems such as sorting and path planning in their papers - perhaps because they are easy to understand and show a major improvement over other ML approaches. IMHO they should focus more on "real" problems - e.g. in Table 1 of this paper it seems to be state of the art on the bAbl tasks, which is amazing.

Once you have a learning machine that can solve simple problems. You can scale it up to solve very complex problems. Its a first step to true AI imho. Al lot of small steps are needed to go towards this goal. Integrating Memory & Neural Nets is a big step imho.