Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Differentiable Neural Computers

deepmind.com

Post reply on HN
Print view View original

31–40 of 77 posts

Re: Differentiable Neural Computers

#31

Post by TeeWEE »

svantana wrote: post #21
modeless wrote: post #12

Earlier quoted context omitted.

The reason other researchers haven't jumped on NTMs may be that, unlike commonly-researched types of neural nets such as CNNs or RNNs, NTMs are not currently the best way to solve any real-world problem. The problems they have solved so far are relatively trivial, and they are very inefficient, inaccurate, and complex relative to traditional CS methods (e.g. Dijkstra's algorithm coded in C). That's not to say that NT…

They sure put a lot of focus on "toy" problems such as sorting and path planning in their papers - perhaps because they are easy to understand and show a major improvement over other ML approaches. IMHO they should focus more on "real" problems - e.g. in Table 1 of this paper it seems to be state of the art on the bAbl tasks, which is amazing.

Once you have a learning machine that can solve simple problems. You can scale it up to solve very complex problems. Its a first step to true AI imho. Al lot of small steps are needed to go towards this goal. Integrating Memory & Neural Nets is a big step imho.

Re: Differentiable Neural Computers

#32

Post by igravious »

vintermann wrote: post #28
outsideline wrote: post #13

Earlier quoted context omitted.

Alan Turing's tape machine + neuron model. In the human brain, Neurons store an incredible amount of information. Neuron models in neural networks only did so with weights. There is still a lack of understanding on how the human brain does it. Deep Mind grabbed a proven memory model from Alan Turing's work and applied it to the feature barren neuron models in use. Sprinkle magic ... They are not operating on another…

In order to use a turing machine in a neural network - or at least to train it, in any way that isn't impractical and/or cheating - you need to make it differentiable somehow. Graves and co. have been really creative in overcoming problems in their ongoing program to differentiate ALL the things.

In this context what does differentiable mean?

Re: Differentiable Neural Computers

#33

Post by outsideline »

gjm11 wrote: post #24
outsideline wrote: post #19

https://en.wikipedia.org/wiki/Bio-inspired_computing Present day Neuron models lack an incredible number of functional features that are clearly present in the human brain. NTMs = representing memory that is stored in neurons https://en.wikipedia.org/wiki/Neuronal_memory_allocation Decoupled Neural Interfaces using Synthetic Gradients = https://en.wikipedia.org/wiki/Electrochemical_gradient Differentiable Neural Comp…

You're not getting downvoted for being mean about DeepMind, you're getting downvoted for making overconfident pronouncements about things you don't understand. "Neural Turing machines" are not the same thing as neuronal memory allocation: NTMs' memory is external and neuronal memory allocation is all about how memory is stored in neurons in the brain. The "synthetic gradients" in that paper have nothing to do with th…

Overconfident pronouncement by indicating that they are making computational models of natural processes that no one can confidently state are correct or are the most efficient?

Making statements that allow people to see behind the curtains and maybe go off and make their own competitive models... Yes, this is a disservice to the advancement of A.I and should be downvoted : Removing the prestigious veil and illusion from published works.

NTMs memory is external in what sense? Please detail what this means in a 'functional' sense. It's biologically inspired. Neurons maintain memory beyond synaptic weights. The neuron models of present day A.I were basic. Someone comes along and sees the obvious : There is no computational model for how neurons utilize memory and suddenly they're thinking on another level? Give me a break..

Synthetic gradients have everything to do w/ electro-chemical gradients : http://www.nature.com/articles/srep14527 http://www.pnas.org/content/110/30/12456.full.pdf So, where is your establishment that I am incorrect. It is nowhere to be found. Again, biologically inspired computational models.

Oh look, someone published a paper back in June that is an implementation of Differentiable Neural Computers: https://arxiv.org/abs/1607.00036

It's hype and that is a disservice to the community of people completing similar work and taking similar approaches.

It would be an interesting discussion to have. That discussion was terminated in favor of downvoting me.

They're feeble to someone who isn't well informed on neuroscience. Thus, you'd rather be wow'd and believe in the fantasy that only a small segment of people can write computational models of biology.

Continue believing the hype. Rarely will someone be truthful and honest about where they got their ideas when hype follows. An interesting conversation could have transpired. Enjoy the feels from the downvotes.

Re: Differentiable Neural Computers

#34

Post by TeeWEE »

outsideline wrote: post #13
rkaplan wrote: post #5

This paper builds off of DeepMind's previous work on differentiable computation: Neural Turing Machines. That paper generated a lot of enthusiasm when it came out in 2014, but not many researchers use NTMs today. The feeling among researchers I've spoken to is not that NTMs aren't useful. DeepMind is simply operating on another level. Other researchers don't understand the intuitions behind the architecture well enou…

Alan Turing's tape machine + neuron model. In the human brain, Neurons store an incredible amount of information. Neuron models in neural networks only did so with weights. There is still a lack of understanding on how the human brain does it. Deep Mind grabbed a proven memory model from Alan Turing's work and applied it to the feature barren neuron models in use. Sprinkle magic ... They are not operating on another…

Dont forget you first need to understand the mathemetical theory of how a brain does computation and pattern recognition. Off course they look into how a brain does it. But the mathematical underpinnings, and how the information flows is much more important than how an individual neuron works in real live. Abstraction and applying it to real data is what they are doing.

Re: Differentiable Neural Computers

#35

Post by TeeWEE »

outsideline wrote: post #33
gjm11 wrote: post #24

Earlier quoted context omitted.

You're not getting downvoted for being mean about DeepMind, you're getting downvoted for making overconfident pronouncements about things you don't understand. "Neural Turing machines" are not the same thing as neuronal memory allocation: NTMs' memory is external and neuronal memory allocation is all about how memory is stored in neurons in the brain. The "synthetic gradients" in that paper have nothing to do with th…

Overconfident pronouncement by indicating that they are making computational models of natural processes that no one can confidently state are correct or are the most efficient? Making statements that allow people to see behind the curtains and maybe go off and make their own competitive models... Yes, this is a disservice to the advancement of A.I and should be downvoted : Removing the prestigious veil and illusion…

Even if they would do copy-pasta from nature.. Even if they copy everything..

They are the first who have a machine learn to solve problems that require memory. They are the first. These are the stepping stones to artificial Intelligence.

Note: The whole point of the Synthetic gradients, is to learn a network in parallel. This allows Google to make computers learn recognize things in images even better. To recognize human speech even beter... To make self driving cars even better.....

I don't know if they are copied or not from nature (doesnt look like). The point is that they are improving mankind.

Re: Differentiable Neural Computers

#36

Post by outsideline »

znah wrote: post #25
outsideline wrote: post #19

https://en.wikipedia.org/wiki/Bio-inspired_computing Present day Neuron models lack an incredible number of functional features that are clearly present in the human brain. NTMs = representing memory that is stored in neurons https://en.wikipedia.org/wiki/Neuronal_memory_allocation Decoupled Neural Interfaces using Synthetic Gradients = https://en.wikipedia.org/wiki/Electrochemical_gradient Differentiable Neural Comp…

Ideas are cheap, making them work is hard.

The brain's architecture is laid bare for anyone to see. Making models of features is cheap. Anyone can do it. There are loads of white papers and benchmarks. Some approaches beat others depending on the benchmark. Claiming it 'works' by tweaking it until it fits a canned benchmark is not hard work. Not having any explanation as to why it works is not hard work.

Coming up with a functional systems architecture that ties the bits and pieces together is hard work. Understanding what is really happening in the human brain, how/why it is performing various functions, and how this provides for an intelligent architecture is hard work. Creating an 'aware' platform is hard and elusive work which is why people chase the low hanging fruit of optimization algorithms.

*Cheers

Re: Differentiable Neural Computers

#37

Post by TeeWEE »

outsideline wrote: post #33
gjm11 wrote: post #24

Earlier quoted context omitted.

You're not getting downvoted for being mean about DeepMind, you're getting downvoted for making overconfident pronouncements about things you don't understand. "Neural Turing machines" are not the same thing as neuronal memory allocation: NTMs' memory is external and neuronal memory allocation is all about how memory is stored in neurons in the brain. The "synthetic gradients" in that paper have nothing to do with th…

Overconfident pronouncement by indicating that they are making computational models of natural processes that no one can confidently state are correct or are the most efficient? Making statements that allow people to see behind the curtains and maybe go off and make their own competitive models... Yes, this is a disservice to the advancement of A.I and should be downvoted : Removing the prestigious veil and illusion…

An "Electro chemical gradient" is an ion. That works on small scale within cells. The gradient here is the "electrochemical potential" of the ion.

An Synthetic Gradient is a way to allow learning Forward Propagated Neural Nets in a parallel way. The gradient here is referring to the 'error' backpropagation that is part of the training process of an neural net (im talking about computer science neural nets).

They have nothing todo with each other. The papers that you are referring to have nothing todo with the process of training a neural net.

Re: Differentiable Neural Computers

#38

Post by outsideline »

TeeWEE wrote: post #35
outsideline wrote: post #33

Earlier quoted context omitted.

Overconfident pronouncement by indicating that they are making computational models of natural processes that no one can confidently state are correct or are the most efficient? Making statements that allow people to see behind the curtains and maybe go off and make their own competitive models... Yes, this is a disservice to the advancement of A.I and should be downvoted : Removing the prestigious veil and illusion…

Even if they would do copy-pasta from nature.. Even if they copy everything.. They are the first who have a machine learn to solve problems that require memory. They are the first. These are the stepping stones to artificial Intelligence. Note: The whole point of the Synthetic gradients, is to learn a network in parallel. This allows Google to make computers learn recognize things in images even better. To recognize…

> They are the first who have a machine learn to solve problems that require memory.

Incorrect. It was named a Neural (Turing) machine for a reason. Maybe people should go back and dust off the white papers from the 70s like those who are borrowing from that era and respectfully giving credit where credit is due.

They do great work and they are making great progress in Artificial Intelligence. Many people are. Everything is a stepping stone. It serves no good to over-hype one person's stones over another's or ignore/downplay where they were inspired from. Notable visionaries of a past time were visionaries because they detailed the depths of their thinking and centered on the hows/whys. It seems it is fashionable now-a-days to do the exact opposite. This is to a disservice to learning and progress.

The whole point of the human brain is parallel processing. Extra-cellular chemical Gradients function the same way in the human brain and serve the same purposes. Take a look at the papers I linked.

> I don't know if they are copied or not from nature (doesnt look like).

Extra-cellular chemical Gradients. I linked to white papers that explain how memory is stored in them and shared across neurons. This is how it works in nature and biology.

They named their approach 'Synthetic Gradients'. An artificial form of the biological Gradient that is decoupled and lies outside of a neuron. They are clearly giving credit to nature.

They and many other people are improving mankind. Many others can improve mankind if there was less hype and more of a focus on where the ideas originated.

That was my point..

The behavior of people regarding selective 'hype' is one of the big reasons why a tremendous amount of deeply functional work that centers on hard intuitions and ideas for this area will remain closed source when a real break is made.

Enjoy the hype train I guess... They're operating on another level than anyone else.

Post reply on HN