Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Differentiable Neural Computers

deepmind.com

Post reply on HN
Print view View original

51–60 of 77 posts

Re: Differentiable Neural Computers

#51

Post by Senji »

empath75 wrote: post #49
modeless wrote: post #12

Earlier quoted context omitted.

The reason other researchers haven't jumped on NTMs may be that, unlike commonly-researched types of neural nets such as CNNs or RNNs, NTMs are not currently the best way to solve any real-world problem. The problems they have solved so far are relatively trivial, and they are very inefficient, inaccurate, and complex relative to traditional CS methods (e.g. Dijkstra's algorithm coded in C). That's not to say that NT…

The brain is not a single universal neural network that does everything well. It's a collection of different neural networks that specialize in different tasks, and probably use very different methods to achieve them. It seems like the way forward would be networking together various kinds of neural networks to achieve complex goals. For example, an NTM specialized in formulating plans that has access to a CNN for im…

If we're going with a brain metaphor. What would be the those neural networks' version of synesthesia?

Re: Differentiable Neural Computers

#52

Post by vinay427 »

empath75 wrote: post #49
modeless wrote: post #12

Earlier quoted context omitted.

The reason other researchers haven't jumped on NTMs may be that, unlike commonly-researched types of neural nets such as CNNs or RNNs, NTMs are not currently the best way to solve any real-world problem. The problems they have solved so far are relatively trivial, and they are very inefficient, inaccurate, and complex relative to traditional CS methods (e.g. Dijkstra's algorithm coded in C). That's not to say that NT…

The brain is not a single universal neural network that does everything well. It's a collection of different neural networks that specialize in different tasks, and probably use very different methods to achieve them. It seems like the way forward would be networking together various kinds of neural networks to achieve complex goals. For example, an NTM specialized in formulating plans that has access to a CNN for im…

This is being done using various types of networks. See these slides on image captioning by Karpathy for an example using a CNN and RNN: http://cs.stanford.edu/people/karpathy/sfmltalk.pdf

Re: Differentiable Neural Computers

#53

Post by Senji »

chriswarbo wrote: post #44
dharma1 wrote: post #41

Earlier quoted context omitted.

He has done a lot of pioneering work, to be honest. I recommend seeing him talk (or watch a video), I think his humour comes across better that way

It's interesting to think that Schmidhuber's actually applying machine learning methods to the field of machine learning, e.g. see the opening of http://people.idsia.ch/~juergen/deep-learning-conspiracy.htm... If AGI is the goal and machine learning research is the search algorithm, then Schmidhuber's attempting to perform backpropagation by pushing rewards back along the connections :)

We should use back propagation with government.

Re: Differentiable Neural Computers

#54

Post by empath75 »

Senji wrote: post #51
empath75 wrote: post #49

Earlier quoted context omitted.

The brain is not a single universal neural network that does everything well. It's a collection of different neural networks that specialize in different tasks, and probably use very different methods to achieve them. It seems like the way forward would be networking together various kinds of neural networks to achieve complex goals. For example, an NTM specialized in formulating plans that has access to a CNN for im…

If we're going with a brain metaphor. What would be the those neural networks' version of synesthesia?

Feeding mp3s to an image recognition neural net. And as soon as I typed that, I want to try it.

Re: Differentiable Neural Computers

#55

Post by [deleted] »

orthoganol wrote: post #16

It appears they are touting 'memory' as the key new feature, but I know at least in the deep learning NLP world there already exists models with 'memory', like LSTMs or RNNs with dynamic memory or 'attention.' I can't imagine this model is too radically different than the others. Maybe I just feel a bit uneasy with a claim such as: > We hope DNCs provide a new metaphor for cognitive science and neuroscience.

[deleted]

Re: Differentiable Neural Computers

#56

Post by praccu »

orthoganol wrote: post #16

It appears they are touting 'memory' as the key new feature, but I know at least in the deep learning NLP world there already exists models with 'memory', like LSTMs or RNNs with dynamic memory or 'attention.' I can't imagine this model is too radically different than the others. Maybe I just feel a bit uneasy with a claim such as: > We hope DNCs provide a new metaphor for cognitive science and neuroscience.

LSTMs are very different. Think of it this way. LSTMs store information about the current problem you're solving. DNCs store information about the world.

LSTMs are designed to capture long range dependencies, e.g., "this word at the start of the sentence interacts with this word at the end of the sentence."

DNCs are designed to incorporate outside information, e.g., "i happen to know (from background knowledge) that these two people in this sentence are married"

Re: Differentiable Neural Computers

#57

Post by akssri »

outsideline wrote: post #19

https://en.wikipedia.org/wiki/Bio-inspired_computing Present day Neuron models lack an incredible number of functional features that are clearly present in the human brain. NTMs = representing memory that is stored in neurons https://en.wikipedia.org/wiki/Neuronal_memory_allocation Decoupled Neural Interfaces using Synthetic Gradients = https://en.wikipedia.org/wiki/Electrochemical_gradient Differentiable Neural Comp…

Do you know if CompNeuro models have been trained to do things interesting to CS folk ?

Re: Differentiable Neural Computers

#59

Post by dj-wonk »

outsideline wrote: post #14
visarga wrote: post #8

Earlier quoted context omitted.

Deep mind is breaking new ground in number of directions. For example, "Decoupled Neural Interfaces using Synthetic Gradients" is simply amazing - they can make training a net async and run individual layers on separate machines by approximating the gradients with a local net. It's the kind of thing that sounds crazy on paper, but they proved it works. Another amazing thing they did was to generate audio by direct sy…

Decoupled Neural Interfaces using Synthetic Gradients is a fancy name for the electro-chemical gradient that lies outside the cell wall of neurons : https://en.wikipedia.org/wiki/Electrochemical_gradient It's decoupled yet stores transient local information regarding previous neuron activity. Another bio-inspired copy-pasta.

Please choose derogatory phrases like 'copy pasta' intentionally and carefully.

Many algorithms are bio-inspired -- good artists borrow, the best steal.

Re: Differentiable Neural Computers

#60

Post by [deleted] »

outsideline wrote: post #14
visarga wrote: post #8

Earlier quoted context omitted.

Deep mind is breaking new ground in number of directions. For example, "Decoupled Neural Interfaces using Synthetic Gradients" is simply amazing - they can make training a net async and run individual layers on separate machines by approximating the gradients with a local net. It's the kind of thing that sounds crazy on paper, but they proved it works. Another amazing thing they did was to generate audio by direct sy…

Decoupled Neural Interfaces using Synthetic Gradients is a fancy name for the electro-chemical gradient that lies outside the cell wall of neurons : https://en.wikipedia.org/wiki/Electrochemical_gradient It's decoupled yet stores transient local information regarding previous neuron activity. Another bio-inspired copy-pasta.

[deleted]
Post reply on HN