Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Differentiable Neural Computers

deepmind.com

Post reply on HN
Print view View original

61–70 of 77 posts

Re: Differentiable Neural Computers

#61

Post by iandanforth »

gugagore wrote: post #45
igravious wrote: post #32

Earlier quoted context omitted.

In this context what does differentiable mean?

I think the easiest way to see this is by an example of a non-differentable architecture. Let's suppose on the current training input, the network produces some output that is a little wrong. It produced this output by reading a value v at location x of memory. In other words, output = v = mem[x] It could be wrong because the value in memory should have been something else. In this case, you can propagate the gradien…

Turing didn't specify how reads and writes happened on the tape. For the argument he was making it was clearer to assume there was no noise in the system.

As for "digital" computers remember they are built out of noisy physical systems. Any bit in the CPU is actually a range of voltages that we squash into the abstract concept of binary.

Re: Differentiable Neural Computers

#62

Post by hyperbovine »

pmatev wrote: post #6
ericjang wrote: post #3

Earlier quoted context omitted.

here you go http://rdcu.be/kXdj

This is great, thanks. Anyone know how those charts / graphs have been generated?

By hand, I imagine. There is an author credited specifically with the graphics and nothing else.

Re: Differentiable Neural Computers

#63

Post by modeless »

orthoganol wrote: post #18
modeless wrote: post #17

Earlier quoted context omitted.

The "memory" in a typical RNN is akin to a human's short term working memory. It only holds a few things and forgets old things quickly as new things come in. This new memory can hold a large number of things and stores them for an unlimited amount of time, more like a human's long term memory or a computer's RAM. It's a big difference, and the implementation is completely different too.

I was not referring to typical RNNs, but LSTMs or RNNs with 'attention'. They are designed to overcome vanishing/ exploding gradient problems and hold arbitrary memory lengths.

The memory provided by an attention architecture is immutable and has only one addressing mode. The memory in this paper is mutable (hence "Neural Turing Machine") and has several different addressing modes.

Re: Differentiable Neural Computers

#64

Post by modeless »

empath75 wrote: post #54
Senji wrote: post #51

Earlier quoted context omitted.

If we're going with a brain metaphor. What would be the those neural networks' version of synesthesia?

Feeding mp3s to an image recognition neural net. And as soon as I typed that, I want to try it.

Actually, in the architecture you described, if there is a planning net that's connected to image net and an audio net, rather than feeding audio to the image net I think synesthesia would be better modeled by feeding the output of the audio net into the image net's input on the planning net. If that makes sense.

Re: Differentiable Neural Computers

#65

Post by dharma1 »

empath75 wrote: post #54
Senji wrote: post #51

Earlier quoted context omitted.

If we're going with a brain metaphor. What would be the those neural networks' version of synesthesia?

Feeding mp3s to an image recognition neural net. And as soon as I typed that, I want to try it.

CNNs can actually be used for audio tasks too, on spectrograms

Re: Differentiable Neural Computers

#66

Post by Senji »

modeless wrote: post #64
empath75 wrote: post #54

Earlier quoted context omitted.

Feeding mp3s to an image recognition neural net. And as soon as I typed that, I want to try it.

Actually, in the architecture you described, if there is a planning net that's connected to image net and an audio net, rather than feeding audio to the image net I think synesthesia would be better modeled by feeding the output of the audio net into the image net's input on the planning net. If that makes sense.

Not the output. Making several single connections from intermediate layers from the different nets.

Re: Differentiable Neural Computers

#67

Post by igravious »

gugagore wrote: post #45
igravious wrote: post #32

Earlier quoted context omitted.

In this context what does differentiable mean?

I think the easiest way to see this is by an example of a non-differentable architecture. Let's suppose on the current training input, the network produces some output that is a little wrong. It produced this output by reading a value v at location x of memory. In other words, output = v = mem[x] It could be wrong because the value in memory should have been something else. In this case, you can propagate the gradien…

No wonder Google built (is building) custom accelerators in hardware. This points to a completely different architecture from Von Neumann, or at least it points to MLPUs, Machine Learning Processing Units.

Re: Differentiable Neural Computers

#69

Post by tvural »

The idea of using neural networks to do what humans can already write code to do seems a bit wrong-headed. Why would you take a system that's human-readable, fast, and easy to edit, and make it slow, opaque, and very hard to edit? The big wins for ml have all been things that people couldn't write code to do, like image recognition.

Re: Differentiable Neural Computers

#70

Post by AgentME »

tvural wrote: post #69

The idea of using neural networks to do what humans can already write code to do seems a bit wrong-headed. Why would you take a system that's human-readable, fast, and easy to edit, and make it slow, opaque, and very hard to edit? The big wins for ml have all been things that people couldn't write code to do, like image recognition.

Even if it only works for problems that are trivially solved by people, the fact that it can be done automatically is useful. A system could react to changes and some new problems automatically by continually retraining itself.
Post reply on HN